What is gene expression, and why does controlling it matter?

Gene expression is the process of using the information in a gene to make a functional product. The usual way this works is by converting the DNA into RNA (called transcription, using a RNA polymerase) and then a ribosome converting that RNA into a protein (called translation). A gene that is said to have expressed usually means that a lot of it is converted to RNA and the corresponding proteins. However, this has caveats as some genes can produce functional RNAs that are never translated into proteins. Also usually when people say a gene is highly expressed, they are talking about high RNA corresponding to this DNA, but another RNA which is quickly converted to protein might be harder to measure. Controlling gene-expression matters for the proper functioning of biology, for e.g. pancreatic cells should express the insulin gene while other cells should not.

How can cells with essentially the same DNA behave differently?

Gene expression in a cell is controlled by several factors like (a) transcription factors which are specific proteins that bind to particular DNA sequences and help increase or decrease the expression of nearby (sometimes distant) genes, (b) DNA is packaged with chromatin in a cell's nucleus and this packiing is organized differently across cell types allows access to different parts of the DNA for the transcription process (not all accessible regions are transcribed tho), and (c) cellular context like hormones, nutrients, infection, developmental signals can change which gene and how much it expresses.

How does DNA regulate gene expression?

There are several factors that assist gene expression:

a. Transcription Start Site (TSS): It is the first nucleotide where RNA transcription begins (note that this is different from the protein translation begins as mRNA has a stretch of RNA before its protein-coding instructions). A gene can use different TSSs under different circumstances.

b. Promoter: Its a DNA region around a TSS that helps assemble the machinery needed for translation (RNA polymerase works with other proteins to start making RNA).

c. Transcription factors are proteins that regulate transcription -- they recognize short DNA sequence patterns (called binding motifs) which can stabilize or inhibit the transcription machinery.

c. Enhancer is a DNA region that can increase transcription from a target promoter under appropriate context. Enhancers often contain binding sites for several transcription factors, that when binded can help a target promoter initiate transcription more frequently.

Non-coding parts of a DNA might not directly produce an RNA, but can still impact the trancription process (like a promoter and enhancer).

When a DNA mutation happens, that can change the protein-coding region (and therefore change the protein produced) or change the regulatory region (which change how much the RNA is produced) or no change if its unrelated to this gene.

Why do distant parts of the DNA matter? How can an enhancer influence a gene thousands of DNA bases away?

Distant parts of a DNA in the sequence position might become very close in the 3D space because of how the chromatin folds the DNA, in this case an enchancer 50 kb away from a promoter can come close and increase RNA production of a gene (called as promoter-enhancer looping). Note that this looping is not rigid, chromatin is dynamic and physical proximity does not always impact RNA production. So for gene regulation modeling, a model should not only consider the immediate neighbor sequence positions, but farther positions as well.

What biological problem is Enformer trying to solve, and why is it useful?

Enformer aims to can predict gene activity in different cell types from the surrounding DNA. The central thesis is to have a long-context view of gene regulation, rather than only looking at a gene's nearby promoter. There are two other kinds of predictions they want to make: (b) predicting impact of mutations, so say if there is a change in enhancer of a gene, we want to be able to predict the impact on gene expression (sau that change weakens the enhancer's binding site), and (c) identity the parts of the surrounding DNA that impact gene's expression (this can help researchers rank and prioritize enhancer-gene relationships for experimental verification). A disease connection comes useful via use case (b) and (c) when a change in the noncoding part of the DNA impacts gene expression and thereby affect biology.

What exactly are the model’s inputs and outputs? What is a “genomic track”? How are predictions organized along the DNA, and how does the model distinguish cell types when its input is only sequence?

The model inputs are DNA sequences that are capped at 196K length, each DNA nucleotide is one of 4 (A, T, G, C), so the input is a one-hot vector of size 196K * 4. The target (or the output) is derived from the experimental data. Say there are bunch of experiments done in labs across the world measuring various activities in some cell types that can be tied to different parts of the human genome. Such a measurement is called a genomic track. Now for a particular input, the model's goal will be to output the same experimentally observed measurements of those properties. However, the loss is only calculated for the central 114K input base pairs of the DNA from the 196K pairs, from each side 40K pairs are excluded from the loss as the model might not have enough context about them (even though they might still influence the central bit's prediction). Also the central 114K base pairs are binned into 128 sized base pairs bins giving 896 bins and these measurements are a target for that bin, here is an example of what that might look like

Genomic bin	A: CAGE, liver	B: CAGE, neurons	C: accessibility, liver	D: TF binding	E: histone mark	…
Bin 1	0	0	1.2	0.4	3.1	…
Bin 2	45	2	8.6	0.2	7.5	…
Bin 3	8	31	3.4	6.1	2.8	…
…	…	…	…	…	…	…
Bin 896	0	4	0.5	1.7	0.9	…

Here the columns are the different experiments that were done at various labs and the dataset was collected (over 5313 tracks) and the values are non-negative numerical targets. Overall there were 638 CAGE experiments, 684 DNA accessibility experiments, 2131 transcription-factor ChIP-seq, and 1860 histone modification chIP-seq. Now for these experiments, we might not have a signal for each bin (or even the entire 196K long input) as the experimenters might have focused on a different region or sometimes regions cannot be mapped accurately.

The last question about distinguishability of a cell from the DNA sequences comes from the combination of patterns learned in the data. So there might be specific patterns of DNA that a transcription protein gets attached and increase RNA production for it. Such patterns can be learned by the model that can then predict like "regions containing this combination of DNA patterns leads to high CAGE signal in liver cells, but low in neuron cells" for example.

Where do the training data come from, and what do the experiments actually measure? What do CAGE, DNA-accessibility assays, and ChIP–seq tell us?

The training data for Enformer comes from the public datasets that were assembled for a previous paper, Basenji2. Lets understand in more depth each of these measurements:

a. CAGE: stands for cap analysis of Gene expression, it captures and sequences the beginning of RNA molecules carrying a particular chemical "cap". An RNA has a beginnning called 5'-end and the goal of CAGE is to identify how many RNA sequences start in a region, more RNA starting means strong CAGE signal or otherwise low CAGE signal.

b. DNAse-seq /ATAC-seq: DNA in a cell is packaged with chromatin, and therefore some regions are easier to access than other. This measures the accessibility of a region which can be a promoter, enhancer for e.g. High signal means accessible DNA

c. Transcription-factor ChIP-seq: this measures which parts of a DNA are associated with a particular transcription factor (protein).

d. Histone-modification ChIP-seq: DNA wraps around proteins called histones and cells can attach specific groups to histones: which are modifications associated with different regulatory states. This experiment (similar to TF ChIP-seq) find regions of the DNA that are associated with a specific histone marker.

What does one training example represent, and what is the learning objective? How are DNA regions paired with thousands of experimental tracks?

One training example is a pair of $(X, Y)$ , where $X \in \{0, 1\}^{196\mathrm{K} \times 4}$ and $Y \in \mathbb{R}^{896 \times 5313}$ . Therefore there are 4.76M scalar targets per human example (although these are not IID because neighboring bins are related tracks are correlated). The model compues $\hat{y} = f(X)$ and the loss penalizes differences with the experimental assay. The loss is Poisson negative log-likelihood, $l(y, hat{y}) = hat{y} - y\log \hat(y)$ (the loss is averaged over each bin and each track) and a softplus (similar to relu) ensures positive predictions. Poisson is chosen because many biological measurements start as a count and Poisson is better at modeling counts than L2 and Poisson weighs errors differently than L2 (L2 has the same value irrespective of the value of y, so an error of 10 at value 100 and 10000 have the same L2 error, whereas Poisson takes into account that an error of 10 is worse when y=10 than when y=10000).

Enformer also separately trains in mouse examples using 1643 tracks but it shares the model's body. Human and mouse batches alternate during training, with a final finetuning on human data.

What is the central computational contribution, and how does the architecture implement it?

Enformer combined comvolutions (that learn local DNA patterns) with attention (combines information across distant genomic regions). Here are the stages:

Stage	Output shape, excluding batch
One-hot DNA input	196608 * 4 (in next stage 128 bp combine to give 1536 bins)
Convolutions and pooling	1536 * 1536 (one is the number of genomic positions and other is the feature size)
11 transformer blocks	1536 * 1536
Crop 320 positions from each side	896 * 1536
Per-position feature transformation	896 * 3072
Human output head and softplus	896 * 5313

a. Convolutions can learn filters specific to certain motifs and then apply it across the entire input, and therefore is a good inductive bias. Successive layers combine features and after 7 rounds of pooling we get, $196608/2^{7} = 1536$ . Each vector incorporates information from neighboring sequences, so its not the information from exactly those 128 base pairs.

b. Attention just as in LLMs helps distant regions to attend to each other and relative position embeddings help the attention account for how far they are in the sequence (note that position embeddings account for the distance along the DNA sequence, not the 3D distance).

c. Once attention has mixed information from all the regions, the 320 bins from each side are cropped as those regions might require even more regions for accurate prediction, but they can still influence the predictions of the central region. Finally, the contextual representation is projected to the tracks (humans or mouse).

How are training and evaluation separated?

The training and evaluation are done on distinct genomic regions but for the same genomic tracks used during training, so the expected generalization is to new regions rather than new tracks. The split of regions cannot be just random because nearby regions can share similarities in measurements and this can even be true for human genomic regions and mouse genomic regions because of shared ancestry. Therefore the data split is coordinated between humans and mouse: entire genome is divided into regions with 1M bp and a human and mouse region is said to be related if they share more than 100K bp of alignment. Now an entire 1M region is assigned to either training, val, or test. Final split are of 196K base pair windows are:

Species	Training	Validation	Test
Human	34,021	2,213	1,937
Mouse	29,295	2,209	2,017

How is gene-expression prediction evaluated, and what does the improvement mean?

The first step is to convert the bin-level signal to gene-level expression estimates. This is done by summing up the signal in each bin containing all the TSS (transcription start sites) for genes that are known to researchers. This gives us 638 CAGE track value for each gene. Now image a table like this:

	Liver CAGE	Neuron CAGE
Gene A	…	…
Gene B	…	…
Gene C	…	…

There are two axes of evaluation: for a specific track, does the model capture the correlations between different gene expressions (down the column) and for a specific gene, does the model capture the correlations between different biological contexts, and these are distinct measurements for understanding the model capability. They compute the average pearson correlation across the CAGE tracks and found that compared to the baseline Basenji2 model (0.81), Enformer captured more correlations (0.85). Based on some repeated tracks, authors estimate the upper limit of this correlation to be 0.94 (as the same tracks give different measurements under repeated experiments).

Which design choices actually explain the improvement?

Authors perform several ablations experiments to test this part: a. Replace attention with dilated convolutions: attention performs better b. receptive field: they use attention but restrict its receptive field to 44K bp -- the model performs worse, so the model might be using distant parts of the DNA for this task. c. Increasing model hidden dimension also helped (so larger model helped) d. Positional embeddings also help performance and their custom relation embeddings perform better than standard embeddings like sine and cosine, meaning relative distance is a better informant to the model.

What evidence suggests that Enformer uses biologically relevant regulatory elements?

Here the authors perform an experiment to show that Enformer's prediction are correlated with experimental evidence of an enhancer's impact on gene expression. For this, the take a lab experiment where a candidate enhancer K562 in a human cell line was suppressed and that lead to statistically significant gene expression change. And now they perform a experiment where they randomly replace the a candidate region with a random DNA sequence and study its impact on the K562 CAGE signal (this is called as in-silico mutagenesis). The score is $S(E, G) = |\hat{y}(original DNA) - \hat{y}(DNA with region X randomized) |$ . They evaluate candidate regsionw with 2-kb windows around this gene's TSS. They use the predicted score to rank enhancer-gene pairs and evaluate how well the ranking recovers the experimentally validated pairs (they use precision-recall curve for this). Authors find Enformer's scores to outperform the baseline Basenji's score (which only evaluated much larger candidates at 20K bp). This experiment alludes to Enformer's prediction depending on biologically relevant regulatory elements.

Can Enformer predict what happens when the DNA sequence changes?

Authors test this using naturally occuring genetic variations in humans and controlled lab experiments that target specific DNA changes. The difference in the prediction scores before and after the change is computed, a positive difference means increased expression and vice-versa. For the first case, they use expression quantitative trait locus (or eQTL) whcih is a genomic location where genetic variation is statitically linked with variation in gene expression, for e.g. "Across people, those carrying allele G tend to have higher expression of gene Z in liver than those carrying allele A." For this the authors use the GTEx project. For the second, they use lab experiments that directly mutate DNA and measure its effect using massively parallel reporter assays (MPRAs). For both cases, Enformer performs better than Basenji2. However, both models struggle with predictions when variants are more than 1kb from the TSS, limiting confidence in predicting impact of distant regulatory mutations.

What has the paper established, and what remains unsolved?

The paper shows that use long-range DNA information improves predictions of measured gene regulation and even show usage of biologically relevant regulatory factors. There are several limitations of the approach: a. the model is only given DNA sequence as input and therefore it cannot base its predictions on cellular contexts like transcription-factor concentration, signaling state, or chromatin state.

b. The differences in predictions do not establish causal impact.

Mainly the paper allows better prioritization: plausible enhancers and variants to test.

My questions:

Since the neighboring bins and tracks are related, how can we use a loss that models that part too?
What if we replace convolutions with attention like used in deepseek for long-context modeling?

The collection