bilby

Predicting genome browser tracks — RNA-seq, CAGE, DNase-seq, ATAC-seq, ChIP-seq coverage — directly from DNA sequence is one of the central sequence-to-function problems in regulatory genomics. The lineage runs from the convolutional Basenji through Enformer, which added attention to capture long-range regulatory interactions, to Borzoi, which predicts base-resolution RNA-seq coverage. Attention is what makes these models work at scale, and also what makes them expensive: its cost grows quadratically with sequence length, which caps the genomic context a model can see and pushes training onto multi-GPU hardware.

bilby, from Ian Holmes at UC Berkeley with Johannes Linder and David Kelley at Calico Life Sciences, asks whether sub-quadratic sequence operators can stand in for attention on this task. It is a Jax/Flax library plus a set of trained models implementing convolutional, attention, bidirectional Hyena, bidirectional Mamba, and striped architectures that interleave Mamba and attention layers, all trained on an identical functional genomics corpus so the architectures can be compared head to head.

The headline result is that convolution plus bidirectional Mamba, with attention layers striped in, gives a small but consistent accuracy gain over the convolution-and-attention reference at comparable parameter counts and training times. Hyena, as originally specified, was not competitive; MultiResNet proved too slow to be practical. The striped Mamba configuration is the model the authors carry forward as their best.

#Key Features

• Striped Mamba backbone: Alternating selective state-space and attention layers combines Mamba's linear-time sequence scan with attention's explicit pairwise mixing, and outperformed either operator used alone.
• Bidirectional state-space layers: Because track prediction depends on flanking sequence on both sides, the authors built bidirectional variants of Mamba and Hyena, doubling the Mamba scan over forward and reverse-complemented sequence with tied projection weights.
• Multi-task coverage prediction: A single model emits all output tracks at 32 bp resolution from a 393,216 bp input window, spanning four assay types and 49 tissues.
• Single-GPU training: Every training run used one 24 GB NVIDIA L4, putting architecture experiments in this family within reach of a modest compute budget.
• Frozen-backbone variant scoring: Model outputs for reference and alternate alleles feed a random forest that classifies fine-mapped GTEx eQTLs, with no retraining of the sequence model.

#Technical Details

All architectures share a five-round convolutional stack with twofold max-pooling that maps a one-hot (393216, 4) input to a (12288, 768) embedding at 32 bp resolution; the attention, Hyena, and Mamba blocks then replace one another in the middle of the network, with U-Net-style pooling and upsampling around the attention layers. Training used a Poisson total-count loss combined with a multinomial within-track positional loss, upweighted fourfold on the multinomial term, matching Borzoi's objective. The corpus is 2,194 human genome-wide coverage datasets across 49 tissues: 724 RNA-seq and 321 DNase-seq tracks from ENCODE, 35 GTEx RNA-seq tracks via recount3, 434 3' RNA-seq tracks from Tabula Sapiens, and 680 FANTOM5 CAGE tracks. The genome was split into 13,851 windows grouped into a 4:1:1 train/validation/test split, repeated over four distinct splits.

Mamba-based models improved Pearson R by 3–4% and r² by 1–2% averaged over all tracks and splits, with the best configurations using six Mamba layers, hidden dimension N = 8, and expansion factor E = 1. Depth beyond six Mamba layers degraded peak accuracy. On the GTEx eQTL classification benchmark the transformer reached 0.7919 mean AUROC against 0.7853 for the best Mamba model, while Spearman correlation ran the other way at 0.2645 versus 0.2702 — close enough that the accuracy gains on the modeling task did not carry through to variant scoring.

#Applications

bilby serves two audiences. For regulatory genomics groups, the trained models predict expression and chromatin accessibility tracks from sequence and score candidate non-coding variants for regulatory impact. For methods developers, the library is a controlled testbed: a common convolutional trunk, a common training corpus and loss, and interchangeable sequence-operator blocks, which makes it a practical starting point for anyone evaluating a new architecture on functional genomics data without rebuilding the surrounding pipeline.

#Impact

The study is one of the more carefully matched comparisons of attention against state-space and long-convolution alternatives on a real functional genomics task, and its conclusion is deliberately measured: selective state-space models warrant further exploration rather than immediate replacement of attention. The models are trained at a smaller scale than Borzoi — a 393 kb rather than 524 kb window, 2,194 rather than 7,611 human tracks, a 768 rather than 1,536 embedding dimension — so the reported gains are architecture-level rather than absolute state of the art. The GitHub repository carries no license file, which limits reuse of the released code and parameters, and the preprint has not been peer reviewed.