Helix-mRNA

Helix-mRNA is a foundation model for full-length mRNA sequences, released in February 2025 by Helical, a Luxembourg-based AI drug discovery company. It combines state-space and attention layers in a hybrid backbone so that a whole transcript — 5' UTR, coding sequence, and 3' UTR — can be encoded in one pass at single-nucleotide resolution.

The properties that determine whether an mRNA therapeutic works are distributed across the transcript. Codon choice in the coding region shapes translation efficiency and expression; the untranslated regions govern stability, degradation, and ribosome recruitment. Existing sequence models tend to specialize: some optimize codons and ignore the UTRs, and region-specific models such as 5' UTR-LM cover one segment in depth. Adapting a region-specific model to another region generally means retraining from scratch. The other constraint is length — full mRNA transcripts are long, and pure attention models handle them by chunking or by coarse tokenization that groups nucleotides together and discards resolution.

Helix-mRNA resolves both constraints with architecture rather than scale. Mamba-2 layers carry the long-sequence load while a small number of attention layers supply in-context retention, which allows a 12,288-token pretraining context — six times that of the transformer baseline the authors compare against — at 5.19 million parameters, a tenth of that baseline's size.

#Key Features

• Hybrid state-space and attention backbone: The layer mix is 44.4% Mamba-2, 44.4% MLP, and 11.1% attention, with every attention layer preceded by a Mamba-2 layer so the recurrence encodes the positional information attention needs.
• Single-nucleotide tokenization with codon structure: Each of A, C, U, and G maps to its own token, and a special separator character marks codon boundaries, so reading frame is preserved without coarsening resolution.
• Whole-transcript coverage: One model handles coding regions and both UTRs, and its predictive capability extends across all mRNA regions rather than requiring a region-matched checkpoint.
• Two-stage pretraining: A first stage over mixed-quality, taxonomically diverse data builds a general base model; a second stage under a decaying learning rate refines it on human-only sequences for human-specific tasks.
• Cross-phyla pretraining corpus: RefSeq mRNA from vertebrates, plants, invertebrates, fungi, and 238 clinically relevant viruses gives the model both deep eukaryotic conservation signal and viral sequence composition.

#Technical Details

The 5.19M-parameter model is trained autoregressively under a Warmup-Stable-Decay learning rate schedule, which is what makes the two-stage design work: the stable phase absorbs the broad mixed-quality corpus, and the decay phase concentrates on high-quality human transcripts. The architectural ratio of Mamba-2, MLP, and attention layers follows published hybrid-model ablations rather than a new search.

After the first pretraining stage alone, UMAP projections of embeddings for unseen coding sequences separate cleanly by phylum, with viral sequences forming a distinct cluster — evidence that unsupervised pretraining captured codon usage and compositional structure without labels. On downstream benchmarks, reported as Spearman rank correlations against CodonBERT, Transformer HELM, and Transformer XE, Helix-mRNA scores 0.79 on MLOS flu vaccines, 0.86 on mRFP expression, 0.52 on mRNA stability measured by iCodon, 0.64 on the Tc-Riboswitch task, and 0.84 on vaccine degradation. It leads on four of the five, with mRNA stability the exception where it trails HELM's 0.53 by a statistically negligible margin.

#Applications

The immediate audience is mRNA therapeutics and vaccine development, where sequence design decisions — which codons to use, which UTRs to pair with a payload — currently rest on heuristics and expensive iterative testing. Embeddings from the model support ranking candidate constructs for expression, stability, and degradation before synthesis, and the whole-transcript scope means a single model serves the UTR and coding-region questions that would otherwise need separate tools. The small parameter count lowers the barrier for fine-tuning on in-house assay data, and the model ships inside Helical's publicly released package alongside other biological foundation models, so it can be composed into existing pipelines without bespoke inference code.

#Impact

Helix-mRNA is a useful demonstration that hybrid state-space architectures suit biological sequences whose informative structure spans thousands of positions, and that resolution need not be traded for context length. Its efficiency argument matters practically: a 5.19M-parameter model that outperforms substantially larger mRNA models is cheap to fine-tune and cheap to serve. The release terms are worth noting, because the model is non-commercial end to end: the Helix-mRNA module inside the Helical package ships its own CC BY-NC-SA 4.0 licence covering the modelling code, the tokenizer, and the fine-tuning wrapper — only the surrounding framework is AGPL-3.0 — and the Hugging Face weights and the arXiv paper carry the same non-commercial terms, which constrains exactly the industrial mRNA design setting the model targets. Benchmarks are correlational sequence-to-property tasks on existing datasets rather than prospective validation of designed constructs, and the second pretraining stage's human focus means performance on non-human transcripts rests on the general base model.