HELM

GCT, GCC, GCA and GCG all encode alanine. A language model trained on mRNA with ordinary cross-entropy is told that predicting GCC where the answer was GCT is exactly as wrong as predicting a stop codon — the loss has no way to express that the first mistake changes nothing about the protein and the second truncates it. That flat treatment of a 64-way vocabulary discards the most basic fact about the genetic code: codons are grouped under the amino acids they specify.

HELM (Hierarchical Encoding for mRNA Language Modeling) restores that grouping inside the training objective rather than the architecture. Codons are arranged as a tree — a root splitting into coding and non-coding branches, the coding branch splitting into the 20 amino acids, and each amino acid into its synonymous codons — and a hierarchical cross-entropy loss factorizes the token distribution along paths through that tree, weighting each level by exp(-alpha*h) in the node's height h. Errors high in the tree are penalized heavily, errors between synonymous siblings lightly. Tokenizer, vocabulary and network are unchanged, so the method drops into an existing masked or causal language modeling pipeline as a loss swap.

The work comes from Johnson & Johnson Innovative Medicine with Mehdi Yazdani-Jahromi of the University of Central Florida, who led it as an intern there; it was posted to arXiv in October 2024 and published at ICLR 2025. It fills a gap the authors identify: RNA-FM covers non-coding RNA and SpliceBERT pre-mRNA, leaving protein-coding mRNA — where codon choice is the whole point — without a model built around it.

#Key Features

• Hierarchy-weighted loss: Hierarchical cross-entropy over the codon tree replaces flat cross-entropy, so residual errors fall on synonymous codons instead of scattering across unrelated amino acids.
• Codon tokenization: The vocabulary is 64 codon tokens plus start, stop, padding, mask and unknown symbols, chosen over nucleotide and 6-mer tokenization in a like-for-like comparison.
• Objective- and architecture-agnostic: The same loss applies to masked and causal language modeling, and to transformer, Mamba and Hyena backbones at equal parameter count.
• Generative counterpart: HELM-CLM samples full mRNA sequences conditioned on a prefix, not only embeddings for a downstream head.
• Curated antibody mRNA corpus: Pretraining uses a de-noised nucleotide-level curation of the Observed Antibody Space database, previously curated only at the protein level.

#Technical Details

All models hold to roughly 50M parameters: the headline transformer is a GPT-2 backbone with 10 layers, hidden size 640 and intermediate size 2560, over a 70-token vocabulary and a 444-token context, with positional embeddings sized to 2048. Pretraining runs 40 epochs on 8 NVIDIA A100 GPUs at an effective batch size of 1024 with AdamW and alpha = 0.2, over 15.3M sequences curated from OAS by ANARCI filtering, restriction to productive complete VDJ recombinations and redundancy removal. Evaluation probes the frozen representation with a 1M-parameter TextCNN head on seven property-prediction datasets — two antibody expression sets, MLOS viral expression, iCodon thermostability across human, mouse, frog and fish, Tc-Riboswitch switching factors, mRFP production in E. coli, and COVID-19 vaccine degradation — scored by Spearman rank correlation.

Against its own non-hierarchical twin the masked HELM model improves on every dataset: 0.626 versus 0.569 on Tc-Riboswitches, 0.822 versus 0.753 on mRFP, 0.833 versus 0.801 on COVID-19 vaccine degradation, an average gain near 8%. The margin tracks codon usage bias, largest where codon-usage entropy is lowest. The 50M models beat RNA-FM, SpliceBERT and CodonBERT baselines by 5–17 percentage points despite being smaller, and HELM-CLM achieves better Frechet Biological Distance than the flat-loss baseline at every sampling temperature while preserving functional properties with 2–31% lower squared error.

#Applications

The frozen encoder is a representation extractor for mRNA engineering: ranking candidate constructs for expression, thermostability and degradation before synthesis, the gating step in mRNA vaccine and therapeutic design. The annotation head labels each nucleotide of an antibody-encoding transcript as signal peptide, V, DJ or constant region — a task with no prior machine-learning tool — suiting repertoire-sequencing pipelines that work at the nucleotide rather than protein level. The causal checkpoint supports generative codon optimization, proposing synonymous variants inside the observed distribution.

#Impact

HELM's contribution is that a biological prior can enter a language model through the loss alone, at no architectural or tokenization cost — cheap enough that 50M-parameter models outperform 100M-parameter RNA foundation models on mRNA tasks. The same group extended the idea in HyperHELM, which moves the hierarchy into a hyperbolic prediction head. Real limits bound the claims: pretraining is antibody-encoding mRNA specifically, so performance on other transcripts rests on transfer rather than in-domain data; evaluation is entirely computational; and the released code is a training and evaluation pipeline, shipped without pretrained checkpoints or the curated 15.3M-sequence corpus, so reproducing the model means repeating the pretraining run.