HyperHELM

Language models for biological sequences almost always place their tokens in Euclidean space, yet the relationships those tokens encode are frequently hierarchical. In mRNA, 64 codons map onto 20 amino acids plus start and stop signals — a tree whose node count grows exponentially with depth, while Euclidean volume grows only polynomially, so flat embeddings distort the very relationships a codon-level model is meant to learn. HyperHELM moves the masked language modeling (MLM) objective itself into hyperbolic space, whose volume expands exponentially and which admits low-distortion tree embeddings.

Developed by researchers at Johnson & Johnson Innovative Medicine and the University of Amsterdam and posted to arXiv in September 2025, HyperHELM starts from HELM, the group's earlier Euclidean hierarchy-aware mRNA language model, and changes only the prediction head. The transformer backbone stays Euclidean; final-layer token representations are projected onto the Poincaré ball, where a prototype classifier scores masked-token predictions against fixed prototypes placed directly from the codon–amino-acid hierarchy. Keeping the backbone flat preserves hardware efficiency: pretraining runtime is essentially unchanged at roughly 72 minutes per epoch versus 71 for the Euclidean baseline. Because backbone, pretraining corpus, and tokenization are held identical across HyperHELM and its Euclidean counterparts, the reported differences isolate the effect of geometry rather than of scale or data.

#Key Features

• Hyperbolic masked language modeling: Token representations are mapped onto the Poincaré ball with an exponential map and a hyperbolic linear layer, and the MLM objective is optimized in that space rather than in the usual flat embedding space.
• Hierarchy-derived prototypes: Codon prototypes are generated from the codon–amino-acid tree with a low-distortion hyperbolic embedding method, so the representation space is structured by biology before any sequence data is seen.
• Entailment-cone similarity: Beyond distance-to-prototype scoring, the model uses hyperbolic entailment cone energy as a similarity function, extending cones from a device for stating hierarchical relations to a scoring rule for MLM.
• Out-of-distribution robustness: Where the Euclidean baseline degrades on sequences longer than those seen in pretraining, both prototype variants improve, and they hold up better on high-GC sequences.

#Technical Details

The backbone is a 10-layer transformer with hidden size 640 and intermediate size 2560, about 50M parameters, trained with codon-level tokenization over a 70-token vocabulary (64 codons plus special tokens). Pretraining masks 15% of tokens and uses the hierarchical cross-entropy loss over the codon hierarchy, running 40 epochs at batch size 1024 on eight NVIDIA A100 GPUs with AdamW. The pretraining corpus is the curated set of antibody-coding mRNA from the Observed Antibody Space database used by HELM; context length is 444 tokens, with positional embeddings sized for 2048 to accommodate longer downstream sequences. Prototypes are 128-dimensional at curvature 1.0, implemented with the HypLL library.

Evaluation freezes the backbone and trains a TextCNN probe on 10 downstream datasets spanning antibody expression, mRFP production, COVID-19 vaccine degradation, protein abundance in Drosophila melanogaster, Saccharomyces cerevisiae and Pichia pastoris, fungal expression, E. coli expression class, and iCodon thermostability. The prototype variants win on 9 of 10 datasets, improving on average by about 10% and by 2.8–35.5% over a non-hierarchical transformer baseline, and outperforming RNA-FM, SpliceBERT, and CodonBERT where those models can process the sequences. On antibody region annotation the distance-based variant reaches 76.48% accuracy against HELM's 73.48%. Gains track codon usage bias: datasets with a lower effective number of codons benefit most. A plain hyperbolic multinomial logistic regression head underperforms the Euclidean baselines, indicating the prototype formulation does the work rather than hyperbolic geometry alone.

#Applications

The frozen encoder is a drop-in representation extractor for mRNA engineering tasks — predicting protein expression, transcript abundance, degradation and thermostability, and annotating antibody-coding mRNA into signal peptide, V, DJ, and constant regions. That profile fits mRNA therapeutic and vaccine design, where candidates must be ranked for expression and stability before synthesis, and antibody discovery pipelines working from repertoire sequencing at the nucleotide level. Robustness to long and high-GC sequences matters for constructs outside the length range typical of pretraining corpora.

#Impact

HyperHELM's contribution is methodological: it shows that the geometry of the output space is a usable inductive bias for biological sequence models, and that a hybrid Euclidean-backbone/hyperbolic-head design captures the benefit without the training cost of a fully hyperbolic network. It is a controlled ablation of geometry at fixed scale, unusual in a field where gains typically come from more parameters or more data. Several limits bound the claims: the work is a preprint awaiting peer review, and no code or weights have been released for HyperHELM itself, though the Euclidean HELM backbone is public. Prototypes are fixed rather than learned, evaluation is entirely computational and uses frozen-backbone probing rather than fine-tuning, and the pretraining corpus is antibody-focused, so performance on other mRNA populations rests on downstream transfer rather than in-domain pretraining. The authors identify causal language modeling and other biological modalities as the next targets.