LAMAR

LAMAR — a Language Model for RNA Regulation — is a transformer encoder pretrained on roughly 15 million genomic and transcriptomic sequences drawn from 225 mammals and 1,569 mammal-infecting viruses. It was built by the Rnasys Lab and the Bio-Med Big Data Center at the Shanghai Institute of Nutrition and Health together with the Southern University of Science and Technology, and posted as a preprint in October 2024.

The regulatory information that governs an RNA's fate is spread across its whole life cycle: where a pre-mRNA is spliced, how efficiently a mature transcript is loaded onto ribosomes, how long it survives before decay, and whether it carries an internal ribosome entry site (IRES) that permits cap-independent translation. Each of these questions has historically had its own specialist predictor, trained from scratch on its own labeled dataset. LAMAR asks whether a single self-supervised backbone can be fine-tuned to all four, and reports that it can.

That framing distinguishes LAMAR from its closest neighbors. RNA-FM pretrains on non-coding RNA, and 5' UTR-LM covers only 5' untranslated regions. LAMAR's corpus deliberately mixes genome and transcriptome, so intronic context needed for splicing and untranslated regions needed for translation and decay are both represented in one model.

#Key Features

• One backbone, four regulatory tasks: The same pretrained encoder is fine-tuned into LAMAR-SS for splice sites, LAMAR-TE for translation efficiency, LAMAR-DR for mRNA half-life, and LAMAR-IRES for internal ribosome entry sites.
• Two context lengths: LAMAR-2k and LAMAR-4k share an architecture but are pretrained at maximum lengths of 2,048 and 4,096 tokens, with rotary position embeddings carrying positional information into long transcripts.
• Single-nucleotide tokenization: Sequences are tokenized one base at a time rather than in k-mers, which keeps per-position resolution and lets fine-tuned models score individual point mutations.
• Variant effect readout: On a near-saturation mutagenesis of the CXCL2 3' UTR, LAMAR-DR's predicted change in half-life between mutant and wild-type tracked the measured effect on mRNA level, and recovered a strong degradation element.
• Cap-independent translation screening: LAMAR-IRES, trained on 1,901 experimentally validated IRESs from IRESite, IRESbase, and Rfam, assigns significantly higher scores to putative IRESs from independent high-throughput screens.

#Technical Details

The encoder follows the ESM-2 design: 12 bidirectional transformer layers, 12 attention heads, hidden size 768, feed-forward intermediate size 3,072, and rotary position embeddings, for roughly 85 million parameters. Pretraining uses masked language modeling with 15% of tokens masked, and the pretrained models reconstruct over 50% of masked bases. The corpus combines RefSeq release 221 transcripts, RNAcentral release 23 non-coding transcripts, 6,033,179 genes extracted from mammalian reference genomes, and viral reference genomes, totaling 15,000,406 sequences after clustering.

On benchmarks, LAMAR-SS exceeded SpliceAI-2k by 0.029 PRAUC and 0.037 top-k accuracy, and RNA-FM by 0.041 PRAUC; a frozen encoder with only a prediction head already reached PRAUC 0.926 and top-k accuracy 0.886. LAMAR-TE, fine-tuned on 10,903 HEK293T transcripts, reached MSE 0.409 and Spearman 0.652, improving on RNA-FM by 7% and 5' UTR-LM by 18% in correlation. LAMAR-DR, trained on a reporter assay of 1,967 human 3' UTRs, reached MSE 0.176 and Spearman 0.647, 8.0% above a comparably fine-tuned RNA-FM. Both LAMAR-IRES and its RNA-FM counterpart reached AUC near 0.98.

#Applications

The fine-tuned heads map onto concrete design problems in RNA therapeutics. A group optimizing an mRNA vaccine or protein-replacement construct can rank candidate 5' UTRs by predicted translation efficiency and candidate 3' UTRs by predicted half-life before committing to synthesis. Groups working on circular RNA — which lacks free ends and must be translated through an IRES — can use LAMAR-IRES to triage sequence libraries in silico. The splice-site head supports transcript annotation and interpretation of intronic variants in clinical sequencing.

#Impact

LAMAR's contribution is evidence that RNA regulation across splicing, translation, and decay shares enough sequence grammar for one pretrained model to serve all three, rather than requiring a specialist model per layer. Its margins over task-specific baselines are largest for translation efficiency and half-life and narrower for splice sites, where the convolutional SpliceAI remains close. The preprint has not yet been peer reviewed, and it is posted under CC BY-NC-ND 4.0, so its benchmark tables cannot be redistributed in adapted form even though the software is permissively licensed. Pretrained and fine-tuned weights are published under the MIT license alongside notebook-level fine-tuning and evaluation code, which makes the four downstream heads reproducible and the backbone reusable for regulatory tasks the authors did not test.