RiboSpan

A mature human mRNA routinely runs several thousand nucleotides, but the dense bidirectional encoders RNA biology leans on — RNA-FM, RiNALMo, AIDO.RNA — were pretrained at context lengths of about 1,024 tokens. Anything longer has to be truncated, which severs the 5′ UTR from the coding sequence and the 3′ UTR it shares a transcript with. Those regions jointly set translation, folding, stability, localization and RNA-binding-protein contacts, and a substitution in one regularly moves a phenotype attributed to another. A model that never sees the whole transcript cannot represent that coupling — the coupling that decides whether a therapeutic mRNA works.

RiboSpan answers this with native long-context pretraining rather than an inference-time patch. Developed at the Shanghai Innovation Institute with Shanghai Jiao Tong University and the CAS Center for Excellence in Molecular Cell Science, and posted as a preprint in August 2026, it is a 1.61-billion-parameter bidirectional Transformer encoder trained by masked language modeling at a native context of 10,240 nucleotides, one token per nucleotide, with all-to-all self-attention in every layer. The paper sets that design against the two cheaper routes to long RNA and reports what each gives up: rescaling a 1K model's rotary embeddings with YaRN at inference restores contextual differentiation but lets perturbations diffuse far down the transcript, while replacing dense attention with state-space mixing, as HydraRNA does, confines propagation tightly at the cost of weaker context-dependent differentiation.

#Key Features

• Native 10,240-nucleotide pretraining: The long-context checkpoints learn transcript-scale dependencies during pretraining instead of having their positional encodings stretched afterwards, and hold their reconstruction accuracy at full length where extrapolated short-context models fall away.
• Single-nucleotide tokenization: Every nucleotide occupies one position, so representations stay position-aligned — no byte-pair merging of adjacent bases, the compromise other long-RNA encoders make to fit more sequence in.
• Attention-isolated sequence packing: Several RNAs share one training sequence, but attention is masked and positional indices reset at each boundary, so packing buys throughput without leaking context between unrelated transcripts.
• Frozen-representation transfer: Mean-pooled final-layer states separate RNA biotypes directly, with no projection head, classifier or fine-tuning in the loop.
• Conditional discrete diffusion on the same backbone: A masked-diffusion generator conditioned through AdaLN-Zero blocks writes full-length mRNAs, and a synonymous-codon mode rewrites a CDS under whole-transcript context while preserving the protein.

#Technical Details

The backbone is a 32-layer pre-norm encoder with model dimension 2,048, 32 attention heads, SwiGLU feed-forward networks of intermediate size 5,440, rotary position embeddings over 64 dimensions, and a 16-token vocabulary. Pretraining used Megatron-LM in BF16 with AdamW at a global batch of 2,048 sequences — six epochs at 15% masking, then two more at 40%. The corpus draws RNAcentral v26.0 with Ensembl release 115 and Ensembl Genomes release 62, keeping only complete protein-coding transcripts annotated with start and stop codons and both UTRs; after deduplication it holds 67.7 million sequences and 85.9 billion nucleotides, with 90,000 each held out for validation and test. Four matched checkpoints span 1,024 versus 10,240 nt native context at 15% versus 40% masking.

On the held-out mRNA test subset at 10,240 tokens under 15% masking, RiboSpan-10K-15 recovers 94.8% of masked nucleotides against 93.1% for its own extrapolated 1K counterpart and 91.6% for AIDO.RNA-CDS; under 40% masking the continued 10K-40 checkpoint reaches 85.9%. Frozen embeddings scored by leave-one-out 10-NN over 89,955 held-out sequences in 25 biotypes give 0.899 accuracy for 10K-15, ahead of RNA-FM at 0.865, HydraRNA at 0.861, RiNALMo at 0.857 and AIDO.RNA-CDS at 0.805; on the 17,130 sequences longer than 1,024 nt the margin holds at 0.889 against 0.884 and 0.810.

#Applications

The frozen encoder is the immediately usable half: biotype and family assignment, long non-coding transcript triage, and embedding-based retrieval on sequences longer than a 1K encoder can hold. For mRNA engineering, whole-transcript context lets a UTR change and a codon change be scored against each other rather than optimized in separate passes, and the synonymous-codon diffusion mode constrains codon optimization to preserve the protein while still seeing the untranslated regions. Vaccine and protein-replacement programs are the obvious consumers, alongside groups building RNA property predictors on a long-context representation rather than training one.

#Impact

RiboSpan's contribution is the controlled comparison as much as the checkpoint: it isolates native long-context pretraining from the inference-time and architectural shortcuts and measures what each gives up, evidence the long-RNA encoder question has lacked. The limits are worth stating plainly. All three evaluations come from the model's own held-out pretraining split rather than external downstream benchmarks, so transfer to tasks such as secondary-structure or variant-effect prediction is not yet demonstrated. The generative leg is evaluated computationally only, with the design framework's checkpoints and any experimental validation deferred to a later journal publication. And while the code is Apache-2.0, the weights carry a non-commercial licence, the 10K-15 checkpoint is released on request rather than openly, and 10K-40 has not been posted.