BSM

A gene, its transcript, and the protein it encodes are the same biological object written three ways. A DNA language model and a protein language model, each trained on its own alphabet, never see the correspondence — so when a task hands them a DNA and a protein sequence together, the usual remedy is a dual tower: two frozen encoders and a classifier asked to recover a relationship neither backbone was taught. BSM removes the tower. It tokenizes nucleotides and amino acids into a single vocabulary and joins the two sequences with a <sep> token, so the model evaluates their relationship inside one context.

The second argument is that this does not require scale. Rather than adding parameters, BSM adds data types that carry the cross-modal relationship explicitly: NCBI RefSeq records, which pair a gene with its RNA and protein products along the central dogma; NCBI Gene collections linking a gene to related genes and protein records; and web passages where several biological sequences naturally co-occur. That mixture lowers validation loss for both genes and proteins against a matched single-modal run.

BSM was developed at Microsoft Research Asia with the Chinese Academy of Agricultural Sciences and posted as a preprint in October 2024. It sits alongside two much larger unified models: LucaOne, a 1.8B multi-task encoder over nucleic acids and proteins, and Evo, a 7B genomic model whose DNA training implicitly spans other modalities.

#Key Features

• One vocabulary for three modalities: A single-nucleotide tokenizer covers nucleotides, amino acids, and special tokens, so a DNA-protein pair is one token stream rather than two separately encoded inputs.
• Mixed-modal pretraining corpora: 9.2B tokens of RefSeq DNA/RNA/protein pairs across 15 species, 8.3B tokens of gene-related sequences, and 33M tokens of interleaved sequences filtered out of FineWeb-Edu.
• In-context learning on cross-modal tasks: Without fine-tuning, accuracy on the Central Dogma DNA-protein association task climbs from 0.6216 at four demonstrations to 0.8179 at 32, approaching the fine-tuned model's 0.8245.
• Annealed three-round curriculum: Single-modal pretraining first, then upsampled RefSeq and gene-related data, then interleaved web data; the mixing ratio at each stage was chosen by short annealing probes rather than fixed in advance.
• Two scales from one recipe: The 110M and 270M checkpoints share a tokenizer, data mix, and schedule, isolating the effect of size.

#Technical Details

BSM is a decoder-only autoregressive transformer trained on next-token prediction. BSM-110M uses 12 layers, 12 attention heads, and a hidden dimension of 768; BSM-270M uses 20 layers, 16 heads, and 896. Both use rotary position embeddings at base frequency 100,000, a 1,024-token context, and flash attention. Round one consumes 100B single-modal tokens sampled from LucaOne's corpus (sequences only, from RefSeq, UniProt, and ColabFold); round two adds 24B RefSeq pair and 12B gene-related tokens to 120B single-modal tokens at 10:2:1; round three upsamples a small high-quality mixture at 1:1:10 across RefSeq, gene-related, and web data.

On ncRNA-protein interaction prediction, BSM-270M reaches 0.9603 accuracy against 0.9380 for LucaOne and 0.9332 for a DNABERT-2 plus ESM-2 3B dual tower. It leads on protein-protein interaction (0.9788) and prokaryotic subcellular localization (0.9713, versus 0.9452 for LucaOne), and matches LucaOne on protein stability (SRCC 0.7681 versus 0.7718). Zero-shot ncRNA fitness prediction reaches SRCC 0.261, above Evo-7B's 0.243 and RNA-FM's 0.187; zero-shot protein fitness reaches 0.441, above ProGen2-large but below ESM-2 650M's 0.512. Held-out protein perplexity is 9.47, better than ProtGPT2 700M's 9.75.

#Applications

The tasks BSM is evaluated on are ones a molecular biology group actually runs: screening candidate ncRNA-protein and protein-protein interactions, assigning prokaryotic proteins to subcellular compartments, classifying ncRNAs into families, ranking mutations by predicted effect, and testing whether a DNA region and a protein belong together. Feeding the gene sequence alongside the protein is itself worth something: it raises BSM-110M's zero-shot protein fitness SRCC from 0.407 to 0.423. Two properties make the model practical outside large compute groups — a 270M checkpoint fine-tunes on a single commodity GPU, and the few-shot route lets a group with a handful of labelled examples prompt it instead of assembling a training set. The RefSeq species list, spanning buffalo, camel, macaques, and several fungi alongside human, reflects the agricultural side of the collaboration.

#Impact

BSM's contribution is an argument about data composition rather than architecture: the cross-modal structure a unified sequence model needs can be supplied by curating the right pairings, buying performance otherwise reached by adding an order of magnitude of parameters. The in-context learning result is the sharper one, a capability the larger unified models had not shown. The paper states its limits plainly: only partial RefSeq and gene-related data were used, gene-protein interaction data was left out, and the interleaved web corpus is 33M tokens. The work carries no code or data availability statement, and neither the training code nor the checkpoints have been released, so the results have not been independently reproduced.