BiooBang

A gene and the protein it encodes are the same information written twice, and the two writings are not interchangeable. The genetic code is degenerate, so one amino acid sequence corresponds to an astronomical number of coding sequences, and which one a cell uses changes mRNA stability, ribosome speed and protein yield. Protein language models read only the amino acid version; codon models such as CaLM read only the nucleotide one. Neither can consult the other half while it works.

BiooBang trains on both halves at once, in one set of parameters. Every pretraining example is a protein sequence and its own coding sequence concatenated into a single token string, so reconstructing a masked codon can draw on the amino acid it encodes and vice versa. The architecture that makes this practical is a transformer prefix decoder with an adjustable attention-mask matrix: an all-zero block over the first sequence gives bidirectional attention for understanding, a lower-triangular block over the second gives causal attention for generation, and both train under one masked-language-modeling objective. The same checkpoint is therefore an encoder when you want embeddings and an autoregressive generator when you want new sequences.

The model was developed at the University of Science and Technology of China, across its Schools of Life Sciences, Information Science and Technology, and Computer Science, and posted to bioRxiv in October 2024. It joins a small group of models that refuse to pick one alphabet — cdsFM pairs a codon encoder with a codon decoder, ProDMM extends the pairing to flanking non-coding DNA — but BiooBang uses one shared parameter set rather than paired specialist models.

#Key Features

• One vocabulary for two alphabets: Protein tokens inherited from ESM-2 are joined by five nucleotide characters (A, U, C, G, N), giving a 41-symbol vocabulary and one embedding table covering both modalities. The vocabulary is inherited; the weights are trained from scratch.
• Attention mask as a task switch: Following the UniLM construction, concatenating a zero matrix with a strictly lower-triangular one lets bidirectional understanding and unidirectional generation be learned from the same objective.
• Frozen-parameter evaluation: On the property tasks the pretrained weights are held fixed and only a ConvBERT head is trained, so the scores measure the representation rather than the fine-tune.
• Wet-lab validated generation: Coding sequences for mCherry and GFP were produced by beam search, then synthesized, transcribed in vitro and transfected into HEK293T cells.
• Transfer beyond the coding region: Although pretraining sees only CDS and protein, fine-tuning on 5′ UTR inputs still outperforms a model pretrained on 5′ UTRs.

#Technical Details

BiooBang is a 33-layer transformer with 1,280-dimensional hidden states, 20 attention heads, a 5,120-dimensional feedforward width and roughly 650 million parameters, using rotary position embeddings, post-layer normalization and a RoBERTa-style prediction head, with a context of up to 3,079 tokens. Pretraining pairs come from RefSeq, truncated by a sliding window that preserves codon correspondence, with 20% of tokens masked. Under frozen-parameter fine-tuning it reaches Spearman ρ of 0.905 on GB1 fitness, 0.662 on GFP fluorescence and 69.1% fold-classification accuracy, above the 1.15-billion-parameter Ankh. On the CodonBERT mRNA suite it reports 0.545 accuracy for E. coli expression and Spearman ρ of 0.775, 0.838 and 0.827 for mRFP expression, fungal expression and SARS-CoV-2 vaccine degradation. For translation efficiency it exceeds 5′ UTR-LM by about 37% in Spearman ρ on CDS inputs and about 13% on 5′ UTR inputs, and on mean ribosome loading it gains 7% while cutting mean absolute error to roughly a fifth. In HEK293T, sequences generated under a causal language modeling fine-tune expressed about 45% more mCherry and GFP than a conventional CAI-of-1.0 design.

#Applications

The immediate use is codon optimization for recombinant protein production and mRNA therapeutics, where the released HEK293T-fine-tuned checkpoint generates coding sequences tuned to that host rather than to a species-level codon usage table. The frozen embeddings also serve protein property prediction — solubility, fold, subcellular localization, secondary structure — and mRNA property prediction, so one checkpoint covers both the design and the screening halves of an expression-engineering workflow. Groups building reporter constructs, therapeutic mRNAs or high-yield expression systems are its natural users.

#Impact

BiooBang argues that the two languages of the Central Dogma augment each other rather than duplicating information: adding CDS to pretraining improved protein tasks, which single-modality reasoning does not predict, and the transfer to 5′ UTRs points to shared structure across the coding boundary. The work remains a preprint with a single version and has not been peer reviewed, the pretraining corpus and evaluation scale are modest beside larger codon models, and the generative validation rests on two reporter genes in one cell line. Code and both checkpoints were released after the preprint, under an academic-use license and CC BY-NC 4.0 terms that exclude commercial deployment.