SynCodonLM

The genetic code is degenerate: most amino acids are encoded by several synonymous codons, so one protein corresponds to an astronomical number of coding DNA sequences. The choice is not neutral — synonymous substitutions change translation speed, mRNA stability, protein yield, and folding, which is why codon optimization is standard practice in producing biotherapeutics and mRNA vaccines. Codon language models learn these patterns from data, but nearly all train with ordinary masked language modelling over the full 64-codon vocabulary. A model recovering a masked alanine codon is penalised for predicting a leucine codon, so it spends capacity first inferring which amino acid belongs at the position and only then which codon. Codon-level and protein-level semantics end up entangled, visible in the way these models' codon embeddings cluster by amino acid identity.

SynCodonLM, from Boehringer Ingelheim with the Rensselaer Polytechnic Institute, removes the confound at the objective level. Before the softmax, logits for every non-synonymous codon are set to negative infinity, so a masked codon can only be recovered from the synonymous options that encode the same residue. The amino acid identity is given, not inferred, and every gradient the model receives is about codon choice.

The constraint mirrors the deployment setting exactly: in codon optimization the protein sequence is known in advance and only the DNA is free to vary. Training under that same condition removes a mismatch that unconstrained models such as CaLM, CodonBERT, cdsBERT, CodonTransformer, and Mistral-Codon carry into every downstream task.

#Key Features

• Synonymous-constrained masking: a precomputed vocabulary-by-vocabulary mask matrix maps each codon to its synonymous set; the corresponding row is added to the logits before softmax, restricting the prediction space to codons encoding the same amino acid.
• Species-aware token type embeddings: relative synonymous codon usage is computed per species and k-means clustered within each RefSeq organismal group, assigning 501 token type IDs so species with similar translational machinery share an embedding.
• Nucleotide-structured representations: codon embeddings cluster by wobble base and by suffix dinucleotide rather than by amino acid, a separation confirmed by PERMANOVA at p < 0.001.
• Broad pretraining corpus: 66,503,469 coding sequences from 34,769 species, the largest corpus used for a codon language model to date.
• Openly released artifacts: weights, the pretraining corpus, the benchmark datasets, and the inference code are all public under Apache 2.0, with a model card and a dataset card.

#Technical Details

The backbone is DeBERTa-v2, whose disentangled content/position attention and relative positional embeddings suit coding sequences where context is both local and distal. It uses hidden size 768, intermediate size 3072, 12 layers, 12 attention heads, 1024 maximum positions, and a 69-token vocabulary of 64 codons plus 5 special tokens, totalling 102.6M parameters. Pretraining masked 15% of codons and ran for 2 epochs and 76,734 steps at an effective batch size of 1,560 across 12 NVIDIA RTX A6000 GPUs in 16-bit mixed precision — about 12 days and roughly 56.6 billion tokens — using AdamW with warmup to 2e-4 then cosine decay. Synonym-constrained masked prediction accuracy exceeded 60%.

Evaluation deliberately used seven datasets in which the encoded protein is held constant and only synonymous codons vary: two measuring mRNA toxicity from codon-optimized GFP in E. coli, one measuring GFP expression driven by the first eight codons, one measuring mRFP expression after random synonymous substitution, and three S. cerevisiae mRNA abundance datasets. With all base layers frozen, mean-pooled embeddings fed a regularised linear regression head under five-fold cross-validation repeated across 20 to 80 seeds. SynCodonLM had the highest mean R² on six of the seven datasets, significantly ahead of all five comparison models (p < 0.05, paired t-test), and ranked second on the remaining one.

#Applications

The immediate use is codon optimization for biologics and mRNA therapeutics, where synonymous choices materially affect expression titre, folding, and stability, and where the species-conditioned token type embeddings let a sequence be tuned for a specific expression host. Beyond design, the embeddings serve as a feature extractor for any regression over coding sequences whose readout depends on nucleotide-level rather than protein-level properties — reporter expression, transcript abundance, toxicity — where protein language models supply no signal at all.

#Impact

SynCodonLM's contribution is that a single objective-level constraint, costing nothing at inference, produces representations aligned with the biology the model is meant to capture. The work was peer reviewed and published in Nucleic Acids Research. Its evaluation is honest about scope: performance is measured through frozen-embedding linear probes on small reporter datasets confined to E. coli and S. cerevisiae, not end-to-end generative design, and generalisation to mammalian expression systems remains to be shown. A second checkpoint, SynCodonLM-V2, is trained on a redundancy-reduced corpus of over 43 million sequences clustered at 90% nucleotide identity per species with MMseqs2.