Loading…
Codon language model trained with synonymous-codon-constrained masking, so its embeddings encode nucleotide-level signal, not amino acid identity.
No providers recorded yet. Browse all providers
The genetic code is degenerate: most amino acids have several synonymous codons, so one protein corresponds to an astronomical number of coding DNA sequences. The choice is not neutral: synonymous substitutions change translation speed, mRNA stability, protein yield, and folding, which is why codon optimization is standard practice for biotherapeutics and mRNA vaccines. Codon language models learn these patterns from data, but nearly all train with ordinary masked language modelling over the full 64-codon vocabulary, where a model recovering a masked alanine codon is penalised for predicting a leucine codon. It spends capacity first inferring which amino acid belongs at the position and only then which codon, so codon-level and protein-level semantics end up entangled — visible in codon embeddings that cluster by amino acid identity.
SynCodonLM, from Boehringer Ingelheim with the Rensselaer Polytechnic Institute, removes the confound at the objective level. Before the softmax, logits for every non-synonymous codon are set to negative infinity, so a masked codon can only be recovered from synonyms of the same residue. The amino acid identity is given, not inferred, and every gradient the model receives is about codon choice.
The constraint mirrors deployment exactly: in codon optimization the protein sequence is known in advance and only the DNA is free to vary. Training under that condition removes a mismatch that unconstrained models such as CaLM, CodonBERT, cdsBERT, CodonTransformer, and Mistral-Codon carry into every downstream task.
The backbone is DeBERTa-v2, whose disentangled content/position attention suits coding sequences where context is both local and distal. It uses hidden size 768, intermediate size 3072, 12 layers, 12 attention heads, and a 69-token codon vocabulary, totalling 102.6M parameters. Pretraining masked 15% of codons and ran for 2 epochs and 76,734 steps at an effective batch size of 1,560 across 12 RTX A6000 GPUs — about 12 days and roughly 56.6 billion tokens. Synonym-constrained masked prediction accuracy exceeded 60%.
Evaluation used seven datasets in which the encoded protein is held constant and only codons vary: two measuring mRNA toxicity from codon-optimized GFP in E. coli, two measuring reporter expression under synonymous variation, and three S. cerevisiae mRNA abundance datasets. With all base layers frozen, mean-pooled embeddings fed a regularised linear regression head under five-fold cross-validation. SynCodonLM had the highest mean R² on six of the seven datasets, significantly ahead of all five comparison models (p < 0.05, paired t-test), and ranked second on the remaining one.
The immediate use is codon optimization for biologics and mRNA therapeutics, where the species-conditioned token type embeddings let a sequence be tuned for a specific expression host. A follow-up from the same group applies the frozen checkpoint directly, editing an already-optimized sequence one synonymous codon at a time — each edit taken where synonym-constrained likelihood improves most, then rescored in context — under a 10% cap on positions changed. Across 24 antibody-based clinical-stage therapeutics, this raised recombinant expression in CHO cells for 18, a 75% responder rate, without detectable compromise of product quality or biophysical attributes. Beyond design, the embeddings serve as a feature extractor for regressions over coding sequences whose readout is nucleotide-level rather than protein-level — reporter expression, transcript abundance, toxicity — where protein language models supply no signal.
SynCodonLM's contribution is that a single objective-level constraint, free at inference, produces representations aligned with the biology the model is meant to capture. The work was peer reviewed and published in Nucleic Acids Research. Its evaluation is honest about scope: performance is measured through frozen-embedding linear probes on small reporter datasets confined to E. coli and S. cerevisiae, not end-to-end generative design. The directed-evolution work is a downstream use of the same frozen checkpoint rather than a new model, but it carries the evidence into a mammalian host: the change in model likelihood predicted those expression gains better than heuristic or mRNA-structure descriptors, and codon-level likelihood tracked sampling year across influenza A H1N1 sequences. A second checkpoint, SynCodonLM-V2, is trained on a redundancy-reduced corpus of over 43 million sequences clustered at 90% nucleotide identity with MMseqs2.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.