cdsBERT

The six codons that encode serine — UCU, UCC, UCA, UCG, AGU and AGC — collapse into a single letter S before a protein language model ever reads the sequence. That collapse throws away information the cell demonstrably uses: synonymous codon choice sets translation speed through tRNA availability, varies systematically across phylogeny and organelle, and in documented cases changes the domain structure the finished protein adopts. cdsBERT asks whether a protein language model given codons instead of amino acids recovers any of it.

Training such a model from scratch is impractical: curated coding-sequence corpora are far smaller than amino acid databases, and starting over forfeits everything a pretrained protein language model already knows. cdsBERT converts an existing model instead. Developed by Logan Hallee, Nikolaos Rafailidis and Jason P. Gleghorn at the University of Delaware and posted to bioRxiv in September 2023, it takes ProtBERT-BFD from the ProtTrans suite and replaces its 30-token amino acid vocabulary with a 69-token codon vocabulary through a three-stage procedure the authors name MELD — Masked Extended Language Distillation.

The seeding stage doubles as the experiment's own control. Every synonymous codon is initialized to exactly the embedding of the amino acid it encodes, so any distance that later opens between UCU and AGC must have been learned from coding sequences. Measuring that movement is how the paper argues codons carry signal the amino acid alphabet erases.

#Key Features

• Codon vocabulary on a pretrained protein encoder: The token embedding matrix widens from 30 entries to 69 — 64 codons plus five special tokens — while the transformer body keeps ProtBERT-BFD's learned protein knowledge.
• MELD vocabulary extension: Seeding, masked language modeling on coding sequences, then contrastive student-teacher distillation — a recipe the authors present as general to any pair of translatable vocabularies.
• Seeded synonymous codons as a built-in control: Because all six serine codons start at the same point, separation measured afterwards is attributable to the data, not to initialization.
• Frozen-embedding readout: Mean-pooled last hidden states feed classifiers directly, so function prediction needs no fine-tuning of the encoder.
• Single-character codon encoding: A custom code maps each of the 64 codons to one ASCII character, cutting corpus storage by a third and rendering open reading frames visible as matched bracket pairs.

#Technical Details

cdsBERT is a 420-million-parameter BERT encoder: 30 transformer layers, 1024-dimensional embeddings, 16 attention heads, and a 69-token vocabulary. The corpus combines the NIH Consensus CDS project with over 300 additional Ensembl genomes, filtered to sequences that begin with ATG, end in a stop codon, have a length divisible by three, and encode at most 1000 amino acids — roughly 9.5 million sequences, of which about 4 million were seen during masked language modeling, less than a single epoch. Distillation then contrasted 500,000 codon sequences against a frozen Ankh-base teacher reading the matching amino acid sequences, a trainable linear layer projecting Ankh's 768 dimensions into cdsBERT's 1024, yielding the cdsBERT+ checkpoint.

Evaluation used enzyme commission number prediction over 2,629 CCDS proteins carrying a single EC annotation, scored by 10-fold cross-validation on KNN and SVM classifiers fit to frozen embeddings. After masked language modeling alone, cdsBERT reached 0.624 KNN accuracy on full EC numbers, well below ProtBERT's 0.747 — consistent with an undertrained new vocabulary. Distillation lifted cdsBERT+ to 0.787, ahead of the identically distilled ProtBERT+ at 0.781, and it led every KNN metric plus SVM accuracy. ProtBERT+ stayed ahead on SVM F1 and on the easier first-EC-digit task, and Ankh-base outscored both students throughout. Codon embedding movement correlated moderately negatively with average usage frequency across phylogeny: frequently used codons stayed closest to their parent amino acid.

#Applications

The two released checkpoints serve as feature extractors wherever the coding sequence, not just the translated protein, is in hand — genome annotation, enzyme function assignment, and property prediction on recombinant constructs whose synthesized DNA is known exactly. The base checkpoint also performs codon-level mask filling. Beyond the model, MELD is a reusable way to retarget an existing language model to a related vocabulary without repeating pretraining.

#Impact

cdsBERT established that a codon vocabulary can be grafted onto a protein language model cheaply, and that synonymous codons then separate in embedding space in ways that track usage bias across phylogeny. The authors are candid that the downstream gains are small: the codon-vocabulary advantage on full EC prediction is under a percentage point, most protein information is adequately summarized by amino acids, and both models appear to hit a ceiling set by architecture and dataset size. They present cdsBERT as a proof of concept and call for a purpose-built codon foundation model alongside better CDS coverage in protein databases. It has since become a reference point for that line: codon language models including SynCodonLM cite it among the unconstrained codon models they measure themselves against.