TEDlm

TEDlm is a family of protein language models from the UCL Bioinformatics Group that rethinks what a protein language model should read during pretraining. Conventional models such as ESM are trained on full-length protein sequences, which interleave multiple structural domains with linkers and intrinsically disordered regions. Mixing these signals dilutes the fold-specific patterns that carry most of the evolutionary and structural information, and it forces the model to spend capacity learning where one domain ends and another begins.

Instead of full chains, TEDlm pretrains masked language models on structurally-defined domain segments drawn from The Encyclopedia of Domains (TED), a UCL and CATH-Gene3D resource that parses hundreds of millions of domains from the AlphaFold structure database. Training on clean, single-domain units concentrates the learning signal on fold-defining residues. The base model, TEDlm, uses a standard masked-language-modelling objective over domain sequences alone. Its variant, TEDlm3D, adds a structural pretraining signal: a Cα distance-guided contact loss that supervises the model's attention maps so that geometric contacts are learned directly during pretraining.

Developed by Wei, Kandathil, Buchan, and Jones, TEDlm sits in the ESM lineage of sequence-only protein language models but demonstrates that how the training corpus is segmented can matter more than raw model scale. The work is available as a preprint under a CC BY license and awaits peer review.

#Key Features

• Domain-centric pretraining: Trains on structurally-defined domain segments from TED rather than full-length chains, concentrating the objective on fold-specific signal instead of linkers and disordered regions.
• Optional structural pretraining: TEDlm3D augments the masked-language objective with a Cα distance-guided contact loss that supervises attention maps, injecting geometric information directly into the representations.
• Structure signal in the representations: Attention-map and categorical Jacobian probes show the contact signal is encoded in the internal representations themselves, not merely recovered by a trained output head.
• Scale-efficient design: A 650M-parameter TEDlm outperforms the 3B-parameter ESM-2 on remote-homology detection, showing corpus segmentation can outweigh parameter count.
• Stronger zero-shot function prediction: TEDlm variants substantially improve zero-shot Molecular Function annotation over ESM-2 while matching it on biophysical property tasks.

#Technical Details

TEDlm uses a transformer encoder trained with a masked-language-modelling objective on TED domain segments. TEDlm3D shares this architecture but adds a Cα distance-guided contact loss that supervises the attention maps during pretraining. On CATH S40 remote-homology detection at under 40% sequence identity, domain-centric pretraining has a larger effect than model scale: at the final layer, a 650M-parameter TEDlm reaches a remote-homology AUROC of 0.28 versus 0.22 for the 3B-parameter ESM-2, while TEDlm3D reaches 0.50, approaching the structure-based search tool Foldseek at 0.53 from sequence alone at inference time. Because the structural signal is embedded in the representations, TEDlm3D recovers much of a structure-search tool's discriminative power without ever computing or aligning explicit 3D coordinates at query time.

#Applications

TEDlm targets tasks where fold identity, not just local sequence composition, drives the answer. Its representations improve remote-homology detection, letting researchers link proteins that share a fold but have diverged past the point where sequence-based search works, all without running structure prediction or structural alignment. The same domain-intrinsic signal strengthens zero-shot molecular function annotation, supporting functional characterization of uncharacterized proteins and metagenomic sequences. The compact model sizes make these embeddings practical to compute at scale for downstream classification, similarity search, and property prediction pipelines.

#Impact

TEDlm reframes a core assumption in protein language modelling: that ever-larger models trained on full-length sequences are the primary route to better representations. By showing that a 650M-parameter model trained on cleanly segmented domains can beat a model roughly five times larger, and that a lightweight structural loss can bring sequence-only inference close to structure-search performance on remote homology, the work argues for compact, structurally informed models built on curated domain corpora. It also demonstrates a concrete downstream use for The Encyclopedia of Domains beyond cataloguing. As a recent preprint, its benchmarks are in-silico and awaiting peer review, and no dedicated code or model weights release has yet accompanied the paper.