ProtTeX

Protein language models have overwhelmingly treated the amino-acid sequence as their only tokenizer. That is a real limitation, because most of the questions biologists ask about a protein — what it binds, where it sits in the cell, whether a mutation destabilizes it — depend on three-dimensional structure. The usual workaround attaches a structure encoder to a frozen language model through a projection layer, which lets the decoder read geometry but never write it: structure stays in a continuous side channel outside the output vocabulary.

ProtTeX, from Changping Laboratory with Soochow University and Peking University, takes the opposite route and discretizes structure into the same token space the language model already uses. A vector-quantized tokenizer maps each residue's local backbone geometry to one of 512 codes, and twenty further tokens cover the amino-acid alphabet. Those 532 new tokens are added to a Meta-Llama-3-8B base model, after which sequence, structure, and natural-language text are trained together under nothing but next-token prediction — no adapter, no contrastive alignment stage, no task-specific head.

Putting structure in the output vocabulary as well as the input makes reasoning and generation symmetric. The same decoder reads a PDB file as text, uses structure tokens as intermediate reasoning steps before committing to an answer, and emits new structure tokens that decode back to coordinates. ProtTeX is the backbone that ProtTeX-CC later compresses to fit worked examples into the context window.

#Key Features

• Unified discrete vocabulary: Sequence, backbone conformation, and text occupy one token stream, so a single autoregressive objective covers understanding, prediction, and generation without modality-specific machinery.
• Structure as chain of thought: The model can be prompted to predict a protein's structure tokens first and then answer from them; doing so raised subcellular localization accuracy by 49.6% over answering from sequence alone.
• Multi-conformation sampling: Sampling repeatedly from the same input recovered two distinct folds, each above 0.7 TM-score, for six of nine fold-switching proteins tested — behavior single-answer structure predictors do not exhibit.
• Controllable design and editing: Because structure tokens are generated as output, the model produces new sequence-structure pairs conditioned on a functional description or an existing scaffold.
• Standard LLM tooling: Training and inference use ordinary decoder-only pipelines, and the released repository runs function prediction on an arbitrary PDB file through two scripts.

#Technical Details

Pretraining used 3.36 million paired sequence-structure proteins drawn from AlphaFold DB v4 (2.27 million single-chain models), Swiss-Prot (541,327), and the RCSB PDB (551,957). Supervised fine-tuning covered four task corpora: 429,201 protein function question-answer samples, 192,617 design samples, 264,370 structure-aware annotation samples, and 2,821,238 structure prediction samples. The released checkpoint carries roughly 8.03 billion parameters.

On protein function question answering across six inquiry types — molecular function, subcellular location, biological process, domains and motifs, feature overview, and multi-attribute annotation — ProtTeX reaches 71.73 exact-match Jaccard index and 57.89 ROUGE-L, against 65.40 and 56.98 for the fine-tuned ProtT3 baseline, 22.06 and 38.15 for Llama2-molinst, and 11.31 and 14.89 for BioMedGPT. Structure prediction uses beam search ranked by perplexity, which correlates negatively and significantly with TM-score. Design case studies on cytidylate kinase and dITP/XTP pyrophosphatase generated twenty sequences each with under 0.8 identity to training entries; refolding with AlphaFold 3 confirmed the intended folds and showed conserved active-site residues.

#Applications

The practical entry point is annotation of a structure with no reliable functional label. A researcher tokenizes an experimental or predicted PDB file and asks free-form questions about function, localization, or biological process, receiving text answers grounded in geometry rather than sequence homology. The generative direction supports the reverse workflow: specifying a desired function or a scaffold to preserve, then sampling sequence-structure pairs that satisfy it. The conformational sampling behavior is useful for proteins suspected of adopting more than one stable fold, where a single predicted structure is misleading.

#Impact

ProtTeX's contribution is architectural rather than incremental: it demonstrates that a general decoder-only language model, given a discrete structural alphabet, handles protein reasoning, structure prediction, and design under one unmodified training recipe. That framing has been taken up directly — ProtTeX-CC builds its in-context learning capability on this exact backbone. The work is peer reviewed, published in the Journal of Chemical Information and Modeling in June 2025; the ACS version of record sits behind a subscription, so the March 2025 arXiv posting remains the freely readable text. Its artifacts are public, with code under Apache-2.0 and weights on HuggingFace, though the weights repository ships an unfilled model-card template, leaving their license terms unstated. Evaluation is entirely computational, and the design results rest on two case-study proteins validated by refolding rather than experiment.