ProtTeX-CC

Protein large language models that discretize both amino-acid sequence and backbone structure into token sequences of residue length — ProtTeX is the canonical example — let a single decoder reason over both modalities in one stream. The design carries two structural costs. Concatenating sequence and structure tokens roughly doubles the length of every protein input and severs the residue-level alignment between the two views. And because such models are trained on single-protein inputs inside a bounded context window, they cannot accept worked examples at inference time, which forecloses the in-context learning that gives general-purpose LLMs much of their generalization.

ProtTeX-CC, from Soochow University with Changping Laboratory, Zhejiang University, and Peking University, is a two-stage instruction-compression framework that removes both constraints without touching the backbone weights. The first stage fuses the sequence and structure representations of each residue into a single joint embedding, halving protein input length. The second stage compresses an entire demonstration — protein, question, and answer — into the latent space of its last few linguistic tokens, so that sixteen worked examples fit in the space the original model needed for one.

The result is a protein multimodal LLM that answers function questions from retrieved neighbors rather than from fine-tuning on the target distribution. That matters most on proteins outside the training corpus, where the paper's fine-tuned baselines degrade sharply while the compressed few-shot model gains double-digit percentages.

#Key Features

• Joint sequence-structure compression: Residue-level fusion of the amino-acid token and the structure token into one embedding halves the protein input length while holding in-domain accuracy flat against the uncompressed backbone.
• Demonstration self-compression: Each full in-context demonstration is aggregated into the last few linguistic tokens of its own encoding, taking the average demonstration from 751 tokens to 16 or fewer — roughly a 93.68% reduction in total prompt length at 16 shots.
• Frozen backbone: Stage one adapts the backbone with LoRA; stage two adds a single trainable projection layer and freezes everything else, so the added parameter count is small relative to the 7B-parameter decoder.
• Retrieval-ready latents: The compressed demonstration embeddings are semantically meaningful enough to drive similarity search, matching or exceeding BM25 lexical retrieval when selecting which examples to include.
• Out-of-domain transfer without fine-tuning: Trained only on one benchmark, the model improves on a second, unseen protein QA benchmark purely by retrieving compressed demonstrations.

#Technical Details

The underlying ProtTeX backbone encodes amino-acid sequences with a 3B-parameter ESM-2 encoder and backbone conformations with the ProToken structure tokenizer, decoding with BioMedGPT-LM-7B. ProtTeX-CC trains both compression stages on PFUD, a protein question-answering corpus of 426,928 QA pairs built from Swiss-Prot entries, totaling roughly 320.8 million tokens after ProtTeX tokenization and split 90/5/5 into train, validation, and test.

Evaluation covers five protein understanding subtasks — molecular function, subcellular localization, protein captioning, and multi-attribute annotation among them — scored with an exact-match Jaccard index (EMJI) alongside BLEU and ROUGE. On the in-domain PFUD test set, ProtTeX-CC reaches 72.93 EMJI against 71.73 for the uncompressed backbone, and the joint-embedding stage alone holds 71.59 EMJI at half the input length. Naive in-context prompting of the original model collapses to 17.42 EMJI, since it can fit at most one example and was never trained on interleaved multimodal inputs. On the out-of-domain UniProtQA benchmark, evaluated with no additional fine-tuning, ProtTeX-CC gains 11.26% in EMJI, 11.89% in BLEU-2, and 10.23% averaged across ROUGE-1/2/L over the backbone. The reported configuration retains 16 tokens per demonstration across 16 demonstrations.

#Applications

The target use is functional annotation of proteins that existing databases describe poorly or not at all. A researcher supplies a sequence and predicted or experimental backbone coordinates and asks free-form questions about molecular function, subcellular localization, or catalytic activity; the model conditions its answer on compressed examples retrieved from an annotated corpus. Because the retrieval corpus is swappable and the backbone is frozen, the same checkpoint can be pointed at a new protein family or annotation vocabulary by changing the demonstration pool rather than retraining.

#Impact

ProtTeX-CC makes a specific and transferable argument: the barrier to in-context learning in multimodal protein LLMs is prompt length, not model capability, and it can be lifted with two lightweight modules rather than a larger context window or a bigger backbone. The larger gains out of domain than in domain support that reading — demonstrations help most where the model has least memorized knowledge. The work is an arXiv preprint, but its artifacts are public: the implementation, the joint-embedding model and projection checkpoint released under MIT on HuggingFace, and the PFUD corpus on Zenodo under CC-BY-4.0. The code repository carries no license, leaving reuse terms undefined. Results are also confined to protein question answering; whether the same compression carries to structure generation or design tasks built on the same tokenization remains untested.