Protein-Llama-3-8B

A protein generator steered by class tokens has to be retrained whenever the vocabulary of properties grows, and a model trained on amino acids from scratch has no way to read a request written in English. Protein-Llama-3-8B avoids both by never discarding the language model it starts from: the request for a ligase is typed as [Generate Ligase enzyme protein] Seq=< and the model continues the string in amino acids. Because the base weights, the 128,256-token vocabulary and the instruction-following behaviour of Meta's Llama-3-8B are carried through adaptation rather than replaced, the property is expressed as words rather than as an index into a fixed label set.

Aayush Shah and Shankar Jayaratnam at Esperanto Technologies released the model in November 2024 together with a smaller sibling, Protein-Phi-3-mini, built the same way from Microsoft's Phi-3-mini-4k-instruct. Both follow the two-stage recipe introduced by ProLLaMA — continual pretraining with low-rank adapters on raw sequences, then instruction tuning on sequence-property pairs — and the paper's claim is that the recipe carries over to newer, smaller base models at a fraction of the compute. The released checkpoint is the merged product of that training, not a bare adapter, so nothing has to be composed at load time.

The paper's second argument is about energy. The same checkpoint runs on Esperanto's ET-SoC-1, a RISC-V inference chip drawing 25 W against an A100's 300 W, trading throughput for a 60% gain in tokens per second per watt with controllable-generation quality essentially unchanged.

#Key Features

• Property prompts in natural language: Ten classes are supported — the six EC enzyme classes (oxidoreductase, transferase, hydrolase, lyase, isomerase, ligase) plus the SAM-MT, TPHD, thioredoxin-like and CheY-like superfamilies. Prompts outside that set are neither supported nor evaluated.
• Unconditional sampling from the same checkpoint: Prompting with a bare Seq=< produces de novo sequences with no property constraint, so one model covers both modes.
• Parameter-efficient adaptation: LoRA adapters of rank 128 and alpha 256 on the query, key, value, output, up, gate and down projections hold trainable parameters at 4% of the model, letting both stages run on a single NVIDIA A100 80GB in 37 and 48 hours.
• Stock inference tooling: The checkpoint loads through a transformers text-generation pipeline, with no custom tokenizer, alphabet or modelling code to install.

#Technical Details

The architecture is unchanged Llama-3-8B: 32 decoder layers, hidden size 4096, 32 attention heads over 8 key-value heads, an 8192-token context and 8.03B parameters stored in fp16. Stage one is causal language modelling over 2 million sequences drawn from the 2024_03 release of UniRef50; stage two instruction-tunes on roughly 780,000 sequence-property pairs, with the enzyme classes taken from ECPred40 and the four superfamilies reused from ProLLaMA's open-sourced instruction dataset for comparability. Generation uses top-k 40, top-p 0.9 and a repetition penalty of 1.2.

Evaluation is structural rather than sequence-based: 100 sequences per class are folded with ESMFold, candidates below pLDDT 60 are discarded, and the remaining predicted structures are aligned with Foldseek against the Swiss-Prot subset of the AlphaFold Protein Structure Database and the full PDB. Unconditional generation reaches an average pLDDT of 69.75 ± 12.74, above ProLLaMA at 66.49, ProGen2 at 61.07 and ProtGPT2 at 56.32, with an AFDB TM-Score of 0.60. Controllable generation averages a TM-Score of 0.84, but the spread across classes is wide — 0.97 for thioredoxin-like and 0.96 for CheY-like sequences against 0.73 for oxidoreductase and transferase, the classes with the least training data.

#Applications

The model suits the earliest stage of enzyme engineering, where the goal is a diverse pool of plausible starting sequences for a named catalytic class rather than a single optimised design. Sampling is cheap enough to over-generate and filter, and the paper's own pipeline — fold, discard low-confidence candidates, rank the rest by structural similarity to known members of the class — is a reasonable triage template before anything reaches expression and assay. The low-power deployment path matters most where generation runs continuously or on-premises rather than in bursts on rented GPUs.

#Impact

Protein-Llama-3-8B is a preprint result with no peer-reviewed version, no released training or evaluation code, and no license declared on the weight repository, which leaves the Meta Llama 3 Community License inherited from the base weights unaddressed. No demo of this checkpoint has been published either: the HuggingFace Space named for it loads the sibling Protein-Phi-3-mini instead, with no way to switch models. Its contribution is a demonstration rather than a benchmark win: an off-the-shelf instruction-tuned LLM becomes a usable protein generator with 4% of its parameters trained, and runs on hardware an order of magnitude below a datacenter GPU in power draw. The unconditional pLDDT of 69.75 sits at the low end of the band where structures are considered ordered, so the controllable mode is where the model is strongest.