Protein-Phi-3-mini

A generator that encodes protein properties as reserved control tokens can only answer requests phrased as an index into its own label set. Protein-Phi-3-mini keeps that vocabulary in language instead: a request for a ligase is typed as [Generate Ligase enzyme protein] Seq=< and the model continues the string in amino acids. Nothing in the base model is discarded to make room for the amino-acid alphabet — the 32,064-token Phi-3 vocabulary, its embedding matrix and its instruction-following behaviour all survive adaptation, and protein sequences are treated as more text.

Aayush Shah and Shankar Jayaratnam at Esperanto Technologies released the model in November 2024 alongside a larger sibling, Protein-Llama-3-8B, built the same way from Meta's Llama-3-8B. Both follow the two-stage recipe introduced by ProLLaMA — continual pretraining with low-rank adapters on raw sequences, then instruction tuning on sequence-property pairs — and both reuse four of ProLLaMA's superfamily classes so the numbers line up against it. The argument specific to this checkpoint is that the recipe survives a much smaller base: it converged in 26 and 35 hours across the two stages, against 37 and 48 for the sibling.

The paper's second argument is energy. The same checkpoint runs on Esperanto's ET-SoC-1, a RISC-V inference chip drawing 25 W where an A100 draws 300 W: throughput falls from 36 to 10 tokens per second, but tokens per second per watt rises from 0.12 to 0.40, with controllable-generation quality unchanged.

#Key Features

• Property prompts in natural language: Ten classes are supported — the six EC enzyme classes (oxidoreductase, transferase, hydrolase, lyase, isomerase, ligase) plus the SAM-MT, TPHD, thioredoxin-like and CheY-like superfamilies. Prompts outside that set are neither supported nor evaluated.
• Unconditional sampling from the same checkpoint: A bare Seq=< with no property clause produces de novo sequences, so one set of weights covers both modes.
• Parameter-efficient adaptation: LoRA adapters of rank 128 and alpha 256 on the attention and MLP projections hold trainable parameters at 4% of the model, so both training stages ran on a single NVIDIA A100 80GB.
• Small inference footprint: Generation occupies 7.36 GB in FP16, which is what puts it within reach of low-power accelerators.
• Stock inference tooling: The checkpoint loads through a transformers text-generation pipeline, with the tokenizer pulled from microsoft/Phi-3-mini-4k-instruct because none ships in the weight repository.

#Technical Details

The architecture is unmodified Phi-3-mini-4k-instruct: 32 decoder layers, hidden size 3072, 32 attention heads over 32 key-value heads, intermediate size 8192, a 4096-token context with a 2047-token sliding window, and 3,821,079,552 parameters in bfloat16. What is released is the merged product of LoRA training rather than a bare adapter. Stage one is causal language modelling over 2 million sequences from the 2024_03 release of UniRef50; stage two instruction-tunes on 779,935 sequence-property pairs, with the enzyme classes taken from ECPred40 and the four superfamilies reused from ProLLaMA's open-sourced instruction dataset. Generation uses temperature 0.2, top-k 40, top-p 0.9 and a repetition penalty of 1.2.

Evaluation is structural rather than sequence-based: 100 sequences per class are folded with ESMFold, candidates below pLDDT 60 are discarded, and the survivors are aligned with Foldseek against the Swiss-Prot subset of the AlphaFold Protein Structure Database and the full PDB. Controllable generation averages a TM-Score of 0.81, identical on the A100 and on ET-SoC-1, with a wide spread across classes: 0.95 for CheY-like sequences and 0.86 for thioredoxin-like and isomerase, against 0.69 for oxidoreductase, the class with the least training data at 23,901 sequences. Unconditional generation records the lowest RMSD in the paper's comparison table, 3.87 against AFDB and 4.38 against the PDB, with TM-Scores of 0.56 and 0.48; the accompanying pLDDT is stated twice and inconsistently, as 60.41 ± 4.56 in that table and 51.30 ± 6.90 in the surrounding text.

#Applications

The model suits the opening stage of enzyme engineering, where the goal is a diverse pool of plausible starting sequences for a named catalytic class rather than one optimised design. Sampling is cheap enough to over-generate and filter, and the paper's own pipeline — fold, discard low-confidence candidates, rank the remainder by structural similarity to known members of the class — is a workable template for that triage before expression and assay. The small memory and power envelope matters most where generation runs continuously or on-premises rather than in bursts on rented GPUs.

#Impact

Protein-Phi-3-mini is a preprint result with no peer-reviewed version, no released training or evaluation code, and no license declared on the weight repository, which leaves the MIT terms covering Microsoft's base weights unaddressed for the derivative. Its contribution is a demonstration rather than a benchmark win: a general-purpose instruction-tuned LLM of under 4 billion parameters becomes a usable property-conditioned protein generator with 4% of its parameters trained, matching its 8-billion-parameter sibling on controllable generation at half the inference footprint.