LLaMA-Gene

A promoter-detection dataset arrives as a sequence and a binary label; a secondary-structure dataset as a sequence and a per-residue string; a protein-interaction dataset as a pair of sequences. Each has conventionally required its own classification head bolted onto a pretrained encoder, which is why a group running five gene tasks ends up maintaining five fine-tuned models. LLaMA-Gene's premise is that all five become one task once they are written as instructions: a question in English, a sequence as input, and an answer in text.

Wang Liang at Huazhong University of Science and Technology built the model by teaching the 7B LLaMA checkpoint to read DNA and protein alongside English rather than in place of it. Byte-pair-encoding vocabularies of 30,000 tokens each were trained separately over DNA and over protein sequence, then merged into LLaMA's native 32,000-token vocabulary for a combined size of roughly 91,000. Because one tokenizer covers all three languages, sequences need no wrapper markers such as <protein>…</protein> to signal what the model is looking at; the token identities carry that distinction themselves.

That places LLaMA-Gene between two established groups. Encoder-style nucleotide models such as DNABERT-2 and HyenaDNA are strong per task but need a new head for each one, while single-modality LLaMA adaptations such as ProLLaMA and Protein-Llama-3-8B converse about protein but not DNA. It belongs to the same author's line as GPT2-gene-eng and OmniGene-4, and pursues the same goal as LucaOne — one model spanning nucleic acid and protein — by a chat-shaped route rather than a shared encoder.

#Key Features

• Merged tri-lingual vocabulary: DNA and protein BPE tables are folded into LLaMA's English vocabulary, so the embedding matrix is extended rather than replaced and natural-language competence survives the transition.
• LoRA continued pretraining: adapters on the attention and feed-forward projections, with the token embeddings and language-model head saved, put roughly 10% of parameters in play and make 7B-scale continued pretraining affordable on a single eight-GPU node.
• Unified instruction format: classification, structure prediction, multi-sequence association, function prediction, and regression are all rewritten into Alpaca-style instruction/input/output triples, so one checkpoint serves every task without a task head.
• Regression by binning: continuous targets such as gene expression level are converted to labelled bands around the median, letting a generative model answer numeric questions in the same text format as the rest.
• Three released checkpoints: the DNA-only variant ships in both continued pretrained and instruction-tuned form and the DNA-plus-protein variant in continued pretrained form, alongside the converted instruction corpus.

#Technical Details

The released DNA-plus-protein checkpoint keeps stock LLaMA-7B geometry — 32 layers, hidden size 4096, 32 attention heads — and totals 7,226,634,240 parameters once the enlarged 91,597-token embedding is counted. Vocabulary training used about 1 GB each of model-organism genomes and UniProt sequence. Continued pretraining consumed roughly 16 GB each of DNA fragments of 300 to 1000 bp, Swiss-Prot and TrEMBL protein sequence, and biology papers from S2ORC, the last to limit catastrophic forgetting. Instruction tuning used about 500 MB of converted data: DNA downstream tasks, protein downstream tasks, and UniProt function annotations. Continued pretraining took roughly one week on eight L20 GPUs; instruction tuning runs 2 to 8 epochs in about ten minutes. Across eleven task-and-modality pairs scored by accuracy, LLaMA-Gene lands close to reported task-specific results on DNA work — 0.83 against 0.84 on DNA classification, 0.83 against 0.87 on DNA regression — and well behind on protein association, at 0.63 against 0.87 for protein-protein interaction and 0.71 against 0.91 for DNA-protein pairs. Evaluation drew 2 to 3 datasets per task from the DNABERT-2, LucaOne, InstructProtein, and BiomedGPT benchmark collections.

#Applications

The practical draw is a single conversational endpoint for mixed genomic and proteomic queries: asking whether a sequence is a promoter, what secondary structure a residue window adopts, whether a DNA and a protein sequence correspond, or what GO function a gene carries — all through prompts rather than separate inference scripts. Because the backbone remains a general instruction-following LLM, the checkpoint slots into ordinary LLM tooling such as retrieval-augmented generation and agent frameworks, embedding sequence questions in larger analysis workflows without a bespoke serving path.

#Impact

LLaMA-Gene is best read as a feasibility study for ChatGPT-style interaction with gene sequence, and it is candid about the cost: protein tasks trail specialist models by a wide margin, and the author attributes the gap to limited compute and training data rather than to the method. Reproducing the reported numbers is harder than the release suggests: the instruction-tuned DNA-plus-protein checkpoint the benchmark table rests on was never actually uploaded — its HuggingFace repository contains no weights, config, or tokenizer, and the repository's own SFT test notebook loads the model from a local directory rather than the Hub — leaving the continued-pretrained DNA-plus-protein model and the two DNA-only checkpoints as what a reader can download. The base checkpoint is an early conversion of LLaMA-1, which leaves the redistribution status of the released weights unsettled even though the code carries an MIT license, and adoption remains small. Its durable contribution is the recipe: merge sequence vocabularies into a text model's own tokenizer instead of inventing markers, then unify downstream gene tasks as instructions — an approach the author's later work continues to build on.