HAD (Hybrid Architecture Distillation)

DNA foundation models have followed the same scaling logic as language models: more genomes, more parameters, better downstream accuracy. That trend leaves genomics labs fine-tuning half-billion-parameter encoders for tasks — histone-mark classification, enhancer annotation, promoter detection — whose inputs are under a kilobase. Smaller alternatives exist, but they have generally traded away several points of accuracy for their compactness.

HAD (Hybrid Architecture Distillation), from the University of Chinese Academy of Sciences, the Institute of Automation, CAS, Zhongguancun Academy, and Nanjing University, narrows that gap by distilling Nucleotide Transformer v2-500M into a 1.1M-parameter student — a roughly 450-fold compression — and, on a majority of benchmark tasks, ending up ahead of the teacher rather than behind it. The student is a hybrid: a bidirectional Gated Delta Net backbone, which gives linear-time sequence mixing with a gated delta-rule memory update, plus one self-attention layer for precise long-range lookups.

The distillation objective is what makes the compression work. Rather than distilling only at masked positions, HAD runs two branches at once during masked language modeling: it aligns the student's hidden states to the teacher's on the visible tokens while simultaneously reconstructing the masked ones. An ablation confirms that both branches and the hybrid backbone are necessary — removing either the attention layer or the visible-token distillation degrades downstream accuracy.

#Key Features

• Grouping masking across tokenizers: The teacher uses 6-mer tokens and the student is character-level, so masks are chosen as whole 6-mer units on the teacher side and then mapped down to the corresponding nucleotide positions, which keeps alignment consistent and prevents information leakage across the two vocabularies.
• Dual-branch pretraining: Visible-token feature distillation and masked-token reconstruction are optimized together, giving the student both the teacher's representation geometry and its own generative pretraining signal.
• Hybrid linear-plus-attention backbone: Four bidirectional Gated Delta Net blocks handle sequence mixing efficiently while a single Flash Attention layer supplies content-based retrieval that the recurrent blocks approximate less precisely.
• Reverse-complement equivariance: Pretraining uses reverse-complement data augmentation and fine-tuning applies post-hoc conjoining, matching the inductive bias established by Caduceus.

#Technical Details

The student has 4 GDN blocks with a hidden dimension of 128, one self-attention layer, and a gated MLP, totalling approximately 1.1 million parameters against the teacher's 498.3M. A projection layer maps the student's 128-dimensional states to the teacher's 1,024 dimensions, and student representations are pooled over non-overlapping 6-mer windows to match sequence lengths. Pretraining uses the human reference genome with the same data and train/validation split as HyenaDNA and Caduceus, at a sequence length of 1,026 nucleotides — chosen for divisibility by 6 and to cover the downstream task inputs.

Evaluation adopts the Caduceus fine-tuning protocol so baseline numbers are directly comparable: MCC for histone-marker tasks, F1 for enhancer, promoter, and splice-site annotation, and top-1 accuracy on the Genomic Benchmark. On the Nucleotide Transformer Benchmark, HAD takes the top position on 11 of 18 tasks despite being the smallest model evaluated, and it exceeds NTv2-500M on all ten histone-marker tasks — by 24.3 points on H3K4me2, 23.3 on H3K4me3, and 15.9 on H4ac — and on both enhancer-annotation tasks. It falls short of the teacher on promoter annotation (about 0.8 points) and splice-site annotation (7 to 12 points). On the Genomic Benchmark it is best on five of eight tasks with an average of 0.875, the highest among reported baselines including HyenaDNA at 1.6M parameters, Caduceus at 1.9M, and DNABERT-2 at 117M.

#Applications

The practical draw is deployment cost. A 1.1M-parameter encoder fine-tunes on a single consumer GPU in minutes, which suits labs running many task-specific classifiers over regulatory genomics data — chromatin-mark prediction, enhancer and promoter calling, open-chromatin-region classification — and settings where inference must run at genome scale or on constrained hardware. The uneven task profile matters for model selection: HAD is the stronger choice for chromatin and enhancer work and the weaker one for splice-site prediction, where the teacher's larger capacity still wins.

#Impact

HAD is evidence that the accuracy of a genomic language model is not simply a function of its parameter count, and that a well-matched architecture combined with a richer distillation signal can extract more usable representation from a large teacher than the teacher itself deploys on short sequences. That result is most relevant to a field where downstream inputs are frequently sub-kilobase and where compute is a real constraint on how many tasks a group can attempt. The caveats are concrete: pretraining is on the human reference genome only, so multi-species and long-range capabilities are untested; the gains do not extend to splice-site tasks; and the preprint carries no code or weight availability statement, with no public repository or checkpoint accompanying it, so the results cannot currently be reproduced or the model reused.