SPACE (Species-Profile Adaptive Collaborative Experts)

Most DNA foundation models copy the recipe that worked for text: mask or autoregressively predict nucleotides across a large genomic corpus, then fine-tune the learned representation for downstream tasks. SPACE, from Renmin University of China, argues that this transfers the method without transferring its premise. Natural language carries its meaning in the token sequence; a stretch of DNA does not, because its regulatory function depends on chromatin state, transcription factor occupancy, and histone marks that the sequence alone underdetermines. The paper's claim is that supervised prediction of measured genomic profiles is a better pretraining objective than self-supervised sequence modeling, and that the representation this produces is a stronger general-purpose DNA encoder.

Framing genomic profile prediction as pretraining raises a structural problem that the Enformer line of models handles only implicitly: the training signal spans multiple species and several categorically different assay types, and a single dense trunk has to absorb all of them at once. SPACE — Species-Profile Adaptive Collaborative Experts — answers with two Mixture-of-Experts stages, one routing on species and one routing on profile type, so that shared regulatory grammar and assay-specific idiosyncrasies can occupy different parameters.

The model was posted as a preprint in June 2025 by Zhao Yang, Jiwei Zhu, and Bing Su, and presented as a poster at ICML 2025. Code is on GitHub under MIT and pretrained weights are on Hugging Face, loadable through a from_pretrained call once the repository is installed.

#Key Features

• Supervised pretraining objective: The representation is learned by predicting measured chromatin accessibility, transcription factor binding, histone modification, and CAGE tracks rather than by reconstructing masked nucleotides.
• Species-aware expert routing: A Mixture-of-Experts encoder with four experts and top-3 selection lets human and mouse share most computation while specializing where their regulatory landscapes diverge.
• Profile-grouped decoder: A second MoE stage groups experts by assay type, capturing that accessibility and CAGE signals behave similarly to each other and differently from TF ChIP-seq.
• Sparse activation: At 588M sparsely activated parameters the model beats the 2.5B dense multispecies Nucleotide Transformer on most downstream tasks.
• Released training data: The Basenji corpus, previously distributed in TensorFlow format behind requester-pays cloud storage, is republished in H5 form on Hugging Face.

#Technical Details

SPACE processes 131,072 bp input windows through three stages: a convolutional local-context aggregator following Enformer's design, a species-aware transformer encoder whose feed-forward blocks are replaced by a 4-expert MoE with top-3 routing, and a profile-grouped enhancement decoder using eight cross-profile-type shared experts with two expert-selected groups per profile type. Training uses the same human and mouse data as Enformer, derived from Basenji: 34,021 human and 29,295 mouse training sequences covering 5,313 human tracks (684 DNase/ATAC, 2,131 TF ChIP-seq, 1,860 histone ChIP-seq, 638 CAGE) and 1,643 mouse tracks. The objective is a Poisson negative log-likelihood over track counts plus a mutual-information regularizer, weighted at 0.01, that encourages the species experts to specialize. On the 18-task Nucleotide Transformer downstream benchmark, evaluated by Matthews correlation with the original 10-fold protocol, SPACE sets the best score on 11 of 18 tasks and beats Enformer on all 18. Cross-species transfer on the GUE benchmark shows the largest gains — H3K36me3 rises from 42.41 to 54.82 MCC on yeast and COVID classification from 61.33 to 70.26 F1 relative to Enformer — and on Genomic Benchmarks, mouse enhancer classification improves from 0.835 to 0.905.

#Applications

SPACE is useful wherever a regulatory readout is wanted from sequence alone: prioritizing non-coding variants by their predicted effect on accessibility or transcription factor binding, annotating enhancers and promoters in a newly assembled genome, or supplying a frozen encoder for a downstream classifier on a small labeled dataset. The cross-species results make it particularly relevant to groups working outside human and mouse, where a model whose representation transfers to evolutionarily distant genomes saves the cost of species-specific pretraining. The republished H5 Basenji dataset independently lowers the barrier for anyone training profile-prediction models in PyTorch.

#Impact

SPACE's contribution is an argument as much as a model: it provides direct evidence that supervised functional-genomics pretraining yields better DNA representations than sequence-only self-supervision at equal or smaller scale, which cuts against the prevailing scaling narrative in the field. The Mixture-of-Experts treatment of species and assay type also gives an interpretable handle on what the model has learned, since expert-selection frequencies recover known mechanistic relationships among profile types. The main limitation is coverage: pretraining spans only human and mouse, and the model is far smaller than the largest DNA language models, so how the advantage holds up with more species or more parameters is untested. The released checkpoint also requires the GitHub repository for its model class, and the weights themselves carry no license declaration separate from the MIT-licensed code.