Loading…
Scientific multitask language model whose byte-level genome modeling beats Evo 7B on DNA perplexity and mutation-effect prediction at 1.5B params.
No providers recorded yet. Browse all providers
BigBang-Proton is an autoregressive sequence model from SuperSymmetry Technologies that treats genome modeling as one task inside a single cross-disciplinary scientific learner. Instead of training a dedicated DNA language model, the developers mixed 27.3 billion nucleotide tokens from the OpenGenome corpus with simulated particle-physics jets, crystal structures from MPtrj, 300 million synthetic arithmetic problems, sensor time series, stock prices, Python code, and the SlimPajama text corpus, then trained one 1.5-billion-parameter model on all of it by next-token prediction.
The design rests on a specific diagnosis: that generic language-model tokenization is what breaks large language models on scientific data. Byte-pair encoding fragments numbers inconsistently and shatters DNA's four-letter alphabet into arbitrary subwords. BigBang-Proton replaces BPE with Binary Patch Encoding, in which every input — English prose, a lattice vector, or a nucleotide string — is consumed as raw bytes grouped into fixed-size patches, and replaces full self-attention with Monte Carlo Attention, a layer-wise inter-patch delegation scheme whose effective context grows as P^(N+1) for patch size P and depth N while compute stays linear in sequence length.
For biologists, the reason to look at this model is the head-to-head against Evo, which was pretrained on the same OpenGenome data. The work is a self-published technical report from a Shanghai AI company, released with code and checkpoints, and follows its earlier BigBang-Neutron particle-physics model. It sits alongside general scientific language models such as Galactica, but on a different premise: numerical and sequence data as first-class training modalities rather than text about science.
.pth checkpoints, including a biology checkpoint, are public, and the UniverseBench multitask test datasets are released separately under Apache 2.0.The genomics portion uses OpenGenome — over 80,000 bacterial and archaeal genomes plus millions of predicted prokaryotic phage and plasmid sequences, with eukaryote-infecting viruses excluded — sampled at 27.3 billion tokens for pretraining and a further 82.8 billion during downstream fine-tuning, for 110.1 billion nucleotide tokens total against Evo's 300 billion. Maximum sequence length for the genomic runs is 16,384 tokens. In the compute-optimal scaling comparison, BigBang-Proton 1.5B reaches evaluation perplexity 2.8 at 7×10^19 FLOPs, below Evo 7B's saturated 3.1, while LLaMA 3 1.2B and Qwen2.5 1.5B fine-tuned with LoRA on 3 billion OpenGenome tokens reach only 6.23 and 6.89. On next-base prediction with 512 bp of context, the model scores 56% versus roughly 25% (chance) for DeepSeek-R1, ChatGPT o1, Claude 3.5 Sonnet, and Kimi.
Three fitness-prediction benchmarks are reported, each as a supervised fine-tune on one deep mutational scanning dataset: Spearman 0.785 on IF-1 coding-sequence DMS in E. coli, 0.68 on a ribozyme ncRNA DMS, and 0.72 on promoter-driven gene expression. The Evo figures quoted for comparison (0.67, 0.65, 0.68) are that model's best zero-shot results across six or seven datasets, so the two settings differ.
The genomics capability is prokaryotic and sequence-level: nucleotide-resolution sequence completion, ranking coding and non-coding mutations by predicted fitness, and estimating expression from bacterial promoter sequences. That suits protein and RNA engineering in microbial hosts, synthetic-biology promoter tuning, and variant triage in bacterial genomes. The broader appeal is to groups wanting one model spanning genomics and physical-science modalities in a single instruction-driven interface.
BigBang-Proton is an early, concrete test of whether a tokenizer-free byte model trained across disciplines can match domain-specific genomic foundation models, and its perplexity advantage over a model seven times its size on identical data is a real argument that BPE, not scale, limits LLMs on DNA. Several caveats bound that claim: the report is not peer reviewed, the fitness benchmarks are fine-tuned single-dataset results compared against zero-shot baselines, training data covers prokaryotes only, and the BigBang-Proton Community License restricts use to non-commercial research. Adoption so far is modest, and the released checkpoints ship without the training pipeline.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.