BigBang-Proton

BigBang-Proton is an autoregressive sequence model from SuperSymmetry Technologies that treats genome modeling as one task inside a single cross-disciplinary scientific learner. Instead of training a dedicated DNA language model, the developers mixed 27.3 billion nucleotide tokens from the OpenGenome corpus with simulated particle-physics jets, crystal structures from MPtrj, 300 million synthetic arithmetic problems, sensor time series, stock prices, Python code, and the SlimPajama text corpus, then trained one 1.5-billion-parameter model on all of it by next-token prediction.

The design rests on a specific diagnosis: that generic language-model tokenization is what breaks large language models on scientific data. Byte-pair encoding fragments numbers inconsistently and shatters DNA's four-letter alphabet into arbitrary subwords. BigBang-Proton replaces BPE with Binary Patch Encoding, in which every input — English prose, a lattice vector, or a nucleotide string — is consumed as raw bytes grouped into fixed-size patches, and replaces full self-attention with Monte Carlo Attention, a layer-wise inter-patch delegation scheme whose effective context grows as P^(N+1) for patch size P and depth N while compute stays linear in sequence length.

For biologists, the reason to look at this model is the head-to-head against Evo, which was pretrained on the same OpenGenome data. The work is a self-published technical report from a Shanghai AI company, released with code and checkpoints, and follows its earlier BigBang-Neutron particle-physics model. It sits alongside general scientific language models such as Galactica, but on a different premise: numerical and sequence data as first-class training modalities rather than text about science.

#Key Features

• Byte-level tokenizer-free encoding: Binary Patch Encoding maps all inputs to raw bytes over a 259-symbol vocabulary (256 byte values plus BOS, EOS, and padding), so nucleotides, digits, and text share one representation and numbers are never fragmented across subword boundaries.
• Monte Carlo Attention: Each layer generates delegate tokens per patch via 1×1 convolution and distributes them to neighboring patches, giving 20 layers an effective context on the order of 10^30 bytes at linear cost — aimed at physical structures far larger than any natural-language context window.
• Theory-Experiment Learning: Textual corpora and large-scale numerical experimental data are aligned in a single context window, letting downstream tasks be issued as natural-language instructions that resolve to classification, regression, or generation through next-token prediction.
• One architecture across five domains: A single pretraining run feeds downstream fine-tuning for jet-origin identification, inter-atomic potential prediction, water-quality forecasting, 50-digit arithmetic, and genome modeling, rather than a separate model design per discipline.
• Released weights under a community license: Inference code and .pth checkpoints, including a biology checkpoint, are public, and the UniverseBench multitask test datasets are released separately under Apache 2.0.

#Technical Details

The genomics portion uses OpenGenome — over 80,000 bacterial and archaeal genomes plus millions of predicted prokaryotic phage and plasmid sequences, with eukaryote-infecting viruses excluded — sampled at 27.3 billion tokens for pretraining and a further 82.8 billion during downstream fine-tuning, for 110.1 billion nucleotide tokens total against Evo's 300 billion. Maximum sequence length for the genomic runs is 16,384 tokens. In the compute-optimal scaling comparison, BigBang-Proton 1.5B reaches evaluation perplexity 2.8 at 7×10^19 FLOPs, below Evo 7B's saturated 3.1, while LLaMA 3 1.2B and Qwen2.5 1.5B fine-tuned with LoRA on 3 billion OpenGenome tokens reach only 6.23 and 6.89. On next-base prediction with 512 bp of context, the model scores 56% versus roughly 25% (chance) for DeepSeek-R1, ChatGPT o1, Claude 3.5 Sonnet, and Kimi.

Three fitness-prediction benchmarks are reported, each as a supervised fine-tune on one deep mutational scanning dataset: Spearman 0.785 on IF-1 coding-sequence DMS in E. coli, 0.68 on a ribozyme ncRNA DMS, and 0.72 on promoter-driven gene expression. The Evo figures quoted for comparison (0.67, 0.65, 0.68) are that model's best zero-shot results across six or seven datasets, so the two settings differ.

#Applications

The genomics capability is prokaryotic and sequence-level: nucleotide-resolution sequence completion, ranking coding and non-coding mutations by predicted fitness, and estimating expression from bacterial promoter sequences. That suits protein and RNA engineering in microbial hosts, synthetic-biology promoter tuning, and variant triage in bacterial genomes. The broader appeal is to groups wanting one model spanning genomics and physical-science modalities in a single instruction-driven interface.

#Impact

BigBang-Proton is an early, concrete test of whether a tokenizer-free byte model trained across disciplines can match domain-specific genomic foundation models, and its perplexity advantage over a model seven times its size on identical data is a real argument that BPE, not scale, limits LLMs on DNA. Several caveats bound that claim: the report is not peer reviewed, the fitness benchmarks are fine-tuned single-dataset results compared against zero-shot baselines, training data covers prokaryotes only, and the BigBang-Proton Community License restricts use to non-commercial research. Adoption so far is modest, and the released checkpoints ship without the training pipeline.