3D molecular pretraining model scaled from 84M to 1.1B parameters on 884M conformations, with power laws linking loss to size, data and compute.
No providers recorded yet. Browse all providers
Language and vision models improved predictably as parameters and data grew, and that predictability — a power law relating loss to scale — is what justified training them at billions of parameters. Molecular pretraining had no such evidence. The pretrained 3D encoders in wide use sat in the tens of millions of parameters, trained on corpora of around 20 million molecules, and the few data-scaling studies that existed used small graph networks. Whether a larger molecular encoder would actually be better, or merely more expensive, was an open question.
Uni-Mol2, from DP Technology with the AI for Science Institute and Peking University, answers it by building the data first and then measuring. The team assembled roughly 884 million 3D conformations spanning 73 million Murcko scaffolds, trained a family of models from 42 million to 1.1 billion parameters on the same objective, and fitted how validation loss falls with model size, training steps and compute. The fitted law predicted the loss of the two largest models from the smaller ones, and downstream accuracy on quantum-chemical property prediction improved with size on most targets.
The architecture departs from the original Uni-Mol. Rather than a single atom-level transformer with a pair-distance bias, Uni-Mol2 follows the two-track design of Uni-Mol+, carrying an explicit pair representation alongside the atom representation and updating both in every layer. The paper was posted in June 2024 and presented at NeurIPS 2024.
The corpus combines the roughly 19 million molecules of the Uni-Mol dataset with the standard-reactivity subset of ZINC20, about 884 million compounds, with conformers generated by RDKit ETKDG and MMFF optimization. Temperature sampling selects 838 million conformations for training, and 520,000 molecules are held out for validation. Pretraining follows Uni-Mol: 15% of atom tokens are masked and predicted, and all coordinates receive Gaussian noise of standard deviation 0.2 Å, with a position-prediction head recovering the clean coordinates and pair distances under L1 losses. Model sizes run 42M, 84M, 164M, 310M, 570M and 1.1B parameters, of which the five from 84M up were released as checkpoints and the 42M model appears only as a point in the scaling study. The largest has 64 layers, embedding dimension 1536 and 96 attention heads, and all use batch size 1024 with AdamW at a peak learning rate of 1e-4. Models up to 310M were trained on 32 NVIDIA A100 GPUs and the two largest on 64.
On QM9 under a scaffold-similarity split, the 1.1B model beat Uni-Mol on all six property groups, an average 27% reduction in error; the 84M model beat it on only four. HOMO, LUMO, gap and ZPVE errors converged as size grew, and heat capacity did not improve monotonically. On COMPAS-1D, a polycyclic aromatic hydrocarbon benchmark, the 1.1B model with atom and bond features reduced error by 14% relative to Uni-Mol. In low-data fine-tuning on HOMO-LUMO gap with training sets half to twice the test-set size, the largest model gave the lowest error at every size.
The checkpoints serve as initializations for molecular property regression and
classification, particularly quantum-chemical and optoelectronic properties of small organic
molecules. The unimol_tools package wraps Uni-Mol2 for representation extraction and
fine-tuning without the Uni-Core training stack, which suits computational chemistry and
screening groups working from SMILES. The released size range lets users trade accuracy against
inference cost; the evaluations cover property prediction only, not docking or conformer
generation.
Uni-Mol2 carried scaling-law analysis over to 3D molecular pretraining and, at 1.1 billion parameters, was the largest molecular pretraining model reported at its release, giving molecular modeling the kind of loss-versus-scale evidence that had guided language modeling. Code and all five published checkpoints (84M to 1.1B) are public under the MIT license, and the model now serves as a pretrained component elsewhere in the catalog: Uni-Mol3 initializes its molecular tokenizer from the 84M Uni-Mol2 checkpoint, Boltz2ESI uses its 3D embeddings for substrate chemistry, and ASOCompass encodes modified oligonucleotide monomers with it.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.