Loading…
Molecular foundation model for drug discovery that masks entire data modalities and predicts their latent embeddings from the remaining ones.
No providers recorded yet. Browse all providers
Self-supervised learning on molecules usually borrows its recipe from vision: make two views of the same input, then train the encoder to represent them alike. For a molecule that means masking an atom or a bond — and molecular property landscapes are discontinuous. A single substituent swap can move potency by orders of magnitude, the phenomenon medicinal chemists call a property cliff, so the augmented pair the model is told to collapse together may behave nothing alike in an assay.
Mol-JEPA's answer is to stop perturbing the molecule. A compound is instead represented as a set of parallel modalities — its graph, computed descriptors, embeddings borrowed from other pretrained chemistry models, a cell-painting phenotype vector, a binding-affinity profile, a block of measured ADMET endpoints — and pretraining masks whole modalities at a time. A transformer predictor reconstructs the missing modality's latent representation from the ones that remain, so the model is never asked to accept a chemically invalid structure as an equivalent view. It comes from Boehringer Ingelheim with the University of Tübingen, UT Austin and Brown University.
Predicting in latent space rather than input space is the Joint Embedding Predictive Architecture formulation, which avoids the negative sampling contrastive molecular models depend on; Sketched Isotropic Gaussian Regularization, the anti-collapse term from LeJEPA, keeps those latents from collapsing to a trivial subspace.
Pretraining merges 4.66 million unique compounds from ChEMBL bioactivity, Therapeutics Data Commons ADMET measurements, the PCBA-1328 screening subset, the ∇²DFT quantum-chemistry set and two million unlabelled Enamine REAL molecules. Fourteen modalities fall into three groups: embeddings distilled from other pretrained models, including Boltz-2 binding-affinity features computed against nine targets from the Bowes-44 off-target panel; computed descriptors such as 227-dimensional MOE features, ECFP4 fingerprints and GFN2-xTB simulations; and raw experimental label vectors spanning 306 ChEMBL, 1,328 PCBA and 672 TDC endpoints. Three learnable encoders — atom, graph and vector — project everything into a shared space, and a transformer predictor is trained under mean-squared error on the masked modalities plus a SIGReg term over 1,024 random one-dimensional projections. The released checkpoint holds 45.4 million parameters after an Optuna hyperparameter search.
Evaluation uses blind challenges and industrial data rather than the criticized legacy benchmarks: OpenADMET ExpansionRx, the ASAP/Polaris antiviral challenge, the OpenADMET pregnane-X receptor challenge and Biogen ADME, split by Taylor-Butina clustering at a 0.65 Tanimoto threshold. Averaged mean absolute error is 0.402 for the best Mol-JEPA variant, against 0.453 for TabICLv2 on AlvaDesc, 0.468 for LightGBM, 0.471 for CheMeleon, 0.519 for Chemprop, 0.559 for random forest and 0.576 for CLAMP. Binned by nearest-neighbour Tanimoto similarity to the training set, the gap widens as similarity falls.
The practical use is as a featurizer for small-data problems in early drug discovery — potency, solubility, microsomal clearance, permeability, plasma protein binding — where a project has hundreds of measured compounds, not tens of thousands. A team embeds its series once and fits a light probe on the CLS token; a tabular in-context model or a two-layer transformer over the modality tokens beats a linear probe, and LoRA fine-tuning underperforms both. Because the pretraining corpus carries off-target and phenotypic signal, the embeddings also suit early safety triage.
Mol-JEPA answers a persistent negative result: pretrained molecular representations have repeatedly failed to beat well-tuned descriptor baselines, and augmentation strategies inherited from other domains are a plausible cause. Ablations support the framing — downstream error falls as modalities are added from two to four to eight, and leave-one-modality-out analysis ranks the trainable graph encoder first, then the MOE and ECFP4 fingerprint modalities, with binding-affinity and cellular-phenotype channels contributing less but measurably. The authors are candid about the limits: most ablations ran on a 100,000-molecule subset, removing a modality at inference barely changes results because the objective encourages redundancy, and adding some modalities can hurt. Code, the checkpoint and the assembled multimodal dataset are all released, but under a non-commercial license, and the work remains a preprint.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.