Loading…
Chemical language model that tokenizes each atom by its functional group, giving transferable embeddings for molecular property prediction.
No providers recorded yet. Browse all providers
Salicylic acid and aspirin share a skeleton and differ by one functional group — a hydroxyl replaced by an acetyl — and that substitution is what lets aspirin irreversibly inhibit cyclooxygenase. A conventional SMILES tokenizer cannot see that difference in kind: it spells both molecules out of the same dozen atom symbols, and the token O stands equally for a hydroxyl, an ester and a ketone oxygen. FARM intervenes there, labeling every atom with the functional group it belongs to before any learning happens, so the string carries O_hydroxyl and O_ketone as separate tokens and the chemistry a medicinal chemist reads off a structure is in the input rather than something the model must rediscover.
The annotation is then used twice. The labeled string feeds a BERT masked language model that learns atom-level context, while the same labels fragment the molecule into a functional-group graph in which a ketone is one node rather than two atoms. A graph neural network trained on that graph by link prediction learns which groups tend to be connected — the molecular backbone — and contrastive learning pulls the two views into one embedding space.
FARM was developed at the University of Illinois Urbana-Champaign — the Siebel School of Computing and Data Science with the chemistry and chemical engineering departments — and Texas A&M University, and released in October 2024 under the NSF Molecule Maker Lab Institute. It sits between SMILES language models such as MoLFormer-XL and ChemBERTa and graph contrastive models such as MolCLR and GROVER, keeping both views but making the functional group, not the atom, the unit each is built on.
The language branch is a 103M-parameter BERT encoder with 768-dimensional hidden states trained on masked atom prediction; masking 35% of tokens gave the best average downstream result. Pretraining uses a purpose-built corpus of 20 million molecules — all of ChEMBL25, thirteen commercial supplier libraries and a ZINC15 subset — assembled because ZINC15 alone, filtered toward Lipinski-style drug-likeness, spans far fewer functional groups; a 200-million-molecule version is also published. The structural branch trains ComplEx knowledge-graph embeddings, then a graph convolutional network by link prediction, with a margin-based contrastive loss aligning them.
Under an 8:1:1 scaffold split across three seeds, FARM reaches state-of-the-art on 8 of the 13 MoleculeNet tasks, including 93.3 ROC-AUC on BBBP, 89.6 on BACE and 83.5 on HIV, and 1.097 RMSE on FreeSolv. Generalization was probed on a donor–bridge–acceptor oligomer photostability set of 25 quantum-mechanical property tasks, under a split withholding three donor–bridge and fifteen acceptor building blocks outright: R² exceeds 0.9 on 16 of the 25 targets and never falls below 0.815. On seven ADMET tasks from the Therapeutics Data Commons, FARM matches or beats the leaderboard best on three — aqueous solubility, plasma protein binding and Ames mutagenicity.
The practical use is property prediction on small datasets typical of early discovery — blood-brain barrier penetration, solubility, toxicity endpoints, ADMET triage — where a few thousand labeled compounds are all that exist and the pretrained encoder supplies the rest. Because that encoder stays frozen, embeddings can be extracted once and reused across assays. The photostability results extend the same recipe to functional materials, screening light-harvesting oligomers for excited-state properties that would otherwise need quantum chemistry calculations.
FARM's contribution is an argument about tokenization rather than scale: at 103M parameters it is far smaller than billion-scale chemical language models, and the gains come from spending vocabulary on chemically meaningful units. The knowledge graph and the reversible annotation scheme are reusable independently of the encoder, and the weights and both corpora are MIT-licensed. The stated limitation is that FARM carries no 3D representation, so stereochemistry and conformation-dependent properties are out of reach. It remains a preprint, revised four times between October 2024 and April 2026 with no journal version.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.