SMI-TED

Encode CCO and CCCO with a masked-language chemical model and you get two vectors. Nothing in the objective guarantees that the step between them matches the step separating CCCO from CCCCO, or that either vector retains enough to write the molecule back out. Masked-token prediction only has to fill blanks in a string the model can already see, so knowing which tokens co-occur is enough to score well — and that is not the same as representing the molecule.

SMI-TED makes reconstruction part of pretraining. A bidirectional transformer encoder is first trained on masked SMILES tokens; a learned pooling function then compresses the token embeddings into a single molecule-level vector, and a decoder regenerates the full SMILES string from that vector alone. Because the pooled embedding is the only channel between encoder and decoder, it has to carry the structural detail — ring closures, branch points, heteroatom placement — that a classification head would let the model discard. The consequence shows up in the geometry of the space: across six homologous carbon-chain families, a linear model fitted on SMI-TED embeddings predicts a combined molecule's embedding from its parts with R² = 0.99, against 0.55 for the encoder-only MoLFormer. Partitioning QM9 embeddings by the presence of nitrogen gives a Davies–Bouldin index of 2.82, against 4.28 for MoLFormer, with no fine-tuning.

IBM Research released SMI-TED as a preprint in July 2024 and published it in Communications Chemistry in July 2025; it ships as the SMILES arm of IBM's FM4M multimodal materials suite. Downstream generators have since adopted it as a fixed chemical prior, including LiFT, which conditions 3D ligand generation on frozen SMI-TED embeddings.

#Key Features

• Reconstruction-constrained pretraining: a two-stage recipe — masked language modeling for the encoder, then joint encoder-decoder training for SMILES reconstruction — forces the pooled representation to be invertible, not merely discriminative.
• Learned pooling instead of mean or max: a trained pooling function compresses token embeddings, which is what lets one fixed-width vector decode back to the original molecule.
• A decoder you can actually use: encode() and decode() are both exposed, so an embedding produced by interpolation or arithmetic can be turned back into a SMILES string rather than only scored.
• Mixture-of-OSMI-Experts: the 8 × 289M variant composes eight task-fine-tuned copies of the base model with top-2 routing, improving every regression benchmark tested without a bigger backbone.
• Few-shot latent arithmetic: fitting the compositional relation on a handful of molecule triples lets the model construct and decode unseen combinations, reaching Tanimoto similarity up to 0.92.

#Technical Details

The base model totals 289 million parameters — a 47M encoder and a 242M decoder — with 12 transformer layers, hidden size 768, 12 attention heads and dropout 0.2. The tokenizer uses a curated 2,993-token vocabulary and molecules are truncated at 202 tokens. Pretraining ran on 91 million canonicalized, curated PubChem SMILES, roughly 4 billion molecular tokens, on 8 A100 GPUs. Evaluation covers 11 MoleculeNet datasets — six classification, five regression, original splits, ten seeds — where the fine-tuned model leads on four of six classification sets and all five regression sets, and places best or second-best on 11 of QM9's 12 quantum property tasks. Decoder quality is measured on MOSES (1,936,962 molecules, 176k held-out Bemis-Murcko scaffolds), where scaffold and nearest-neighbor similarity exceed the CharRNN, SMILES-VAE, JT-VAE, LIMO and MolGen-7b baselines. On 3,955 Pd-catalyzed Buchwald–Hartwig C–N cross-coupling reactions, yield prediction beats DFT-descriptor random forests and Yield-BERT, and holds up trained on just 2.5% of the reaction set.

#Applications

The practical use is as a drop-in molecular featurizer for teams with small labeled datasets: encode a compound library once, then fit a light head for solubility, lipophilicity, permeability or toxicity. The reaction-yield results extend that to process chemistry, where screening data is expensive and the low-data regime is the normal one. Because the decoder ships too, the embedding space also supports generative workflows — interpolating in latent space and reading out valid SMILES — and downstream models use the frozen encoder as a chemical prior rather than training their own.

#Impact

SMI-TED argues that a reconstruction objective changes what a chemical language model represents, not just how many parameters it has, and its latent-space comparison against IBM's own encoder-only MoLFormer is the evidence. Adoption so far is mostly as a frozen encoder in other pipelines. Two caveats matter for anyone building on it. Only the 289M base checkpoint ("smi-ted-Light") is published on HuggingFace under Apache 2.0; the 8 × 289M mixture is a per-task ensemble of fine-tuned copies rather than a separately pretrained release, and its weights are not distributed. And the 91-million-molecule pretraining corpus is not released — the curation constraints are documented, but the paper directs data requests to the corresponding author, so pretraining cannot be reproduced from the public artifacts alone.