Mamba state-space encoder-decoder pretrained on 91 million PubChem SMILES that embeds molecules for property prediction and decodes them back.
No providers recorded yet. Browse all providers
A transformer's self-attention compares every token with every other token, so its cost grows with the square of the sequence length, and anything beyond its fixed context window is invisible to it. Many PubChem SMILES are long, and screening tens of millions of compounds multiplies whatever each string costs. SMI-SSED (SMILES-based State-Space Encoder-Decoder, also called MoLMamba, and published as OSMI-SSM-336M) replaces attention with Mamba, a selective state-space model that carries a compressed running state along the sequence and scales linearly with its length.
The rest of the design follows its transformer sibling SMI-TED. The Mamba encoder is trained with masked-token prediction, then a learned projection folds the token embeddings into a single 768-dimensional molecule vector, and a second projection plus a language head unfolds that vector back into the SMILES string. Mean pooling, the usual way to get one vector per molecule, has no inverse; this learned bottleneck does, so one embedding serves as both a featurizer and a decodable latent point.
IBM Research teams in Rio de Janeiro and Almaden released the code and weights in December 2024 alongside a NeurIPS 2024 AI4Mat workshop paper, "A Mamba-Based Foundation Model for Chemistry". The journal version appeared in npj Artificial Intelligence in June 2025 as "A Mamba-based foundation model for materials". It is the state-space member of IBM's FM4M foundation-model suite and a successor to the lab's encoder-only MoLFormer.
encode()/decode() interface and fine-tuning scripts, so the two architectures are directly swappable.SMI-SSED has 336 million parameters: a 94M Mamba encoder and a 242M decoder, hidden size 768, state dimension 16, convolution width 4 and expansion factor 2. The pretraining corpus began as 113 million PubChem SMILES, deduplicated, canonicalized and sanitized with RDKit down to 91 million unique molecules, about 4 billion tokens. The tokenizer yields a 2,993-token vocabulary, and inputs are capped at 202 tokens, which covers 99.4% of PubChem. Training combined a masked-language-model loss with a mean-squared-error reconstruction loss and ran for 130 epochs at a fixed learning rate of 3 × 10⁻⁵ with batches of 128 molecules on 24 NVIDIA V100 GPUs.
On six MoleculeNet classification sets, fine-tuned SMI-SSED scores best on three: BBBP (92.81 ROC-AUC), HIV (83.14) and Tox21 (83.84). It is second-best on four of five regression sets. On MOSES reconstruction of 176,000 held-out molecules it reaches scaffold similarity 0.9994, nearest-neighbour similarity 0.9960 and Fréchet ChemNet Distance 0.0025. On 3,955 Buchwald–Hartwig cross-coupling reactions, yield prediction reaches R² 0.846 when trained on only 2.5% of the data and averages 0.983 across the four out-of-sample additive splits. Predicting HOMO and LUMO energies for 10 million PubChem molecules on one V100 took about 9,700 seconds per property, against about 20,600 for SMI-TED.
SMI-SSED suits medicinal and process chemists who have small labeled datasets and large compound libraries. The library can be encoded once and a light model fitted on the embeddings for solubility, lipophilicity, blood-brain barrier penetration or toxicity. Because the encoder runs in linear time, the frozen model is practical for property screens across tens of millions of structures. The reaction-yield results extend this to high-throughput experimentation, where few reactions in a matrix have measured yields. The decoder also lets generative workflows move through the latent space and read valid SMILES back out.
SMI-SSED tests whether a state-space backbone can match transformers as a chemical foundation model. Its most useful evidence is the controlled comparison: it shares a corpus, tokenizer, bottleneck design and benchmark suite with SMI-TED, and differs mainly in architecture. That comparison shows Mamba staying competitive with attention on property tasks while cutting inference time roughly in half. Code and the 336M checkpoint are public under Apache 2.0. The model card also lists an 8 × 336M variant, but only the base checkpoint is distributed. The 91-million-molecule pretraining set is described in detail but not released as a file, so pretraining can be re-run only by repeating the curation scripts on PubChem. All reported evaluations are computational benchmarks.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.