CheMatE

Chemistry has two text-shaped representations that rarely meet in one model. SMILES strings encode molecular structure as a compact line notation, and the scientific literature describes what those molecules do in ordinary prose. Encoders pretrained hard on SMILES become excellent molecular featurizers but lose the ability to represent general scientific language — a form of catastrophic forgetting — while general text encoders treat a SMILES string as an unfamiliar token soup. Because the two live in separate embedding spaces, a molecule and the paragraph describing it cannot be compared directly.

CheMatE (Chemical Embedder with Matryoshka Embedding) is an encoder-only model that places both in a single space. It was developed at the Laboratory of Artificial Chemical Intelligence (LIAC) at EPFL, with a co-author from Roche, and released as a preprint in August 2026. The core idea is not a new architecture but a new corpus: an automated pipeline detects chemical entities in scientific documents and splices their canonical SMILES into the text at the point of mention, so a single masked-language-modeling objective forces the model to predict masked SMILES tokens from surrounding prose and vice versa.

Two checkpoints are published. CheMatE-v0 is the contrastively trained feature extractor and is the artifact intended for downstream use; CheMatE-v0-MLM is the mid-stage checkpoint from before the contrastive phase, useful mainly as an ablation reference.

#Key Features

• Bi-semantic embedding space: A SMILES string and a natural-language description of the same chemistry land near each other, enabling cross-modal retrieval without a separate alignment model.
• Inline SMILES injection: Structural notation is woven into ~14.4 million scientific and educational documents rather than trained as a separate modality, grounding molecular tokens in long-form context.
• Matryoshka embeddings: Contrastive training optimizes 768, 512, 256, 128, and 64 dimensional prefixes simultaneously, so vectors can be truncated for cheaper indexing with graceful degradation.
• Long context: The 8,192-token window covers entire papers, in contrast to the short input limits typical of SMILES-only encoders.
• Drop-in frozen encoder: The checkpoint loads through sentence-transformers and is used as a fixed feature extractor, with no task-specific fine-tuning required.

#Technical Details

The backbone is ModernBERT-base (149M parameters). Continued MLM pretraining runs for 3 epochs at a 15% masking rate across 16 GH200 GPUs over a 21.9-billion-token corpus: 11.45B tokens of chemistry papers from ChemPile plus 10.41B tokens of FineWeb-Edu filtered by a word-frequency text chemistry score. Annotation is a four-stage cascade — ChemDataExtractor 2 for chemical named-entity recognition, OPSIN for IUPAC-to-SMILES conversion, a PubChem lookup fallback, and RDKit canonicalization with valence validation. A custom BalancedTokenBatchSampler equalizes per-GPU quadratic attention cost, cutting mean padding overhead by up to 50%. The second stage applies Multiple Negatives Ranking Loss under a Matryoshka wrapper on 19,638 synthetic anchor-positive pairs built by Tanimoto similarity over Morgan fingerprints; the released checkpoint is taken at the end of the first contrastive epoch.

Evaluation uses a frozen-embedding protocol — a logistic regression or Ridge probe on top of fixed vectors — over 48 datasets (26 classification, 22 regression) against 11 baselines. CheMatE takes the lowest mean rank on both panels, 1.9 across 31 SMILES tasks and 3.2 across 17 scientific NLP tasks. It falls in the top statistically indistinguishable group on 30 of 31 SMILES datasets and 13 of 17 text datasets — 43 of 48 overall — for a modality-balanced bi-semantic score of 86.7%. SMILES-specialized encoders such as ChemBERTa and MoLFormer score well on molecules but sit at the bottom of the text panel; the MLM-only checkpoint reaches 55.5%, isolating the contrastive stage as the source of the cross-modal gain.

#Applications

The natural uses are retrieval and featurization. A single index can hold both compounds and passages, supporting literature search by structure, structure lookup by description, and retrieval-augmented generation over chemistry corpora where a query may be either kind of object. As a frozen featurizer, the embeddings feed linear probes for molecular property prediction, materials-property regression, and chemistry text classification without fine-tuning, which suits small labeled datasets common in medicinal chemistry and process development.

#Impact

CheMatE argues that structural and linguistic competence are not mutually exclusive in an encoder, and that the lever is corpus construction rather than architecture. Both code and weights carry an MIT license, and the SMILES-injection pipeline is reusable independently of the model. Several caveats are worth noting: this is an arXiv preprint that has not been peer-reviewed; the annotation cascade can silently propagate wrong tautomers or stereoisomers into training data; the contrastive stage uses a modest 20k-pair budget whose tradeoffs the authors have not explored; and most evaluation datasets are short, so the long-context window is not directly stressed. The annotated corpus itself is available on request, with a public dataset release stated as planned.