Autoregressive set transformer generating 3D molecules atom by atom without a canonical atom ordering, trained on QM9 and GEOM-Drugs.
No providers recorded yet. Browse all providers
Autoregressive transformers are cheap to sample from and handle variable-length outputs naturally, which makes them attractive for building 3D molecules atom by atom. The obstacle is that they expect their tokens in an order, and a molecule supplies none. Earlier work supplied one by fiat: the atom order in an XYZ file, a canonical ordering from a cheminformatics toolkit, or a designated "focus atom" at each step. Chemically, every atom bonded to the partial fragment is an equally legitimate continuation, so a model supervised on one designated successor learns the bookkeeping convention alongside the chemistry. The cost appears when the convention is violated — hand such a model a scaffold to grow, and it falters unless the scaffold matches the first atoms the ordering would have produced.
NEAT replaces the convention with a different training signal. Given a randomly sampled connected fragment of a training molecule, the model is supervised against the fragment's entire boundary — every 1-hop neighbour that could come next — rather than one atom. The authors call this neighborhood guidance, and it lets them drop positional embeddings entirely and encode the partial molecule with a bidirectional set transformer, which is permutation invariant to atom order by construction. A flow-matching head places the new atom's coordinates in continuous space, so no voxel grid or coordinate tokenizer is needed.
NEAT was released in December 2025 by Daniel Rose, Roxane Axel Jacob, Johannes Kirchmair and Thierry Langer of the Department of Pharmaceutical Sciences and the Christian Doppler Laboratory for Molecular Informatics in the Biosciences at the University of Vienna. It generates molecules unconditionally; the pocket-conditioned extension NEAT-POCKET builds on its GEOM-Drugs checkpoint.
Both released models are 12-layer, 12-head transformers of hidden dimension 768 with a six-block adaptive layer-norm MLP at hidden dimension 1536, totalling 165M parameters. Atom types are embedded linearly and positions with Fourier features; hydrogens sit in the vocabulary alongside a stop token, and bonds are inferred after generation. Two checkpoints were trained on a single RTX Pro 6000 Blackwell GPU: 10,000 epochs on QM9 (100,000 training molecules) and 1,000 epochs on GEOM-Drugs (1,172,436 conformers across 243,451 unique molecules). Gaussian noise injected into a fraction of the source-set coordinates trains in robustness to the model's own accumulated error.
On GEOM-Drugs, NEAT reaches 99.7% lookup-based validity and 93.9% xyz2mol validity at 96.9% uniqueness — the best validity among FlowMol3, SemlaFlow, Megalodon-FM, QUETZAL and Geo2Seq, all re-evaluated through one bond-assignment pipeline. On QM9 it records 95.2% xyz2mol validity, 91.6% uniqueness and 53.5% novelty, leading the autoregressive baselines there while trailing them on atom and molecule stability. Sampling 1,000 molecules runs 5 times faster than SemlaFlow or QUETZAL and 16 times faster than FlowMol3. The prefix experiment demonstrates the design most sharply: across 100 GEOM-derived ring scaffolds, NEAT scores 97.8% lookup validity against QUETZAL's 27.8%.
The immediate use is exploratory generation of drug-like 3D structures, and the prefix mode is what makes it practical for medicinal chemistry: supply a ring system with its substitution sites marked, and the model elaborates it in three dimensions while holding the given geometry fixed. Because it never conditions on a target, NEAT is a chemistry prior rather than a design tool on its own — the role it plays for NEAT-POCKET.
NEAT makes the case that the autoregressive route to 3D molecule generation was held back by its ordering conventions rather than anything intrinsic, and that removing them buys both quality and a large speed advantage over transport-based generators. The authors are explicit about what remains open: sequential construction can propagate errors into larger molecules, pooling atom embeddings risks oversquashing, the architecture relies on rotation augmentation rather than enforcing SE(3) equivariance, the model does not extrapolate beyond the largest molecules seen in training, and its reliance on bond information rules out domains such as inorganic crystals where connectivity is ambiguous. The work is a preprint awaiting peer review; source code and both checkpoints are available under the MIT license.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.