FragLlama

A medicinal chemist optimizing a lead does not redraw the molecule atom by atom. They swap a morpholine for a piperazine, hang a new ring off a known growth vector, or stretch the chain between two binders. A language model that reads SMILES one character at a time has no representation of those units: a benzene ring arrives as six separate decisions, and nothing in the token stream marks where the ring starts or stops. The generative move the chemist actually makes is invisible to the model making the prediction.

FragLlama, from YDS Pharmatech, moves the unit of prediction up to the fragment. Molecules are pre-segmented into chemical fragments before byte-pair encoding is applied, so the resulting 8,000-token vocabulary is populated by rings, linkers and functional groups rather than by arbitrary character runs. Next-token prediction over that vocabulary becomes next-fragment prediction, and generation proceeds by appending one chemically meaningful piece at a time to whatever the user supplies as a starting point.

That framing sets the model's scope: it is conditioned on one or two input fragments rather than sampling molecules from nothing. One fragment plus a growth anchor is fragment growing or scaffold modification; two fragments is linker design or scaffold hopping. It is a different posture from fragment-space samplers such as FragAtlas-62M, which enumerates the fragments themselves; FragLlama consumes fragments and decides what comes next.

#Key Features

• Fragment-level tokenization: Molecules are fragmented before BPE training, so tokens correspond to recognizable substructures. The published example encodes a GLP-1R inhibitor in 61 tokens against 105 for an atom-level tokenizer, 86 for GPT-2's and 71 for GPT-4's.
• Conditional generation from anchors: Generation starts from user-supplied fragments and designated growth anchors, mapping directly onto fragment growing, scaffold hopping and linker design rather than unconstrained de novo sampling.
• Two-stage training: A pre-training stage learns conditional probabilities over fragment tokens; a separate alignment stage fine-tunes that checkpoint toward desired properties, optionally with special tokens that encode an activity range.
• Chemistry-tuned architecture: The Llama 3 decoder is modified for short molecular sequences — standard positional encoding replaces RoPE, and full Flash Attention replaces grouped-query attention, which the authors judge a poor trade below the 7B-parameter scale.

#Technical Details

FragLlama is a 779M-parameter decoder-only transformer with SwiGLU feed-forward layers and RMS normalization, pre-trained on roughly 70 billion fragment tokens across a cluster of seven NVIDIA A100 GPUs using DeepSpeed ZeRO Stage 3. Decoding uses a beam-search variant. Attention maps compared against GPT-2 on the same shared tokens show FragLlama concentrating on a small number of heads where GPT-2 spreads attention diffusely.

Three design campaigns exercise the model. For molecular glues, growth anchors identified in the Protein Data Bank for S-pomalidomide, S-lenalidomide and S-thalidomide produced 5,834 molecules after sanitization and deduplication; atom-pair fingerprints and Tanimoto distances place them partly inside and partly outside a curated reference set of 73,443 cereblon-binder analogues from PubChem, and one generated structure is identical to the reported IMiD derivative PT-179. For PROTAC linkers, the model generated 156, 537 and 52 linkers for SP27, ARV-471 and CP07, with best-matching designs scoring 0.93, 0.78 and 0.74 Tanimoto similarity to the published linkers. For EGFR, fine-tuning on 8,532 PubChem inhibitors with active bioassay results and valid IC50 values — with and without IC50 special tokens — shifted generated molecules into chemical space occupied by 1,769 known 4-anilinoquinazoline inhibitors, which the un-tuned model did not reach.

#Applications

The model targets the hit-to-lead and lead-optimization stages, where a starting structure already exists and the question is what to attach to it. Its three demonstrated settings — enumerating molecular glue libraries around a cereblon binder, proposing PROTAC linkers between two known ligands, and biasing a scaffold toward a target's known active series — are the routine tasks of a degrader-focused medicinal chemistry group. Because the fine-tuning stage accepts activity labels as tokens, teams with in-house assay data can steer the pre-trained checkpoint toward their own potency range.

#Impact

FragLlama is a company flagship model rather than a community release: the preprint has no code or data availability statement, no training corpus is published, and the weights sit in a private repository behind a hosted Gradio demo. Its contribution is the argument that the tokenizer, not the architecture, is where chemical prior knowledge should enter a molecular language model — fragment-level tokens shorten sequences, cut the number of generation steps, and align the model's atomic decision with the edit a chemist would make. Evaluation is retrospective and computational throughout: similarity to expert-designed compounds and chemical-space overlap, with no synthesis or assay results reported for any generated molecule. The preprint has not been peer reviewed.