DoctoBERT

Masked language modeling learns from whatever token it happens to hide. On a web page that names a drug twice in 800 words of navigation furniture and lifestyle advice, nearly every masked position lands on ordinary French prose, and the encoder spends its gradient budget on syntax it already knows. What makes medical text worth pretraining on is not the document's topic but its per-token terminology density — the quantity DoctoBERT, a family of French medical encoders released by Doctolib in June 2026, maximizes in its corpus.

Domain encoders are conventionally built from small, hand-assembled corpora: PubMed abstracts, clinical guidelines, a handful of canonical repositories, as in PubMedBERT or, for Chinese, MC-BERT. That sourcing caps corpus scale and, less obviously, stylistic range, and the ceiling is lower still outside English, where the canonical repositories barely exist. DoctoBERT instead borrows the web-scale data curation now routine for decoder LLM pretraining and adapts it to encoder MLM through two complementary levers. Medical-term density filtering scores each document by the fraction of its characters falling inside extracted medical-entity spans, and beats the widely used educational-quality filter on downstream medical tasks. Signal-amplifying rephrasing has an LLM rewrite a document into a denser, strictly meaning-preserving variant that widens the co-occurrence context around each medical concept. The corpora this produces — FineMed and its densified variant FineMed-rephrased — are released alongside the encoders.

#Key Features

• Medical-term density as the filter signal: Density is medical-term characters over total characters, a ratio that exposes the spans it scores and generalizes across document lengths better than a learned quality regressor.
• Faithful LLM densification: Rephrasing is gated so documents with too little medical content are dropped rather than embellished, and the rewriter must preserve every medical fact while replacing non-medical PII with varied fictional values.
• Two backbones, two context windows: DoctoBERT-fr-base is a 111M-parameter RoBERTa encoder with a 512-token window; DoctoModernBERT-fr-base (150M) and DoctoModernBERT-fr-large (395M) follow the ModernBERT recipe and read up to 8,192 tokens.
• Tokenizer trained on the dense subset: A SentencePiece BPE vocabulary (32,768 tokens for the RoBERTa member, 50,368 for the ModernBERT members) is fit on the entity-rich filtered subset, so dense medical vocabulary survives as whole tokens.
• The whole pipeline is released: Weights, the corpora, the three distilled annotators, the reproducible curation code and per-stage checkpoints are all public.

#Technical Details

FineMed draws on FineWeb-2, FinePDFs and FineWiki, prefiltered to French health content with a multilingual domain classifier that retains 5.3%, 7.7% and 1.5% of documents respectively. Each survivor is then labeled along three axes by lightweight annotators distilled from LLM teachers in two stages: a 15-class medical-subdomain classifier, a 0–5 educational-quality regressor, and an 8-class medical entity extractor fine-tuned from GLiNER2 on a UMLS-derived taxonomy. The corpus ships unfiltered at 21.1M documents and 19.2B words; the paper's best filter (educational quality ≥ 4 intersected with medical-term density ≥ 0.1) leaves 2.1M documents and 3.8B words, and rephrasing contributes a further 13.6M documents and 4.5B words at roughly double the density. DoctoBERT-fr pretrains on a mix of the two for 500B tokens, then anneals for 200B on the biomedical and clinical subdomains; DoctoModernBERT-fr runs three phases totalling 240B tokens, extending its context to 8,192 in the second. On a seven-task subset of DrBenchmark, DoctoBERT-fr leads both aggregate metrics (98.17 Min-Max, 97.14 win probability) and five of seven per-task scores, ahead of French medical baselines including DrBERT, CamemBERT-bio and TransBERT-bio-fr. On a proprietary Doctolib clinical NER task covering 12 entity and 9 qualifier classes, DoctoModernBERT-fr records the best precision (79.12) and F1 (79.40).

#Applications

Encoders of this shape are the workhorse of production clinical text processing: tagging diagnoses, drugs and procedures in consultation summaries, structuring free-text patient records, classifying documents by specialty, and de-identification, where the rephrasing step's varied fictional PII is deliberate training signal. Building from heterogeneous web text rather than a narrow curated corpus matters most where real-world clinical writing is noisy and register-inconsistent. The long-context ModernBERT members suit whole-document work — consultation notes, discharge summaries — that a 512-token encoder must chunk.

#Impact

The transferable result is methodological: web-scale curation, studied almost entirely for decoder pretraining so far, also pays off for encoder MLM in a terminology-dense domain, and the two levers compose rather than duplicate each other. The levers are language- and domain-agnostic by construction, so porting them costs re-distilling the small annotators and redesigning the domain-specific instances — though the authors do not test cross-domain transfer empirically. Two limits bound what the release shows. The recipe is instantiated only on French medical NLP, and one of the two evaluations is a proprietary task outside groups cannot reproduce. And no long-context French medical benchmark exists, so DoctoModernBERT-fr's 8,192-token window remains unmeasured on the tasks it was built for.