French biomedical and clinical language model family: RoBERTa encoders pretrained on NACHOS, a billion-word corpus of French medical web text.
No providers recorded yet. Browse all providers
Clinical notes are the richest text medicine produces and the hardest to train on: they carry patient identities, and in France their reuse needs authorization from the national data protection authority. That leaves a French-language team wanting a medical encoder with two imperfect options — pretrain on whatever medical text is openly available on the web, or borrow a model that already knows medicine in another language. DrBERT, developed by Avignon Université (LIA), Nantes Université (LS2N), Nantes University Hospital and Zenidoc and published at ACL 2023, tests both routes and measures them against a model trained on real hospital records.
The release is a family of RoBERTa encoders built on NACHOS, a French medical corpus the authors crawled from 24 public sources, alongside matched ChuBERT models trained on 1.7 million de-identified hospital-stay reports from the Nantes University Hospital data warehouse. Because the public and private corpora were sampled to equal size, the comparison isolates where the text came from. The headline finding is that 4 GB of web-crawled medical text trains an encoder that matches, and often beats, one trained on the same volume of genuine clinical notes. Only the public-data models were released; the ChuBERT weights stay private under GDPR.
Every from-scratch model uses the CamemBERT-base configuration — 12 layers, 768 hidden dimensions, 12 attention heads, about 110M parameters and a 512-token context — trained with masked language modeling at a 15% masking rate. Pretraining ran for 80k steps at a batch of 4,096 sequences of 512 tokens, on 128 NVIDIA V100 GPUs for 20 hours on the Jean Zay supercomputer; the study as a whole consumed about 25,500 GPU-hours. The released checkpoints are DrBERT-7GB (NACHOS-large, 7.4 GB and 1.1B words), DrBERT-4GB (a 4 GB, 646M-word subsample), and the two continued-pretraining variants DrBERT-4GB-CP-CamemBERT and DrBERT-4GB-CP-PubMedBERT. Four of the five carry their weights on Hugging Face; the CP-CamemBERT repository holds only a model card, and that variant is distributed instead as a 1.2 GB tarball from the project site. A 24-layer, 1024-dimensional DrBERT-7GB-Large checkpoint was published on Hugging Face after the paper, which reports only base-size models. Across the benchmark, from-scratch models took the best F1 on every private task and five of seven public ones. DrBERT-7GB reached 81.25 F1 on acute heart failure classification and 92.09 weighted F1 on QUAERO-EMEA nested NER, against 80.13 and 90.71 for CamemBERT trained on 138 GB of general French. The specialization has a cost: all medical models trail CamemBERT on general-domain French tasks, most sharply on natural language inference.
DrBERT is a backbone for French medical text processing: extracting entities such as drugs, dosages and diagnoses from hospital reports, classifying documents by specialty or condition, tagging clinical-case corpora, and answering pharmacy exam questions. It suits hospital informatics teams and clinical NLP researchers who need a French encoder they can fine-tune locally without sending patient records to an external service, and it can serve as a frozen text branch in multimodal clinical models.
DrBERT was the first openly released biomedical encoder for French, and it became a routine French medical baseline for later encoders such as DoctoBERT to report against. Its most durable contribution is evidence rather than a checkpoint: carefully filtered public web text can substitute for scarce, access-restricted clinical data in pretraining, which lowers the barrier for other languages where hospital notes are locked away. The study's limitations are stated plainly. It does not separate the tokenizer's contribution from the data's, it leaves open whether the English-to-French transfer generalizes to other language pairs, and four of its eleven evaluation tasks rest on private hospital data that outside groups cannot reproduce. NACHOS itself is distributed only for academic research, behind a signed usage agreement.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.