D-Rax

Ask a general biomedical vision-language assistant whether a chest radiograph shows a pleural effusion and it will answer fluently, in clinical register, and sometimes incorrectly — nothing in the answer distinguishes the two cases. Narrow chest X-ray classifiers that do this one job well, with measured error rates, already exist. D-Rax is built on the observation that the second kind of model can discipline the first, not by being called at inference time, but by writing its predictions into the text the assistant is trained on.

D-Rax is a conversational radiologic assistant for chest X-rays (CXRs), produced by instruction fine-tuning LLaVA-Med on a curated corpus in which every image's question-answer conversation is augmented with the outputs of pretrained expert models: disease probabilities across 18 thoracic findings, plus predicted patient age, race and radiographic view. Trained this way, the assistant learns to condition its replies on that structured evidence rather than on the language prior alone, which is what the authors target as the source of hallucination and imprecision in medical VLMs.

It was developed at Children's National Hospital with George Washington University, Universidad Politécnica de Madrid and NVIDIA, and published at MedAGI 2024, the second MICCAI workshop on foundation models for general medical AI.

#Key Features

• Expert predictions embedded in the training text: Each training record carries the image identifier, the VQA conversation, and an expert block — diagnosis probabilities, age, race and view — so supervision includes what specialist models believe about the image.
• Open- and closed-ended dialogue: The assistant handles free-text questions ("what abnormalities are seen in this image?") and binary ones ("is there pleural effusion?"), across abnormality, presence, view, location, level and type categories.
• Broad finding coverage: Disease predictions span 18 conditions including cardiomegaly, atelectasis, pneumonia, pneumothorax, edema, effusion, fracture, lung lesion, mass and consolidation.
• Off-the-shelf expert models: The expert block draws on the TorchXRayVision model zoo — a DenseNet-121 diagnosis network, an age regressor, a race classifier and a view classifier — so the recipe needs no new specialist training.
• Practical inference footprint: The fine-tuned model runs in 20 GB of GPU memory and answers 10,000 questions per hour on a single NVIDIA H100.

#Technical Details

The architecture follows LLaVA: a frozen ViT-Large/14 CLIP visual encoder, a trainable projection matrix mapping visual features into the language embedding space, and a Llama2-7B language model, with both the projection layer and the language model updated during instruction tuning. Training data comes from MIMIC-CXR and the derived Medical-Diff-VQA question-answer set, filtered to drop the longitudinal "difference" questions: 429,000 QA pairs over 129,232 images for training, and 13,688 QA pairs over 4,190 images for the held-out test set. Fine-tuning ran for a single epoch at a learning rate of 2e-5 with an effective batch size of 8.

Six runs crossed three initializations — LLaVA, and LLaVA-Med variants fine-tuned on VQA-RAD and on SLAKE — with two corpora, baseline instructions versus expert-enhanced. Expert enhancement improved every initialization: average closed-ended accuracy rose from 77.3% to 78.6% for LLaVA, 77.0% to 79.0% for the VQA-RAD variant and 76.3% to 78.8% for the SLAKE variant, with average open-ended token recall improving from 60.4% to about 61.7% for both LLaVA-Med starts. Gains concentrated in abnormality and presence questions, where the expert block supplies relevant signal, and were flat for location, level and type, where it does not. On abnormality classification the expert models themselves reach 70.4% accuracy, comparable to the VLMs; an extended test set of 32,205 images and 107,379 QA pairs reproduced the pattern.

#Applications

D-Rax targets the reporting workstation: a radiologist queries a radiograph in natural language, asks follow-ups about a suspected finding's side or severity, and gets answers grounded in both the image and specialist model output. The same interface suits teaching, a structured second read, and explaining an examination in plain language to a patient or a referring clinician. Because the recipe only changes the instruction data, it also serves as a template for other imaging domains where reliable narrow classifiers already exist and a conversational front end is what is missing.

#Impact

D-Rax's contribution is a training paradigm rather than an architecture: expert model output as instruction-tuning context is a cheap, reusable way to reduce hallucination in a medical VLM. Its scope is deliberately narrow. All training and evaluation data derive from MIMIC-CXR, with no external cohort and no demonstration of transfer beyond chest radiography, so reported accuracy describes in-distribution conversation quality. Including predicted patient race among the expert signals is a design choice that warrants scrutiny before any clinical use. Neither code nor weights have been released, and the instruction corpus is built from MIMIC-CXR, a credentialed-access resource, so reproduction requires rebuilding the pipeline from the paper's description under a PhysioNet data use agreement.