MedDream

A chest radiograph and the report written about it are two readings of the same physical state of a patient's thorax, but medical imaging AI almost always learns them as two unrelated mappings: a classifier running radiograph to diagnosis, and a generator running report to picture. Neither has any account of the state both describe. MedDream, from Emory University with collaborators at the University of Chicago and Memorial Sloan Kettering, is built on the opposite premise — learn the radiographic state once, and let diagnosis and simulation both read out of it.

The mechanism is one visual pathway carrying two objectives. A frozen chest-radiograph VAE turns each image into a grid of continuous latent tokens. A transformer encoder sees only the unmasked tokens and produces a representation; a CLIP-style contrastive loss ties it to the paired report, while a diffusion head reconstructs the masked latent tokens conditioned on that same representation and on the text. Both losses update the same encoder, so the state used for frozen diagnostic transfer is literally the state that conditions synthesis. The name is shared with an unrelated commercial DICOM viewer product.

#Key Features

• Shared radiographic state: One encoder serves frozen linear probes, few-shot classifiers, segmentation heads and the conditional generator, rather than splitting the two capabilities across separate backbones or staging one before the other.
• Progressive masking curriculum: The mean masking ratio, drawn from a truncated Gaussian, climbs from 0 to 1 across warmup, so one run satisfies both the low-mask regime that stabilizes image-text alignment and the high-mask regime that teaches latent recovery.
• Semantic trajectory selection: Generation launches eight stochastic trajectories, advances each 16 of 256 autoregressive steps, scores the partial latents in the model's own image-text space, and completes only the winner — no rendering of losers, no external reranker.
• Gap-conditioned evidence construction: Prompts can be aimed at a prespecified subgroup performance gap, so synthesis fills a named hole rather than adding undirected volume.
• Leakage-controlled corpus: Perceptual hashing screened pairs within and across sources, with whole patient groups and every evaluation image excluded at subject level.

#Technical Details

MedDream is a continuous-token masked autoregressive model with roughly 636 million trainable parameters, 201.5 million of them in the image encoder. Latent tokenization uses a chest-radiograph VAE on the MedVAE architecture, trained from scratch on the corpus rather than loading released MedVAE weights; Clinical-T5-Large encodes text for generative conditioning, with a separate alignment text encoder on the contrastive branch. Training ran 1,800 epochs on eight NVIDIA A100 GPUs — about 12,720 A100 GPU-hours — then a 512x512 refinement stage. The corpus began as 4.40 million candidate pairs, roughly 3.48 million literature-mined from PubMed Central and the rest from MIMIC-CXR, CheXpert Plus, PadChest, NIH ChestX-ray14, BRAX and IU X-Ray; filtering left 2.65 million.

Under a frozen-encoder linear probe, MedDream reached 81.2% mean AUROC across the 14 NIH ChestX-ray14 findings and 86.3% across 23 endpoints on VinDr-CXR, a cohort excluded from pretraining entirely, ahead of Ark+ at 80.3% and 85.2%. On ChestDR at a 5% label budget it reached 64.8% against 61.9% for BiomedCLIP and 57.2% for AFLoc. For generation it reached an XRV-FID of 0.31 and a CLIP-FID of 1.27, against 1.34 and 5.69 for ChexGen and 2.31 and 4.32 for MINIM.

#Applications

The diagnostic half suits groups adapting a chest radiograph encoder to a local label taxonomy or a rare finding, since transfer runs through probes on frozen features rather than full fine-tuning. The generative half is aimed at development rather than display: synthetic radiographs raised external VinDr-CXR macro-AUROC from 76.4% to 81.4%, and pretraining a classifier on MedDream images before fine-tuning on 5% real data added 4.0 AUROC points over ImageNet initialization. Conditioning on a prespecified subgroup gap raised weighted F1 by 3.1 points in Asian patients where matched-volume unguided augmentation lost 2.3. In a reader study, three residents grading severity with the model's estimates and matched synthetic references moved from 56.3% to 63.0% mean concordance with radiologist consensus.

#Impact

MedDream's contribution is an argument that the usual separation of diagnostic encoders from generative modules is both unnecessary and costly: under matched data, architecture and compute, joint optimization beat an alignment-only specialist across all five diagnostic and localization benchmarks by 2.1 to 3.5 points while also improving distributional fidelity, and sequential training in either order weakened whichever capability came first. It also sharpens how medical image generation should be judged — not by realism alone but by whether generated observations carry diagnostic information that transfers to real patients. The work is a preprint awaiting peer review. Training, generation and evaluation code is public along with the PubMed Central extraction pipeline, but the repository carries no license file and explicitly releases no trained weights, so the pretrained state cannot currently be obtained. The authors note that the model covers only chest radiography and examination-level rather than longitudinal state, that the literature-mined corpus retains an estimated 10.7% residual rate of non-compliant pairs, and that confidence scores are uncalibrated.