MedMax

Most biomedical vision-language assistants consume images and emit text. That covers visual question answering and report generation, but it stops short of the interleaved image-and-text documents that clinical and educational material actually consists of — a case write-up alternating between figures and prose, a teaching set that needs an illustrative image generated alongside an explanation. Models such as LLaVA-Med established the image-in, text-out pattern; a mixed-modal model that emits both is a different architecture and needs different training data.

MedMax, developed at UCLA, is that model. It fine-tunes Anole, an open mixed-modal derivative of Meta's Chameleon architecture, on a 1.47 million-instance biomedical instruction corpus of the same name. Chameleon-family models tokenize images into discrete codes through a VQGAN and interleave them with text tokens in one autoregressive stream, so image generation and text generation are the same operation over one vocabulary. That property is what lets MedMax handle interleaved generation rather than treating images as a read-only input.

The corpus is the paper's headline contribution, and the model is what demonstrates it. MedMax spans interleaved image-text generation, biomedical image captioning, biomedical image generation, visual chat, and report understanding, drawing on radiology and histopathology sources — which is why the entry sits across imaging, pathology, and language modeling rather than in any one of them.

#Key Features

• Mixed-modal generation: The model emits images and text in a single autoregressive stream, so it can produce an illustrative figure inside an explanatory answer.
• Interleaved document handling: Training includes passages that alternate between images and prose, matching the structure of case reports and teaching material.
• Cross-domain coverage: Radiology and histopathology are handled by one checkpoint, rather than by separate domain-specific models.
• Unified evaluation suite: The authors assemble 12 downstream biomedical visual question answering tasks into one benchmark and release it alongside the model.
• Open release: Model weights, the training corpus, an evaluation set, a hosted demo, and inference code are all published.

#Technical Details

MedMax is a 7B-parameter model fine-tuned from Anole-7B, itself derived from Chameleon. Images are encoded to discrete tokens by a VQGAN tokenizer and interleaved with text tokens; both modalities share one next-token objective. The MedMax corpus holds 1.47 million instances built from 725,000 unique images and roughly 1.7 billion tokens, split about 0.7 billion visual and 1 billion textual. Its sources include PubMedVision, PMC-VQA, PMC-OA, QUILT-1M and QUILT-Instruct, MIMIC-CXR, OmniMedVQA, LLaVA-Med instruction data, PathVQA, SLAKE, and VQA-RAD, with tasks labeled as visual chat, VQA, image captioning, multimodal generation, image generation, and report understanding.

Across the 12-task evaluation suite, the authors report a 26% gain over the Chameleon base model and an 18.3% improvement over GPT-4o. Those numbers come from the authors' own assembled suite and are best read as relative results within it. A portion of the corpus is credentialed — MIMIC-CXR and Quilt images require access requests through PhysioNet and the Quilt-1M project respectively — so those entries ship as text and metadata, with users tokenizing the images themselves using the same VQGAN encoder.

#Applications

The mixed-modal output makes MedMax most interesting for education and documentation workflows: generating a labeled illustrative image alongside a written explanation, drafting the figure-and-prose structure of a case report, or building teaching sets where a text description needs a matching image. On the input side it performs the standard tasks — answering questions about a radiograph or a histology tile, summarizing findings, and captioning — with one deployed checkpoint covering both radiology and pathology.

#Impact

MedMax's contribution is showing that mixed-modal generative architectures transfer to biomedicine when given a corpus built for them, and releasing that corpus so others can. The work was accepted at NeurIPS 2025. Several caveats belong with it. Part of the training data is sourced from YouTube lecture videos, which raises provenance and licensing questions that the release does not resolve. The checkpoint repository carries an Apache-2.0 license, but the model inherits the Chameleon and Anole lineage, whose upstream terms should be checked before commercial deployment. The published model card is a stub of a few links, so intended-use and limitation documentation lives in the paper and the repository rather than with the weights, though the dataset card is substantive. Finally, generative medical imagery carries an obvious risk of plausible-looking but clinically wrong output, and the release includes no safeguard specific to that failure mode.