CLIMB-ConvNeXtV2

Clinical AI is fragmented by modality. A chest X-ray classifier, a dermoscopy classifier, and a fundus classifier are typically separate models trained on separate corpora, which leaves understudied modalities such as ultrasound permanently short of data and gives no path to a system that reasons over a patient's full record. CLIMB, from MIT with Harvard Medical School and Stanford University, is the dataset assembled to test whether that separation is necessary: 4.51 million patient samples totaling 19.01 TB across imaging, time series, graph, and text modalities, drawn from 37 medical institutions in 18 countries and spanning 13 clinical domains.

CLIMB-ConvNeXtV2 is the vision encoder that comes out of that benchmark. It is a ConvNeXt V2 backbone trained jointly on every two-dimensional imaging task in CLIMB — chest X-ray, mammography, dermoscopy, fundus photography, ultrasound, CT, brain MRI, and histopathology — using a single classification head over a label vocabulary unified across all of them. The result is one encoder where the field normally deploys a dozen, and the paper's central empirical claim is that this sharing is not a compromise but an improvement: the domains with the least data gain the most from being trained alongside the domains with the most.

The second claim concerns transfer. The authors evaluate the pretrained encoder on out-of-distribution datasets it never saw during training, adapting it with 1, 8, or 32 labeled examples per class, which is the regime a hospital actually faces when standing up a new task. The work was published at ICML 2025.

#Key Features

• One backbone across imaging modalities: A single encoder handles radiographs, mammograms, dermoscopy, fundus photographs, ultrasound, CT, and tissue slides, rather than a separate specialist per modality.
• Unified label vocabulary: Terminology across source datasets is standardized and equivalent concepts merged, so multi-label and multi-class annotations from different institutions can train one classification head without collapsing their clinical meaning.
• Few-shot transfer to new tasks: The pretrained encoder is evaluated on held-out datasets representing novel diagnoses or different label granularities, adapted from as few as one example per class.
• Gains concentrated where data is scarce: Multitask training helps most on the modalities, tasks, and geographic regions that are underrepresented in existing clinical corpora.
• Backbone for multimodal fusion: The same encoder is the fixed vision component in the paper's fusion experiments, paired with time-series and graph encoders over shared patients.

#Technical Details

The encoder is ConvNeXt V2, a fully convolutional architecture with global response normalization, trained supervised on a mixture of 26 imaging datasets with the out-of-distribution evaluation sets filtered out. Two-dimensional images account for 707K of CLIMB's 4.51 million samples. Training ran on eight H200 GPUs with the SOAP optimizer, with learning rate searched between 1e-5 and 1e-3 and weight decay fixed at 1e-3. Across the 26 multitask evaluation datasets the encoder reaches an overall AUC of 0.7867 and accuracy of 0.8770. Multitask training is worth the most where single-task training fails outright: on the COVID-US ultrasound task, AUC rises from 0.5000 to 0.8254. Against clinical-specialist architectures, the general-purpose ConvNeXt V2 outperforms MedViT by 35.9%, which the authors attribute to the broader pretraining distribution available to general-domain backbones. On out-of-distribution few-shot transfer, CT and ultrasound gain 28.7% and 29.1% AUC respectively over encoders initialized from public weights. Compared against LLaVA-Med fine-tuned on the same benchmark, the encoder reaches 0.877 accuracy versus 0.555, a gap widest on chest X-rays and endoscopic images where fine-grained visual discrimination matters most.

#Applications

The practical draw is few-shot deployment: a group with a new imaging task and a few dozen labeled cases can start from this encoder rather than from ImageNet weights or a modality-specific model that may not fit. That covers rare-disease imaging cohorts, prospective studies still accruing cases, and institutions in regions poorly represented in public corpora. The encoder also serves as a drop-in vision component for multimodal systems, where the paper pairs it with ECG and EEG encoders — including ECG-JEPA — and reports that fusion strategy should be matched to task complexity rather than chosen once.

#Impact

The result that cuts against expectation is architectural: general-purpose vision backbones beat purpose-built clinical ones on imaging, while in ECG the specialized architecture wins decisively, implying the right choice depends on how much domain-specific pretraining data exists rather than on domain specialization per se. The comparison against a fine-tuned medical vision-language model is similarly pointed: current LVLMs remain well behind dedicated encoders on clinical classification. The CLIMB repository publishes the dataset download pipeline, encoder implementations, and fusion code, and the paper states that pretrained weights are released there; no license is declared on the repository, while the arXiv paper itself carries CC BY-SA 4.0. Because CLIMB aggregates existing public datasets, users must still obtain access to each source dataset under its own terms.