OmniMRI

OmniMRI is a vision-language foundation model that aims to cover the entire magnetic resonance imaging workflow — reconstruction, segmentation, abnormality detection, diagnostic reasoning, and report writing — inside a single architecture. It was developed by the Intelligent Imaging Innovation and Translation Lab, based in the Department of Radiology at Massachusetts General Hospital and the Athinoula A. Martinos Center for Biomedical Imaging at Harvard Medical School, with a collaborator from Università Campus Bio-Medico di Roma.

The problem it addresses is structural rather than algorithmic. Deep learning has produced strong MRI models for individual steps, but each is typically bound to one anatomy, one contrast, or one task, so a working clinical pipeline becomes a chain of separately trained, separately validated systems that cannot share representations. Existing generalist medical segmentation models such as MedSAM and VISTA3D unify segmentation across anatomies but stop short of the language side of radiology — the findings, impressions, and recommendations that constitute the actual clinical product.

OmniMRI's answer is to treat every step as an instruction-following problem over a shared token stream of image and text. The same checkpoint reconstructs an undersampled brain acquisition, segments knee cartilage, localizes a lesion with a bounding box, or drafts a structured report, depending only on the prompt it receives.

#Key Features

• One model across the full MRI pipeline: Reconstruction, anatomical and pathological segmentation, abnormality localization, diagnostic suggestion, and report generation are all expressed as instructions to a single backbone, with no per-task retraining.
• Anatomy-agnostic coverage: Training data spans brain, prostate, breast, knee, heart, liver, and more than a dozen other regions, along with scanners from five vendors at field strengths from 0.3T to 3T.
• Dual decoding heads: A diffusion-based image decoder produces pixel-level outputs such as reconstructions and masks, while a text decoder produces detection coordinates, diagnostic reasoning, and free-text reports.
• Structured annotation schema: Paired image-text supervision was generated by prompting a vision-language model against a fixed JSON schema covering sequence parameters, visible anatomy, tissue signal characteristics, pathological findings, and diagnostic impression.
• Instruction-tuned interface: A final multi-task instruction-tuning stage lets clinicians and researchers invoke capabilities in natural language rather than through task-specific APIs.

#Technical Details

Images are tokenized by a Swin Transformer vision encoder that captures hierarchical features across resolutions; text is tokenized by a lightweight encoder initialized from Qwen2.5. Both streams feed a unified autoregressive Transformer backbone, also adapted from Qwen2.5, that applies multimodal self-attention over the joint sequence and uses a mixture-of-experts feedforward network to increase capacity without proportional compute. Training proceeds in four stages: self-supervised vision pretraining combining masked image modeling with a patch-level contrastive objective; vision-language alignment against a frozen vision encoder; unified multimodal pretraining; and multi-task instruction tuning.

The training corpus was curated and harmonized from 60 public datasets, totaling 47,917 patients, 70,580 scans, 224,194 volumes, and over 19 million slices. Task annotations are dominated by segmentation (53.6%) and diagnosis (19.6%), followed by reconstruction (17.9%), detection (5.4%), and report generation (3.6%). Reported results are qualitative demonstrations across the five task families — including reconstruction of 6× accelerated brain acquisitions — and the authors identify large-scale quantitative benchmarking and radiologist reader studies as work still to be done. Parameter counts are not disclosed.

#Applications

The intended use is consolidation: replacing a chain of narrow MRI tools with one model that a radiology department can prompt for accelerated reconstruction, organ and lesion segmentation, triage of suspicious findings, and first-draft reports. For imaging researchers, a single backbone pretrained across anatomies and contrasts offers a starting point for new MRI tasks without assembling an anatomy-specific pretraining corpus, and the instruction interface makes it straightforward to add tasks by adding prompts rather than architecture.

#Impact

OmniMRI is an argument that the fragmentation of MRI pipelines is a modeling choice rather than a necessity, and it backs that argument with one of the largest curated multi-anatomy MRI corpora assembled from public sources. Its practical status is more limited than its scope suggests: the preprint reports qualitative examples rather than benchmark numbers, no comparison against task-specific state of the art is provided, and the public repository carries a BSD-2-Clause license and documentation but no code or pretrained weights. Independent evaluation therefore depends on a future release, and the work has not yet been peer reviewed.