xvr

Millions of interventions each year are guided by live X-ray fluoroscopy, but an X-ray is a projection and carries no depth information. Aligning the 2D intraoperative image to the 3D CT or MRI acquired beforehand recovers that missing dimension, letting a surgical plan or an anatomical target be reprojected onto the live view. In practice this alignment has been the bottleneck: intensity-based optimizers need per-subject hyperparameter tuning and fail when initialized more than a few centimetres from the true pose, while supervised deep learning needs hand-annotated anatomical landmarks and stays confined to the anatomy it was trained on.

xvr, from MIT CSAIL and Harvard Medical School with clinical collaborators at Massachusetts General Hospital and Boston Children's Hospital, sidesteps the annotation problem entirely. A differentiable X-ray renderer generates unlimited synthetic radiographs, with exactly known C-arm poses, from the patient's own preoperative scan. A pose-regression network is then trained purely on this simulated data and transfers to the patient's real intraoperative X-rays.

The framework's practical contribution is the pretrained backbone. Rather than training a network from scratch for every case — a process that previously took hours — the authors pretrained a whole-body foundation model and showed that five minutes of patient-specific fine-tuning recovers the same accuracy. Deployment is therefore a warm start, not zero-shot inference: the released checkpoint is specialized to each patient before the procedure.

#Key Features

• Annotation-free supervision: Training pairs come from physics-based simulation rather than from clinicians, removing the manual landmark labelling that keypoint methods require for every new scan.
• Five-minute patient adaptation: Fine-tuning the whole-body backbone on one patient's scan matches the accuracy of a from-scratch model trained for 12 hours, a 144-fold reduction in training time.
• Pan-anatomical coverage: One pretrained model has been adapted to pelvis, distal femur, cerebral vasculature, and skull, spanning CT, MRI, and rotational angiography.
• Differentiable pose refinement: The same renderer that produces training data drives a gradient-based optimizer that polishes the network's initial estimate to millimetre accuracy in seconds.
• Interpretable pose parameters: Outputs are expressed in the LAO/RAO and craniocaudal conventions and source-to-object distance that C-arm operators already use.

#Technical Details

Every xvr network is a ResNet34 backbone (21,278,400 parameters) with two linear heads (6,669 parameters) that regress the quaternion adjugate — a continuous representation of SE(3) that avoids the discontinuities in Euler angles and quaternions. A ResNet18 backbone was too small to fit the pose distribution. The foundation model was pretrained for 300,000 steps on a single NVIDIA H200 using 2,245 CT and MR volumes aggregated from four public whole-body datasets and rigidly aligned to a common template, with poses sampled over a full 360° about the LAO/RAO axis and ±60° craniocaudally.

Evaluation covers all public human 2D/3D registration datasets with ground-truth poses: DeepFluoro (6 cadaveric pelvis subjects, 362 X-rays), Femur (5 subjects, 49 X-rays), and Ljubljana (10 endovascular neurosurgery patients). After five minutes of fine-tuning plus differentiable refinement, xvr reached a median target registration error of 1.1 mm on DeepFluoro with a 40.9% submillimetre success rate, versus 7.6% for iterative optimization from a fixed initialization and 1.7% for landmark regularization; final errors were 8.0 mm on Femur and 1.4 mm on Ljubljana. Running the foundation model without any patient-specific fine-tuning still reached 31.2%. On 122 X-rays from 50 neurosurgical patients at Brigham and Women's Hospital, a patient-agnostic skull model achieved a median error of 1.2 mm against clinician annotations, compared with 180.3 mm when initialized from the C-arm pose recorded in the DICOM header.

#Applications

xvr targets image-guided intervention and surgical robotics: reprojecting preoperative plans or lesion boundaries onto live fluoroscopy, localizing instruments, and correcting for motion. It has been applied to orthopaedic, endovascular, and neurosurgical imaging, including four neonates with Vein of Galen malformations — a population absent from the adult-only pretraining corpus — where five minutes of fine-tuning yielded 2.3 mm median error. Because it runs automatically on a consumer GPU, it also serves retrospective work, registering a 122-image clinical archive in ten minutes where manual keypoint annotation took over 40 hours of clinician time.

#Impact

By replacing labelled clinical data with patient-specific simulation, xvr converts 2D/3D registration from a per-institution engineering project into a pretrained model plus a short fine-tune. The code and the whole-body checkpoint are released under an MIT license; the reformatted benchmark datasets are not, carrying CC BY-NC-ND 4.0 inherited from their sources — DeepFluoro under CC BY-NC, Femur and Ljubljana under CC BY-NC-ND — so the model is freely reusable while the evaluation data is restricted to non-commercial, unmodified use. The evaluation remains modest in absolute terms — 21 subjects from public datasets and 54 from private clinical sources — reflecting how little paired X-ray/CT data exists publicly, and the authors caution that larger prospective cohorts are needed. Iterative refinement also adds one to eight seconds of lag, so fully real-time pose estimation without refinement is still open. As a preprint, the reported comparisons await peer review.