XPos3R

A pose-regression network that sees only the X-ray has to keep the patient's anatomy somewhere, and the only place left is its own weights. That is why the accurate members of this family are patient-specific: DiffPose trains a CNN from scratch on synthetic X-rays rendered from one patient's CT, taking around twelve hours, and xvr cuts the same step to roughly five minutes of fine-tuning. Either way, a new patient means a new model — a preoperative step that a scheduled arthroplasty can absorb and a trauma case cannot.

XPos3R removes the step by changing what the network is given. Instead of regressing a C-arm pose from the X-ray alone, it takes the (X-ray, CT volume) pair as input, so patient anatomy arrives at inference time as data rather than being baked in during training. The framing is borrowed from DUSt3R, the feedforward model that regresses camera geometry directly from an image pair, but DUSt3R and its successors assume homogeneous inputs and a Siamese encoder with shared weights. A 2D projection and a 3D volume cannot share an encoder, and the authors' central claim is that the answer is a deliberately asymmetric design: different encoders per modality, and attention that flows in one direction only.

XPos3R comes from Monash University and the Australian National University, posted to arXiv in September 2026 and accepted to ECCV 2026.

#Key Features

• Volume as an input, not as training data: The preoperative CT is encoded and attended over at inference, which lets one checkpoint serve unseen patients with no preoperative preparation.
• Asymmetric dual encoder: A pretrained X-ray vision transformer handles the 2D image while a lightweight 3D CNN trained from scratch handles the volume. Pretrained 3D segmentation backbones hurt accuracy, learning sparse organ-localized features that align poorly with dense image tokens.
• Unidirectional cross-attention: Volume tokens outnumber image tokens by nearly ten to one, so the decoder runs self-attention on image tokens only and treats the volume as a key-value memory the image queries. This keeps the fusion affordable.
• Patient-level global embedding: Average-pooling the volume feature grid yields a single token concatenated into the regression head; ablating it degrades every metric, so it supplies context that token-level fusion misses.
• Pose diversity as the scaling axis: The data pipeline renders thousands of viewpoints per volume rather than chasing more scans, on the argument that X-ray appearance varies more with C-arm angle than with anatomy across patients.

#Technical Details

The image encoder is RayDINO, a vision transformer pretrained on large-scale X-ray data, applied to a 224×224 image to give 196 tokens; the volume encoder is a four-stage Conv3D hierarchy over a 96×96×96 CT giving 1,728 tokens. The decoder stacks four self-attention blocks on image tokens followed by two image-to-volume cross-attention blocks, and two 3-layer 1024-channel MLPs regress axis-angle rotation and translation separately, supervised by a geodesic rotation loss and a Huber translation loss with the angular term scaled by focal length. Training used 441 CTPelvic1K pelvic scans (plus 20 for validation), each pre-registered to a canonical template and rendered into 5,000 synthetic radiographs with DiffDRR under random contrast augmentation, for 2.3 million volume-DRR-pose triplets; pretraining ran 100 epochs on eight H100 GPUs over about five days.

On DeepFluoro, a six-subject cadaveric pelvis benchmark, the pretrained model reaches 1.352° rotation error and 24.16 mm target registration error feedforward, against 2.587° and 33.90 mm for a fine-tuned xvr and 3.347° and 43.88 mm for DiffPose — with markedly lower variance across subjects. Adding seconds of intensity-based test-time optimization brings TRE to 3.288 mm and projection error to 0.448 mm, best among all methods compared. Feedforward inference takes 1.52 s per X-ray, or 2.93 s with refinement.

#Applications

The target is image-guided pelvic intervention — trauma fixation, arthroplasty, resurfacing and periacetabular osteotomy — where a live fluoroscopic view must be tied to the preoperative plan. Removing per-patient preparation matters most in emergency presentations, where even minutes of setup are unavailable, and it lowers the operational burden of deployment, since no per-case training infrastructure has to sit in the clinical workflow.

#Impact

XPos3R is the first demonstration that the DUSt3R-style feedforward geometry recipe transfers to heterogeneous 2D/3D medical inputs, and the asymmetric design is offered as a template for other cross-modal geometry problems. Its scope is narrower than the framing suggests: results cover the pelvis only, because that is where public CT collections and registration benchmarks exist, and the authors state that cerebral vessel registration is out of reach for lack of public vessel datasets. Volumetric input is restricted to CT, since MRI and ultrasound would require modality translation before DRRs could be synthesized. Evaluation is on cadaveric fluoroscopy rather than live surgery, and neither code nor trained weights have been released, so independent replication is not yet possible.