TopKSigLIP

A screening mammogram can exceed 12 megapixels, and a lesion inside it often covers less than 5% of the frame. CLIP-style pretraining was built for 224- to 512-pixel natural images, so the standard recipe resizes the mammogram down until precisely the feature the report describes is smoothed away. Keeping full resolution breaks instead: encoding a whole 1536 x 1536 breast image runs out of memory at batch size 32 on a 96 GB card, and contrastive pretraining needs large batches for its negatives.

TopKSigLIP, from the Department of Radiology at Emory University, resolves that tradeoff by refusing to encode the whole image. A lightweight encoder scores a downsized copy for patch importance, a differentiable top-k module samples the handful of full-resolution patches most likely to contain a lesion, and only those reach the image encoder. Because the sampler trains end to end rather than sitting in front of the model as a region proposal stage, it needs no dense box annotations, and the scores it learns are themselves a lesion map.

The second change addresses the text side. In the EMBED cohort 75.9% of exams are BI-RADS 1, so their reports differ only in density and demographics, yet the softmax in a contrastive loss assumes every caption in a batch is semantically distinct and actively pushes near-identical reports apart. The Sup-sigmoid objective replaces that identity target with a similarity matrix built from tabular fields — BI-RADS score, breast density, cancer outcome, lesion subtype — and scores each pair independently under SigLIP's sigmoid loss. The name refers to that objective, not to a Google SigLIP checkpoint being fine-tuned; TopKSigLIP is pretrained from scratch and treats Mammo-CLIP as a baseline rather than a predecessor.

#Key Features

• Differentiable patch selection: Gumbel-top-K reparameterization with an annealed softmax relaxation makes sampling K of 256 candidate patches trainable by gradient descent, so the sampler learns what a lesion looks like from report text alone.
• Localization without Grad-CAM: the patch importance map is a model output rather than a post-hoc attribution, reaching 14.7 average precision and 42.1 Pointing Game on EMBED and 28.8 / 61.8 on VinDr, against 7.4 / 18.8 and 18.8 / 47.2 for the best baseline under ground-truth-guided Grad-CAM.
• Soft supervision from structured data: the Sup-sigmoid loss lets two mammograms with the same density and BI-RADS category be treated as similar instead of as negatives.
• Adjustable inference budget: k can be raised past its training value of 15 to 30 or 45, covering roughly 10%, 20% and 30% of the image at test time with no retraining.

#Technical Details

The vision tower pairs two weight-shared ImageNet-pretrained ConvNeXt-Tiny encoders (28.6M parameters) — one truncated to score a 512 x 512 downsized view, one to embed the sampled patches — over a 16 x 16 grid of 181-pixel candidate patches at 50% overlap, taking as input the craniocaudal and mediolateral oblique views of one breast concatenated into one 1536 x 1536 image. The text tower is a pretrained CLIP text encoder with context extended to 170 tokens, fed reports preprocessed by Llama 3 to strip demographics and findings from other modalities. Training ran for 100,000 AdamW iterations at batch size 32 on a single RTX PRO 6000 over EMBED cohorts 1 and 2, drawn from a 364,000-exam US dataset, with soft Dice supervision on the ~4% of exams carrying box annotations. Peak memory stays at 64.2 GB where full-image encoding at the same resolution does not fit at all. Zero-shot evaluation on the external VinDr and RSNA sets gives AUC gains of 29.2 and 17.8 points over the prior best on finding detection and cancer prediction, with zero-shot cancer AUC of 85.0 on EMBED and 78.7 on RSNA. Both ideas carry weight in ablation: supervised contrastive training in place of Sup-sigmoid drops cancer AUC to 70.1, and removing ROI supervision, to 78.8.

#Applications

The zero-shot results matter most where labels are the bottleneck: a site with mammograms and reports but no curated annotations can run density assessment, BI-RADS assignment, finding subtyping and cancer prediction off the shelf, and get a lesion heatmap in the same pass for a reader to check. The frozen encoder also supports linear probes for endpoints the pretraining never saw.

#Impact

TopKSigLIP argues that architectures assumed to be domain-general need rework when a domain's failure modes are specific: resolution and caption diversity are assumptions CLIP inherits from web images that mammography violates. The scope is deliberately narrow — this is a mammography model, not a general medical vision-language model, and its small encoder means the advantage narrows under linear probing, where larger baseline backbones catch up. The paper announces public code and weights, but the repository it names holds only a placeholder README, so neither source nor checkpoints can be obtained.