DINOv2-Adapted Chest X-ray ViT

A hospital archive holds hundreds of thousands of chest radiographs, but the labels that make them trainable arrive a few thousand at a time, one physician reading at a time. The usual answer is to fine-tune a large pretrained encoder end to end on whatever labels exist, which gives the backbone enough freedom to memorize a small cohort. This model inverts the budget: it spends its unlabeled data on making the encoder itself radiograph-shaped, then freezes it and trains nothing but a linear classifier on top.

The encoder is a ViT-Small/14 built in three stages at National Taiwan University of Science and Technology with National Taiwan University Hospital, published in September 2026. Stage one initializes from the public DINOv2 weights trained on the 142-million-image LVD-142M corpus of natural images. Stage two continues the same teacher–student self-distillation objective on 5,368 unlabeled chest radiographs from the UCSD pediatric pneumonia collection, with the bottom eight transformer blocks fully frozen so that low-level edge and texture filters survive the jump from color photographs to grayscale, high-contrast films. Stage three attaches a single linear head per diagnosis.

The two demonstrated heads — stroke and heart failure — share that encoder and differ only in the probe. Neither reads the disease directly: stroke detection from a chest film picks up cardiac structural changes that precede cardioembolic events, such as left atrial enlargement and valvular disease, making it an opportunistic risk proxy rather than cerebrovascular imaging.

#Key Features

• Partial-freeze domain adaptation: freezing the bottom eight of twelve transformer blocks during self-supervised adaptation, with layer-wise learning-rate decay above them, confines the update to the semantic layers where the domain gap actually lives.
• Frozen-backbone linear probing: the backbone runs in evaluation mode with no gradients during downstream training, so performance differences measure representation quality rather than adaptation capacity, and a new task costs one linear head.
• Label-free adaptation on out-of-domain films: the adaptation corpus is pediatric pneumonia and normal radiographs containing no stroke or heart failure cases, and the target cohort is adult cardiac patients at a different institution.
• Parameter efficiency: 22M parameters, roughly a quarter of the 86.6M ViT-B/14 used by RAD-DINO, the strongest domain-specific baseline in the comparison.
• Attention-based interpretability: multi-head attention maps and attention rollout localize both heads to the cardiac silhouette and pulmonary fields.

#Technical Details

The backbone is a ViT-Small with 384 embedding dimensions, 12 transformer blocks and 14-pixel patches. Radiographs are converted from DICOM to PNG, resized to 512 × 512, cropped to detected lung contours and contrast-enhanced with CLAHE. Second-stage self-supervision runs 120 epochs at batch size 32, maximum learning rate 1e-4 and 0.65 layer-wise decay, using DINOv2 multi-crop augmentation — global crops over half the image at 224 × 224, local crops under half at 96 × 96 — plus rotation, brightness and blur jitter.

Evaluation uses 2,000 labeled images per task from the National Taiwan University Hospital–integrated Medical Database (NTUH-iMD), balanced 1,000 positive to 1,000 negative, with patient-level 80/10/10 splits across five seeds. Under that protocol the stroke probe reaches 89.60 ± 1.39% accuracy and 89.55 ± 1.40% F1, ahead of RAD-DINO at 89.50 ± 2.69% and CheXzero at 87.10%; the heart failure probe reaches 92.52 ± 1.57% accuracy against RAD-DINO's 91.03% and CheXzero's 88.54%. An ablation separates the contributions: random initialization is weakest, LVD-142M initialization gains substantially, and the UCSD adaptation stage gains again, with the largest increment on heart failure. These figures are specific to a negative class composed 80% of normal radiographs drawn from an external Kaggle COVID-19 database; the authors show accuracy moves with that proportion and caution that it changes the difficulty of the task rather than only the model's skill.

#Applications

The setup fits a hospital with an imaging archive and a small annotation budget: extract features once with the frozen encoder, then train a probe for each new diagnosis on a single consumer GPU. Because the demonstrated tasks read cardiac structure off an examination ordered for other reasons, the natural use is opportunistic screening — flagging patients whose routine films carry cardioembolic or heart failure signal for echocardiography — rather than primary diagnosis.

#Impact

The work is a compact demonstration that a modest, targeted self-supervised adaptation stage can substitute for scale: 5,368 unlabeled films lifted a 22M-parameter general-purpose backbone past radiology encoders several times its size under a matched linear-probing protocol. The evidence is correspondingly narrow: both tasks were evaluated on one proprietary single-institution cohort with no external multi-center validation, the negative classes mix an outside dataset with different acquisition characteristics, and the headline accuracies are not general chest X-ray performance. The paper's data availability statement records that the institutional datasets and the code are not publicly released, so the adapted weights cannot currently be reproduced or reused outside the group.