Lung Masked Autoencoder

Interstitial lung diseases (ILDs) comprise more than 200 distinct entities whose imaging patterns overlap heavily, and definitive diagnosis often requires a multidisciplinary board and sometimes a lung biopsy. Radiologists can reduce that burden by grading chest CT against the Fleischner Society criteria — usual interstitial pneumonia (UIP), probable UIP, indeterminate for UIP, or a pattern most consistent with non-IPF disease — but supervised deep learning has made little headway here because certified, board-consensus ILD cohorts number in the dozens, not the thousands.

The Lung Masked Autoencoder attacks that bottleneck with self-supervision. A team at the University of Bern and Inselspital, Bern University Hospital assembled a corpus of more than 5,000 thoracic CT volumes and pretrained a 3D masked autoencoder on it without any labels, then fine-tuned the encoder for ILD grading. The corpus is deliberately cross-pathological: COVID-19 scans dominate it (4,296 volumes), joined by 577 ILD cases, 330 normal studies, and 60 bacterial pneumonia scans, on the premise that COVID-19 and ILD share reticulation, ground-glass, and consolidation patterns that a reconstruction objective can learn from either source.

Released as a preprint in August 2025 with code and pretrained checkpoints, it sits alongside self-supervised CT backbones such as FM-CT for head imaging and supervised chest CT systems such as M3FM, but targets a far smaller-data regime than either.

#Key Features

• 3D masked-autoencoder pretraining: A ViT-Base encoder from the MONAI library reconstructs 75% masked patches of a 128×128×128 volume, learning lung-parenchyma representations without a single diagnostic label.
• Cross-pathology pretraining corpus: Pooling COVID-19, pneumonia, normal, and ILD scans from in-house and public sources yields a training set roughly 50× larger than the labelled ILD cohort alone.
• Lung-cavity-aware reconstruction loss: An ablation reweights reconstruction error so that patches containing at least 25% lung voxels dominate the objective, preventing empty background from masking poor detail recovery inside the lung.
• Linear probing beats full fine-tuning: With only 105 labelled scans, freezing the encoder and training a linear head outperforms end-to-end fine-tuning, which overfits quickly.
• Released weights and segmentation model: Pretrained MAE checkpoints and a trained nnU-Net for lung pattern segmentation are both distributed with the code.

#Technical Details

The encoder is a ViT-Base — 12 layers, 12 attention heads, hidden dimension 768 — adapted to 3D input. Volumes are resampled to average voxel spacing, cropped to the lung bounding box using the LungMask library, min-max normalised over Hounsfield units −200 to 1200, and resized to 128³ with patch size 16, giving 512 non-overlapping tokens. Pretraining ran for 198,000 iterations at batch size 64 with AdamW, learning rate 3e-4 and weight decay 5e-2, on a single NVIDIA H100. Fine-tuning attaches a class token, normalisation, and a linear layer, trained with class-weighted cross-entropy.

Evaluation uses 105 board-certified scans (27 UIP, 21 probable UIP, 18 indeterminate, 39 non-IPF) over five repeated 70:30 splits, reporting mean and standard deviation. On the binary task the linear probe reaches 72.3% ± 2.2 balanced accuracy and F1 0.71, matching an Inception-ResNet montage baseline (72.3%) and a radiomics pipeline (71.7%). The gap opens on the four-class task, where the linear probe scores 49.1% ± 7.0 balanced accuracy against 42.5% for radiomics, 40.8% for Inception-ResNet, and 28.6% for a ChexZero-initialised ViT. The same 3D ViT trained supervised from scratch manages only 25.0%. The lung-cavity-aware loss lifts binary balanced accuracy to 73.2% ± 4.9 but does not help the multiclass task.

#Applications

The pretrained encoder gives ILD researchers a chest CT backbone that transfers to label-scarce respiratory tasks — diagnostic grading, disease-pattern classification, and, the authors suggest, longitudinal endpoints such as forced vital capacity decline. Clinically, an automated Fleischner-category read functions as a second opinion where subspecialist thoracic radiology is unavailable, potentially narrowing the set of patients referred for biopsy. The accompanying nnU-Net segmentation model and preprocessing pipeline are reusable independently of the MAE.

#Impact

The work is a concrete demonstration that self-supervised pretraining on adjacent pathologies transfers to a rare disease with almost no labels — a result that matters wherever cohort size, not model capacity, is the binding constraint. The caveats are substantial and the authors state them: absolute multiclass balanced accuracy remains below 50%, the labelled cohort is only 105 scans, reported figures come from repeated validation splits rather than an independent cohort, and validation on an external ILD cohort is future work. The preprint has not been peer reviewed, and neither the code repository nor the checkpoint distribution carries a license, which limits reuse.