HeAR (Health Acoustic Representations)

HeAR (Health Acoustic Representations) is a self-supervised foundation model developed by Google Research for health-related sounds such as coughs and breathing. Released as part of the Health AI Developer Foundations (HAI-DEF) collection, it converts short audio clips into low-dimensional embeddings that capture the most salient parts of health-related sounds, giving researchers a reusable feature extractor for building acoustic biomarker models with less data and compute.

The model addresses an underexplored area of medical machine learning. Sounds like coughs, breaths, and speech carry health signals, yet acoustic data are noisy, heterogeneous, and rarely labeled at scale. By pretraining on an enormous, diverse corpus of everyday audio using self-supervision, HeAR learns general-purpose representations of health sounds that transfer to many downstream tasks without requiring task-specific labels during pretraining.

HeAR was created on December 4, 2023, and is described in "HeAR -- Health Acoustic Representations" (Baur et al., 2024). It is distributed with a model card and inference code on Hugging Face and GitHub under the HAI-DEF terms of use, and is intended explicitly as a research tool to accelerate work on acoustic biomarkers rather than as a diagnostic product.

#Key Features

• Health-sound embeddings: Maps two-second audio clips to 512-dimensional float vectors optimized to capture health-relevant acoustic content such as coughs and breaths.
• Masked autoencoding pretraining: Learns by reconstructing masked spectrogram patches from visible ones, a self-supervised objective that requires no labels and scales to very large audio corpora.
• Massive, diverse training corpus: Pretrained on roughly 174,000 hours of audio (313.3 million two-second clips) curated from public non-copyrighted YouTube videos.
• State-of-the-art acoustic transfer: Establishes strong performance across a broad benchmark of health acoustic tasks using simple linear probes on the frozen embeddings.
• Developer-oriented release: Ships with a documented model card, Hugging Face weights, and example code for downstream biomarker research.

#Technical Details

HeAR is a transformer-based masked autoencoder built on a Vision Transformer Large (ViT-L) backbone that operates on audio spectrograms. Input is two-second, 16 kHz mono audio, and the model outputs a 512-dimensional embedding per clip. Pretraining used the YT-NS (YouTube Non-Semantic) dataset, containing about 313.3 million two-second clips (roughly 174,000 hours) selected from three billion public non-copyrighted YouTube videos using a health acoustic event detector, and the model learned by reconstructing masked spectrogram patches from the visible ones. Evaluated via linear probes, HeAR reached state-of-the-art results on a benchmark of 33 health acoustic tasks across six datasets, spanning 13 health acoustic event detection tasks, 14 cough inference tasks, and six spirometry inference tasks.

#Applications

HeAR is aimed at researchers developing acoustic biomarker tools. Its embeddings support downstream models for detecting health acoustic events, inferring information from coughs, and estimating spirometry-related lung-function measures. Because the encoder is frozen and its embeddings are compact, teams can train lightweight classifiers on limited labeled recordings, making it well suited to low-resource settings and to exploratory studies of respiratory and other conditions from sound. It is particularly relevant to global-health research, where audio can be collected inexpensively on mobile devices.

#Impact

HeAR helped bring foundation-model methods to health acoustics, a modality long overlooked relative to imaging and text, and demonstrated that large-scale self-supervised pretraining on everyday audio yields representations that transfer robustly to clinical sound tasks. As part of Google's HAI-DEF collection, it lowers the barrier for teams investigating cough- and breath-based biomarkers. Its limitations are important to state plainly: it is a research tool that has not been cleared for clinical or diagnostic use; embeddings inherit biases from the YouTube-derived training corpus and the specific benchmark datasets; and downstream performance depends heavily on recording quality, capture devices, and the populations represented in fine-tuning data.