AVES2-BEATs

AVES2-BEATs is a self-supervised bioacoustic audio encoder from the Earth Species Project (ESP) that turns raw recordings of animal sounds into transferable, general-purpose embeddings. It is the second generation of AVES (Animal Vocalization Encoder based on Self-Supervision), the bioacoustic encoder introduced by Masato Hagiwara in 2022, and it is distributed on HuggingFace as the checkpoint esp-aves2-sl-beats-all. Because a single labeled dataset is rarely large enough to train a wildlife classifier from scratch, AVES2-BEATs provides a pretrained backbone whose 768-dimensional embeddings can be adapted to many downstream tasks with lightweight probes.

The model targets a long-standing bottleneck in computational bioacoustics: labeled data are scarce and heavily skewed toward birds, so encoders trained on one taxon or task transfer poorly to others. AVES2-BEATs is designed to be broad, spanning birds, marine mammals, and other taxa, and to support species classification, individual identification, detection, and unsupervised vocal-repertoire discovery from one shared representation.

AVES2-BEATs was developed by Marius Miron, David Robinson, and colleagues at the Earth Species Project and introduced in the 2025 study "What Matters for Bioacoustic Encoding" (accepted to ICLR 2026). That paper is a systematic empirical investigation of how architecture, training data, and training strategy affect the quality of bioacoustic encoders, and esp-aves2-sl-beats-all is its strongest resulting checkpoint.

#Key Features

• BEATs transformer backbone: Built on the BEATs audio transformer, self-supervised pretrained on AudioSet and then supervised post-trained on a mixed bioacoustics-plus-general-audio corpus.
• Two-stage training recipe: Combines self-supervised pretraining with supervised fine-tuning (the "sl" in the checkpoint name), the configuration the authors found most robust both in and out of distribution.
• Broad, diverse corpus: Post-trained on roughly 17,760 hours of audio spanning bird, marine-mammal, and mixed-species sources, with additive-noise and mixup augmentation.
• Transferable embeddings: Emits 768-dimensional embeddings that pool into fixed-length vectors, usable with linear, MLP, LSTM, attention, or transformer probe heads.
• Open tooling: Loaded and applied through the MIT-licensed AVEX library (pip install avex), which also ships training and evaluation scripts.

#Technical Details

The encoder uses a BEATs transformer that is first self-supervised pretrained on AudioSet (~5,700 hours of general audio) and then supervised post-trained on a combined corpus of roughly 17,760 hours drawn from Xeno-canto (~10,416 h of bird recordings), iNaturalist (~1,539 h), AudioSet, the Watkins Marine Mammal Sound Database (~27 h), and the Animal Sound Archive (~78 h), with additive noise and mixup augmentation. It outputs 768-dimensional frame embeddings. Across the paper's evaluation suite of 26 datasets — spanning species classification, detection, individual identification, and vocal-repertoire discovery — the all-data variant is the strongest overall, reaching 0.840 probe accuracy on the BEANS benchmark and 0.726 mAP on BirdSet. The paper's central finding is that self-supervised pretraining followed by supervised post-training on a diverse mixed corpus gives the best in- and out-of-distribution performance.

#Applications

AVES2-BEATs is aimed at ecologists, conservation biologists, and bioacoustics researchers who need to analyze large volumes of field recordings but lack the labeled data to train task-specific models. Its embeddings support biodiversity monitoring and passive acoustic surveys, automated species and call-type classification, individual animal identification, and clustering or retrieval for exploratory vocal-repertoire analysis, all via lightweight probes on top of the frozen encoder.

#Impact

AVES2-BEATs extends the original AVES line from a HuBERT-based, largely bird-focused encoder toward a broader, more rigorously benchmarked family of bioacoustic representations, and its parent study offers the field a concrete recipe: pretrain self-supervised, then post-train supervised on a diverse bioacoustics-plus-general-audio mixture. The weights are openly downloadable on HuggingFace and the surrounding AVEX library is MIT-licensed, lowering the barrier to reuse. Practical limitations remain: the training corpus over-represents certain taxa and geographic regions, performance can degrade under large distribution shifts, and the released weights carry a non-commercial CC BY-NC-SA 4.0 license that restricts commercial deployment.