NatureLM-audio

NatureLM-audio is an audio-language foundation model built for bioacoustics: given an audio recording and a natural-language prompt, it answers questions about the animal sounds it contains. Rather than training a separate classifier for each species or task, the model treats bioacoustic analysis as an instruction-following problem, so a single system can name the species vocalizing in a clip, detect whether a target call is present, describe the sound in free text, or count the number of individuals it hears.

The model targets a persistent bottleneck in the field: annotated bioacoustic data is scarce and highly fragmented across taxa, recording conditions, and label schemes, which has limited machine learning to narrow, dataset-specific classifiers. NatureLM-audio addresses this by learning shared audio-language representations across a large, heterogeneous corpus spanning animal vocalizations, human speech, and music, then transferring that knowledge to unseen species and unseen tasks without additional training. This lets it, for example, borrow a speaker-counting ability learned from human speech and apply it to counting animals.

NatureLM-audio was introduced in November 2024 by David Robinson, Marius Miron, Masato Hagiwara, Benno Weck, Sara Keen, Milad Alizadeh, Gagan Narula, Matthieu Geist, and Olivier Pietquin at Earth Species Project, a nonprofit AI research lab, and was published as a conference paper at ICLR 2025. Its weights, training data, benchmark, and code are publicly released.

#Key Features

• Prompt-driven bioacoustics: A single model handles species classification, detection, captioning, life-stage prediction, call-type classification, and individual counting, all specified through natural-language instructions rather than task-specific heads.
• Zero-shot generalization: The model classifies species, genera, and families it never saw during training, predicting the correct scientific name for unseen species 23.8% of the time versus 0.5% for random guessing.
• Cross-domain transfer: Training jointly on bioacoustics, speech, and music lets the model repurpose human-audio skills — such as counting speakers — for bioacoustic tasks like counting individual animals in a recording.
• State-of-the-art benchmark results: On BEANS-Zero, a new zero-shot benchmark, NatureLM-audio surpasses general audio-language models (SALMONN, Qwen2-audio) and bioacoustics baselines (BioLingual, CLAP, BirdNET, Perch) on most tasks.
• Open release: Model weights, the 26-million-pair training dataset, and the BEANS-Zero benchmark are all published, with training and inference code under an MIT license.

#Technical Details

NatureLM-audio connects a self-supervised BEATs audio encoder to the Llama-3.1-8B-Instruct language model through a Q-Former connector that maps audio embeddings into the language model's input space. During training the BEATs encoder is fully fine-tuned and the Q-Former is trained, while the Llama backbone stays frozen and only its adapter layers are updated, giving roughly 0.7 billion trainable parameters. The model is trained on a released corpus of 26.4 million audio-text pairs (over 13,000 hours) aggregated from sources including Xeno-canto, WavCaps, iNaturalist, and the Animal Sound Archive, with language-model-generated captions supplying the text side. Evaluation uses BEANS-Zero, which extends the BEANS benchmark with new zero-shot tasks and comprises 91,965 test samples across classification, detection, and captioning.

#Applications

NatureLM-audio is aimed at ecologists, conservation biologists, and behavioral researchers who process large volumes of passive acoustic monitoring data. Its instruction-following interface lets practitioners query recordings directly — identifying which species are present, flagging rare or endangered calls, labeling call types and life stages, and estimating how many individuals are vocalizing — without building a bespoke classifier for each survey. Because it generalizes to species absent from its training data, it is particularly useful for biodiversity monitoring in understudied taxa and regions where labeled examples are unavailable.

#Impact

NatureLM-audio is the first audio-language foundation model built specifically for bioacoustics, extending the instruction-following paradigm of large language models to the analysis of animal sound. By demonstrating that a single model can generalize across taxa and transfer skills between human-audio and animal-audio domains, it offers an alternative to the narrow, per-dataset classifiers that have dominated the field. The accompanying BEANS-Zero benchmark establishes a shared yardstick for zero-shot bioacoustics, and the open release of weights, training data, and code lowers the barrier for conservation and animal-communication research. Model weights and datasets carry non-commercial licenses, which constrains commercial reuse even though the code is permissively licensed.