Loading…
Bioacoustic encoder for birdsong that resolves individual syllables at 5 ms, using asymmetric spectrogram patches and Voronoi masked pretraining.
No providers recorded yet. Browse all providers
Birdsong syllables turn over roughly ten times faster than human phonemes, and 18.6% of the gaps between adjacent syllables in annotated laboratory recordings are shorter than 20 milliseconds. That is a problem for the encoders bioacoustics has been building on, because they inherited their temporal resolution from speech models: BirdAVES emits one embedding every 20 ms, and Bird-MAE tiles spectrograms with square patches spanning about 160 ms. The boundary a researcher wants falls inside a single token, so however good the representation is, it cannot say where one syllable ends and the next begins.
SongMAE, from George Vengrovski and Timothy J. Gardner at the University of Oregon, is a masked autoencoder trained to reconstruct birdsong spectrograms at 5 ms resolution. Its argument is that patch shape is an inductive bias rather than a tuning knob. Instead of the square patches that vision and audio MAEs inherit, SongMAE varies frequency and temporal extent independently, and the two axes turn out not to be interchangeable: finer temporal patches improve syllable parsing, while pushing frequency resolution past a moderate band buys nothing. Fine temporal patches create a second problem — because the FFT window is wider than a patch, neighbouring tokens overlap in acoustic content and a randomly masked patch can be interpolated from its visible neighbours. SongMAE answers that with Voronoi-based spatial masking, which grows irregular, connected masked regions outward from a few randomly seeded patches until 75% of the spectrogram is hidden.
Recent bioacoustic foundation models — AVES2-BEATs and NatureLM-audio among them — are built and benchmarked around clip-level questions. SongMAE targets the layer beneath while holding the layer above.
SongMAE is a ViT encoder-decoder over 128-bin log-mel spectrograms computed at 32 kHz with a 1,024-sample FFT, a 32 ms Hann window and a 160-sample hop. A shallow convolutional stack precedes the transformer; the decoder is discarded after pretraining. Three sizes were trained — Micro (1.75M parameters), Base (14.89M) and Large (98.65M) — each for 500,000 steps at batch 128 with AdamW and cosine annealing. Pretraining used the XCL subset of BirdSet, 528,422 Xeno-Canto recordings totalling 7,562 hours, minus every recording whose eBird taxon matched an evaluation species.
On syllable classification across canary, zebra finch and Bengalese finch, SongMAE-Large at 32 mels × 5 ms reaches a mean macro frame error rate of 4.91%, against 7.31% for BirdAVES and 8.46% for HuBERT; the gain is almost entirely in parsing error (1.85% versus 4.16%) rather than syllable identity. Unsupervised k-means clustering of embeddings gives a mean V-measure of 0.543, ahead of BirdAVES (0.451) and HuBERT (0.455) — and SongMAE-Micro still beats both. Under the AVEX evaluation protocol on the clip-level BEANS benchmark, SongMAE-Large attains 0.781 classification accuracy, ahead of BEATs (0.774) and the 630M-parameter Bird-MAE-Huge (0.766), with detection mAP of 0.389 for the 20 ms variant.
The immediate use is unsupervised annotation: clustering SongMAE embeddings yields initial syllable labels for species with no human-labelled corpus, the main bottleneck in birdsong research. The same representations support tracking song change across seasons or juvenile development, dialect mapping through shifts in specific song units, and detection of rare species outside a supervised detector's label set. Because the encoder stays frozen and probes are cheap, adoption needs no training infrastructure.
SongMAE is a preprint awaiting peer review, with limits its authors state plainly: three species, all recorded in controlled laboratory conditions, with generalisation to noisy field recordings and highly variable songs untested. Purity was still climbing at 500,000 steps for all three sizes, so none is trained to convergence. What the work establishes is more durable than a leaderboard position — that temporal patch shape, not model capacity, is the binding constraint for unit-level birdsong analysis, and that a 1.75M-parameter model with the right patch geometry can cluster syllables better than encoders fifty times its size. It also names a gap the field has not closed: unit-level annotation datasets are few, and the task has no shared benchmark of the kind that drove progress on species classification.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.