MouseGPT

Behavioral phenotyping is the readout for most rodent neuroscience and psychiatric research, and it has long been squeezed between two unsatisfying options. Supervised classifiers can label predefined actions such as grooming or rearing with expert-level accuracy, but only actions someone already thought to annotate. Unsupervised methods cluster pose and kinematic features without labels, yet their clusters are defined by whichever features were chosen and often fail to correspond to anything a behaviorist would call a distinct behavior, with rare actions disappearing into larger groups.

MouseGPT, from ShanghaiTech University with LumiAni Technology, replaces the label set with language. It is a vision-language model that reads multi-view video of a mouse together with derived pose kinematics and writes a free-text description of what the animal is doing — posture, head orientation, limb placement, overall activity — rather than emitting a class index. Those descriptions are then embedded and used as quantitative features for clustering, fine-grained comparison between experimental groups, anomaly detection, and phenotype prediction. It was posted to arXiv in March 2025 and to bioRxiv in April 2025, and remains a preprint.

The approach parallels what NatureLM-audio does for bioacoustics: take a general-purpose multimodal model, specialize it on a large domain corpus, and let natural language serve as an open-ended label space for animal behavior instead of a fixed taxonomy.

#Key Features

• Open-vocabulary behavior description: The model produces free-text ethological descriptions rather than selecting from a closed label set, so behaviors absent from any annotation scheme can still be characterized.
• Two released model sizes: MouseGPT-Large (76 billion parameters) targets maximum descriptive accuracy; MouseGPT-Lite (8 billion) is intended for resource-constrained laboratories.
• Kinematic-aware prompting: 3D keypoints and limb velocities extracted from multi-view video are folded into the query prompt, grounding the language model's description in measured movement rather than appearance alone.
• Novel behavior discovery: An Isolation Forest run over the description embeddings flags outlier behaviors and compiles them into reports, surfacing atypical activity under unfamiliar drug or genetic manipulations.
• Phenotype prediction: Clustering results, fine-grained behavior statistics, and novel-behavior reports are combined with an expert-knowledge database to predict which treatment group an animal belongs to.

#Technical Details

MouseGPT is a supervised fine-tune of two open vision-language backbones: MouseGPT-Large adapts InternVL2-Llama3-76B and MouseGPT-Lite adapts MiniCPM-V 2.6, trained through the MS-SWIFT framework with DeepSpeed for multi-GPU training and LMDeploy for serving. Two horizontally concatenated perpendicular camera views are resized to 896 × 448 per sample; the InternVL2 model ran 18,000 training iterations and the MiniCPM model 45,900, with 1% of data held out for testing.

The training corpus is the paper's central asset. A multi-view acquisition system recorded 75 videos of 55 mice — 1,486 minutes, 42,822,048 frames across 5,352,756 time points — spanning chronic restraint stress models of depression and hallucination or schizophrenia models treated with psilocybin, LSD, and MK801. A separate pose dataset of 2,161,416 manually labeled keypoints across 90,059 images trained the 3D keypoint system. For language supervision, GPT-4o generated descriptions for 45 video sequences covering 37 mice, which a scoring-and-filtering pass distilled into 270,085 curated frame-level open-vocabulary annotations. Total data volume exceeds 20 TB. Evaluation against GPT-4o, InternVL2, and MiniCPM-2.6 used 15 senior animal-behavior researchers, blinded to model identity, who scored each output for correctness and ran pairwise comparisons — an assessment of description quality rather than a scored classification benchmark.

#Applications

The intended users are neuroscience and psychopharmacology laboratories running long-form behavioral assays, where classical tests such as forced swim or tail suspension capture only brief, coarse proxies for chronic conditions. MouseGPT profiles behavior continuously across a session, quantifies how the composition of behaviors shifts between drug-treated and control groups, and flags actions that no existing ethogram covers — useful when screening compounds whose behavioral signature is not yet known.

#Impact

MouseGPT is a preprint whose central claim rests on expert judgment of description quality rather than a numerical benchmark, which makes independent replication important — and currently impossible. Neither the weights nor the code have been released: no public repository or checkpoint exists, and the dataset is available only on request. The authors commit to publishing MouseGPT-Large, MouseGPT-Lite, the kinematic estimation models, the code, and the 20 TB corpus after peer-reviewed publication, under a CC BY-NC-ND 4.0 license that permits neither commercial use nor derivative models. The described system is therefore not yet a usable one, and the non-derivative terms will constrain reuse even after release.