HeartcareGPT

Electrocardiograms occupy an awkward position for medical multimodal large language models. Clinically, an ECG exists in two forms at once: the digitized 12-lead waveform, which carries precise temporal and amplitude information, and the printed or scanned image, which is what most clinicians actually read and what most hospital archives store. Med-MLLMs built around vision encoders can handle the image but discard the signal; signal-native models handle the waveform but cannot interpret a scanned report. Neither aligns the two.

HeartcareGPT, developed at Zhejiang University, is the model component of Heartcare Suite, a three-part release that also includes the Heartcare-400K instruction dataset and the Heartcare-Bench evaluation benchmark. Its central idea is to treat the waveform as a first-class modality with its own discrete vocabulary — a structure-aware tokenizer called Beat — and then project both the signal tokens and image features into the language model's embedding space through a dual-encoder alignment mechanism the authors call Dual Stream Projection Alignment.

The result sits alongside ECG-language models such as ECG-Chat, which aligns waveforms with report text, and PULSE, which interprets ECG images; HeartcareGPT's contribution is handling both input forms in a single autoregressive model and supporting cross-modal comparison between them.

#Key Features

• Structure-aware ECG tokenizer: Beat applies dual-level vector quantization with a core codebook and a residual codebook, compressing 12-lead recordings into discrete tokens while preserving rhythmic structure.
• Dual signal-image modeling: Separate expert paths encode the digitized waveform and the rendered ECG image, then project into one shared feature space so a single model can answer questions about either.
• Instruction corpus built from clinical records: Heartcare-400K combines the public PTB-XL archive with de-identified hospital ECG reports, assembled through an agentic data pipeline the authors call HeartAgent.
• Five-task benchmark: Heartcare-Bench spans Closed-QA, Open-QA, Comparison-QA, report generation, and signal prediction across signal, image, and cross-modal subsets.
• Native signal forecasting: A dedicated prediction token routes generation back through the Beat module, letting the model emit future cardiac electrical activity rather than only describe it.

#Technical Details

Two variants are reported. HeartcareGPT-3.8B builds on Phi-3-mini-4k-Instruct and HeartcareGPT-7B on Qwen2.5-7B-Instruct; both use SigLIP-So400M-patch14-384 as the image encoder and Beat as the signal encoder, connected by one-layer and two-layer MLP projectors and adapted with LoRA at rank 64. Four special tokens mark signal, image, text, and prediction boundaries in the extended vocabulary. Training proceeds in three stages: Beat is first trained on PTB-XL under joint reconstruction and prediction losses, the projectors are then warmed up on paired ECG images and signals, and the full model is instruction-tuned on Heartcare-400K.

Heartcare-400K draws on 21,799 12-lead recordings from PTB-XL and 12,170 hospital ECG images with structured reports. On Heartcare-Bench Closed-QA, HeartcareGPT-7B averages 83.42% accuracy and the 3.8B variant 83.33%, against 39.97% for the strongest generalist baseline (Yi-VL-6B) and 32.41% for the strongest medical baseline (Lingshu-7B); the comparison set spans twelve models including Qwen2.5-VL-7B, InternVL-2.5-8B, Claude 3.5, LLaVA-Med, and HealthGPT. The advantage carries into Comparison-QA, where HeartcareGPT scores 77.23% average against 57.04% for the best baseline, and into report generation on radiology-style F1.

#Applications

The suite targets automated ECG report generation, conversational triage of cardiac findings, and diagnostic question answering in settings where the available record may be a waveform file, a scanned printout, or both. Comparison-QA addresses a workflow clinicians perform routinely and few models support — contrasting serial ECGs from the same patient to detect change over time. Heartcare-Bench is independently useful as an evaluation harness for anyone building ECG comprehension models.

#Impact

HeartcareGPT extends the Med-MLLM paradigm beyond imaging into physiological time series, and its tokenizer-plus-projection recipe generalizes in principle to other continuous biosignals. The reported margin over general-purpose and medical MLLMs is large, though it reflects a comparison against zero-shot baselines on a benchmark released with the model. Data-processing code, training notebooks, and the PTB-XL-derived instruction files are public on GitHub, but no model checkpoints have been released and the repository carries no license file, so the model cannot currently be run without reproducing training from scratch.