One-sentence Summary

Auton Lab at Carnegie Mellon University proposes JoLT, a method that jointly learns aligned representations from pre-trained time-series and text models via a Querying Transformer (Q-Former) rather than converting time-series to images or graphs, achieving state-of-the-art performance on electrocardiography summarization and question answering for medical time-series data.

Key Contributions

• JoLT is introduced as a neural model that jointly learns representations from pre-trained time-series and text models to generate text from time-series and textual prompts, explicitly modeling clinical time-series and text instead of modeling them separately or converting time-series into images or graphs.
• A Querying Transformer (Q-Former) is incorporated to align time-series and text representations, providing a mechanism for connecting waveform signals with natural language outputs.
• Experiments on the PTB-XL electrocardiography dataset for medical time-series summarization show that JoLT outperforms state-of-the-art image captioning approaches, including BLIP and BLIP-2, supporting explicit time-series modeling.

Introduction

Clinical time-series and text data are common in routine patient care, but interpreting them often requires substantial effort from medical professionals. Prior work has largely treated these modalities separately, relied on rule-based joint methods, or converted time-series data into images or graphs rather than modeling the raw signal directly. The authors introduce JoLT, a neural model that learns joint representations of language and time-series and can generate text from time-series and textual prompts. Evaluated on ECG summarization using PTB-XL, JoLT is presented as one of the first automated ECG interpretation approaches that explicitly models time-series to produce meaningful text, improving performance over image-captioning baselines such as BLIP and BLIP-2.

Dataset

• Source and scale: The authors use the PTB-XL dataset for an ECG interpretation task. It contains 21,837 twelve-lead ECG recordings, each 10 seconds long, collected from 18,885 patients.

• Supervised subset: A subset of recordings is paired with gold-standard clinical interpretations. These paired time-series and text samples are used to train and fine-tune the model. The paper reports 11,319 training samples, 1,636 validation samples, and 1,650 test samples.

• Schema and labels: Each example consists of an ECG waveform record and a corresponding clinical text interpretation. The dataset thus supports time-series-to-text generation, where the model produces clinical summaries from ECG signals.

• How the data is used: The authors train and fine-tune JoLT on the paired ECG and interpretation samples. They also compare against BLIP and BLIP-2, first in a zero-shot setting and then with BLIP-2 fine-tuned on the same PTB-XL paired data.

• Input processing: For the image-captioning baselines, ECG time-series are converted into graphical images using an ECG plotting tool before being fed into the models. JoLT is described as modeling time-series features more directly, while its encoder is pre-trained on a small set of PTB-XL time-series data.

• Additional details: The excerpt does not report a cropping strategy or further metadata construction beyond the fixed 10-second recordings and the available paired clinical text interpretations.

Method

JoLT formulates time-series summarization as a cross-modal generation task. Given a multivariate signal $T \in \mathbb{R}^{C \times L}$T∈RC×L with $C$C channels and length $L$L, the model is trained to produce a textual description of salient temporal patterns in a target domain. The architecture combines three components: a transformer-based time-series encoder, an OPT language decoder, and a Querying Transformer, or Q-Former, which serves as an alignment bridge between time-series and text representations. Refer to the framework diagram:

The training process is divided into three stages. In the time-series pretraining stage, each input signal is segmented into disjoint subsequences, or patches. A small subset of patches is masked uniformly at random, and the encoder is trained to reconstruct the missing values using a mean squared error loss. This objective encourages the encoder to learn informative latent representations from unlabeled time-series data.

In the representation learning stage, the pretrained time-series encoder is frozen, and the Q-Former is trained to learn query embeddings that capture features relevant to the accompanying text. The Q-Former is optimized with three complementary objectives. A contrastive loss aligns time-series and text representations by encouraging mutually informative embeddings. A text generation loss trains the Q-Former to generate text conditioned on the time-series input. A time-series text matching loss provides finer-grained alignment between paired or mismatched time-series and text representations.

In the generative learning stage, the time-series encoder, Q-Former, and OPT decoder are connected into a single generation pipeline. The Q-Former outputs learned query embeddings, which are inserted into the language decoder as soft prompts. These prompts guide the decoder to generate summaries conditioned on the input time-series. The model is then optimized with a causal language modeling loss, enabling the bridging module to translate time-series evidence into coherent natural language descriptions.

Experiment

The experiment evaluates JoLT on the PTB-XL dataset for generating clinical interpretations from ECG recordings, comparing it against image-captioning baselines such as BLIP and BLIP-2 using graphical ECG inputs. Results show that domain-specific fine-tuning is essential for clinical waveform interpretation and that explicitly modeling time-series features produces summaries closer to clinical ground truth than image-based approaches. These findings support the value of dedicated time-series representation and broader pretraining for medical text generation.

JoLT generates ECG interpretations that are closer to gold-standard clinical text than zero-shot and fine-tuned image captioning baselines across multiple text quality metrics. Zero-shot BLIP and BLIP-2 show very weak performance, while fine-tuned BLIP-2 improves substantially, highlighting the importance of domain-specific adaptation. The strongest results for JoLT suggest that explicitly modeling time-series features helps produce more meaningful clinical summaries. JoLT outperforms both zero-shot and fine-tuned image captioning baselines on ECG interpretation quality. Fine-tuning is critical because zero-shot BLIP-2 performs poorly compared with its fine-tuned version. Modeling time-series features directly leads to interpretations that are closer to clinical ground truth.

The qualitative comparison shows that JoLT generates an ECG interpretation closely aligned with the ground truth. In contrast, the fine-tuned BLIP-2 model identifies the normal sinus rhythm base class but adds inaccurate findings that make its summary less clinically faithful. JoLT produces summaries that more closely match the ground-truth ECG description. Fine-tuned BLIP-2 captures the base rhythm class but misdescribes additional sample details. The example suggests that direct time-series modeling can support more accurate clinical summaries than image-based captioning.

The experiments compare JoLT with zero-shot and fine-tuned image captioning baselines on ECG interpretation using text quality metrics and qualitative examples. The results show that zero-shot captioning is weak, fine-tuning improves alignment with clinical text, but JoLT remains closest to ground-truth interpretations because it models ECG time-series features directly. These findings indicate that domain-specific adaptation and direct temporal modeling are important for producing clinically faithful ECG summaries.