One-sentence Summary

The researchers from Kyutai, Mirelo AI, and IRCAM-CNRS Sorbonne Univ introduce MuScriptor, an open-weight model for multi-instrument music transcription that overcomes the limitations of prior single-instrument or synthetic-only approaches by combining synthetic pre-training, fine-tuning on real music audio, reinforcement learning post-training, and instrument-presence conditioning to enable accurate transcription of real-world recordings across diverse musical genres.

Key Contributions

• A training strategy combines synthetic data pre-training, fine-tuning on real music audio, and post-training with reinforcement learning (Group Relative Policy Optimization) to substantially outperform the MT3 baseline in multi-instrument transcription.
• Inference-time instrument conditioning is introduced to allow users to specify present instruments and tailor the transcription output.
• MuScriptor is released as the first effective open-weight multi-instrument music transcription model that works across diverse real-world genres and is configurable via instrument conditioning.

Introduction

Automatic Music Transcription (AMT) converts audio recordings into symbolic representations like MIDI, a foundational capability for music analysis, production, and generative modeling. While single-instrument transcription has matured, general-purpose multi-instrument transcription remains challenging because models must handle diverse timbres, overlapping harmonics, and studio effects across a wide range of genres. Prior work, such as MT3, relies heavily on synthetic training data, which causes a severe domain shift when applied to real-world music, making outputs too error-prone for practical use. The authors introduce MuScriptor, an open-weight decoder-only transformer that overcomes these limitations by training on a large dataset of 170,000 real recordings (11,000 hours) with aligned note annotations, augmented by synthetic pre-training, reinforcement learning alignment on high-quality transcriptions, and optional instrument conditioning to stabilize and customize predictions.

Dataset

The authors build a multi-instrument automatic music transcription dataset from three main sources, each serving a distinct role in training and evaluation.

• Synthetic data (D_Synth): Roughly 1.45 million MIDI files are collected from public sources (e.g., Lakh MIDI) and commercial providers, covering pop and Western classical music. During training, an on-the-fly pipeline randomly selects excerpts, applies symbolic augmentations (pitch shift, tempo change, velocity adjustment, instrument randomization), synthesizes the augmented MIDI with a random soundfont (over 250 soundfonts), and adds random detuning. This creates an effectively infinite stream of audio paired with precise note annotations. The synthetic set is used for large-scale pre-training.

• Real music data (D_Real): An internal dataset of 170,000 real audio recordings (over 11,000 hours) with aligned note annotations. Annotations are obtained via audio-symbolic synchronization: linear interpolation between annotated bar lines, or dynamic time warping using chroma and onset features when bar annotations are missing. Low-quality alignments are filtered out by thresholding the time warping distance and enforcing a maximum time dilation factor (e.g., after 8 seconds in one sequence, the other must have progressed at least 1 second). The dataset contains 17 instrument groups present in at least 5% of tracks, indicating diverse instrumentation. D_Real is used for supervised training.

• Reinforcement learning data (D_RL): A subset of 300 tracks from D_Real, manually verified for high annotation quality. These tracks are held out from D_Real and used exclusively for reinforcement learning post-training.

• Test data (D_Test): A separate set of 372 tracks selected from D_Real for their highly accurate annotations. These tracks are removed from D_Real, and any training tracks with similar titles are also excluded to ensure a clean split. D_Test focuses on multi-instrument pieces, matching real-world application scenarios.

All subsets are used for training a general-purpose AMT model: synthetic data for pre-training, real data for supervised fine-tuning, the RL subset for policy improvement, and the test set for final evaluation. The synthetic pipeline constructs excerpts on the fly, while the real data uses full recordings aligned with note-level annotations. No fixed cropping is applied beyond the random excerpt selection in the synthetic pipeline.

Method

The authors leverage a decoder-only Transformer architecture to perform automatic music transcription by autoregressively predicting a MIDI-like token sequence from a mel-spectrogram of a short audio segment. The model is scaled across four sizes, ranging from 60M to 1.3B parameters, varying in the number of self-attention heads, stacked layers, and latent dimension.

The model accepts two primary inputs: a mel-spectrogram representing a 5-second audio excerpt and a list of target instruments to be transcribed. The raw audio waveform (16kHz, mono) is converted into a mel-spectrogram using an STFT with $n_{\text{FFT}} = 2048$nFFT​=2048, a hop size of 160 samples (yielding a 100Hz frame rate), and a mel filter-bank with 512 bins. For instrument conditioning, the model is provided with the set of instruments present in the full track, mapped from 128 MIDI instruments into 36 subgroups. The distribution of these instrument classes in the real music dataset is diverse, as illustrated in the figure below.

To feed these conditioning signals into the model, the mel-spectrogram is projected to the latent dimension of the transformer and concatenated with embeddings for the present instrument classes, obtained via a learned lookup table. This embedding sequence serves as prefix conditioning for the transformer model.

The model is trained with teacher forcing over the tokenized note annotations using a standard cross-entropy loss. The training runs for 1M steps with a batch size of 64, utilizing the AdamW optimizer ($\beta_1 = 0.9, \beta_2 = 0.95$β1​=0.9,β2​=0.95) and a learning rate of $1e-4$1e−4 with a linear warmup of 2000 steps followed by a cosine schedule. Each conditioning signal is independently dropped with a probability of 0.2 to facilitate classifier-free guidance. At inference time, transcriptions are obtained via argmax decoding, applying classifier-free guidance to both conditions with a strength of $\alpha_{\text{CFG}} = 2$αCFG​=2. This instrument conditioning allows for customizing transcription results and obtaining coherent transcriptions across segment boundaries.

To further improve results, the authors post-train the models on a high-quality subset of real music data using a policy gradient algorithm that combines the REINFORCE estimator with group-relative advantage normalization. The distribution of active instruments across the different datasets used in this pipeline is shown below.

At each training step, the model is set to evaluation mode to generate $G$G independent transcriptions for each audio segment in the batch by sampling from the autoregressive distribution with a fixed temperature $\tau$τ (with $\alpha_{\text{CFG}} = 1$αCFG​=1). For each generated sequence $y_{i,g}$yi,g​, a scalar reward is computed against the ground-truth MIDI reference as the sum of three note-level F-scores:

$r_{i,g} = F_{\text{onset}} + F_{\text{frame}} + F_{\text{offset}}$ri,g​=Fonset​+Fframe​+Foffset​

Following the group-relative advantage normalization approach, within-group advantages are computed by standardizing the rewards across the $G$G outputs of the same segment:

$\hat{A}_{i,g} = \frac{r_{i,g} - \mu_i}{\sigma_i + \varepsilon}$A^i,g​=σi​+εri,g​−μi​​

where $\mu_i = \frac{1}{G} \sum_g r_{i,g}$μi​=G1​∑g​ri,g​, $\sigma_i = \sqrt{\frac{1}{G} \sum_g (r_{i,g} - \mu_i)^2}$σi​=G1​∑g​(ri,g​−μi​)2​, and $\varepsilon = 10^{-8}$ε=10−8.

The model is then updated via the standard REINFORCE objective, where the log-likelihood of each generated sequence is weighted by its advantage:

$\mathcal{L}_{\text{RL}} = \frac{1}{B \cdot G} \sum_{i,g} \hat{A}_{i,g} \cdot \mathcal{L}_{\text{CE}}(\theta, \tau; y_{i,g})$LRL​=B⋅G1​∑i,g​A^i,g​⋅LCE​(θ,τ;yi,g​)

where $\mathcal{L}_{\text{CE}}(\theta, \tau; y)$LCE​(θ,τ;y) is the average cross-entropy of the model with temperature $\tau$τ over the generated sequence. In practice, this optimization is performed with $G=8$G=8 over 5-second segments with a temperature $\tau = 0.75$τ=0.75 and a batch size of 8.

Experiment

The evaluation uses standard mir_eval metrics to assess multi-instrument transcription, with models pretrained on synthetic data, fine-tuned on real audio, and further refined via reinforcement learning. Qualitative results show that each training stage improves performance, especially real-data fine-tuning, and RL post-training reduces false negatives while enhancing onset accuracy; classifier-free guidance benefits most models but not the RL-trained one. Ablations indicate that synthetic pre-training is most valuable when real data is limited, mel spectrograms outperform other audio representations, larger models yield better results, and overlapping notes remain a challenge due to tokenization constraints.

Starting from synthetic data alone, MuScriptor already surpasses the YourMT3+ baseline in frame F1. Adding real data lifts all metrics by roughly 20 percentage points, and subsequent reinforcement learning on high-quality data yields the best overall scores, with onset F1 reaching 60.4. Classifier-free guidance improves onset for the synthetic-only model but provides no benefit after full training. The synthetic-only model achieves a frame F1 of 51.3, exceeding the 45.54 of the YourMT3+ baseline. Fine-tuning on real data raises frame F1 to 69.4 and onset F1 from 26.1 to 52.5. Applying CFG (α=2) to the synthetic-only model boosts onset F1 from 26.1 to 34.5, but the fully trained model sees no improvement from CFG. After RL post-training, the best model attains an onset F1 of 60.4, offset F1 of 49.0, and multi-pitch F1 of 48.2.

On the two unseen datasets, Bach10 and Dagstuhl ChoirSet, MuScriptor substantially improves frame-level and multi-pitch F1 scores compared to YourMT3+, while onset and offset F1 are generally lower. The largest frame F1 gain occurs on Dagstuhl ChoirSet, where the score rises from 51.0 to 80.7, and multi F1 climbs from 2.6 to 11.5. On Bach10, frame F1 also increases notably, but onset and offset scores drop, consistent with the challenge of annotating precise temporal boundaries for certain musical styles like chorals. MuScriptor lifts frame F1 on Dagstuhl ChoirSet by nearly 30 points (from 51.0 to 80.7) and multi F1 from 2.6 to 11.5. Onset and offset F1 scores are lower for MuScriptor on Bach10 and remain low on Dagstuhl ChoirSet, underscoring the difficulty of annotating exact note onsets and offsets for choral music.

Supplying instrument conditioning at inference time leads to consistent improvements across all evaluated metrics, including onset, frame, offset, drums, and multi F1 scores. The model is trained with conditioning dropout, so it can operate with or without explicit instrument labels, but providing the true instruments present in the track yields modest gains. The largest benefits are observed for frame and drums F1, while offset F1 shows the smallest increase. Providing ground-truth instrument labels at inference time improves all F1 scores relative to the unconditioned baseline. Frame F1 and drums F1 see the most noticeable gains, while offset F1 improves only slightly. The model can function in both conditioned and unconditioned modes due to conditioning dropout during training, but performance is better when instrument metadata is supplied.

Scaling the MuScriptor model from 60M to 1.3B parameters yields consistent improvements across all transcription metrics, with the largest model achieving the highest scores. The smallest 60M model already delivers a strong frame F1 of 65.7, indicating practical applicability in resource-constrained scenarios. Larger model capacity consistently improves onset, frame, offset, drums, and multi F1 scores, with the 1.3B version reaching the top performance. Even the 60M parameter model obtains a frame F1 of 65.7, demonstrating robust transcription quality suitable for low-resource settings.

Mel-scaled spectrograms yield the highest automatic music transcription scores across onset, frame, offset, and multi F1 metrics. A magnitude constant-Q transform with the same 100 Hz frame rate performs slightly worse, while learned representations like Encodec and MERT show substantially lower performance, suggesting that signal-close representations are more effective for this task. Mel-spectrogram input achieves the best overall F1 scores, outperforming all other representations in onset, frame, offset, and multi metrics. Magnitude CQT, despite having the same frame rate and a similar time-frequency design, shows a consistent but small decline across all metrics. Encodec, a neural audio codec with a 50 Hz frame rate, leads to the largest performance drop, with frame F1 falling from 68.0 to 58.0. MERT embeddings at 75 Hz improve drum F1 slightly over the mel-spectrogram baseline but degrade all other metrics.

MuScriptor is evaluated across multiple dimensions including training data composition, model scaling, input representations, inference conditioning, and generalization to unseen datasets. Synthetic data alone already surpasses a baseline, and subsequent fine-tuning with real data followed by reinforcement learning yields the best overall transcription scores. Mel-spectrogram input proves most effective, larger models consistently improve all metrics, and supplying instrument labels at inference offers modest gains, while generalization to choral music substantially boosts frame-level accuracy but underscores the challenge of annotating precise temporal boundaries.