Jointist: Joint Learning for Multi-instrument Transcription and Its Applications

In this paper, we introduce Jointist, an instrument-aware multi-instrumentframework that is capable of transcribing, recognizing, and separating multiplemusical instruments from an audio clip. Jointist consists of the instrumentrecognition module that conditions the other modules: the transcription modulethat outputs instrument-specific piano rolls, and the source separation modulethat utilizes instrument information and transcription results. The instrument conditioning is designed for an explicit multi-instrumentfunctionality while the connection between the transcription and sourceseparation modules is for better transcription performance. Our challengingproblem formulation makes the model highly useful in the real world given thatmodern popular music typically consists of multiple instruments. However, itsnovelty necessitates a new perspective on how to evaluate such a model. Duringthe experiment, we assess the model from various aspects, providing a newevaluation perspective for multi-instrument transcription. We also argue thattranscription models can be utilized as a preprocessing module for other musicanalysis tasks. In the experiment on several downstream tasks, the symbolicrepresentation provided by our transcription model turned out to be helpful tospectrograms in solving downbeat detection, chord recognition, and keyestimation.