Transcribe any instrument to MIDI. One instrument-agnostic model powered by a Neural Semi-CRF.

日本語版 README | | MIT License

Transcription example	Original video

Click either image to open the video on YouTube.

The easiest way to get started is Colab_Inference.ipynb: upload a song, run the cells, and download the MIDI. No local setup or checkpoint search required.

To run locally:

git clone https://github.com/anime-song/tsumugi.git
cd tsumugi
uv sync --locked
uv run python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav

The checkpoint is downloaded from Hugging Face on the first run.

tsumugi is an instrument-agnostic Automatic Music Transcription (AMT) model that converts audio to MIDI.

Like Basic Pitch, it works across instruments. Piano, guitar, bass, vocals, strings, brass—if it has pitch, tsumugi can transcribe it with the same model.

The architecture extends the Neural Semi-CRF approach introduced by Transkun (Yujia Yan et al.), originally designed for piano transcription, to instruments of all kinds.

tsumugi works best on clean, sustained material where each stem is monophonic. Accuracy falls in areas underrepresented in the training data.

• Electric guitar, especially with distortion, does not generalize well.
• Traditional instruments such as shamisen and sitar are underrepresented in the training data.
• Fast vocal runs and swing phrasing can produce unstable pitches and note boundaries.
• A sustained note may be split into several shorter notes (over-segmentation).
• The stem-separated workflow can introduce small timing offsets between stems, which may cause drift after the MIDI files are merged.

Instrument classification and multi-track output are available, but classification accuracy remains limited. Slap and synth bass often collapse into electric bass; acoustic and electric piano are frequently confused; and instruments such as sitar or banjo that bleed into a separated guitar stem may be assigned to the wrong class. The drum models (--type drums_v1_5 and the legacy --type drums) are experimental and may change. The core instrument-agnostic pitch transcription is not experimental.

• 🎹 Any pitched instrument — transcribe piano, guitar, bass, vocals, strings, winds, and more with one model
• 🎚️ Per-note velocity — a dedicated post-processing model estimates dynamics from separated stems
• 🎯 Beat, chord, and key — optional MIDI-frame models, disabled by default
• 🧪 [Experimental] Instrument classification — a 36-class head for per-instrument MIDI tracks

Requirements

• Python 3.10–3.14
• uv
• PyTorch 2.13.0 / torchaudio 2.11.0, installed automatically from uv.lock
• An NVIDIA GPU (12 GB+ VRAM recommended), an Apple Silicon Mac, or CPU

Linux and Windows resolve to the CUDA 13.0 wheels. Apple Silicon macOS resolves to the platform wheels. PyTorch 2.13 does not provide Intel macOS wheels.

git clone https://github.com/anime-song/tsumugi.git
cd tsumugi

# Core inference dependencies pinned by uv.lock
uv sync --locked

# Optional workflows
uv sync --locked --extra stem        # stem-separated inference
uv sync --locked --extra evaluation  # evaluation scripts
uv sync --locked --extra training    # training

uv sync creates .venv/. Activate it with source .venv/bin/activate, or prefix commands with uv run.

.python-version selects Python 3.12 as the development default without narrowing the supported 3.10–3.14 range. If 3.12 is not installed, uv downloads a managed CPython unless downloads are disabled or the machine is offline.

Tested on: Apple Silicon (M4 Pro, macOS / MPS) and CUDA (Colab Tesla T4).

Tests:

uv sync --locked --all-extras
uv run pytest

MPS-specific tests are skipped when MPS is unavailable. The compile regression test is opt-in:

RUN_ACCELERATOR_COMPILE_TEST=1 uv run pytest tests/test_mps_inference.py

python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav

If --checkpoint is omitted, the model is downloaded from Hugging Face.

--device defaults to auto and selects the first available backend in this order: CUDA → MPS → CPU. If an explicitly requested backend is unavailable, inference fails instead of silently falling back.

python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav                # auto
python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav --device mps   # Apple Silicon GPU
python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav --device cpu

python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav --compile
python -m instrument_agnostic_amt.amt.cli.infer --audio input_song.wav --compile --compile-mode reduce-overhead

The first window pays the compilation cost, so a short one-off transcription may be slower. Compilation is most useful when the loaded model processes many songs. --compile-mode also accepts default, reduce-overhead, and max-autotune-no-cudagraphs.

Colab_Inference.ipynb includes a workflow that separates a song into stems, transcribes them, reclassifies each note's instrument, and predicts velocity.

It is slower than a single pass over the mix, but usually more accurate because each stem is acoustically simpler and contains fewer overlapping instruments. The benefit is greatest for dense mixes, band recordings, and arrangements with sustained chords under a melody.

With predict_beat_chord=True, beat_source chooses where beats and time signatures come from. "sheetsage2" (default) runs the SheetSage2-Refined beat tracker (no decoder) on the original mix; "tsumugi" uses the built-in beat model. Chords and keys always come from tsumugi. Install with uv sync --locked --extra stem --extra sheetsage. The SheetSage2-Refined weights (2.7 GB) are CC BY-NC 4.0 (non-commercial use only).

Reclassifies each note in an existing MIDI file using the stem it was transcribed from. Timing and pitch are preserved; only the track program and name change.

python -m instrument_agnostic_amt.instrument_refinement.cli.infer \
  --audio separated_stems/song_other.wav \
  --midi stem_midis/song_other.mid \
  --stem-name other \
  --output-midi song_other_refined.mid

--stem-name limits candidates to instruments plausible for that stem. --mode cluster (the default) groups notes by timbre embedding and labels each group; --mode single assigns one instrument to the entire stem.

This is a separate post-processing model. Given a MIDI file and its stems, it replaces fixed velocities with per-note dynamics while preserving tracks, pitches, and Note On/Off timing.

python -m instrument_agnostic_amt.velocity.cli.infer_velocity \
  --midi output.mid \
  --stems-dir separated_stems \
  --output-midi output_velocity.mid

Name files in the stem directory after their stems, such as vocals.wav, bass.wav, drums.wav, and other.wav. --compile-velocity regionally compiles the velocity backbone independently of the core AMT --compile. See instrument_agnostic_amt/velocity/README.md for training and data preparation.

Model variants (--type)

Place stems and their matching MIDI files in this structure:

stems/          # Audio (.wav / .flac)
  ├── song1__piano.wav
  ├── song1__guitar.wav
  ├── song2__vocal.wav
stem_midis/     # Matching MIDI
  ├── song1__piano.mid
  ├── song1__guitar.mid
  ├── song2__vocal.mid

Use the naming convention <song_name>__<instrument_name>.wav. A double underscore separates the song and instrument names. Stems with the same song name are treated as one song.

python preprocess/prepare_dataset.py \
  --stems_dir ./stems \
  --midis_dir ./stem_midis \
  --npz_dir ./stem_npz \
  --manifest_path ./manifest.csv

This creates stem_npz/, containing preprocessed note arrays (start, end, pitch, velocity, and instrument ID), and manifest.csv, the dataset index.

If the audio is not 22050 Hz, resample it in place while preserving the container format and sample subtype:

python -m preprocess.resample_only --input ./stems --resample-rate 22050

python -m recipes.amt.train \
  --manifest_path manifest.csv \
  --batch_size 8 \
  --lr 5e-4 \
  --epochs 3000 \
  --save_dir checkpoints \
  --wandb

With all augmentations enabled:

python -m recipes.amt.train \
  --dataset_config dataset_config.yaml \
  --batch_size 8 --lr 5e-4 --warmup_steps 1000 --epochs 3000 \
  --ir_folder ./IRs --noise_folder ./noise --drum_folder ./drum_stems \
  --p_augment 1.0 --p_intra_drop 0.3 --p_cross_mix 0.5 --p_drum_mix 0.1 \
  --sa_p 0.5 --sa_freq_max 10 --sa_time_max 20 --sa_num_freq 2 --sa_num_time 2 \
  --wandb --project_name tsumugi

dataset_config.yaml mixes datasets with different weights:

datasets:
  - name: main
    manifest: manifest.csv
    weight: 0.2
    use_for_cross_aug: true

  - name: maestro
    manifest: other_db/maestro_manifest.csv
    weight: 0.05
    use_for_cross_aug: true

  - name: musicnet
    manifest: other_db/musicnet_manifest.csv
    weight: 0.5
    use_for_cross_aug: false  # Do not use for cross-stem mixing

Use group when stems rendered from the same songs are stored in separate manifests:

datasets:
  - name: rendered_piano
    group: single_stems
    manifest: piano_stem_manifest.csv
    allow_multi_stem_same_song: true

  - name: rendered_strings
    group: single_stems
    manifest: strings_stem_manifest.csv
    allow_multi_stem_same_song: true

Entries with the same group and CSV song_name form one virtual song, allowing allow_multi_stem_same_song: true to select stems across manifests. Weights, augmentation settings, and cross-augmentation eligibility remain per entry. If group is omitted, name is used and each entry remains isolated.

Stem level — drop stems from the same song to simulate sparse arrangements, mix stems from different songs into new combinations, and add drums to drumless mixes.

Audio level — 7-band EQ, ±0.2-semitone micro pitch shift, IR reverb from real impulse responses, Gaussian noise and background sounds, channel swapping and random panning, and ±6 dB gain per stem.

Spectrogram level — SpecAugment time/frequency masking on CQT features, plus harmonic dropout that randomly removes harmonic channels while always preserving the fundamental.

# Pretrain beat detection from MIDI
python -m recipes.beat_chord.pretrain_beat \
  --pretrain_midi_dir beat_chord_dataset/beat_pretrain_dataset/midis

# Joint beat/chord training
python -m recipes.beat_chord.train --midi_dir midi_dataset/merged

# Beat/chord inference
python -m instrument_agnostic_amt.beat_chord.cli.infer --checkpoint path/to/checkpoint.pth --midi_path song.mid

instrument_agnostic_amt/
├── amt/                        # Core AMT model, inference, data, and CLI
├── taxonomy/                   # Instrument classes and GM mappings
├── beat_chord/                 # Public beat/chord models and inference
├── instrument_refinement/      # Public refinement model and inference
├── velocity/                   # Public velocity model and inference
└── cli/                        # Composed stem inference pipeline

recipes/
├── amt/                        # AMT train, dataset, targets, and losses
├── beat_chord/                 # Beat/chord datasets and training
├── instrument_refinement/      # Refinement dataset preparation and training
├── velocity/                   # Velocity data preparation and training
└── common/                     # Training-only shared augmentation

preprocess/                     # Standalone dataset preparation tools

The Neural Semi-CRF formulation and much of the decoding design come from Transkun by Yujia Yan et al. This project extends that work from piano to arbitrary instruments.

MIT