A user uploads an MP3. Forty seconds later they download a MusicXML file ready to open in Sibelius, MuseScore, or Dorico. What happens in between?
Stage 1 — Separation (Demucs)
The first problem is that real recordings are mixed. A drum-transcription model fed a full stereo master gets confused by bass, vocals, and guitars sitting in the same frequency bands. We use Demucs (Hybrid Transformer) to separate the input into four stems: drums, bass, vocals, and other.
Demucs runs on a GPU worker in our audio-workers Kubernetes namespace. A typical 4-minute track takes ~15–25 seconds depending on the GPU we land on. The drum stem moves forward; the rest get cached for downstream workers (bass transcription, piano transcription).
Stage 2 — Transcription (instrument-specific models)
Each instrument has its own specialist:
- Drums → ADTOF, a CNN-based onset/instrument classifier trained on the ADTOF dataset.
- Piano → Transkun-v2, a transformer that predicts note-on/off + velocity from spectrograms.
- Bass (jazz) → BassUNet, a pitch-contour model tuned for upright bass timbres.
- Bass (general) → FCPE, fast fundamental-frequency estimation for electric bass.
Each worker subscribes to a Pub/Sub topic, processes the stem, and publishes a completion event with the MIDI artifact stored in GCS. The orchestrator chains them according to the workflow definition.
Stage 3 — Quantization & cleanup
Raw model outputs are messy. A snare hit detected at 487.3 ms doesn't map cleanly to a sixteenth-note grid at 120 BPM. We run a quantizer that:
- Estimates tempo if not provided (or accepts a user-supplied BPM).
- Snaps onsets to the nearest grid value with a tolerance window.
- Filters out spurious double-hits within a configurable refractory period.
- Maps detected instruments to standard General MIDI drum pitches.
Aggressive quantization makes the score readable but kills feel; loose quantization preserves feel but produces unreadable notation. We default to a middle setting and expose ghost-note sensitivity + groove-preservation knobs in the UI.
Stage 4 — Engraving (midi2score)
MIDI is not sheet music. To produce MusicXML we use a custom midi2score worker that handles voice splitting (kick on stem-down, snare/hi-hat on stem-up), tuplet detection, rest insertion, and measure layout. The output is valid MusicXML 4.0 which most notation software opens cleanly.
Why microservices?
The whole pipeline could in theory live in one Python process. We split it because:
- Independent scaling. Demucs is GPU-bound; midi2score is CPU-bound. They don't need the same hardware or replica count.
- Independent deployment. Swapping ADTOF for a better drum model doesn't risk breaking the piano path.
- Backpressure isolation. A queue of slow piano jobs doesn't block fast drum-only jobs.
The orchestrator publishes jobs to Pub/Sub; workers subscribe and ack on completion. The frontend polls for status via a single endpoint.
The honest numbers
- 3:30 song, drums only: ~25 seconds end-to-end.
- 3:30 song, drums + bass + piano: ~60 seconds.
- Cost per credit: 1 credit = 1 second of audio. A free-tier user gets 1800 credits / month (~30 minutes).
The whole point is that you don't think about any of this. You upload, you download, you play.
