Abstract
Bridging the semantic gap between perfect symbolic musical structure and severely degraded acoustic signals offers a powerful new paradigm for high-fidelity audio restoration. In this work, we introduce MiDiff, a diffusion-based generative model for drum audio enhancement that conditions the reverse process on temporally aligned MIDI piano rolls. We extend the Score-Based Generative Models for Speech Enhancement (SGMSE) framework in the complex STFT domain by integrating a pretrained MIDI encoder, multi-scale temporal alignment, and three complementary conditioning strategies: early injection, Feature-wise Linear Modulation (FiLM), and cross-attention. To enforce structural fidelity and control the conditioning strength, we investigate two separate guidance mechanisms: classifier guidance driven by an Automatic Drum Transcription (ADT) model, and classifier-free guidance. We train our model on the StemGMD dataset using a non-additive degradation pipeline that simulates room acoustics, electret capsule response, and codec compression. MiDiff achieves a Fréchet Audio Distance (FAD) of 0.29, substantially outperforming the audio-only baseline (0.56). Ablation studies demonstrate that combining all three conditioning strategies yields the best results and that authentic MIDI velocity dynamics are critical for effective symbolic conditioning. Our method demonstrates the immense potential of leveraging symbolic priors to reconstruct high-fidelity percussive audio.
MIDI Conditioning Methods
We investigate three approaches: early injection, FiLM, and cross-attention. Empirical comparisons are in the ablation study.
Early Injection. \([\mathbf{x}_t, \mathbf{y}]\) and MIDI embeddings (from the piano roll via the encoder) are summed before feeding into the U-Net.
FiLM. We employ FiLM layers with scaling/shifting parameters:
\[[\boldsymbol{\gamma}, \boldsymbol{\beta}] = \mathrm{Conv1D}_{1\times 1}(\mathbf{M}), \quad \mathrm{FiLM}(x) = (1 + \boldsymbol{\gamma}) \odot x + \boldsymbol{\beta}\]
A \(1{\times}1\) convolution projects aligned MIDI embeddings \(\mathbf{M}\) into scale \(\boldsymbol{\gamma}\) and shift \(\boldsymbol{\beta}\), broadcast across frequency; the \((1 + \boldsymbol{\gamma})\) form initializes near identity. FiLM follows each NCSN++ residual block—we use frame-wise FiLM (which achieved lower FAD than temporal FiLM in our experiments).
Cross-Attention. Spectral features serve as queries; aligned MIDI embeddings are keys and values. Cross-attention blocks are inserted after every self-attention layer in the encoder, bottleneck, and decoder (at resolutions \(\{4,8,16,32\}\) time steps).
MiDiff integrates all three strategies within a single architecture.
Dataset
Our training dataset consists of triplets containing (clean, noisy, MIDI) samples for drum audio enhancement. The clean audio serves as ground truth, noisy versions simulate realistic degradation scenarios, and MIDI data provides symbolic musical context including note onsets, velocities, and timing information.
This multimodal structure enables the model to learn relationships between symbolic musical information and audio quality, allowing for musically-informed enhancement decisions. The MIDI component serves as a strong conditioning signal, providing explicit knowledge about drum patterns and timing for the enhancement process.
The following samples provide a direct perceptual comparison. The "Clean Original" and "Noisy Input" are provided as references.
Clean Reference Audio
Noisy Input Audio
Condition - MIDI Input File
Results
We report Fréchet Audio Distance (FAD), computed from VGGish embeddings following Kilgour et al.: a lower FAD indicates generated audio closer in distribution to clean references. Validation curves below compare our MIDI-conditioned MiDiff training run to the audio-only SGMSE baseline (StemGMD setup as in the paper).
Audio Samples
Compare the audio outputs of our baseline model and MIDI-conditioned model across different training epochs. Use the interactive tool below to select any two outputs for side-by-side comparison.
Best FAD Comparison Tool
Compare baseline vs. MiDiff outputs. Select an audio file to compare side-by-side.
Ablation Studies
Impact of MIDI Velocity Information
To assess the impact of MIDI velocity information on model performance, we conducted an ablation study comparing two variants of our MIDI-conditioned diffusion model: one utilizing full MIDI data including velocity, and another using MIDI data with velocity information removed (i.e., all note velocities set to a constant value).
Velocity Sweep
Select a velocity condition to view its spectrogram and listen. Source: drummer1_1_funk-groove1_138_beat_4-4_bluebird (last 5 seconds).
Conclusion
We introduced MiDiff, a MIDI-conditioned diffusion framework for drum enhancement that bridges symbolic and acoustic representations via early injection, FiLM, and cross-attention, with classifier-free guidance at inference to control conditioning strength without an external classifier. On StemGMD, MiDiff reaches FAD 0.29 vs. 0.56 for the audio-only baseline; ablations show all three conditioning paths together work best and that authentic MIDI velocities matter. Future work includes extending the framework toward automatic transcription and other instruments.