INSTRUCTFX2FX DAFx26

$ cat abstract.md

INSTRUCTFX2FX▮

A Multi-turn Text-to-Effect System for Sequential Audio Effect Refinement

AUTHORS
Song-Ze Yu*·Milan Liessens Dujardin·Yuxuan Cai·Wantong Zhang·Brian Cruz·Jeremy Wagner·Carmine-Emanuele Cella*
LAB
UC Berkeley · CNMAT
VENUE
Proc. 29th Int. Conf. Digital Audio Effects (DAFx) · 2026
TAGS
text-guided audio editing · multi-turn FX refinement · LLM · CLAP · audio production

* Corresponding authors

READ PAPER CODE
[01]Abstract · Sequential FX Refinement

$ background

Existing text-to-effect systems are largely single-shot: one descriptor maps to one preset. Real audio engineering is instead sequential — engineers refine an existing FX chain through successive instructions {I₁, I₂, …}.

$ problem

This poses a stateful problem that single-shot systems do not address: given a current chain, parameter state P and a new instruction, update the sound toward the user's evolving intent while preserving what earlier turns already achieved. We name this problem sequential FX refinement.

$ answer

This is why we propose InstructFX2FX, a hybrid that works like a single audio engineer — a brain paired with an ear. The LLM is the brain: it selects the effects and proposes the initial parameters. CLAP-guided optimization is the ear: it listens to the rendered audio and adjusts the parameters to pull it toward the instruction, turn after turn. The LLM sets the starting point; CLAP sets the refinement direction.

[02]Architecture · Three-Layer Harness

FIG. 01 · HARNESS OVERVIEW

Session state history (prompt, FX chain, params) | current params read history read params update state Input audio, text LLM Planner FX selection via tool calls Routing Module init | reuse | mixed Gradient Descent differentiable FX Bayesian Optimization non-differentiable FX Output params, audio CLAP-guided optimization

FIG. 02 · ROUTING MODULE EXPLAIN

turns instruction llm planner routing module session chain › “make it brighter” eq comp rev dist delay pitchshift bitcrush I · INIT-ONLY eq › “make it sound like a church” comp rev dist delay pitchshift bitcrush III · MIXED eq rev › “too harsh, soften it” comp dist delay pitchshift bitcrush II · REUSE-ONLY eq rev available fx tools pool newly init reused from prior state
[03]Listen · Gradient-Descent Sessions3 SESSIONS

Real sessions on the gradient-descent path (EQ & reverb). Each turn shows what LLM Planner decided, then plays the result. Click a waveform to seek, toggle dry / result to A​/B, or drag the gradient-descent track to hear the optimization converge.

Loading sessions…
[04]Evaluation · By the Numbers

EXP.01 · SEQUENTIAL MMD

Sequential MMD across directed prompt pairs: InstructFX2FX beats the LLM-only baseline on 9 of 10
InstructFX2FX lowers MMD versus an LLM+LLM initialize-then-reprompt baseline on 9 of 10 pairs.

EXP.02 · MMD CONVERGENCE TRAJECTORY

Contact sheet of MMD trajectories for all 10 directed descriptor pairs, source and target panels, piano and violin vs. LLM-only baseline
Refinement drives the target MMD (right) below the LLM+LLM initialize-then-reprompt baseline while the initialized MMD (left) does not rise, showing that the audio moves toward the target without discarding earlier instruction.

EXP.03 · LLM INITIALIZATION ABLATION

CLAP similarity rising and MMD falling across optimization iterations
Departing from LLM initialization, CLAP-guided refinement achieves lower DSP-feature MMD (left) and Fx-Encoder++ MMD (right).
[05]Cite
@misc{yu2026instructfx2fxmultiturntexttopresetdemo,
  title         = {InstructFX2FX: A Multi-turn Text-to-Preset Demo for
                   Iterative Audio Effect Refinement},
  author        = {Song-Ze Yu and Milan Liessens Dujardin and
                   Yuxuan Cai and Wantong Zhang and Brian Cruz and
                   Jeremy Wagner and Carmine-Emanuele Cella},
  year          = {2026},
  eprint        = {2606.22005},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2606.22005}
}
INSTRUCTFX2FX / DAFx26 UC BERKELEY · CNMAT
▲