One-Step Is Enough to Judge Hybrid Noise Skimming for Full-band Text-to-Audio Generation

Anonymous authors · Paper under double-blind review

Abstract

Text-to-audio (TTA) systems based on latent diffusion and flow matching achieve high fidelity in the full-band scenarios, but their iterative sampling nature hinders practical deployment in real-world applications. Although few-step generation methods such as MeanFlow substantially reduce the number of function evaluations (NFEs), two fundamental challenges still remain in extremely low-NFE models. The first is unreliability: in extremely low NFE environments, quality of generated samples becomes highly sensitive to the initial noise. The second is quality inconsistency: increasing the number of ODE steps does not guarantee monotonic improvements in general sample quality. We propose a hybrid noise skimming framework to overcome both issues through better initial noise selection in few-step generation. The proposed framework combines two complementary components: a learnable Noise Booster amortizes the initial noise selection at the distribution level by refining random noise toward more favorable regions and Single-Step Search which improves the sample quality as instance-wise adaptation using only a single additional search step. The learnable Noise Booster restores the reliability and raises the quality floor by refining initial noise and in contrast to increasing ODE steps, enlarging the search budget yields monotonic improvement in sample quality. Experiments at 44.1 kHz show that our framework achieves SOTA performance with only a one-step generation.

Hybrid Noise Skimming

Overview of the Hybrid Noise Skimming framework: a Noise Booster refines Gaussian noise; for NFE ≤ 3 the boosted noise is fed directly to a K-step generator, while for NFE > 3 N candidates are verified with one-step generation and a score verifier before a two-step final generation.
Overview of the proposed Hybrid Noise Skimming framework. When NFE ≤ 3, search is skipped and the boosted noise is directly used for K-step generation Gθ[K], where K = NFE. When NFE > 3, the Noise Booster refines N Gaussian noise into favorable noise candidates, and Single-Step Search verifies them with one-step generation Gθ[1] to select the best one. The boosted noise is then used for two-step final generation Gθ[2].

Audio samples

Proposed 1 NFE / 16 NFE (44.1 kHz) TangoFlux 50 NFE (44.1 kHz) MeanAudio-L 1 NFE (16 kHz) SoundCTM-DiT 1 NFE (44.1 kHz) SAO small-ARC 8 NFE (44.1 kHz)

01Crickets chirping very loudly

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

02A metal clack followed by a horse neighing as gusts of wind blow and birds chirp in the background

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

03A man talking as a door slams shut followed by a door creaking

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

04Rain is falling continuously

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

05A few gunshots followed by a man speaking

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

06A series of compressed air spraying as a motor hums in the background

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

07Metal clinking with faint clip-clops of a horse

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

08A motor humming as wood is being scrapped and sanded

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

09A motorbike engine running as multiple car horns honk and a car alarm rings in the distance

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

10Dog panting followed by footsteps

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

11A drone whirring followed by a crashing sound

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

12Machinery banging and hissing

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

13Several faint clicks are followed by a person sneezing and coughing

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

14Humming of a large engine with wind blowing

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

15A sewing machine operating as a machine motor hisses loudly in the background

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

16A race car engine is idling and then the car accelerates away quickly

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

17Continuous quacking and rustling

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

18A train horn honks with a train racing by

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

19Synthesized rumbling followed by an explosion

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

20A sewing machine is in use as well as knocking and bleeping noises

Proposed1 NFE
Proposed16 NFE
TangoFlux50 NFE
MeanAudio-L1 NFE
SoundCTM-DiT1 NFE
SAO small-ARC8 NFE

Results

Table 1Objective comparison with representative TTA models on the AudioCaps test set.

Model NFE SR
(kHz)
FD ↓ IS ↑ KL ↓ CLAP ↑ PQ ↑ CE ↑ CU ↑ PC ↑ AQA ↑
Diffusion / Flow TTA
GenAU-L2001610.0315.631.510.3175.693.395.003.170.862
TangoFlux5044.116.3215.131.370.3205.753.565.023.230.910
Resonate5044.111.8012.831.380.4065.753.484.993.320.911
Accelerated TTA
AudioLCM21621.798.851.610.2425.533.264.873.000.819
ConsistencyTTA11619.569.881.480.2625.853.504.943.500.842
MeanAudio-L11611.0212.841.340.3175.523.364.913.030.882
SoundCTM-UNet21615.719.291.500.2525.663.314.893.300.836
SoundCTM-DiT144.122.1310.921.690.3465.653.234.873.060.779
SAO small-ARC844.147.929.232.420.1816.283.555.402.720.656
Proposed144.114.3510.861.410.3915.843.494.963.520.865
213.4411.701.410.3965.883.545.013.410.872
413.0812.061.380.4075.983.595.103.400.878
812.7412.761.350.4276.133.665.243.390.885
1612.7912.961.340.4386.223.725.343.400.886
3212.3513.251.300.4456.293.765.413.380.890
6412.2913.211.310.4506.353.795.463.380.889

Table 2Human evaluation results. OVL = overall quality, REL = relevance to the caption.

Model NFE OVL ↑ REL ↑
TangoFlux502.87 ± 0.123.73 ± 0.11
SAO small-ARC82.99 ± 0.122.55 ± 0.13
MeanAudio-L12.66 ± 0.113.04 ± 0.13
SoundCTM-DiT12.50 ± 0.112.51 ± 0.12
Proposed12.76 ± 0.113.21 ± 0.13
163.00 ± 0.113.42 ± 0.12