🎧 VoxZip

arXiv ACM MM 2026

Official code for the ACM MM 2026 paper "VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference".

✨ Overview

Long-context audio inference on multimodal LLMs (e.g., Qwen3-Omni) is bottlenecked by the KV cache: audio frames dominate the sequence length, so memory and latency grow quickly with conversation duration. VoxZip compresses this audio-side KV cache along the temporal axis, anchored to ASR segment boundaries.

The idea is simple. Run Whisper ASR on each history audio to obtain segment-level timestamps — one [start, end] per ASR segment, not per-word timestamps. Each segment is aligned to its text-token span and audio-frame span, the audio and text embeddings are fused at segment granularity, and a temporal heavy-hitter policy then merges / delays redundant KV entries. The result is a shorter fused sequence that preserves semantic coverage while cutting cache size.

Why segment-level, not word-level? Per-word timestamp fusion aligns audio frames to individual tokens, which is noisy and underperforms. Segment-level fusion uses one timestamp per ASR segment — coarser, cleaner, and empirically stronger. The word-level variant was removed from this repo for that reason.

🔥 Highlights

🎯 Segment-anchored fusion — ASR segments (not words) define the alignment
🧠 Temporal KV compression — Heavy-hitter + time merge / delay (ours)
🧩 Drop-in on Qwen3-Omni — uses only the public transformers==4.57.2 API, no fork needed
📊 6 benchmarks — Vox-Infinity, IEMOCAP, AudioMarathon, MMAR, MMAU, MMSU, SPIRAL
🔇 Noise robustness — built-in SNR sweep via the *_noise drivers
🧹 Clean & English-commented — dead code removed, comments translated, env-var paths

🗂️ Project Structure

VoxZip-ACMMM2026/
├── kv_compression/ # KV-cache methods
│ ├── ablation/ # ours: temporal_kv_merge / temporal_kv_delay / temporal_kv_delay_merge
│ ├── embedding_compression/ # ours: ASR segment-fusion (EmbedAudio2Text, ...)
│ ├── snapkv/ pyramid/ streamingLLM/ segKV/ cam/ l2norm_cluster/ # baselines
├── utils/ # custom_processor, custom_processor_embed, filter_word
├── eval/ # scoring scripts (GPT-judge, exact-match, ABCD, ...)
├── scripts/ # run_*.sh per benchmark
├── results/ # output JSONL (created on run)
├── setup_env.sh # exports VOXZIP_* path env vars
├── requirements.txt
├── README.md
└── *.py # benchmark drivers (root, so `from kv_compression...` works)

🚀 Installation

Step 1: Create environment

conda create -n voxzip python=3.10 -y
conda activate voxzip

Step 2: Install dependencies

VoxZip uses only the public API of transformers 4.57.2 (it inherits from Qwen3OmniMoe* classes and rewrites the attention layer), so the upstream wheel works — no patched fork required.

cd VoxZip-ACMMM2026
pip install -r requirements.txt

Step 3: Configure paths

Edit setup_env.sh so the VOXZIP_* fallbacks point to your local copies of the model weights, Whisper weights, and datasets. Then source it:

source setup_env.sh

Every driver reads these variables via os.environ.get("VOXZIP_...", "<fallback>"), so on the original development host the defaults already work.

📦 Data & Weights (not bundled)

This repo ships code only. Download separately and point the env vars at them:

Asset	Source	Env var
Qwen3-Omni-30B-A3B-Instruct	`Qwen/Qwen3-Omni-30B-A3B-Instruct`	`VOXZIP_MODEL_PATH`
Whisper large-v3-turbo	`openai/whisper-large-v3-turbo`	`VOXZIP_WHISPER_WEIGHT`
Vox-Infinity (+QA)	`vox-infinity/Vox-Infinity`	`VOXZIP_DATA_DIR`
MMSU	`ddwang2000/MMSU`	`VOXZIP_DATA_DIR`
MMAR	`BoJack/MMAR`	`VOXZIP_MMAR_DATA`
MMAU-test-mini	`gamma-lab-umd/MMAU-test-mini`	`VOXZIP_MMAU_DATA`
SPIRAL	(see original `spiral_datasets/`)	`VOXZIP_SPIRAL_DATA`
AudioMarathon	(see original `AudioMarathon/`)	`VOXZIP_AUDIOMARATHON_DATA`

🎯 Running Experiments

All commands assume you have run source setup_env.sh. Pick GPUs with --visible_devices.

Part 1: Vox-Infinity (main benchmark)

VoxZip temporal component — KV-compression ablation across all baselines + ours:

bash scripts/run_vox_infinity.sh # all 4 splits, temporal_kv_merge (ours)
bash scripts/run_vox_infinity.sh Conversational snapkv # single split + a baseline

VoxZip full method — ASR segment fusion + temporal merge (the complete pipeline):

bash scripts/run_voxzip_full.sh Ultra-Multi-Turn-Dialogues

Part 2: IEMOCAP

bash scripts/run_iemocap.sh h2o_time_merge

Part 3: AudioMarathon

bash scripts/run_audiomarathon.sh seg RACE # VoxZip full method
bash scripts/run_audiomarathon.sh fullkv GTZAN # full-KV baseline
bash scripts/run_audiomarathon.sh baselines RACE # SnapKV / PyramidKV / ...

Part 4: MMAR / MMAU / MMSU / SPIRAL

Each benchmark has a kv_cache_* driver (baselines) and an audio2text_seg_* driver (VoxZip full method):

bash scripts/run_bench.sh mmar seg # VoxZip full method on MMAR
bash scripts/run_bench.sh spiral kv_cache h2o # H2O baseline on SPIRAL
bash scripts/run_bench.sh mmsu seg_noise # + white-noise robustness (20 dB)

Part 5: Noise robustness

Any *_noise driver accepts --noise_snr_db:

python Ablation_test_multi_noise.py --temporal_kv_merge=True --noise_snr_db=20 ...

📊 Evaluation

# Generic GPT-4o-mini judge (Vox-Infinity, MMAR, MMAU, SPIRAL) — needs an OpenAI-compatible API key
bash scripts/run_eval.sh results/Vox/*_temporal_kv_merge.jsonl
# Benchmark-specific scorers
python eval/eval_iemocap.py --input_file <iemocap.jsonl> # per-emotion accuracy
python eval/evaluate_mmsu.py --input_file <mmsu.jsonl> # ABCD by category
python eval/evaluate_Audio_Marathon.py --input_file <audiomarathon.jsonl>
python eval/summary_results.py --input_file <vox.jsonl> # duration-bucketed accuracy
python eval/plot_niah.py # NIAH heatmap (paper figure)
python eval/statics.py # KV-length statistics

The GPT judge in eval/GetGPTScore_meituan_api.py calls an OpenAI-compatible endpoint; set the API key / base URL inside that file.

🔑 API Key (GPT judge)

# Linux / macOS
export OPENAI_API_KEY="your_api_key_here"
bash scripts/run_eval.sh results/Vox/*.jsonl

# Windows PowerShell
$env:OPENAI_API_KEY="your_api_key_here"
python eval/evalute_output.py --input_file results/Vox/test.jsonl

⚙️ Key configuration

Flag	Meaning	Typical value
`--temporal_kv_merge`	VoxZip temporal merge (ours)	`True`
`--temporal_kv_delay`	VoxZip temporal delay (ours)	`True`
`--heavy_budget` / `--recent_budget`	Temporal KV cache size	900 / 300 (Vox-Infinity)
`--asr_thresold`	Whisper word confidence filter	0.3
`--embed_compression`	enable ASR segment fusion	`True`
`--noise_snr_db`	white-noise SNR (noise drivers)	20 / 10 / 5
`--visible_devices`	GPU ids	`"0,1"`

Per-benchmark budgets differ — see each driver's --help.

🧪 Minimal usage

Run a single Vox-Infinity split with the VoxZip temporal method:

source setup_env.sh
python Ablation_test_multi.py \
 --datasets_content $VOXZIP_DATA_DIR/Vox-Infinity/Conversational \
 --datasets_qa $VOXZIP_DATA_DIR/Vox-Infinity-QA/Conversational \
 --output_jsonl $VOXZIP_OUTPUT_DIR/Vox/Conversational_temporal_kv_merge.jsonl \
 --visible_devices 0,1 \
 --temporal_kv_merge=True \
 --heavy_budget 900 --recent_budget 300

🧠 Methods at a glance

Component	What it does	Code
Semantic-Anchored	Whisper ASR → segment alignment → interleave fusion	`audio2text_embedding_compression_seg.py`, `kv_compression/embedding_compression/`
Temporal	H2O heavy-hitter + KV merge / delay	`kv_compression/ablation/h2o_time_merge.py`, `temporal_kv_delay.py`, `temporal_kv_delay_merge.py`
Baselines	H2O / SnapKV / PyramidKV / StreamingLLM / CAM / L2Norm / ChunkKV	`kv_compression/{ablation/temporal_kv_base, snapkv, pyramid, streamingLLM, cam, l2norm_cluster, segKV}`

D2O and TOVA baselines were removed per request.

📚 Citation

If this repository is helpful to your research, please cite:

@inproceedings{voxzip2026,
 title = {VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference},
 author = {Jia, Wenxu and Fu, Dongjie and Cheng, Xize and Feng, Fangming and Li, Linjun and Chen, Wenshi and Li, Yingming and Zhao, Zhou and Jin, Tao},
 booktitle = {Proceedings of the 34th ACM International Conference on Multimedia (ACM MM)},
 year = {2026},
 doi = {10.1145/3767308.3835719},
 eprint = {2608.08569},
 archivePrefix = {arXiv},
 primaryClass = {cs.AI},
}

Name		Name	Last commit message	Last commit date
eval		eval
kv_compression		kv_compression
scripts		scripts
utils		utils
.gitignore		.gitignore
Ablation_test_iemocap.py		Ablation_test_iemocap.py
Ablation_test_iemocap_jsonl.py		Ablation_test_iemocap_jsonl.py
Ablation_test_multi.py		Ablation_test_multi.py
Ablation_test_multi_noise.py		Ablation_test_multi_noise.py
AudioMarathon_em_debug.py		AudioMarathon_em_debug.py
AudioMarathon_em_debug_noise.py		AudioMarathon_em_debug_noise.py
AudioMarathon_em_debug_noseg.py		AudioMarathon_em_debug_noseg.py
LOCAL_SETUP.md		LOCAL_SETUP.md
README.md		README.md
Test_on_AudioMarathon0104.py		Test_on_AudioMarathon0104.py
audio2text_embedding_compression_seg.py		audio2text_embedding_compression_seg.py
audio2text_embedding_compression_seg_noise.py		audio2text_embedding_compression_seg_noise.py
audio2text_seg_mmar.py		audio2text_seg_mmar.py
audio2text_seg_mmau.py		audio2text_seg_mmau.py
audio2text_seg_mmsu.py		audio2text_seg_mmsu.py
audio2text_seg_spiral.py		audio2text_seg_spiral.py
audio2text_seg_spiral_noise.py		audio2text_seg_spiral_noise.py
kv_cache_mmar.py		kv_cache_mmar.py
kv_cache_mmau.py		kv_cache_mmau.py
kv_cache_mmsu.py		kv_cache_mmsu.py
kv_cache_spiral.py		kv_cache_spiral.py
kv_cache_spiral_noise.py		kv_cache_spiral_noise.py
requirements.txt		requirements.txt
setup_env.sh		setup_env.sh
snapkv_test_AudioMarathon.py		snapkv_test_AudioMarathon.py
snapkv_test_AudioMarathon_noise.py		snapkv_test_AudioMarathon_noise.py
time_merge_test.py		time_merge_test.py
visual_tsne.py		visual_tsne.py

Navigation Menu

Uh oh!

Latest commit

History

Folders and files

Repository files navigation