CLAP features for Audio Moment Retrieval

This page includes CLAP features of three datasets used in Language-based audio moment retrieval [1].

Clotho-Moment

UnAV100-subset

TUT Sound Events 2017

Raw wav files are also publicly available here.

[1] H. Munakata, T. Nishimura, S. Nakada, T. Komatsu, "Language-based Audio Moment Retrieval", 2024, under review.

How to Use

We can train/evaluate audio moment retrieval models using these features in Lighthouse.Please check the instructions of Lighthouse.

Unzip the file with the following commandsClotho-moment: 

for file in clotho-moment_features.tar.part-*.gz; do gunzip "$file"; done

clotho-moment_features.tar.part-* > clotho-moment_features.tar

tar -xvf clotho-moment_features.tar

UnAV100-subset, TUT Sound Events 2017: 

tar -xvf tut2017_features.tar.gz

tar -xvf unav100-subset_features.tar.gz

Set symbolic links in Lighthouseln -s features/{dataset_name} {lighthouse_dir}/features

Train the modelpython training/train.py --model qd_detr --dataset clotho-moment --feature clap

Evaluate the model

model=qd_detr

dataset=unav100-subset

feature=clap

model_path={lighthouse_dir}/results/qd_detr/clotho-moment/clap/best.ckpt

eval_split_name=val

eval_path=data/unav100-subset/unav100-subset_test_release.jsonl
python training/evaluate.py \

--model $model \

--dataset $dataset \

--feature $feature \

--model_path $model_path \

--eval_split_name $eval_split_name \

--eval_path $eval_path