S-VideoXum: A Dataset for Script-driven Video Summarization

This is S-VideoXum, a dataset for Script-driven Video Summarization, derived from the existing VideoXum large-scale dataset (the latter is used for cross-modal video summarization). The original VideoXum dataset includes 14,000 open-domain videos up to 12.5 min. long with diverse visual content, from the ActivityNet Captions dataset. Each video is associated with multiple ground-truth video summaries - in the form of frame-level binary scores which denote the inclusion (label "1") or not (label "0") of a frame in the video summary - that were obtained by 10 different human annotators. The existence of multiple ground-truth summaries per video (10 in total) makes the VideoXum dataset well-suited for extending it to the script-driven video summarization task, as it allows to train a summarization method to generate different summaries for a given video driven by a different script about the content of each individual summary. To make VideoXum suitable for training and evaluation of script-driven video summarization methods, we extended it by producing natural language descriptions of the different ground-truth summaries that are available per video. These serve as the scripts that can drive the summarization process. For this, we employed the publicly-available state-of-the-art Large Multimodal Model LLaVA-NeXT-7B. Since some of the videos from the VideoXum dataset were not publicly-available during this work, the extended S-VideoXum dataset for script-driven video summarization differs from the original one in terms of number of videos, including data for 11,908 videos. These we split in 6,782 samples for training, 1,707 for validation and 3,419 for testing. The generated natural language descriptions for the ground-truth summaries, i.e., the "scripts" (as well as similarly-derived natural language descriptions for the full-length videos) of the S-VideoXum dataset, along with the extracted (CLIP-based) embeddings from visual and textual data and the used data splits in our experiments, are publicly-available in the present repository. More information on the structure of the provided data can be found in the dataset_readme.md file and in https://github.com/IDT-ITI/SD-VSum.