S-MrHiSum and S-VideoXum (Multimodal Extension) Datasets for Script-driven Multimodal Video Summarization

The S-MrHiSum and S-VideoXum (Multimodal Extension) are two large-scale datasets suitable for training and evaluation of methods for script-driven multimodal video summarization. The original MrHiSum dataset (Sul et al., 2024) was constructed from a curated subset of YouTube-8M videos, where highlight annotations were derived from YouTube’s “Most Replayed” statistics. These video replay statistics, aggregated from at least 50 unique viewers per video, serve as a reliable indicator of audience engagement. Each video was annotated at the frame level with importance scores, representing highlight intensity. Ground-truth video summaries were generated based on a predefined temporal segmentation of the videos and by solving the Knapsack problem for a given time-budget about the summary duration, ensuring that the obtained summaries are concise while covering key highlights. In total, the dataset contains 31,892 videos and the associated ground-truth annotations, supporting the training and evaluation of methods for video highlight detection and summarization. To make MrHiSum suitable for script-driven multimodal video summarization, we extended it by producing textual descriptions of the human-annotated summaries and extracting audio transcripts, forming the S-MrHiSum dataset. For this, the visual content of each ground-truth video summary (sampled at 1 fps) was described by LLaVA-NeXT-Video-7B which was prompted to "describe the important scenes in this video". Audio transcripts were extracted through a two-step pipeline: the speech was isolated from background noise using a pretrained model of Silero for voice activity detection, and then speech-to-text was performed using a pretrained model of Whisper, which outputs a series of timestamped transcripts. The created S-MrHiSum dataset contains 29,918 videos, where each video is associated with: a) ground-truth summary, b) a textual description of this summary, and c) a set of timestamped audio transcripts. The S-VideoXum dataset (Mylonas et. al, 2025) as an extension of the VideoXum dataset for cross-modal video summarization, that is suitable for training and evaluation of methods for script-driven video summarization. The multiple ground-truth summaries that are available per video of VideoXum, are associated with textual descriptions of their visual content, generated using LLaVA-NeXT-Video-7B. We created S-VideoXum (Multimodal Extension) by further extracting timestamped audio transcripts from each full-length video of S-VideoXum, following the approach described above for the videos of the S-MrHiSum dataset. In our implementations and experiments, all the visual, textual, and transcript data of the S-MrHiSum and S-VideoXum datasets have been represented using CLIP-based embeddings. The details of the scripts, embedings and all other data that we release as part of this repository are reported in SD-MVSum_Datasets_readme.md More information on the released datasets, along with technical details of the SD-MVSum script-driven multimodal video summarization method that we developed, can be found in the following preprint: https://arxiv.org/abs/2510.05652