Multimodal Music Recommendation Dataset

Overview

This dataset is a multimodal enrichment of the publicly available LastFM-1K dataset, designed for session-based music recommendation research. It combines user listening histories with audio embeddings, lyric embeddings, LLM-generated semantic metadata, and engagement signals to enable evaluation of multimodal recommendation systems.

Dataset Statistics

Total Users: 814

Total Listening Events: 4.21 Million

Unique Tracks: 295,957

Unique Artists: 43,406

Total Sessions: 421,396

Top-50K Song Catalog: 50,029 tracks

Non-English Songs: 27.1% across 42 language codes

Unique Genres: 81

Date Range: February 2005 to June 2009

Dataset Structure

The dataset contains three folders: Completion Ratio, Embeddings, and Metadata.

Completion Ratio

File: user_sessions_with_completion.csv

Each row represents a single listening event with the following columns:

user_id: Anonymized user identifier

timestamp: UTC timestamp of the listening event

session_id: Session identifier, where sessions are split by 20 minutes of inactivity

artist_name: Artist name

track_name: Track name

track_id: MusicBrainz track identifier

track_index: Remapped integer index in the Top-50K catalog

song_duration_seconds: Total duration of the track in seconds

user_listening_duration_seconds: Actual duration listened by the user in seconds

completion_ratio: Ratio of listening duration to song duration, ranging from 0 to 1

A completion ratio of 1.0 indicates the user listened to the full track. Missing values indicate unavailable duration metadata.

Embeddings

Pre-computed dense vector representations for all tracks in the Top-50K catalog, organized by modality and encoder.

audio_embeddings/clap/ - CLAP, 512 dimensions

audio_embeddings/mert/ - MERT, 256 dimensions

audio_embeddings/music2vec/ - Music2Vec

audio_embeddings/encodec/ - EnCodec

audio_embeddings/mfcc/ - MFCC handcrafted descriptors, 63 dimensions

lyrics_embeddings/minilm/ - MiniLM, 384 dimensions

lyrics_embeddings/bge_m3/ - BGE-M3

lyrics_embeddings/mpnet/ - MPNet

lyrics_embeddings/multilingual/ - Multilingual Sentence Encoder

lyrics_embeddings/bert/ - BERT

Each folder contains an index file, embeddings.csv, with the following columns:

track_index: Integer index mapping to the Top-50K catalog

artist_name: Artist name

track_name: Track name

lyrics_source: Source of lyrics, such as lrclib or whisper

Individual track embeddings are stored as PyTorch tensor files (.pt), named by their track index, for example, audio_clap_0.pt. Each file can be loaded using torch.load() and returns a one-dimensional tensor. Index 0 is reserved as zero-padding for missing tracks. Files can be joined across modalities using the key: artist_name.lower().strip() + "||" + track_name.lower().strip()

Metadata

File 1: annotations_no_meta_few_shot.csv

Contains MGPHot perceptual annotations generated via few-shot prompting using GPT-5. Each track is rated on 58 musicological attributes across 7 categories, normalized to the range 0 to 1:

Lyrics: Angry, Sad, Happy, Humorous, Love, and other mood dimensions

Vocals: Register, Timbre, Breathiness, Smoothness

Harmony: Minor and Major Key Tonality, Harmonic Sophistication

Rhythm: Tempo, Swing Feel, Syncopation, Danceability

Instrumentation: Drum Set, Electric Guitar, Acoustic Guitar, Piano

Sonority: Live Recording, Acoustic, Electric

Composition: Focus on Lead Vocal, Focus on Melody, Focus on Lyrics

Annotations were generated using Azure OpenAI GPT-5 via the Batch API with two few-shot anchor tracks representing opposite musical extremes: Mrs. Robinson by Simon and Garfunkel and Master of Puppets by Metallica. Annotations were validated against ground-truth MGPHot ratings with Spearman correlation above 0.50 for most attribute categories.

File 2: top_50k_full_augmented_v2.csv

Contains extended musicological features and structured audio attributes for the Top-50K catalog:

Lyric Structure: Rhyme scheme, internal rhyme density, vocabulary richness, repetition ratio, profanity intensity

Narrative: Dominant perspective, perspective shift

Novelty: Genre subversion score, humor and irony presence, spoken word ratio

Audio Attributes: Valence, energy, danceability, and tempo retrieved via the ReccoBeats API

Extended features were generated using a three-model consensus of LLaMA-3.3-70B-Instruct, Qwen2.5-7B-Instruct, and Mistral-Nemo-12B-Instruct. Audio attributes cover 47,113 tracks representing 94.2% of the catalog.

How to Use

Load the embeddings.csv index file from the desired embedding folder to get the track index to song mapping. Use the track index to load the corresponding .pt file using torch.load(). Join embeddings across modalities using the normalized artist and track name key described above. Use user_sessions_with_completion.csv to incorporate engagement signals per user interaction. Join metadata files using artist_name and track_name to enrich track representations with musicological annotations.