See the Movie, Hear the Song, Read the Book: Extending MovieLens-1M, Last.fm 2K, and DBBook with multimodal Data

Datasets Structure This folder contains the multimodal features of the three state-of-the-art we have extended (`MovieLens-1M`, `DBbook`, `Last.FM-2K`). For each folder, we provide both the interaction data in the original format (in the folder `interaction_data`) and the multimodal features in several formats, based on the needs (in the `multimodal_data` folder). In the following, we provide all the information needed to work with such data. Note that, although some dataset-specif details mght change, the general strucuture is common to all the three datasets. Dataset statistics CF data ML1M DBbook LFM2k Users 6040 6181 1892 Items 3706 7672 17642 Interactions 1000209 140360 92834 Interaction data The `interaction_data` contains the interaction data provided in the original version of each datasets. We prefer sharing the original version so that each one can pre-process it in the way they prefer (e.g., apply a certain k-core filtering, adapt the task to sequential recommendation by exploiting temporal information - when available -, and so on). ML1M In `MovieLens-1M`, interaction data includes user information (`users.dat`), movie information (`movies.dat`), and user ratings (`ratings.dat`); in order to work with this data, we suggest to read those files with the `pandas` python library, by using the `ISO-8859-1` encoding (if using other encoding, like `utf-8`, the reading will raise an error); the default separation character sequence is `::`. For example, in order to read ratings and movie information, one should use: ratings = pd.read_csv('interaction_data/ratings.dat', sep='::', names=['user', 'item', 'rating', 'timestamp'])movies = pd.read_csv('interaction_data/movies.dat', sep='::', names=['id', 'name', 'genres'], encoding='ISO-8859-1') DBbook In `DBbook`, interaction data includes training and testing data (already split, as in the original version); unfortunately, such version cannot be download anymore as the original web page is no longer accessible; using tools like [waybackmachines, it possible to access that page and download some files, but only the training data is available in the backups that have been made, while test data is not obtaibale.For these reasons, we considered the version of the dataset that have been used in other works listed below and reachable at the public repository of our SWAP Research Group: - https://dl.acm.org/doi/abs/10.1145/3523227.3551484- https://dl.acm.org/doi/abs/10.1145/3565472.3592965- https://dl.acm.org/doi/abs/10.1145/3627043.3659548- https://link.springer.com/article/10.1007/s11257-024-09417-x This way, we have been able to reconstruct the full verison of this dataset.Similarly to `MovieLens-1M`, interaction data contains user ratings in the `train.tsv` and `test.tsv` files, and book information in the `DBbook_Items_DBpedia_mapping.tsv` file. We suggest to load such data using `pandas` as follows: train = pd.read_csv('interaction_data/train.tsv', sep='\t', names=['userID', 'itemID', 'rating'])test = pd.read_csv('interaction_data/test.tsv', sep='\t', names=['userID', 'itemID', 'rating'])books = pd.read_csv('interaction_data/DBbook_Items_DBpedia_mapping.tsv', sep='\t') Last.FM-2K In `LFM2K`, interaction data is encoded in the `user_artists.dat` file; this file encodes the listening counts for each pair (user,item) available (from this information, it is possible to derive the user ratings); the file `artist_info` encodes information assiciated to the artists, including the name of the artist, the URL of the associated Last.FM resource, and the link to the image (not available anymore); the file `tags.dat` contains the set of all the possible tags users attributed to artists, while all the tags attributed to specific artists is encoded in the `user_taggedartists.dat` file (the `user_taggedartists-timestamps` contains, in addition, the timestamp of the attribution). In order to read data, we suggest to use `pandas` as follows: interactions = pd.read_csv('original_data/user_artists.dat', sep='\t')artist_info = pd.read_csv('original_data/artists.dat', sep='\t')usertag = pd.read_csv('original_data/user_taggedartists-timestamps.dat', sep='\t')tags = pd.read_csv('original_data/tags.dat', sep='\t', encoding='latin-1') Multimodal data Each dataset is also provided with with multimodal data, in the `multimodal_features` folder. In this folder, we include the data source data we considered (plain text and links to image/audio/video files), with the pre-trained multimodal features. Here is the coverage of multimodal information w.r.t. the datasets considered: Multimodal item coverage ML1M DBbook LFM2K Text 3667 (Plots) 4197 (Abstracts) 2813 (Tags) Image 3197 (Movie posters) 7588 (Book covers) 2820 (Top-5 Album Covers) Audio 3104 (Trailer audio) - 2742 (Top-5 album songs) Video 3105 (Trailer video) - - As depicted in the table, for `ML1M` we have gathered movie plots (text), movie posters (images), and movie trailers (for audio and video); in the `movielens_1m/multimodal_features` folder, we provide an extended mapping named `ml1m_full_extended_mapping`, in which we report which are the links to download `covers` and `trailers`, while `text` is available in the `text_ml1m.tsv` file. For `DBbook`, we have gathered book abstracts (text) and book covers (images); in the `dbbook/multimodal_features` folder, we provide an extended mapping named `full_extended_dbbook_img_links.tsv`, in which we report which are the links to download the `book covers`, while `text` is available in the `dbbook_text.tsv` file. For `LFM2K`, we have gathered artist tags (text), the top-5 most popular album covers (images), and the top-5 most popular audio songs (audio); in the `lfm2k/multimodal_features.tsv` folder, we report extended mappings, named `lfm2k_song_extended_mapping.tsv` and `lfm2k_covers_extended_mapping.tsv`, tha encode the top-5 most popular `songs` and `album covers` for each artist, respectively; on the other hand, the `lfm2k_text.tsv` encode the `text` we considered, obtained from the user tags. With this information, anyone can donwload the raw features and use them in their recommendation scenario; in our case, to carry out our experiments, we considered the following state-of-the-art multimodal encoders: Text: we considered `MiniLM` and `MPNET` (for `ML1M`, `DBbook`, and `LFM2K`) Image: we considered `ResNet152`, `VGG`, `ViT_AVG`, `ViT_CLS` (for `ML1M`, `DBbook`, and `LFM2K`) Audio: we considered `VGGish` and `Whisper` (for `ML1M` and `LFM2K`) Video: we considered `I3D` and `R(2+1)D` (for `ML1M`) The resulting features have been dumped as `dict` (`item_id` -> `np.float32` embedding) in a pickle `.pkl` file, that can be found in the `multimodal_features/dict` folders (one for each dataset); moreover, to avoid any error in reading such files, we have also saved the embeddings in `.json` files, in the `multimodal_features/json` folders (one for each dataset); finally, to reproduce our experiments, we report the same data as `.npy` files (as required by `MMRec`), that can be found in the `multimodal_features/npy` folders (one for each dataset). Encode multimodal features In order to learn the multimodal features by exploiting the encoders we considered in our experimental analysis, please refer to the GitHub reporisory associated to out work (`1_learn_mm_feat` folder, that contains all the instruction to set up the environments, download multimodal data, and extract multimodal features).