Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding

LilyBench is an evaluation framework for large language models on LilyPond, the textual, code-like score format used by the engraving system of the same name. This archive is its companion data release: it bundles the curated in-domain corpus, the deterministic work-level splits, the prompt bank, the few-shot configurations, and the JSON output of every generation and understanding evaluation cell reported in the accompanying manuscript. Unpacked under the data/ directory of the LilyBench source repository, it is the only artifact you need to reproduce Tables 1 and 2 of the paper from the LilyBench command-line interface — no GPU required. The accompanying paper is currently under review at Ital-IA 2026, the 6th National Conference on Artificial Intelligence of the Italian CINI consortium (Rome, Italy): Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding. Matteo Spanio, Mohammad Torabi, Andrea Poltronieri, Antonio Rodà. Submitted to Ital-IA 2026. Source code: CSCPadova/lilybenchLilyBERT checkpoint (used by the Fréchet Music Distance metric): csc-unipd/lilybert What is inside bmdataset/ — the Baroque Music Dataset, a musicologically curated collection of 391 baroque and early classical works (71 composers, 16 musical forms, 25 MIDI instruments) transcribed in LilyPond from original manuscript sources by the team behind BaroqueMusic.it. The folder contains the raw multi-file LilyPond projects with their rendered MIDI and PDF, plus 2,645 preprocessed single-file scores in Dutch (nederlands) note names, alongside per-work metadata, a file-level manifest, and the label taxonomy. Distributed under CC BY-NC-ND 4.0; see bmdataset/README.md for full attribution. splits/ — deterministic work-level train, validation, and test splits over the Baroque Music Dataset (train.jsonl, val.jsonl, test.jsonl). No work crosses the train/val/test boundary, so memorisation of a sibling part is impossible. The test split is the in-domain reference used by the Jensen–Shannon similarity and the Fréchet Music Distance metrics. fullfile_dataset/all_examples.jsonl — the chunked union of every preprocessed Baroque Music Dataset score before splitting, as a single JSONL of metadata-prefixed text records. prompt_bank/ — the byte-stable prompt banks produced by the LilyBench prompt-bank builder and reused across every (model, regime) evaluation cell. bank_200_short.jsonl is the 200-prompt bank used in the manuscript; bank_1000.jsonl is a larger bank for additional experiments. configs/ — the two few-shot configurations used in the submission: fewshot_train_distribution.txt (few-train regime, demos drawn from the bmdataset training split) and fewshot_ablation_amin.txt (few-ablation regime, hand-written A-minor fragments). eval_generation/<model>_<regime>/ — per-cell generation outputs. Each folder ships summary.json (compile rate, MusPy descriptor averages, Jensen–Shannon similarity against the in-domain test split and against Mutopia, plus a per-sample breakdown), fmd_test.json and fmd_mutopia.json (the LilyBERT-based Fréchet Music Distance against the two references), and where applicable loss.json (LoRA cells only — HuggingFace Trainer eval loss on the test split) and rescue_stats.json (counts of truncated decodings repaired in the lora_rescued_v2 cells). Four backbones — phi4, qwen-coder, deepseek-coder, codestral — are crossed with three regimes — zero-shot, few-shot from the training distribution, and an earlier few-shot configuration kept for transparency. LoRA cells are released as supplementary material only. eval_understanding/ — per-cell understanding outputs. eval_l40s/<model>/summary.json holds the eight ABC-Eval-adapted tasks derived from Mutopia (bar_count, metadata_qa, bar_sequencing, next_bar_prediction, metadata_prediction, music_captioning, composer_recognition, genre_recognition). eval_emotion/<model>/summary.json holds the EMOPIA emotion-recognition task (four valence/arousal quadrants with a confusion matrix). eval_errors/<model>/summary.json holds the error-detection task on synthetically corrupted scores (five corruption categories, macro-F1 and per-category F1). All four backbones are evaluated; decoding is greedy, so on identical hardware and library versions the numbers are deterministic. Generation cells at a glance Backbone HuggingFace identifier Cells phi4 microsoft/phi-4 phi4_zero, phi4_few, phi4_few_old_simple_demos, plus phi4_lora and phi4_lora_rescued_v2 qwen-coder Qwen/Qwen2.5-Coder-7B-Instruct qwen-coder_zero, qwen-coder_few, qwen-coder_few_old_simple_demos, plus qwen-coder_lora and qwen-coder_lora_rescued_v2 deepseek-coder deepseek-ai/deepseek-coder-7b-instruct-v1.5 deepseek-coder_zero, deepseek-coder_few, deepseek-coder_few_old_simple_demos codestral mistralai/Codestral-22B-v0.1 codestral_zero, codestral_few, codestral_few_old_simple_demos, plus codestral_lora and codestral_lora_rescued_v2 How to use it Clone the LilyBench repository, install the package, and unpack the archive under data/: git clone https://github.com/CSCPadova/lilybench.git cd lilybench pip install -e . unzip lilybench_zenodo.zip -d data/ The numbers reported in Tables 1 and 2 of the manuscript can be read directly from the JSON summaries in eval_generation/ and eval_understanding/ without re-running inference. To re-run inference, follow the Quickstart section of the repository README using prompt_bank/bank_200_short.jsonl and the configurations under configs/; the prompt bank is byte-stable, so generations will line up one-to-one with the released summary rows by sample identifier. What this archive does not contain The Mutopia out-of-domain corpus and the EMOPIA emotion-recognition corpus are not redistributed here. They are reconstructed from their upstream sources by two helper scripts shipped in the LilyBench repository: scripts/convert_mutopia.py upgrades a raw Mutopia tree with convert-ly (the upstream corpus carries roughly fifteen years of LilyPond syntax drift) and emits dataset_mutopia.json under data/mutopia/; scripts/prepare_emopia.py downloads EMOPIA, runs midi2ly on every clip, and emits the manifest CSV under data/emopia/. The two scripts are only needed by users who want to extend the benchmark — consumers of this archive who only intend to reproduce the released numbers do not need to run them. The raw .ly decodings and the per-task prediction JSONL files produced during the cluster sweep are also not bundled. They total tens of gigabytes, they are not required to verify the released numbers, and they can be regenerated deterministically (for understanding) or stochastically (for generation, with do_sample=True and known seeds) from the LilyBench command-line interface and the shipped prompt bank. Determinism The prompt bank is built with seed 1234; per-prompt inference seeds are seed_base + i. Generation uses do_sample=True, so numeric metrics drift slightly between runs and trends rather than exact text should be compared. Understanding decoding is greedy (do_sample=False, temperature 0, max_new_tokens 20) following ABC-Eval, so understanding predictions and metrics are deterministic on identical hardware and library versions. The work-level splits guarantee that no part of the same piece crosses the train, validation, or test boundary.