RNGBench-Game-Trajectories

# RNGBench-Game-Trajectories SFT (supervised fine-tuning) trajectory data accompanying **RNGBench · Reconstructive Non-Markov Games** — an evaluation framework that tests whether multimodal language models can reconstruct hidden state from memory and act on it in closed-loop environments (the "remember-to-act" setting, where the current observation alone is not enough and the model must recall relevant history before deciding). - 📄 **Paper:** https://arxiv.org/abs/2606.19338 - 🌐 **Project page:** https://internlm.github.io/RNGBench/ - 💻 **Code (GitHub):** https://github.com/InternLM/RNGBench ## Games The data currently covers the **two** games in RNGBench: - **Matching Pairs (MP)** — a static, categorical hidden-state game: a memory card-matching game where card identities must be recalled by location. - **3D Maze** — a dynamic, spatial hidden-state game: egocentric first-person views that must be assembled into a map to reach the goal. For each game we release two kinds of multi-turn agent trajectories: - **Optimal** — trajectories produced by a rule-based optimal policy. - **Roll** — rollout trajectories collected from strong models, namely **Kimi-K2.5** and **Qwen3.5-397B**. ## Files | File | Game | Type | Modality | | -------------------------- | -------------- | -------- | ------------ | | `data/MP_Optimal.jsonl` | Matching Pairs | Optimal | image + text | | `data/MP_Roll.jsonl` | Matching Pairs | Roll | image + text | | `data/MP_Roll_text.jsonl` | Matching Pairs | Roll | text only | | `data/3D_Maze_Optimal.jsonl` | 3D Maze | Optimal | image + text | | `data/3D_Maze_Roll.jsonl` | 3D Maze | Roll | image + text | Images for each image+text subset are shipped as a matching archive under `images/` (`MP_Optimal.zip`, `MP_Roll.zip`, `3D_Maze_Optimal.zip`, `3D_Maze_Roll.zip`). `MP_Roll_text` is text-only and has no image archive. > Note: more games / trajectories may be added over time, so this card intentionally omits exact > per-split counts. ## Format Each line of a `data/*.jsonl` file is one trajectory in [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) **ShareGPT** format: ```json { "conversations": [ {"from": "human", "value": "Remaining pairs: 4. Score: 0.\nRound 1 start ...\n<image>\nChoose a face-down card ..."}, {"from": "gpt", "value": "Thought: ...\nAction: aA"} ], "images": ["MP_Optimal/2x4_seed_11_composite/round_001_start.jpg", "..."], "system": "You are playing a Memory Match card game. ..." } ``` - `conversations`: alternating `human` / `gpt` turns. Each `<image>` placeholder in a turn's `value` corresponds, in order, to one entry of the top-level `images` list. - `images`: image paths of the form `<subset>/<relative_path>`, matching the layout inside the corresponding `images/<subset>.zip` (each archive stores files under a top-level `<subset>/` folder). - `system`: the system prompt (game rules and output format). - `MP_Roll_text` is text-only and has no `images` field. ## Usage 1. Download the repo and unzip every image archive into a single directory: ```bash mkdir -p images_extracted for z in images/*.zip; do unzip -q "$z" -d images_extracted; done ``` After extraction, an `images` entry like `MP_Optimal/2x4_seed_11_composite/round_001_start.jpg` resolves to `images_extracted/MP_Optimal/2x4_seed_11_composite/round_001_start.jpg`. 2. With LLaMA-Factory, register the subsets via the provided `dataset_info.json` (point `--dataset_dir` / `media_dir` at the directory holding `data/` and the extracted images), then train on e.g. `rngbench_mp_optimal`, `rngbench_3d_maze_optimal`, `rngbench_mp_roll`, `rngbench_mp_roll_text`, `rngbench_3d_maze_roll`. ## License MIT, following the [RNGBench](https://github.com/InternLM/RNGBench) repository. ## Citation ```bibtex @article{rngbench2026, title = {Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games}, author = {Ding, Shengyuan and Wei, Xilin and Fang, Xinyu and Duan, Haodong and Lin, Dahua and Wang, Jiaqi and Zang, Yuhang}, journal = {arXiv preprint arXiv:2606.19338}, year = {2026}, } ```