visres_bench

# VisRes Bench [](https://visres-bench.github.io/) [](https://arxiv.org/abs/2512.21194) **VisRes Bench** is a benchmark for evaluating the **visual reasoning** capabilities of Vision-Language Models (VLMs) in naturalistic settings without contextual language supervision. It is introduced in the paper [*VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs*](https://arxiv.org/abs/2512.21194). ## Paper Summary Vision-Language Models excel at captioning and VQA, but it is unclear how much they rely on **visual reasoning** versus **linguistic priors**. VisRes addresses this by using **image-only, four-choice** tasks on **real-world images** (~19,000 samples) so that performance reflects visual reasoning rather than textual shortcuts. The benchmark is organized in **three levels** of increasing complexity: - **Level 1 — Perceptual grounding:** Local patch completion (masked tile + 4 candidate patches) under perturbations (blur, brightness, rotation, edges, location) and global occlusion (50% or 80% of the image masked). Tests robustness and amodal completion. - **Level 2 — Single-attribute rule:** Raven-style 3×3 grids with one missing cell; one attribute (color, count, or orientation) follows a row-wise rule. Includes uniform, 3-different, 2-similar-1-different, count progression, arithmetic, and min-max subtasks (~5,956 samples). - **Level 3 — Multi-attribute composition:** Same 3×3 format but multiple attributes (color, count, orientation, object identity) with row-wise, grid-wise, or spiral rules (~2,522 samples). **Main findings:** State-of-the-art VLMs perform near **random (25%)** on many subtasks under subtle perceptual changes. Performance is stronger on color than count, and weakest on orientation. When the same logical structure is given as **text**, models do much better, indicating a **visual-to-symbolic** bottleneck rather than a pure reasoning limit. Higher resolution and guided/thinking prompts help but do not close the gap to human baselines. --- ## Main Results (Guided Prompting, Thinking Mode When Available) Accuracy (%) across levels and subtasks. Random chance = 25%. <table> <thead> <tr> <th>Setting</th> <th>GPT-5</th> <th>GPT-4o</th> <th>Gemini-2.5</th> <th>Qwen3-VL-4B</th> <th>Qwen3-VL-30B</th> <th>Mimo-VL-7B</th> </tr> </thead> <tbody> <tr><td colspan="7"><strong>Level-1</strong></td></tr> <tr><td>Edges</td><td>27.17</td><td>23.91</td><td>25.00</td><td>16.67</td><td>25.00</td><td>22.30</td></tr> <tr><td>Location</td><td>23.71</td><td>20.62</td><td>26.00</td><td>23.16</td><td>22.40</td><td>25.77</td></tr> <tr><td>Rotation</td><td>35.42</td><td>26.04</td><td>34.38</td><td>37.50</td><td>36.05</td><td>29.17</td></tr> <tr><td>Brightness</td><td>25.26</td><td>27.37</td><td>27.37</td><td>31.52</td><td>29.47</td><td>27.37</td></tr> <tr><td>Blur</td><td>31.18</td><td>25.26</td><td>26.32</td><td>24.73</td><td>24.28</td><td>26.32</td></tr> <tr><td>Global@50%</td><td>42.86</td><td>20.88</td><td>57.14</td><td>37.50</td><td>47.25</td><td>48.35</td></tr> <tr><td>Global@80%</td><td>32.61</td><td>22.83</td><td>36.96</td><td>25.88</td><td>35.87</td><td>30.43</td></tr> <tr><td><strong>Level-1 Average</strong></td><td><strong>31.10</strong></td><td><strong>23.86</strong></td><td><strong>33.28</strong></td><td><strong>28.17</strong></td><td><strong>31.20</strong></td><td><strong>29.22</strong></td></tr> <tr><td colspan="7"><strong>Level-2</strong></td></tr> <tr><td>Uniform Color</td><td>96.00</td><td>21.00</td><td>97.00</td><td>66.20</td><td>88.00</td><td>78.95</td></tr> <tr><td>Uniform Count</td><td>61.00</td><td>25.00</td><td>90.91</td><td>40.82</td><td>59.00</td><td>52.75</td></tr> <tr><td>Uniform Orientation</td><td>22.22</td><td>25.25</td><td>26.53</td><td>26.00</td><td>23.00</td><td>19.19</td></tr> <tr><td>Count Progression</td><td>50.00</td><td>13.00</td><td>77.00</td><td>37.20</td><td>48.00</td><td>36.96</td></tr> <tr><td>Count Arithmetic</td><td>52.00</td><td>22.00</td><td>75.76</td><td>43.20</td><td>49.00</td><td>33.33</td></tr> <tr><td><strong>Level-2 Average</strong></td><td><strong>49.79</strong></td><td><strong>24.12</strong></td><td><strong>62.29</strong></td><td><strong>37.18</strong></td><td><strong>46.75</strong></td><td><strong>39.15</strong></td></tr> <tr><td colspan="7"><strong>Level-3</strong></td></tr> <tr><td>Independent Color-Object-Orientation</td><td>34.00</td><td>25.25</td><td>38.00</td><td>27.39</td><td>32.60</td><td>19.00</td></tr> <tr><td>Independent Count-Object-Color</td><td>34.00</td><td>24.00</td><td>44.00</td><td>29.45</td><td>36.34</td><td>29.00</td></tr> <tr><td>Coupled Color-Orientation</td><td>24.24</td><td>24.00</td><td>16.33</td><td>26.13</td><td>29.43</td><td>20.00</td></tr> <tr><td>Coupled Color-Count</td><td>30.00</td><td>22.00</td><td>21.21</td><td>27.46</td><td>33.33</td><td>28.00</td></tr> <tr><td>Spiral Color-Count-Object</td><td>56.00</td><td>30.00</td><td>54.17</td><td>28.63</td><td>36.00</td><td>33.00</td></tr> <tr><td><strong>Level-3 Average</strong></td><td><strong>34.39</strong></td><td><strong>23.86</strong></td><td><strong>33.73</strong></td><td><strong>26.31</strong></td><td><strong>31.36</strong></td><td><strong>25.17</strong></td></tr> </tbody> </table> --- ## Finetuning on Level-1 (Qwen2.5-VL-3B) <table> <thead> <tr> <th>Setting</th> <th>Original</th> <th>Finetuned</th> <th>Human Baseline</th> </tr> </thead> <tbody> <tr><td>Location</td><td>24.3</td><td>42.8</td><td>94.1</td></tr> <tr><td>Blur</td><td>23.9</td><td>37.5</td><td>84.3</td></tr> <tr><td>Brightness</td><td>23.7</td><td>39.8</td><td>85.6</td></tr> <tr><td>Rotation</td><td>25.5</td><td>50.8</td><td>92.0</td></tr> <tr><td>Edges</td><td>25.1</td><td>33.2</td><td>82.6</td></tr> <tr><td>Global (50%)</td><td>24.9</td><td>52.2</td><td>96.1</td></tr> <tr><td>Global (80%)</td><td>23.9</td><td>38.6</td><td>98.0</td></tr> <tr><td><strong>Average</strong></td><td><strong>24.5</strong></td><td><strong>43.7</strong></td><td><strong>90.4</strong></td></tr> </tbody> </table> --- ## Single-Attribute Recognition (Perceptual Grounding) Accuracy (%) when models are asked to report a single attribute (color, orientation, or count) for one grid cell. <table> <thead> <tr> <th>Attribute</th> <th>GPT-4o</th> <th>GPT-5</th> </tr> </thead> <tbody> <tr><td>Color</td><td>84.6</td><td>97.6</td></tr> <tr><td>Orientation</td><td>39.8</td><td>49.6</td></tr> <tr><td>Count</td><td>72.4</td><td>94.2</td></tr> </tbody> </table> --- ## Impact of Thinking Mode Accuracy (%) with thinking mode enabled (✓) vs disabled (✗). Open-source models improve substantially with thinking. <table> <thead> <tr> <th>Level</th> <th>GPT-5 (high)</th> <th>GPT-5 (low)</th> <th>Mimo-VL ✓</th> <th>Mimo-VL ✗</th> <th>Qwen3-4B ✓</th> <th>Qwen3-4B ✗</th> <th>Qwen3-30B ✓</th> <th>Qwen3-30B ✗</th> </tr> </thead> <tbody> <tr><td>Level-1</td><td>32.61</td><td>31.43</td><td>29.22</td><td>23.91</td><td>28.17</td><td>23.16</td><td>31.20</td><td>23.60</td></tr> <tr><td>Level-2</td><td>49.79</td><td>47.01</td><td>39.15</td><td>26.68</td><td>37.18</td><td>24.08</td><td>46.75</td><td>28.25</td></tr> <tr><td>Level-3</td><td>34.39</td><td>32.89</td><td>25.17</td><td>25.23</td><td>26.31</td><td>23.50</td><td>31.36</td><td>24.00</td></tr> </tbody> </table> --- ## Impact of Image Resolution (GPT-5) Accuracy (%) at different input resolutions. All levels improve with higher resolution. <table> <thead> <tr> <th>Resolution</th> <th>Level-1</th> <th>Level-2</th> <th>Level-3</th> </tr> </thead> <tbody> <tr><td>512×512</td><td>45.17</td><td>42.83</td><td>31.63</td></tr> <tr><td>1024×1024</td><td>54.01</td><td>49.61</td><td>35.48</td></tr> <tr><td>2048×2048</td><td>56.51</td><td>48.99</td><td>40.07</td></tr> </tbody> </table> --- ## Citation ```bibtex @article{visres2025, title={VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs}, author={Malagurski T{\"o}rtei, Brigitta and Dahou, Yasser and Huynh, Ngoc Dung and others}, journal={arXiv preprint arXiv:2512.21194}, year={2025} } ```