OMIBench: Olympiad Problems Spread Across Images Stump AI Models
A new benchmark spreads Olympiad problems in biology, chemistry, math and physics across several images. The best model tested, Gemini-3-Pro-Preview, scored about 50 percent.
TL;DR
- OMIBench collects over 1,000 Olympiad problems in biology, chemistry, mathematics and physics whose evidence is spread across several images [^1].
- The strongest model tested, Gemini-3-Pro-Preview, reached 50.53 percent, while human experts topped 80 percent on a 52-problem sample [^1].
Background
Large vision-language models (LVLMs) read images and text together and answer in text. Chain-of-thought prompting, which makes a model write out its intermediate steps, has lifted their scores on Olympiad-level benchmarks built around a single image [^1]. Real scientific problems often depend on several interdependent figures, diagrams and experimental setups, the authors argue, yet existing Olympiad benchmarks largely stick to one image per question [^1]. The authors work at institutions including Harbin Institute of Technology and Fudan University [^1].
What happened
The benchmark contains over 1,000 problems, with an average of 3.07 images each [^1]. They come from contests such as the International Mathematical Olympiad, the International Chemistry Olympiad, the Chinese Physics Olympiad and the Australian Science Olympiads, plus problems adapted from existing benchmarks such as OlympiadBench, Mv-MATH and EMMA [^1]. Every problem has an expert-checked solution path. Gemini-2.5-pro-thinking first generated up to 16 candidate solutions per problem and kept those with the correct final answer, then annotators corrected errors, added missing steps or rewrote flawed solutions [^1]. Answers are scored by exact-match accuracy and by GPTScore, where a language model judges whether an answer agrees with the reference [^1].
Gemini-3-Pro-Preview led with a GPTScore of 50.53 percent, ahead of GPT-5 at 48.11, GPT-5-mini at 47.73 and OpenAI-o4-mini at 45.97 [^1]. The best open-source model, Qwen3-VL-32B-Instruct, reached 35.78 [^1]. The same Gemini model scores 75.67 percent on the single-image OlympiadBench [^1]. More images made things worse: accuracy reached 40 percent on problems with one image and dropped below 15 percent with six or more, and cutting each problem down to a single image cost at least 10 points [^1].
Two checks put the numbers in context. In one, the team pasted all images of a problem into one composite picture, keeping the same information and the same question; that version consistently scored higher than the original, which the authors read as evidence that linking separate images, not the amount of visual material, causes the gap [^1]. In a small human study on 52 problems, two domain experts scored above 80 percent and two trained non-experts above 57 percent, while Gemini-3-Pro reached 48.08 percent [^1].
Why it matters
The paper tests the usual fixes and finds few that help much. Different chain-of-thought prompts showed no statistically significant differences [^1]. The "thinking" variant of Qwen3-VL beat its "instruct" variant by about 10 percent, and raising the reasoning budget from 512 to 16,384 tokens brought a near-monotonic gain [^1]. Bigger models did not reliably help: InternVL and QwenVL plateaued at about 25 and 35 percent GPTScore across sizes from 1B to 235B parameters, a plateau the authors tie mainly to mixture-of-experts models [^1]. Visual tools such as Visual Sketchpad made GPT-4o worse and gave GPT-5 only modest gains, and fine-tuning on existing multi-image data added less than about 1.5 to 2 percent [^1]. The authors conclude that progress will likely need more fundamental advances in model architecture and training [^1].
A fluent answer is not a correct one either. When reviewers checked solutions from o4-mini and Gemini-3.0-Pro on 100 problems, they found logical errors in 46 percent of the key steps [^1]. The authors also note limits of their own: GPTScore relies on a model judge and may under-credit creative or partly correct answers, and the benchmark does not yet cover every kind of multi-image problem [^1].
Practical example
Say Lena coaches a school team for a chemistry Olympiad and wants a vision-language model to check the team's practice solutions. A typical task shows a reaction scheme in one image, a spectrum in a second and a data table in a third. The model can describe each image, but the answer depends on linking all three.
That linking is where OMIBench finds models weakest. On problems with one image, accuracy reached 40 percent; with six or more it fell below 15 percent [^1]. Adding "think step by step" to the prompt is unlikely to rescue Lena, since different chain-of-thought prompts made no statistically significant difference [^1]. A model built for long reasoning is a better bet: the "thinking" variant of Qwen3-VL did about 10 percent better than its "instruct" sibling [^1].
Lena can also try a trick from the paper's control experiment: merging all images of a task into one picture scored consistently better than sending them separately [^1]. Even then, Lena reads the model's full reasoning instead of trusting the final answer. In the paper's review of 100 problems, 46 percent of key steps in otherwise fluent solutions contained logical errors [^1].
Related gear
We recommend this book because it teaches the strategies behind Olympiad-style mathematics problems, the kind of competition tasks OMIBench draws on.