The OPTICS Benchmark Shows VLMs Still Can't Count

Vision-language models still can't count. That's the blunt finding behind OPTICS, a new benchmark on arXiv that tests VLMs on object reasoning, and the numbers are worse than most vendor demos would lead you to believe: in zero-shot settings, state-of-the-art models score below 40% on counting questions and around 70% on comparison questions. Even the newer reasoning-focused models leave a 20% gap to human performance.

What The Numbers Actually Show

The benchmark itself is small and deliberately controlled: 360 images paired with 1,080 multi-level, count-based questions (OPTICS-CNT), plus a separate set of 2.1k comparison questions (OPTICS-CMP). Twelve state-of-the-art VLMs were evaluated. The multi-level design matters here — questions aren't just "how many X are there" but build up through object properties, so a model that passes the first level can still fall apart when the question asks about physical or functional attributes.

The failure modes are more interesting than the headline scores. Models did worst on photographic images (as opposed to simpler synthetic ones), counterfactual reasoning, physical properties like weight or material, functional properties like what an object is for, and higher counts. That last one tracks with what practitioners already know: counting beyond a handful of objects remains a genuine weak spot even in frontier models.

Where The Benchmark Is Strong — And Where It's Thin

Credit where it's due: this is a clean evaluation design, the data and code are released for others to build on, and the focus on object reasoning targets something specific rather than chasing aggregate leaderboard scores. A benchmark that isolates which reasoning tier breaks — recognition vs. property inference vs. counterfactuals — tells you more than another generic VQA score.

The part worth being skeptical of is scale. 360 images and roughly 3,200 total questions is enough to demonstrate a gap, not enough to characterize it precisely per model. A single model swapping two or three answers shifts percentages noticeably at this size. And zero-shot-only evaluation means we don't learn whether these failures are fixable with better prompting or few-shot examples — which matters if you're deciding whether to deploy these models in a pipeline that needs reliable counting today.

What This Actually Points To

My read: this benchmark is less an indictment of any one model and more evidence that visual grounding — actually parsing what's in an image at the object level — lags well behind the language half of these systems. The pattern of failures (counterfactuals, function, physical properties) suggests models are pattern-matching from captions and co-occurrence statistics rather than reasoning over scene structure.

In practice this means if your application depends on accurate counts or property comparisons — inventory checks, scene auditing, quality control — you should not assume a strong general VLM benchmark score transfers to your task. Test it on your own data before trusting it. The realistic near-term path is probably hybrid: VLM for open-ended understanding plus a specialized tool for precise counting, rather than waiting for the next model release to close a gap this structural.