DecoyCount

• Images1,660real-world scenes
• Decoys2,203pre-registered, 7 mechanisms
• Prompts3per image, same count
• Best unaided43.2%GPT 6 Astra, exact match
• Answer the decoy5 of 6more often than the truth

Counting is a fundamental step in complex problem solving. Models today can’t count reliably.

The task

How many bottles are there?

What answering it actually takes

1. Interpret the prompt
2. Review the scene and locate the target
3. Decide what counts as one instance, and in which unit
4. Tell physical objects apart from any similar objects
5. Identify depictions and printed text
6. Enumerate what is left

Each of those steps can fail on its own, and a single final number answer collapses all of the step grading into one score. So a wrong count tells you almost nothing on its own. This case is printed 12 and holds more than twelve bottles. Answer of twelve shows the model read a number and trusted it over the scene.

DecoyCount is uniquely built to separate perception from the counting task. It is 1,660 real-world images, each carrying a pre-registered decoy. The decoy is a specific wrong count that the scene invites, recorded in advance along with the mechanism that invites it. When a model answers with the decoy rather than an arbitrary wrong number, it has told us which interpretation it chose.

Seven ways a scene lies about its own count

The taxonomy was fixed before image collection and balanced so that a per-family score is not a proxy for one recurring setting.

The families intervene at different stages. Lookalike, Duplicate, and Wrong Unit change which instances belong in the counted set. Text and Prior supply a competing answer. Structure and Estimate challenge enumeration itself. That split matters further down.

One item from every family

Each item carries all three prompts. The cue names the process and never the answer; the scenario implies the same rule without stating it. One worked item per mechanism, as the paper lays them out in Figure 1.

Text — six bottles, a packet printed “24”

Neutral. How many water bottles are there? Answer 6, decoy 24.

Cue. Count the objects themselves and ignore any number written in the image.

Scenario. Each guest gets one bottle. How many more must be bought?

Duplicate — one woman, one mirror

Neutral. How many faces are there? Answer 1, decoy 2.

Cue. Do not count any reflections.

Scenario. How many women would there be if you put 5 more in front of this mirror?

Prior — a familiar number that is not the pictured one

Neutral. How many NYC boroughs are there? Answer 4, decoy 5.

Cue. Don’t rely on the usual number of boroughs, only count the ones pictured.

Scenario. I buy a watch for each borough I visit. How many more do I need?

Wrong Unit — counted in the wrong thing

Neutral. How many pairs of sails are there? Answer 3, decoy 4.

Cue. Count complete pairs, checking which items actually form one, rather than dividing the total.

Scenario. If each boat is assigned one complete pair, how many pairs remain?

Lookalike — something similar admitted to the set

Neutral. How many red cars are there? Answer 4, decoy 5.

Cue. Pay close attention to the build of each vehicle and determine if it is a car or another type of vehicle.

Scenario. If more red cars parked here, how many would there be?

Structure — part of the structure missed

Neutral. How many window panes are there? Answer 27, decoy 9.

Cue. Look closely for window panes located behind the large window in the foreground.

Scenario. To break every pane with a different hammer, how many more hammers?

Estimate — enumeration gives way to approximation

Neutral. How many amber beads are there? Answer 101, decoy 100.

Cue. Count carefully rather than estimating.

Scenario. A conservator puts 25 beads on each dish. How many are left over?

The scenario version is not simply a harder question, it is a control. It supplies the real-world context a human would bring, without ever naming the rule. If the gap were caused by the neutral question being underspecified, this is the condition that would close it.

Each image carries up to five decoys, 2,203 across the benchmark and every decoy must be supported by the image, differ from the truth, and actually fool a model. Items enter the set only if all three of Claude Opus 5, GPT 5.6 Sol, and Gemini 3.6 Flash miss the neutral answer, and at least one returns a registered decoy exactly. Independent reviewers then counted every item blind, adjudicated disagreements, and removed anything ambiguous, occluded or needing specialist knowledge. That retained 1,660 of 2,100 candidates, corrected 121 counts and replaced 79 images.

The reasoning-alignment gap

Every image comes with three prompts, and the design lives in the difference between them.

1. Neutral. “How many water bottles are there?”
2. Cued. The same question, plus one sentence naming the intended rule without revealing the answer: “Count the objects themselves and ignore any number written in the image.”
3. Multi-hop. The same count, grounded in a scenario that implies the rule rather than stating it: “Each guest gets one bottle. How many more must be bought?” — often with a second image. 842 items are multi-image.

The image never changes. The right answer never changes. The cue adds no visual evidence and no arithmetic. All it does is specify which counting process was intended.

The reasoning-alignment gap is cued accuracy minus neutral accuracy. It measures how much of a model’s failure was never about seeing at all but how much was about choosing the wrong rule when nobody said which rule to use. It isolates one component of visual-reasoning misalignment, not every perceptual or reasoning limitation.

The name may sound odd, because counting looks like it has one determined answer. Two findings argue otherwise. A one-sentence cue moves accuracy by 11.8 to 22.1 points. And changing only the words “are there” to “are visible” touches 1,654 of 1,660 prompts, adding nothing answer-relevant and moves Gemini 3.8 Flash by 6.69 points, repairing 246 answers while breaking 135.

If a two-word change can do that, the intended process was never fully specified by the question. It was supplied by human interpretation all along.

Frontier models answer the decoy more often than the truth

Six models, identical items, three repeats averaged, with no system prompt and no output-format instruction.

No model clears 50% neutral accuracy, and five of six return the planted decoy more often than the true count. These are not scattered miscounts. They are the specific wrong answers each scene was built to invite.

Cues fix 305–437 items while breaking only 63–134. Between 2.6 and 6.7 times more repaired than broken, at p < 10⁻²⁰ for every model and release between 43.3% and 66.7% of exact decoy hits.

The cue has to name the actual trap

The obvious objection is that any extra instruction would help assuming that the cue is just a nudge to concentrate. So the paper ran two controls alongside it: a generic cue (“count carefully”) and a mismatched cue naming a different family’s trap.

Generic and mismatched cues recover at most 18% of the gain. At least 82% of the effect depends on naming the trap that is actually there. The matched cue separates from both controls at p < 10⁻¹², while the two controls do not separate from each other.

Nor is it thinking time. On Gemini the generic and matched prompts spend almost identical token budgets; on GPT 6 Astra the generic cue spends more and delivers an eighth of the gain.

One family is an exception worth knowing: Text gains about 11 points even from a generic or mismatched instruction, which suggests that almost any request to count can displace reading a printed numeral. The 82% is an aggregate bound, not a per-family one.

The strongest model has the largest gap. GPT 6 Astra leads on neutral accuracy and gains another 22.09 points when told the rule. Greater capability improved what it could execute under guidance without improving what it selected on its own.

Which mechanism, and how much of the error is the trap

The aggregate hides the useful part. Broken out by mechanism, the cue’s effect ranges from enormous to nothing and that range lines up with how often the wrong answer was the planted decoy rather than an ordinary miscount.

That split is the benchmark earning its keep. Text, Duplicate, Prior and Wrong Unit combine frequent decoy hits with large gaps, which is what you would expect if the scene was read and the wrong rule applied, though a cue can also sharpen an ambiguous question, and the gap does not separate those two on its own. Estimate is the opposite on both measures . No model gains more than 3.36 points on it, and only a fifth of its errors land on the decoy. Those are ordinary miscounts, and guidance does not touch them.

Structure sits in between, at 48.6% of errors on the decoy with a 5.9-point gap. Its small gap is not the same as Estimate’s: it mixes a repairable shortcut with genuine miscounting, where Estimate is consistent with a perception or enumeration bottleneck throughout.

The ordering is not universal, though. It holds across all six models for the big split. Five large-gap families against Structure and Estimate. But the ranking within the five differs by model. Duplicate swings from +43.9 points on GPT 6 Astra to +16.7 on Gemini 3.8 Flash. An aggregate score hides failure modes that are large and specific to one model.

Scale, thinking and tools all widen the gap

The obvious objection is that this is a temporary artifact of models not yet being good enough. If so, making them better should close the gap. That was tested three ways, and it went the other direction every time.

Scale. Across Qwen 3.5 at 4B, 9B and 27B with image processing and decoding held fixed, neutral accuracy rises from 14.08% to 19.38% while the gap widens from 8.84 to 13.35 points. More striking is where the remaining errors land: neutral decoy-hit rises from 43.76% to 50.74% while other errors fall from 42.17% to 29.88%. The bigger model makes fewer mistakes, and a larger share of what is left is the planted trap, 50.92% of errors become 62.94%.

Thinking budget. With Qwen 3.8 FlashNext held fixed, going from thinking off to native unbounded thinking more than doubles neutral accuracy, 12.65% to 27.17% while the gap grows from +8.96 to +20.80, peaking at +21.71 under a 30k-token cap.

Agentic tooling. Here the paper measures the change in the gap itself: Δ²R, the gap with the tool minus the gap without it. Positive widens, negative narrows.

Every general harness widens the gap. The one row that narrows it is the one that adds a targeted visual tool: SAM3 segmentation raises neutral accuracy by 3.61 points and pulls the gap down 4.52, helping all seven families and Estimate most of all, by 10.8 points the family guidance could never reach.

A caveat the paper is careful about and we will be too: a positive Δ²R does not prove tools harm alignment. Code execution also lengthens and restructures the reasoning, which can let the cued arm benefit more even while neutral accuracy rises. And the aggregate hides exceptions GPT 6 with CI narrows the gap for Wrong Unit by 0.8 points, and Opus with CI narrows Duplicate by 1.7. General execution occasionally supplies evidence that fits a particular failure, just not reliably.

That contrast is the most useful result here. Giving a model more general reasoning capacity makes it better at executing a process and no better at choosing one. Giving it a tool that supplies the missing visual evidence does close the gap on exactly the family where the bottleneck was perceptual in the first place.

Context alone substitutes for neither. The multi-hop condition grounds the same count in a real scenario without ever naming the rule. It does not improve aggregate performance, and rescues only 11.6% to 32.6% of neutral failures per family. Underspecification explains some of the gap. It does not explain the gap.

They assumed. We counted.

DecoyCount plants one trap in every image: a printed number that is wrong, an object that only looks like the target, a familiar pattern with one piece taken out. Our own checkpoints answer in one shot, with no tools.

A newer internal checkpoint reaches 52.17% neutral accuracy on the full released benchmark 866 of 1,660 items, in a single run. That is above every model in the paper, where the best neutral score is 43.17%.

An earlier checkpoint is the one that can speak to the gap, because it has both arms:

The interesting number is not the first column. It is the last one.

Every model in the paper gains between 11.8 and 22.1 points the moment someone names the trap. That gain is the measure of how much the model could already have done and did not. This checkpoint gains 4.70 points. Most of what a cue would have told it, it had worked out unaided.

A higher score alone would not show that. A model can be more accurate and just as badly misaligned, which is what happens to GPT 6 Astra, the strongest model in the paper and the one with the widest gap.

Two honest limits on the above. The 52.17% and the 4.70-point gap come from different checkpoints and cannot be added together; there is no cued run for the newer one. And neither checkpoint has been confirmed as the version behind the deployed product, so these are measurements of our research checkpoints rather than a claim about what you get today.

Get the benchmark

Images come from expert photography, Wikimedia Commons, Openverse, CC0/CC BY Flickr material and government archives, and are edited or generated where a mechanism cannot be found naturally. Provenance, license and credit are recorded for every item and ship with the release as a separate attribution file; the paper notes the fields themselves are not carried into the parquet.

One note on how to read the numbers. Because items were retained partly because frontier models fail them, absolute accuracy here is not an estimate of counting ability on ordinary photographs. What it does preserve is ranking: rank correlation with CountQA is 0.98 on neutral and 0.97 on cued accuracy, and against FSC-147, 0.83 and 0.94. DecoyCount orders models the way established counting benchmarks do, while separating the failure mechanisms those benchmarks average away.