Introducing Humanity's Sixth Sense (HSS)

These questions are among the 522 in Humanity's Sixth Sense (HSS), a new benchmark we built in partnership with Scale AI. HSS measures how a model interprets and reasons through information a visual scene implies but doesn’t explicitly show. When a person looks at something, they can draw countless conclusions without significant thought. As we grow into adults we develop a sort of sixth sense, allowing us to conceptualize real-world scenes, including the spatial layout, context, and structural relationships beyond isolated objects.

This cognitive development grows gradually from infancy through childhood, driven by specialized brain regions and a framework known as “scene grammar.” By their early teens, most people can accurately sketch out a scene, even with limited visual cues and anticipate what will happen next. This is done largely by filling in the blanks and predicting actions based on past experiences. In theory, it seems like frontier models should excel in compiling complex information from a myriad of perspectives, leading to accurate predictions. Most models currently lack fundamental visual skills which would allow them to operate natively with visually-dependent tasks.

The whole picture

Model research has largely focused on text-based problems utilizing code, math, books, and exam questions. Most “visual” benchmarks test the same text-oriented flows. HSS is unique in that it tests simple visual reasoning, across four areas:

• Temporal & causal dynamics What happened before the scene, and what happens next.
• Physical & spatial logic What fits, what’s in reach, and what someone else can see from where they are standing.
• Social understanding Intent, attention, belonging.
• Abstract & contextual inference The patterns and rules a scene implies.

The benchmark is 522 tasks in all: 288 images and 234 short video clips, 17.6 hours of video between them.

What HSS measures

Frontier models write production code and pass professional exams. Given they also edit images and can identify objects in videos, it can seem like these models understand visual tasks better than they can. Most visual benchmarks reinforce that impression.

Humans are capable of visual reasoning well beyond what most visual benchmarks measure. We can usually tell what happened a moment before a photo was taken, whether a box will fit through a door, or why a dog took off running. Every question in HSS is open-ended, and answers are graded against rubrics written by humans. An answer only counts if it meets every point on the rubric, so a model can't pass by guessing.

Leaderboard

Twenty-five models from eight companies. Most score below 40%, and the median model scores 30.9%. The last row is the paper’s blind control — the strongest model, run again with the image or video taken away. It scores 6.6%, which is the floor you get from guessing at the question text alone.

Where models fail

HSS tasks are reasonably easy for the average person, with a baseline score of 93.1%.

Across the 8,573 individual failures the team labeled, 94% trace to seeing or inferring rather than to reasoning. The model misread something in the image in 53% of cases, and in a further 41% it failed to infer something the image implied but did not show. Only 5% came down to faulty logic.

The two most common single mistakes were (1) missing a detail that decides the answer (21% of failures) and (2) misidentifying an object, person, or role (20%). Failures were consistent across the 24 models that analysis covers. Struggling significantly with nuance, perspective, object variations, lighting effects, and causality. On tasks that five or more models failed, a median of 80% failed for the same reason.

The weakest domain is social understanding — reading beliefs, intentions, and who defers to whom. It is the lowest-scoring domain for 21 of the 25 models, averaging 24.4% against 34.1% across the other three. Video is harder than still images for 23 of the 25 models, by 7.3 points on average.

Increasing reasoning effort helps overall, but on some skills it makes things worse. The paper reports a 14-point drop for GPT 6 Astra on retrodiction — working out what happened just before the moment shown — when reasoning effort is scaled from high to xhigh; the score then partly recovers at max. Best scores often came at a middle setting. Increasing reasoning effort on several models frequently led to the model overthinking or focusing on incorrect evidence.

Agent tools such as Claude Code and Codex modestly improved scores by letting a model crop, zoom, and take a second look. On the 388-task subset used for the agentic evaluation, the best setup reached 59.3%. A second look cut failures from 944 to 759 by catching missed cues and misread objects. What a closer look does not fix is depth: errors from reading a 2D overlap as 3D alignment were essentially unchanged, 102 to 101. Agents sometimes talked themselves out of answers they had gotten right the first time.

Humans noted that they mainly answered the questions intuitively. Typically spending more time explaining their reasoning than coming up with the correct answer. Many human failure cases could be traced to sloppiness, misunderstanding the question, and arithmetic errors. Author’s note: definitely never happens to me.

Elorian is building the foundation of visual thinking. HSS is a preliminary step in identifying and understanding how to solve the complex gaps in frontier models. HSS highlights how today's models ability to see and interpret visuals is highly limited and not a quirk of one lab's model.

Read the paper The full leaderboard The dataset

Figures, questions and reference answers in this post are quoted from Humanity’s Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models, by Guo, Gu, Jang et al. — a joint paper from Scale AI and Elorian. Model results are as reported there, for the model versions and reasoning-effort settings used in that evaluation. Scores on this benchmark will move as vendors ship new versions; the leaderboard above is the current picture.