Research
Three results that build on each other
The through line is a single question posed three ways: first measuring where humans
look, then building a model that reproduces it from scratch, then asking how far
today's vision-language models still are from human visual competence.
01
Measurement
Nature Communications · 2026
Eye movements during free viewing to maximize scene understanding
What humans look at when freely viewing a scene is not well understood. We measured
eye movements under different instructions while observers viewed Winograd images:
pairs differing by a small visual alteration that greatly changes how the scene is
interpreted. Free-viewing fixations resemble those of observers describing scenes,
and differ from those counting or searching for objects. They are directed toward the
people and objects whose removal most alters scene interpretation, rather than toward
the most salient or most meaningfully judged regions. Constraining observers to fixate
objects irrelevant to understanding degrades their descriptions, showing that
free-viewing eye movements are functionally important for scene comprehension.
02
Model
Under review · Nature Human Behaviour
Why we look where we look: emergent human-like fixations of a foveated visual language model
When humans view scenes without a specific task, they first direct their eye
movements toward the scene center, then fixate on people, text, objects being looked
at or grasped, and semantically meaningful regions. What these signature fixation
patterns reflect, and whether they optimize an underlying perceptual task, has
remained unknown. We show that a computational agent with simulated foveation,
trained to optimize scene comprehension, exhibits emergent human fixation signature
patterns. Versions of the agent trained instead to search or classify scenes, or
equipped with peripheral vision better or worse than human vision, predicted human
fixation patterns less accurately. Human free-viewing fixation patterns may therefore
emerge as a functional byproduct of optimizing scene comprehension under the
biological constraints of foveated vision.
03
Benchmark
arXiv · 2026
Evolution of accuracy and visual-cognitive errors in a decade of vision-language AI models
Visual reasoning benchmarks mostly use simple scenes, few human descriptions, and
rarely ask what models actually get wrong. We introduce the Complex Social Behavior
dataset, 100 images depicting complex social interactions, and trace scene
description accuracy across a decade of vision-language models against 20 human
descriptions and a gold standard. Multimodal LLMs now reach accuracies similar to the
top-ranked human descriptions, and have closed the gap between simple MS-COCO scenes
and scenes depicting complex behaviors. Across five visual-cognitive error types,
nearly all have been eliminated, except that models still occasionally rely on
different image regions than humans do.
