Tracking the Gaze

Ilya Repin, “Unexpected Visitors” (or “They Did Not Expect Him”), 1884-1888. Oil on canvas.  63.19 x 65.95 in. The Tretyakov Gallery, Moscow, Russia.

When you approach a painting in a gallery, it feels like you’re looking at the entire piece all at once, but what your eyes are actually doing is far more complicated and precise. Since the 1960s, scientists and engineers have developed technology that allows us to actually observe what the eyes are doing when looking at art. What they’re learning may surprise you.

Different parts of our vision have various different functions. When we look at a painting, we’re typically using what is called “foveal vision.” This is the optical function we use to see fine details. At the very center of our gaze, our visual acuity is amazingly sharp. However, the center of the gaze focuses on a relatively small point and at the periphery, our visual acuity (known as “resolution” in neurological circles) drops dramatically. (For a demonstration of how this actually works, try this optical experiment.) Some artists have intuitively understood this phenomenon and have expressionistically recreated the technique in paintings by strategically drawing the eyes to a certain point. For instance, see how Renoir details his subject’s face in Madame Henriot (c. 1876) while everything on the periphery is less defined.

Foveal vision becomes especially interesting when looking at fine art or photography. To further investigate this, a Russian psychologist named Alfred Yarbus invented an apparatus in the 1960s that would provide insight into the phenomenon of eye movement.

Illustrations of Yarbus’ device, circa 1967.

As you can see, it was not pleasant. Mr. Yarbus’s device looks like something out of Clockwork Orange. I doubt that anyone would have recommended wearing this while browsing art galleries, as most patients couldn’t tolerate wearing it for more than a few minutes. Despite being cumbersome, the device revealed exactly what Yarbus was looking for–an exact tracing of the subject’s vision.

By mapping our vision, Yarbus discovered some fundamental patterns of our visual behavior. Eye tracking revealed that the eyes are rarely stationary—they jump around from point to point, like a water strider on the surface of a pond. These micro-adjustments are known as “saccades,” and they are at the core of our visual perception and how we build the world around us. Saccades are essential to vision; as the eyes dart from place to place, landing on the most information-rich areas, it constructs the reality of experience. As Yarbus experimented with patients and paintings, he found that the eyes were drawn to areas of high contrast and fine detail, and especially to human and expressionistic areas, such as eyes and lips. Like a heat seeking missile, our gaze homes in on the points that are most interesting.

Yarbus repeatedly studied one painting titled Unexpected Visitors (pictured on top) by the Russian painter Ilya Repin. Based on the evidence of his eye tracking studies, Yarbus speculated that the eyes would be attracted to areas packed with information. But actually, information alone didn’t hold the attention of his subjects. As it turned out, people were interested in details that explained the narrative of the image: the eyes gravitate towards the clues that help decode the meaning of a painting. Here is an example of what Yarbus’s eye tracking experiment looked like:

An example of Yarbus’s eye tracking studies, circa 1967.

Notice how the eyes jump from one area to the next in the diagram–from the young boy’s face to the visitor’s face to the woman’s face and so on. As our vision darts from one point to the next, we’re visually constructing a story in our mind. Our interpretation may feel instantaneous, but is actually composed of smaller units that make up a whole, like a series of storyboards. If we deconstruct the mechanics of this scene to imitate what is happening in the mind, the slideshow might look something like this:

The way we process visual information is a lot like experiencing a scene in a movie or graphic novel. Essentially, we’re editing the painting in our mind to construct a narrative. What if we looked at art as a dynamic sequence of images rather than as static images? How would that change our interpretation of a piece? The process would be more like how a film editor approaches storytelling. The best editors truly understand how the eyes darts from one piece of information to the next and they leverage that behavior in their own work, in essence recreating what the brain does. (Legendary editor Walter Murch wrote a book on the subject called In the Blink of an Eye.)

Writing for Cabinet magazine in 2008, Sasha Archibald challenged one of the shortcomings of Yarbus’s study: he only investigated one artwork, the painting by Repin. What happens to the eyes when looking a piece that is not as clearly defined as the one Yarbu studied? For instance, what happens when we look at a Jackson Pollock or a Paul Klee or a Hans Hoffman? How do the eyes construct a narrative out of something entirely abstract?

Well, scientists are now beginning to look at abstract art and are coming to interesting conclusions. Subjects gravitate around areas of “salience” or where information is most dense (just as Yarbus observed) but the results tend to be much more personal. Responses tend to be more difficult to articulate and patterns are elusive. Previous knowledge and appreciation vastly influenced how and where subjects fixated on a painting. For more on this, read “How do we see art?“, a study that charted saccade patterns while subjects observed different versions of paintings by Mondrian.

Neuroaesthetics is a nascent field that offers opportunities to reinvestigate and rediscover old and familiar works of art. How can non-scientists take advantage of eye tracking for the humanities? And what other studies are out there that might give us new perspectives?

Additional References

Livingstone, Margaret. Vision and Art: The Biology of Seeing. Harry Abrams, New York, 2002.

Yarbus, Alfred. Eye Movements and Vision. Plenum Press, New York, 1967.