Swartz Mind/Brain Lecture Explores How AI Could Decode and Shape Human Vision

Neuroscientist James J. DiCarlo, MD, PhD delivers his Mind/Brain lecture, AI-derived Mechanisms of Human Vision, at the Staller Center on March 30. Photos by John Griffin.

The annual Swartz Foundation Mind/Brain Lectureat Stony Brook University drew a packed audience for a talk that discussed the mechanics of sight and human emotion.

Neuroscientist James J. DiCarlo, MD,PhD of the Massachusetts Institute of Technology (MIT) described his vision of the brain as a machine that can be modeled, tested and, in the future, possibly guided using artificial intelligence.

"I'm going to start by saying something kind of provocative," began DiCarlo. "The phenomena of the mind - our behaviors, our perceptions, our sense of free will - are emergent phenomena of this device called the brain."

His talk, AI-derived Mechanisms of Human Vision, explored how deep neural networks are analyzing data but also increasingly function as scientific models that mirror how the brain processes the visual world.

The March 30 lecture on the Staller Center main stage was hosted by the Department of Neurobiology and Behaviorand supported by the Swartz Foundation. The Mind/Brain series is designed to "promote a computational perspective on the brain and to bring outstanding neuroscientists to the Stony Brook community," as described by Shaoyu Ge, professor in the Department of Neurobiology and Behavior, who introduced DiCarlo.

Shaoyu Ge, professor in the Department of Neurobiology and Behavior.

DiCarlo framed his first "story" around the question: how do humans recognize objects in a fraction of a second?

"When you and I look at a scene, we don't just take it all in and know everything at once," he said. "We dwell for about 200 milliseconds and that's when we see."

This ability, known as 'core visual perception,' allows humans and other primates to extract meaning from visual input almost instantly. DiCarlo's lab at MIT has spent more than two decades trying to model this process by studying both human subjects and rhesus monkeys, whose visual systems closely resemble those of humans.

"When a mistake is made by a human, it's also typically made by a monkey, and vice versa," he said. "So we see the world in the same way."

The key lies in a network of brain regions known as the 'ventral visual stream,' culminating in the inferior temporal cortex, or IT, where object recognition takes shape. While scientists previously understood the anatomy of this system, they lacked a way to model its function.

That changed with the introduction of deep learning.

Around 2013, DiCarlo and his collaborators began using machine learning techniques to train artificial neural networks on visual tasks. These systems, inspired by biological neurons, contain millions or even billions of adjustable parameters.

"We don't just have a conceptual model," he said. "We have an actual computer model that can take images and make a prediction of what should be happening at each of the levels of this system."

"The really exciting thing for us is that these networks started to align with the neural data far better than any models that we had before," DiCarlo said.

As AI systems improve at tasks like object recognition, they also become better estimations of how the brain works.

"As you make models better, you get better alignment with the brain data," he explained.

He explained that this represents a new scientific paradigm. AI is now a framework for understanding intelligence, instead of simply being a tool to study the brain.

DiCarlo then questioned if scientists can build accurate models of the brain, could they also use those models to influence it?

He described experiments in which researchers "invert" neural networks to design images that produce specific patterns of brain activity. Instead of merely observing how neurons respond to visual input, the models can suggest how to manipulate that input to achieve a desired effect.

"You can then ask the model, could you please find for us an image that moves the neural state into that regime," he said.

In the laboratory, this approach has already shown surprising results, and subtle, nearly imperceptible changes to an image can trigger dramatic shifts in neural activity.

"There are ways to do slight modulations that you might notice are barely perceptible to us, yet can cause large changes deep inside the system," DiCarlo said.

These findings point to what he called "holes in the visual fabric" of the brain, opportunities where targeted inputs can produce outsized effects.

DiCarlo then introduced what he described as a "science fiction story" grounded in emerging data.

"What if it were possible to modulate the activity in some specific set of neurons deep in the human brain, without giving you any drugs, without any electrodes, simply by modulating the pattern of light striking your eyes?" he asked.

The concept relies on technologies like augmented reality glasses that could subtly alter the visual world in real time. By adjusting the images a person sees, this could influence neural activity in targeted ways.

The potential applications are far-reaching, particularly in mental health. DiCarlo pointed to the amygdala, a brain region involved in emotion, as a possible target.

"Could we modulate in a way to lift mood or reduce anxiety?" he asked.

He emphasized that the idea remains unproven and raises significant ethical questions, yet early experiments suggest that the underlying mechanism may be feasible.

"We don't know if we can produce meaningful human health benefits," he said. "But we are intrigued by the possibility."

DiCarlo emphasized the central theme of how the merging of neuroscience and AI is reshaping how scientists think about intelligence.

"This virtuous loop is building a new science of intelligence," he said, describing the feedback between biological discovery and computational modeling.

"All of the things that we call the mind," he said, "can be modeled as an emergent property of a machine called the brain."

- Beth Squire