FAU Researcher Asks: Can AI Understand Space, Time and Pain

Artificial intelligence can do remarkable things, but how much does it actually understand about the world? Researchers have found that surprisingly specific information can be extracted from the inner workings of large language models (LLMs) like ChatGPT. Using mathematical tools to probe a model's internal activity, they can identify where a city is located on a map, when a historical figure lived or what emotion a character feels, even detecting descriptions of pain.

Many have interpreted these findings as evidence that AI models build an internal picture of the world. Some have even linked them to debates about whether AI might have the capacity to suffer. These questions have taken on new urgency as debate grows over whether AI systems could suffer or warrant moral consideration, prompting some companies to restrict harsh treatment of their chatbots.

But a new paper by Elan Barenholtz, Ph.D., of Florida Atlantic University suggests that the evidence may be weaker than it appears. The paper, "World Properties Without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models," is available as a preprint on arXiv. It examines whether the same information attributed to language models can be extracted from much simpler mathematical structures called word vectors.

These vectors capture patterns in which words tend to appear together, without accounting for context. Barenholtz found that they can reveal much of the same information as LLMs.

"This suggests that being able to read something out of an AI system doesn't prove the system actually has a kind of internal model of those things," said Barenholtz, an associate professor of psychology in FAU's Charles E. Schmidt College of Science and a member of the FAU Stiles-Nicholson Brain Institute. "Plain word statistics can often give you the same answer."

To test this, Barenholtz revisited four previously published studies examining how information about places, historical timelines, pain and emotions can be extracted from language models. He applied similar mathematical analyses to static word embeddings, which assign each word a fixed mathematical representation based on its relationships with other words in a body of text.

The results were striking. The simpler method predicted geographic coordinates and historical figures' years of death, distinguished sentences describing pain from similar sentences without pain, and identified 12 emotions in stories written to avoid explicitly naming those emotions. In the geography and historical timeline tests, the word-based method explained a substantial portion of the variation in locations and death dates. In the pain and emotion tests, it distinguished between categories with accuracy that approached the results reported for LLMs.

Although LLMs retained advantages, Barenholtz found that much of their edge came from identifying the correct country for a location or the correct century for a historical figure. These broad classifications rely on information that word associations can readily capture. When the task required distinguishing among places within the same country or historical figures within the same century, all the methods struggled.

The findings suggest that word associations carry a remarkable amount of information about the places and people they describe. Text about places in different regions, for example, may contain distinctive spelling conventions, references to local institutions or mentions of universities associated with particular countries. A system can use those linguistic patterns to infer geographic information without necessarily having anything resembling a map.

The principle dates back decades. In 1957, linguist J.R. Firth observed, "You shall know a word by the company it keeps." Barenholtz's results illustrate how much information can be recovered from the relationships among words alone, even without the contextual processing performed by modern AI systems.

Barenholtz also examined word vectors associated with specific Wikipedia entities, which capture the language patterns linked to a particular person or place rather than just the words in its name. These representations closed much of the remaining performance gap with LLMs, matching one model on geographic coordinates and another on historical figures' years of death. The finding reinforces the possibility that information encoded in language can account for results previously interpreted as evidence of a model's internal representation of the world.

The study does not establish that LLMs lack internal representations of the world. Nor does it settle questions about AI understanding, consciousness or the capacity to suffer. Barenholtz's argument is narrower: The ability to extract information from a model's internal activity, by itself, cannot establish whether the model represents the underlying property or whether the information can be explained by patterns already present in language.

"I'm not saying these models definitely don't represent the world and its properties," said Barenholtz. "I'm saying this kind of test can't tell us, so future studies should check against a simple baseline like this before drawing big conclusions."

His findings point to a practical step for future AI research: compare results from LLMs with simpler, word-based methods before drawing conclusions about what a model knows or represents. Barenholtz also notes that his comparison does not address causal or behavioral evidence, which may provide other ways to investigate how AI systems work.

-FAU-