Classifying Open-Ended Text Responses in Surveys, Part I: Problems with the Dictionary Approach

Classifying Open-Ended Text Responses in Surveys, Part I: Problems with the Dictionary Approach

November 02, 2026

• Article

Asking survey questions with open-ended text responses is a great way to hear directly from your respondents in their own words. It can be especially useful if constructing a reasonable set of response options is prohibitive. On the other hand, reading hundreds of survey responses is burdensome for the analysts on the survey team. Classifying and quantifying those responses is even more burdensome, and subject to analysts' bias.

The purpose of this article is to describe the Economic Survey Research Center's current approach to classifying open-ended text responses to one specific question. In a series of subsequent Survey Practitioner's Corner articles, we will describe other methods we've tested in our search for more efficient analysis methods using newer machine learning and AI packages.

A classification problem

In each quarterly wave of the CFO Survey and every January in the Business Inflation Expectations survey and Survey of Business Uncertainty, we ask a question with an open-ended text response. The question text is: "What are the areas of most pressing concern for your company? Detailed explanations are helpful." Since the first quarter of 2024, the CFO Survey has had, on average, over 500 responses to this question every quarter.

Our goal is to classify each survey response into topics. The topics are a list of terms that describe economic concerns. We use a matching dictionary to classify economic concerns. Each phrase (Item) that might appear in a survey response is associated with a typology that characterizes broad themes like "costs" or "demand" and often will have subcategories (e.g., for "costs," responses can get split into labor and non-labor costs). Figure 1 shows part of the matching dictionary.

The matching dictionary

The dictionary is constructed by reading existing survey responses and extracting words or phrases from each response. We then classify each Item into one of the Category topics. In figure 1, the green highlighted text indicates phrases that one might expect to see quite often in the comments.

The yellow highlighted text are comments that are unlikely to appear again in other survey comments (or anywhere else). The Items are too long and too complex; nobody is likely to say that exact phrase again, nor does it contribute to the robustness of the dictionary.

The entire dictionary contains 98 categories and subcategories. In practice, just 28 different topics have appeared in the top 20 most mentioned topics for each quarter since the third quarter of 2022. Table 1 shows a list of all those topics.

Table 1: List of 28 topics in the matching dictionary

The following is an example of a concern from 2022: "Inflation, supply chain impacts (including China lockdown), pricing, Russian invasion of Ukraine." This survey response could be classified into the following set of topics:

• Inflation
• Supply chain concerns
• Profitability pricing power
• Geopolitical risk

Building the matching dictionary is labor-intensive, as survey staff must read and classify every survey response for several survey waves until we have enough comments to form a robust dictionary. This is prone to user error, as sometimes the comments may be misclassified or poorly classified.

The dictionary is not flexible in the sense that, if the creator uses too many phrases like the yellow highlighted phrases above, it will never be robust enough to capture short phrases or ideas like "inflation."

The dictionary is cumbersome to update-it's just a CSV file. While CSV files are easily opened in Excel, care must be taken to align new entries in the correct columns. Most recently, the word "tariff" and several of its misspellings were added to the dictionary file and classified in the "trade tariffs" category.

The topic of tariffs leads us to the next issue with dictionaries: When new economic conditions arise, like tariffs, or a war begins in a new country, the dictionary must be updated in real time. In an ideal world, survey analysts would anticipate economic developments before they become important enough for respondents to mention them. In practice, that's not always possible since economic conditions change quickly.

Another issue can emerge when categorizing words and phrases into various topics. Arbitrary biases may arise between different evaluators. For example, "inflation" for one analyst might be "cost" to another. This issue isn't necessarily restricted to two different analysts; it can occur for one analyst at two different points in time.

Sometimes simple is good enough

Instead of using many phrases and words from a matching dictionary, we can analyze the frequency of one word appearing in comments over time and across many waves of the survey. It is simple enough to write a script that displays the share of all comments containing a word.

In figure 2, we see that the word "tariff" (misspelled as "tarif") appeared in very few comments leading up to the 2024 election. After the election, "tariff" began appearing more frequently, peaking during the summer of 2025.

This method works well when we are interested in the trends over time for one word at a time.

Upgrading our technology for the future

In a forthcoming Survey Practitioner's Corner post, we will use BERTopic, a topic-modeling method that allows for classification of text into easily interpretable topics, while keeping important words in the topic description. We will refer to our list of topics in table 1 to use zero-shot topic modeling techniques offered by BERTopic.

After that, future articles will focus on another BERTopic classification technique-supervised topic modeling-as well as the use of advanced AI LLMs like Claude and Copilot to classify survey comments.