Participation Bias in Business Panel Surveys: A Study of the SBU

Participation Bias in Business Panel Surveys: A Study of the SBU

November 02, 2026

• Article

Perhaps the greatest challenge in modern survey practice relates to the misalignment between the observed sample, those from whom we have responses, and the target population. When such differences are not accounted for, statistics for the sample may not represent the population, thus introducing a bias. In many fields of study, this issue has grown in prominence as probability samples have seen participation rates plummet and convenience samples have become more prominent due to their relative ease and lower cost.

In short, many sampling schemes have a selection effect that introduce systematic compositional shifts away from the population. It is often assumed that both likelihood of survey participation and the measured variable are linked to known, measurable demographics. Then, realigning the sample-either by weighting or modeling-removes any bias. A much more insidious bias comes when the selection effect is directly linked to the variable of interest (Missing Not at Random in the literature). In general, because one does not observe values for those who did not participate in the survey, this phenomenon is hard to detect, let alone account for. However, in this article, we take advantage of the panel structure in the Survey of Business Uncertainty (SBU) to gain insight to this form of participation bias in the context of firms' year-ahead sales projections.

Introduction to the SBU

As a brief background, the SBU is one of three business surveys housed and managed by the Economic Survey Research Center (ESRC) at the Federal Reserve Bank of Atlanta. The SBU is a monthly survey of high-level firm representatives (overwhelmingly financial managers), who are asked about their respective firm's one-year-ahead expectations and uncertainties relating to employment, sales, and prices. Each of these three topics is measured through a separate survey.

Responding firms cycle through the versions in such a way that each survey version is responded to by about a third of the panel every month, and a firm takes each survey version once every three months. The panel is designed to be nationally representative and is frequently updated through the continual addition of newly recruited participants, balanced against natural participant attrition. More information about panel recruitment and composition as well as SBU participation can be found in the Recruitment and Participation in ESRC Business Surveys document.

The focus of this analysis is year-over-year sale projections. While beyond the scope of this article, the relevant survey question has a unique and innovative structure that yields a wealth of information about each respondent's projections. Specifically, each respondent first provides five growth levels-representing the lowest, a low, a middle, a high, and the highest points-and then assigns each a probability. From this 5-point distribution of likelihoods, a simple weighted average yields an overall expectation for each firm. After some routine data cleaning, these can be aggregated to generate an overall estimate for the collection of US firms.

Participation Patterns

In aggregating to the national level, it is natural and, in fact, important, to consider measurable firm demographics such as number of employees, sector, revenue, and even geographic location. But here we focus on whether sales projections alone affect the likelihood of participation. To make this point more precise, we introduce some notation: for firm i, let Yit be the projected sales growth in survey t and let Oit=1 if this value was observed (the firm participated in the survey) and 0 otherwise. The statistical question then centers around the probability of participation given the projected sales growth Prob ( Oit | Yit = y ) . If this is a function of y, there is a selection effect.

In an ideal world, we could simply compare projected sales growth ( Yit ) for the set of observed and unobserved firms. Quite obviously, this is impossible. Instead, we make use of the fact that within-firm sales projections are very strongly correlated temporally. In other words, if t−1 represents the previous survey, the correlation between Yit and Yi,t−1 is 0.79 . Of course, this is only among firms observed in both survey t−1 and t , but the finding is not surprising and suggests previous projections can serve as a good proxy. So, we can gain insight by considering Prob ( Oit | Yi,t−1 = y ) .

Figure 1 aggregates SBU data from March 2021 to the present to study participation rates as a function of previous sales projections. To this end, sales projections from all included surveys are binned into increments of 0.05 percentage points, and, for each, we calculate the sample proportion to participate in the following survey along with a confidence interval ( ± 2 standard errors). We also fit a logistic, quadratic model with the most likely fit, statistically significant from a flat trend line, shown in figure 1. The data and model make clear that firms with more extreme projections, on either end of the spectrum, are somewhat less likely to participate in the subsequent survey. Just over three-quarters (78 percent) of firms made a year-ahead sales projection between − 0.1 and 0.1 , albeit with a decided positive skew. For those firms, the likelihood of participating in the following survey is roughly 0.62 . A rough approximation, based on a simple stepwise function, is that firms with projections outside this range are about 5 percent less likely to continue survey participation. One plausible explanation is that representatives of these firms, subject to larger changes, have less time or energy to devote to surveys, translating to a slightly lower future participation rate.

Figure 1: A model of participation effects related to year-ahead sales projections in the SBU.

Effects of Participation Patterns

While the pattern described above and showcased in figure 1 is interesting in itself, a larger question is, what are the effects? Of course, the first order statistic of interest is the mean at any point in time, which takes the general form:

θ ^ = ∑ i | O it = 1 w it X it ,

where w it represents a survey weight of some kind. To determine the effect of the participation bias, we based weights as being proportional to the inverse probability of participating (thus adding more weight to the firms which represent firm types less likely to participate):

w it −1 ∝ { Prob ( Oit | Yi,t−1 ) if Oi,t−1 = 1 Prob ( Oit | Yi,t−1* ) if Oi,t−1 = 0 ,

where Y∗ is our best guess of Yi,t−1, given Yit. We use a simple linear regression to make this prediction and trust it to be reasonable based on the strong correlations between subsequent sales projections.

Estimates based on these weights, along with an unweighted version that ignores any participation bias, are shown in figure 2. Also shown are year-over-year growth changes in National Income Accounts, taken from the Bureau of Labor Statistics (BLS). Each set of predictors has an associated root-mean-squared error from the BLS data. While one can argue a lower error does not necessarily mean a better estimate, the main point is that the various versions do not vary much. In other words, factoring in the participation bias does not affect population mean estimates that much. The minor differences arise from the fact that the drop in participation from survey to survey is not that big (5 percent) and affects a relatively small subset of firms. In addition, for the mean, the symmetry of the effect tends to remove firms on both extremes, thus likely balancing effects and not impacting the mean as much.

Adjusting for Participation Bias

Figure 2: A comparison of estimates of year-ahead sales projections along with BLS National Income Accounts data. BLS data are matched to SBU estimates from a year before.

Of course, the selection effect may impact other aspects of the sample distribution, especially when compounded over time. For example, figure 3 shows the proportion of firms projecting sales growths between −10 and 10 percent as a function of survey month. While there is natural volatility, there is also a clear trend, wherein the samples are getting less heterogenous, with roughly a 10 percent increase in the percent of firms hugging this middle ground. It is possible that this phenomenon is simply brought about by shifting economic landscape and more conservative projections. However, it is also consistent with the selective participation patterns noted above, and thus worth noting.

Prop. Reporting Sales Growths Between (−10%, 10%)

Figure 3: Trends in the proportion of firms with relatively neutral sales projections.

Overall, there seems to be little danger in ignoring the studied participation bias, especially when considering the mean. At the same time, it is worth monitoring and thinking about, especially if a change in economic conditions leads to a greater share of firms having more extreme sales projections. It should be noted that the analysis here makes a slew of assumptions; more advanced analysis is beyond the scope of this article. If you have any thoughts, questions, or would like to share your own experiences with similar selection effects, we'd love to hear them. Please contact Marcin Hitczenko.