Understanding Risk Assessment in SNAP Payment Accuracy

Key Findings

Most state agencies do not use risk assessment tools.

Of the 43 state agencies and one local agency that responded to the study's online survey from July to November 2024 (Figure 1):

• Fifteen state agencies currently use an RA tool (34%);
• Three state agencies previously used an RA tool but have since discontinued its use (7%)-one found their tool to be ineffective and the other two had recent case management system changes that required a temporary pause;
• One state agency developed but has not yet implemented an RA tool (2%); and
• Twenty-four state agencies and one local agency have neither developed nor implemented an RA tool (57%).
  

Figure 1. Risk assessment tool status by state agency.

Most state agencies that use risk assessment tools do so to address payment errors and allocate resources.

• Among the 15 state agencies that use RA tools, most developed the tools to help address high payment error rates (PER) (10 state agencies), which is the average of absolute differences between the dollar amount of SNAP benefits received in error divided by the total accurate benefit amount, and/or to help them concentrate their available resources on suspected cases at higher risk of payment errors (seven state agencies).
• Six state agencies reported that they developed their tools to create a formal process for identifying at-risk cases, and two reported that they developed their tools to address audit findings.
• Four state agencies reported other motivations for their development of RA tools, which can be categorized as (1) being proactive in maintaining accuracy rates, (2) building in quality at the beginning of the process, and (3) helping staff better identify at-risk cases to improve the case review process (Figure 2).

Figure 2. Motivation(s) for developing RA tools currently in use.

The effectiveness of risk analysis tools was mixed for state agencies that provided additional data.

• Three state agencies provided additional data to statistically examine the risk-identification effectiveness of their RA tools: Kansas, Missouri, and Rhode Island. The risk analysis tools used by Kansas and Missouri were not effective at identifying cases with errors but were effective in helping caseworkers avoid re-reviewing cases that were unlikely to have errors. Conversely, Rhode Island's tool was effective at identifying cases with errors but was also prone to identifying as high-risk cases that had no errors, which can lead to misallocations of labor.
• The study team used a run chart to examine Minnesota's quarterly payment error data to determine whether there were any identifiable trend breaks after implementation of the RA tool. Based on the assumption that eight quarters of payment error rates below the state's quarterly median rate indicate evidence of success in the RA tool, Minnesota did not experience a shift in PERs after it implemented its RA tool, since it saw only three consecutive quarters with a payment error rate below the median (Figure 3).
• A difference-in-differences analysis, a statistical method for comparing observed changes in a treatment group over time against that of a control group over the same period, for Minnesota provides some suggestive, statically significant evidence of a modest reduction in Minnesota's quarterly payment error rate after implementing its RA tool. However, the model may have lacked appropriate statistical power.

Figure 3. Minnesota's quarterly payment error rates, October 2014 (FY 2015, quarter one) to December 2019 (FY 2020 quarter one).

Why FNS Did This Study

Risk assessment tools offer human services organizations, such as SNAP state agencies, potential opportunities to efficiently improve payment accuracy. Thus, is it important that we have information on state agencies' use of RA tools, how the tools were developed, the effectiveness of the tools, and promising practices for FNS and state agencies to consider in the development and use of RA tools.

How FNS Did This Study

This study involved a four-part data collection approach:

• We conducted a literature review to identify and evaluate RA tool use in human services programs by searching academic databases and publicly available online sources to identify relevant peer-reviewed literature, federal and state reports, publications from business and trade organizations, and vendor-produced materials related to RA tools. These searches yielded 107 documents for review. After deduplication and screening each item for relevance to the study, 23 relevant publications were incorporated into the final review.
• We undertook a web-based survey of all SNAP state agencies, of which 43 state and one local agency responded, to capture information on their use of RA tools. We used Qualtrics as the web-based survey software.
• We conducted and transcribed virtual key informant interviews with staff in six state agencies: five state agencies with current RA tools (Connecticut, Kansas, Rhode Island, Virginia, Wisconsin) and one state agency that formerly had an RA tool but stopped using it (Utah). The study team considered a variety of factors in selecting state agencies for case studies and collaborated with each state agency to identify the appropriate staff members or contractors to describe tool development and implementation, use, and effectiveness.
• We analyzed SNAP QC data to evaluate RA tool performance and effectiveness using two approaches. The first measure of effectiveness is whether the tools can accurately distinguish between cases with and without payment errors and efficiently select cases for review. The state agencies' tools were applied to SNAP QC sample data to identify which cases the tool flagged as high risk and compared these flags against which cases had a payment error. The second measure of tool effectiveness is whether the state agencies that implemented the tools saw a subsequent decrease in their quarterly payment error rate.