21_U. State Director Web Survey

U. State Agency Director Web Survey

OMB Number: 0584-#### Expiration Date: MM/DD/20YY

Introduction

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) is conducting a study to learn about tools used to identify Supplemental Nutrition Assistance Program (SNAP) cases likely to have a payment error. These tools may be known by different names, such as case-profiling tools, risk assessment tools, or error-prone profiling. After cases are flagged as high risk, they undergo a rigorous process to ensure accurate benefit decisions. FNS hired Westat to conduct this research study to identify SNAP State agencies that use case-profiling tools and to understand the development, effectiveness, and perceived value of the tools.

The information collected during the study will be used to highlight best practices, challenges, and solutions for case-profiling tools. These results will inform the development of case-profiling tools at the Federal level and will help FNS provide resources and technical assistance.

Please complete the survey by [date]. You may share the survey link, and other staff in your agency may log in to complete the survey. The survey should take between 15-45 minutes to complete depending on the characteristics of your State. You may log in to the survey as many times as you wish. To save your answers, click “Save and Continue Later” at the bottom of the screen before logging out. There is no cost to you to participate apart from the time you spend to complete the survey, and there is no compensation.

Participation in this study is voluntary. Refusal to participate will not have any impact on your position, your State agency, or your programs. You may also skip questions you do not wish to answer.

Any personally identifying information obtained will be kept private to the extent provided by law. We will use the data we collect only for the purposes we describe. Please note that the final report will present the survey results both in the aggregate and at the individual State level. The raw survey data will be submitted to FNS at the end of the study for research purposes. Any personally identifying information will be removed from the raw survey data submitted to FNS.

If you need additional information, please contact the FNS Project Officer, Eric Williams, at [email protected] or please email us at [study email] or call [study phone].

Thank you.

What Are SNAP Case-Profiling Tools?

Before you fill out the survey, we want to define what we mean when we talk about SNAP case-profiling tools. For the purposes of this study, we are defining anything that meets the following three criteria as a case-profiling tool:

• The tool is used during the application or recertification process or on active cases.

• The tool identifies the risk of a case having a payment error (i.e., high/low risk, high/medium/low risk, etc.)

• Cases may be treated differently based on their risk category (e.g., high-risk applications are subject to additional scrutiny prior to benefit determination; high-risk active cases are prioritized for a quality assurance [QA] review).

What do case-profiling tools look like? SNAP case-profiling tools can range in sophistication from a simple checklist that helps eligibility workers identify cases likely to have a payment error to complex analytics tools incorporated into a State agency’s eligibility system. Three hypothetical examples follow:

Example 1. Checklist of Criteria to Flag Cases for Review

State agency A determined payment errors were concentrated in large households of six or more people and in households with no reported income. State agency leadership requires a second review for all applications meeting either criterion prior to eligibility and benefit determination. When processing applications, eligibility workers flag all such cases for supervisors to review.

Example 2. Integrated Case-Profile Tool

State agency B used several years of Quality Control data to develop a machine-learning model that predicts the likelihood a case has a payment error. The model is integrated into the eligibility system, which automatically assigns a risk score to each case as data are entered. If the score is above a certain threshold, the case is flagged as high risk and undergoes a QA review within 30 days of becoming an active case.

Example 3. QA Review Screener and Procedures Manual Update

State agency C requires supervisors at local offices to conduct at least 10 QA case reviews per month. These reviews are entirely separate from the SNAP QC process. Instead of choosing the 10 cases for QA review at random, the State agency identifies 10 cases at high risk for payment error by passing all active cases through a computerized case-profiling tool. The State agency updated its QA review procedures manual to formally integrate this process into its QA approach.

If you want to ask a clarifying question, please email us at [study email] or call [study phone].

Screener1. Based on the definition and examples provided, which of the following describe your agency’s experience developing and implementing case-profiling tools? Select all that apply.

Our agency currently uses a case-profiling tool(s). (1)

Our agency previously used a case-profiling tool but has since discontinued it. (2)

Our agency developed but never implemented case-profiling tools. (3)

Our agency has never developed or implemented case-profiling tools. (4)

[PROGRAMMER: If SCREENER1_4 is CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: Ask SECTION A If SCREENER1_1 = checked]

Section A. State Agencies With Current Case-Profiling Tools

A1. To the best of your knowledge, how many case-profiling tools does your State agency currently use to identify SNAP cases at risk of payment error?

_________#_________

[PROGRAMMER: IF 1, GO TO A1a; IF 2 OR MORE, GO TO A1b]

A1a. Please provide the name of the tool so that we may refer to it throughout the survey. If the tool does not have an official name, a two- to four-word description may be used (e.g., Essex County’s Tool).

__________________________________________________________________________________

A1b. Of the [INSERT # FROM A1] tools your State agency currently uses, we will ask followup questions about one of them. Please provide a name for one of the tools, preferably the one with which you are most familiar, so that we may refer to it throughout the survey. If the tool does not have an official name, a two- to four-word description may be used (e.g., Essex County’s Tool).

__________________________________________________________________________________

A2a. Please briefly describe what the [INSERT TOOL NAME FROM A1a or A1b] was designed to do.

__________________________________________________________________________________________________________________________________________________________________________A2b. Can local agencies/counties customize (or change) the criteria used by the [INSERT TOOL NAME FROM A1a or A1b] to flag cases at risk of a payment error?

Yes (1)

No (2)

Don’t know (3)

A3. In what month and year was [INSERT TOOL NAME FROM A1a or A1b] first implemented?

Month: __________________________

Year: ____________________________

A4. What was the motivation for creating the [INSERT TOOL NAME FROM A1a or A1b]? Select all that apply.

To address high payment error rates (1)

To address the findings or recommendations of an audit or management evaluation (2)

To concentrate resources (i.e., staff, funds, time) on only those SNAP cases suspected as being at high risk of payment error (3)

To create a formal process for identifying SNAP cases at risk of payment error (4)

Other; explain: __________ (5)

Don’t know (6)

[PROGRAMMER: IF A4_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A5. Who helped design and/or build the [INSERT TOOL NAME FROM A1a or A1b] for SNAP? Select all that apply.

SNAP State agency program/policy staff (1)

State IT staff (2)

Other State-level staff (3)

Local SNAP office staff (4)

Vendor/contractor (5)

FNS Regional or National Office (6)

Other; specify: _________ (7)

Don’t know (8)

[PROGRAMMER: IF A5_5 = CHECKED, GO TO A6, ELSE GO TO A7]

[PROGRAMMER: IF A5_8 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A6. When, if at all, was the SNAP State agency involved in developing the [INSERT TOOL NAME FROM A1a or A1b] with the vendor/contractor? Select all that apply.

Conceptualization or design phase (1)

Development phase (2)

Testing phase (3)

Monitoring and evaluation (4)

Other phase; specify: _________ (5)

The SNAP State agency was not involved in developing the tool (6)

Don’t know or don’t recall (7)

[PROGRAMMER: IF A6_6 | A6_7 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A7. Who helped test the [INSERT TOOL NAME FROM A1a or A1b] to make sure it worked as intended? Select all that apply.

SNAP State agency program/policy staff (1)

State IT staff (2)

Other State-level staff (3)

Local SNAP office staff (4)

Vendor/contractor (5)

Other; specify: _________ (6)

Not applicable, the tool was not tested (7)

Don’t know (8)

[PROGRAMMER: IF A7_7 | A7_8 = CHECKED, NO OTHER OPTIONs CAN BE SELECTED]

A8. When designing the [INSERT TOOL NAME FROM A1a or A1b], what data were analyzed to determine which SNAP cases are at high risk of payment error? Select all that apply.

SNAP Quality Control data (1)

Other SNAP data (2)

Proprietary vendor or contractor data (3)

Other source, describe: _________ (4)

Don’t know (5)

[PROGRAMMER: IF A8_5 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

The next five questions ask about the data elements the SNAP State agency considered and eventually selected for inclusion in the [INSERT TOOL NAME FROM A1a or A1b]. The State agency’s consideration of a data element could mean the agency reviewed the policy related to that data element (e.g., policy related to household size), collected data from staff members on data elements associated with payment error, and/or analyzed SNAP QC or other data to assess the relationship between the data element and payment error.

Each question asks about a different category of data elements, which include (1) household composition, (2) demographic characteristics of the head of the household, (3) economic characteristics, (4) household expenses, and (5) case characteristics.

A9. For the first category, household composition, which of the following data elements were considered, and which were selected, for the [INSERT TOOL NAME FROM A1a or A1b]? Use the checkboxes to the right of each household composition data element to indicate if the data element was considered and/or selected for your State agency’s case-profiling tool.

Household Composition Data Elements	Considered	Selected	Neither	Don’t Know
Total number of household members (1)
Presence/number of children in the household (2)
Presence/number of elderly in the household (3)
Presence of disabled household member (4)
Presence of ABAWD household member (5)
Presence of household member ineligible for SNAP (6)
Child-only unit (7)

Note: ABAWD = able-bodied adults without dependents

[PROGRAMMER: Display A9a on the same web page as A9]

A9a. If any other household composition data elements were included in the [INSERT TOOL NAME FROM A1a or A1b], please list them below.

____________________________________________________________________________________________________________________________________________________________________

A10. The next data category is the demographic characteristics of the household members. Which of the following demographic data elements were considered, and which were selected, for the [INSERT TOOL NAME FROM A1a or A1b]? Use the checkboxes to the right of each demographic data element to indicate if the data element was considered and/or selected for your State agency’s case-profiling tool.

Demographic Data Elements	Considered	Selected	Neither	Don’t Know
Age (1)
Race (2)
Ethnicity (3)
Sex/gender (4)
Student status (5)
Level of education (6)
Employment status (7)
Marital status (8)
Homeless (9)
Residency status (10)
Citizenship status (11)

[PROGRAMMER: Display A10a on the same web page as A10]

A10a. If any other demographic data elements were included in the [INSERT TOOL NAME FROM A1a or A1b], please list them below.

____________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: Include a hover box definition for “Assets/resources” that reads as follows: “Assets/Resources include sums of money owned by the household and usually in an easily accessible financial instrument like savings accounts or stocks. In addition, assets/resources may include the monetary value of household property such as vehicles or land.”

A11. The next data category is economic characteristics of the SNAP household. Which of the following economic data elements were considered, and which were selected, for the [INSERT TOOL NAME FROM A1a or A1b]? Use the checkboxes to the right of each economic data element to indicate if the data element was considered and/or selected for your State agency’s case-profiling tool.

Economic Data Elements	Considered	Selected	Neither	Don’t Know
Presence of earned income (1)
Presence of unearned income (2)
Zero income (3)
Gross income (4)
Net income (5)
Self employment income (6)
Assets/resources (7)

[PROGRAMMER: Display A11a on the same web page as A11]

A11a. If any other economic data elements were included in the [INSERT TOOL NAME FROM A1a or A1b], please list them below.

____________________________________________________________________________________________________________________________________________________________________

A12. The next data category is household expenses. Which of the following household expense data elements were considered, and which were selected, for the [INSERT TOOL NAME FROM A1a or A1b]? Use the checkboxes to the right of each household expense data element to indicate if the data element was considered and/or selected for your State agency’s case-profiling tool.

Household Expense Data Elements	Considered	Selected	Neither	Don’t Know
Medical expenses (1)
Non-utility shelter expenses (2)
Utility expenses (including SUAs) (3)
Standard utility allowance (4)
Dependent care expenses (5)
Homeless shelter deduction (6)
Excess shelter deduction (7)
Legally obligated child support (8)

[PROGRAMMER: Display A12a on the same web page as A12]

A12a. If any other household expense data elements were included in the [INSERT TOOL NAME FROM A1a or A1b], please list them below.

____________________________________________________________________________________________________________________________________________________________________

A13. The next data category is case characteristics. Which of the following case characteristics were considered, and which were selected, for the [INSERT TOOL NAME FROM A1a or A1b]? Use the checkboxes to the right of each case characteristic to indicate if the data element was considered and/or selected for your State agency’s case-profiling tool.

Case Characteristics	Considered	Selected	Neither	Don’t Know
Length of certification period (1)
Reporting requirements (e.g., change reporting, simplified reporting) (2)
Benefit amount (3)
New applicant (4)

[PROGRAMMER: Display A13a on the same web page as A13]

A13a. If any other case characteristics data elements were included in the [INSERT TOOL NAME FROM A1a or A1b], please list them below.

____________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: ASK A14 IF A5_5 = CHECKED]

A14a. Of the household composition data elements you indicated were selected for the [INSERT TOOL NAME FROM A1a or A1b] (shown below), were any recommended by a vendor/contractor to flag cases at risk of payment error? Select “Recommended,” “Not recommended,” or “Don’t recall” for each data element you selected.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A9]

Data Elements Selected	Vendor gave a recommendation on this variable	Vendor did not comment on this variable	Don’t Recall
Total number of household members (1)	1	2	3
Presence/number of children in the household (2)	1	2	3
Presence/number of elderly in the household (3)	1	2	3
Presence of disabled household member (4)	1	2	3
Presence of ABAWD household member (5)	1	2	3
Presence of household member ineligible for SNAP (6)	1	2	3
Child-only unit (7)	1	2	3

A14b. Of the demographic data elements you indicated were selected for the [INSERT TOOL NAME FROM A1a or A1b] (shown below), were any recommended by a vendor/contractor to flag cases at risk of payment error? Select “Recommended,” “Not recommended,” or “Don’t recall” for each data element you selected.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A10]

Data Elements Selected	Vendor gave a recommendation on this variable	Vendor did not comment on this variable	Don’t Recall
Age (1)	1	2	3
Race (2)	1	2	3
Ethnicity (3)	1	2	3
Sex/gender (4)	1	2	3
Student status (5)	1	2	3
Level of education (6)	1	2	3
Employment status (7)	1	2	3
Marital status (8)	1	2	3
Homeless (9)	1	2	3
Residency status (10)	1	2	3
Citizenship status (11)	1	2	3

A14c. Of the economic data elements you indicated were selected for the [INSERT TOOL NAME FROM A1a or A1b] (shown below), were any recommended by a vendor/contractor to flag cases at risk of payment error? Select “Recommended,” “Not recommended,” or “Don’t recall” for each data element you selected.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A11]

Data Elements Selected	Vendor gave a recommendation on this variable	Vendor did not comment on this variable	Don’t Recall
Presence of earned income (1)	1	2	3
Presence of unearned income (2)	1	2	3
Zero income (3)	1	2	3
Gross income (4)	1	2	3
Net income (5)	1	2	3
Self employment income (6)	1	2	3
Assets/resources (7)	1	2	3

A14d. Of the household expense data elements you indicated were selected for the [INSERT TOOL NAME FROM A1a or A1b] (shown below), were any recommended by a vendor/contractor to flag cases at risk of payment error? Select “Recommended,” “Not recommended,” or “Don’t recall” for each data element you selected.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A12]

Data Elements Selected	Vendor gave a recommendation on this variable	Vendor did not comment on this variable	Don’t Recall
Medical expenses (1)	1	2	3
Non-utility shelter expenses (2)	1	2	3
Utility expenses (including SUAs) (3)	1	2	3
Standard utility allowance (4)	1	2	3
Dependent care expenses (5)	1	2	3
Homeless shelter deduction (6)	1	2	3
Excess shelter deduction (7)	1	2	3
Legally obligated child support (8)	1	2	3

A14e. Of the case characteristics data elements you indicated were selected for the [INSERT TOOL NAME FROM A1a or A1b] (shown below), were any recommended by a vendor/contractor to flag cases at risk of payment error? Select “Recommended,” “Not recommended,” or “Don’t recall” for each data element you selected.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A13]

Data Elements Selected	Vendor gave a recommendation on this variable	Vendor did not comment on this variable	Don’t Recall
Length of certification period (1)	1	2	3
Reporting requirements (e.g., change reporting, simplified reporting) (2)	1	2	3
Benefit amount (3)	1	2	3
New applicant (4)	1	2	3

[PROGRAMMER: ASK A15 IF A5_5 = CHECKED]

A15. Does the [INSERT TOOL NAME FROM A1a or A1b] include any variables from datasets maintained by a private company (e.g., LexisNexis LexID)?

Yes (1)

No (2)

Don’t know (3)

[PROGRAMMER: Ask A15a if A15 = Yes]

A15a. Please describe the variables included in [INSERT TOOL NAME FROM A1a or A1b] from a private company .

____________________________________________________________________________________________________________________________________________________________________

A16a. Briefly describe how the decision was made to incorporate the data elements shown below in the [INSERT TOOL NAME FROM A1a or A1b] used to flag cases at high risk of payment error.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A9, A10, A11, A12, or A13 ORGANIZED BY DATA ELEMENT CATEGORY]

__________________________________________________________________________________________________________________________________________________________________________

A16b. Briefly describe how the decision was made to incorporate the other data elements you listed in the [INSERT TOOL NAME FROM A1a or A1b] used to flag cases at high risk of payment error.

[PROGRAMMER: POPULATE RESPONSE OPTIONS FROM VARIABLES CHECKED “Selected” IN A9a, A10a, A11a, A12a, A13a, or A15a ORGANIZED BY DATA ELEMENT CATEGORY]

__________________________________________________________________________________________________________________________________________________________________________

A17. What type of data analysis was used to develop [INSERT TOOL NAME FROM A1a or A1b]? Select all that apply.

Descriptive statistics (1)

Regression modeling (2)

Machine learning (3)

Other; describe: _________ (4)

No data analysis was done when developing the tool (5)

Don’t know (6)

[PROGRAMMER: IF A17_5 | A17_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

The next few questions ask about how and when the [INSERT TOOL NAME FROM A1a or A1b] is used.

[PROGRAMMER: Use pop up informational window for “Hover here for examples of case profiling tools” to show the following text: “

What do case-profiling tools look like? SNAP case-profiling tools can range in sophistication from a simple checklist that helps eligibility workers identify cases likely to have a payment error to complex analytics tools incorporated into a State agency’s eligibility system.

For example, a state agency may determine payment errors were concentrated in large households of six or more people with no reported income. In this instance, the eligibility workers use a checklist to flag all such cases for supervisors to review.

For a second example, a State agency may have used a machine learning model to predict the likelihood a case has a payment error and integrated the results into the eligibility system. This integrated case-profiling tool would automatically assign a risk score to each case as data are entered. If the score is above a certain threshold, the case is flagged as a high risk of payment error and undergoes a QA review. 

A18. Which of the following options best describe the format of the [INSERT TOOL NAME FROM A1a or A1b]? Hover here for examples of case profiling tools. Select all that apply.

Written instructions (1)

Paper checklist (2)

Electronic checklist (3)

Algorithm programmed into the eligibility system (4)

Algorithm programmed into other systems or databases (5)

Other format; describe: _________ (6)

Don’t know (7)

[PROGRAMMER: IF A18_7 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A19. When is the [INSERT TOOL NAME FROM A1a or A1b] used to identify SNAP cases at risk of payment error? Select all that apply.

At the point of application submission for new cases but before the interview (1)

After the interview for new cases but before eligibility determination (2)

After the eligibility determination for new cases but before benefits have been issued (3)

After initial benefit issuance and before recertification (active cases) (4)

At the point of application submission for recertification but before the interview (5)

After the interview for recertification but before eligibility determination (6)

After the recertification determination but before benefits have been issued (7)

After benefit issuance for recertified cases (active cases) (8)

Other time period; please explain: ________ (9)

Don’t know (10)

[PROGRAMMER: IF A19_10 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A20. What data in the SNAP case file, if any, does the [INSERT TOOL NAME FROM A1a or A1b] examine to determine whether the household is at risk of payment error? Select all that apply.

Data from the household application (1)

Data from the household interview (2)

Data from data matches (3)

Other; explain:______ (4)

Don’t know (5)

[PROGRAMMER: IF A20_5 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: IF A20_5 = CHECKED, SKIP TO A22, ELSE CONTINUE TO A21]

A21. Please describe how the [INSERT TOOL NAME FROM A1a or A1b] uses data from the SNAP case file to determine whether the household is at risk of payment error.

__________________________________________________________________________________________________________________________________________________________________________

A22. Which staff use the [INSERT TOOL NAME FROM A1a or A1b] or the results for any purpose? Select all that apply. If county-level staff are the same as local-level staff in your State, please select local-level staff.

Local-level staff (1)

County-level staff (2)

State-level staff (3)

[PROGRAMMER: ASK A23 IF A22_1 = Checked]

A23. Which local-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to flag cases at risk of payment error? Select all that apply.

Administrative staff (e.g., receptionist) (1)

Frontline eligibility workers (2)

Eligibility worker supervisors (3)

Other local office staff; specify:________ (4)

Not applicable (e.g., the tool is fully automated, so no staff need to take any action to flag cases at risk of payment error) (5)

Don’t know (6)

[PROGRAMMER: IF Q23_5 | Q23_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A24 IF A22_1 = CHECKED]

A24. Which local-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to follow up on cases at risk of payment error? Select all that apply.

Administrative staff (e.g., receptionist) (1)

Frontline eligibility workers (2)

Eligibility worker supervisors (3)

Other local office staff; specify:________ (4)

Not applicable (5)

Don’t know (6)

[PROGRAMMER: IF A24_5 | A24_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A25 IF A22_1 = Checked]

A25. How are local-level staff trained on using the [INSERT TOOL NAME FROM A1a or A1b] to flag cases?

In-person (1)

Live virtual session (2)

Online training without a live presenter or facilitator (3)

Written tutorial (4)

Other: ________ (5)

Not applicable; staff are not trained on this (6)

[PROGRAMMER: IF A25_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A26 IF A22_2 = Checked]

A26. Which county-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to flag cases at risk of payment error? Select all that apply.

Quality assurance staff (1)

Quality control staff (2)

Eligibility workers (3)

Eligibility supervisors (4)

Other county-level staff; specify:________ (5)

Not applicable (e.g., the tool is fully automated, so no staff need to take any action to flag cases at risk of payment error) (6)

Don’t know (7)

[PROGRAMMER: IF A26_6 | A26_7 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A27 IF A22_2 = Checked]

A27. Which county-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to follow up on cases at risk of payment error? Select all that apply.

Quality assurance staff (1)

Quality control staff (2)

Eligibility workers (3)

Eligibility supervisors (4)

Other county-level staff; specify:________ (5)

Not applicable (6)

Don’t know (7)

[PROGRAMMER: IF A27_6 | A27_7 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A28 IF A22_2 = Checked]

A28. How are county-level staff trained on using the [INSERT TOOL NAME FROM A1a or A1b] to flag cases?

In-person (1)

Live virtual session (2)

Online training without a live presenter or facilitator (3)

Written tutorial (4)

Other: ________ (5)

Not applicable; staff are not trained on this (6)

[PROGRAMMER: IF A28_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A29 IF A22_3 = Checked]

A29. Which State-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to flag cases at risk of payment error? Select all that apply.

Quality assurance staff (1)

Statisticians (2)

Quality control staff (3)

Other State-level staff; specify:________ (4)

Not applicable (e.g., the tool is fully automated, so no staff need to take any action to flag cases at risk of payment error) (5)

Don’t know (6)

[PROGRAMMER: IF A29_5 | A29_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A30 IF A22_3 = Checked]

A30. Which State-level staff use the [INSERT TOOL NAME FROM A1a or A1b] to follow up on cases at risk of payment error? Select all that apply.

Quality assurance staff (1)

Statisticians (2)

Quality control staff (3)

Other State-level staff; specify:________ (4)

Not applicable (5)

Don’t know (6)

[PROGRAMMER: IF A30_5 | A30_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

[PROGRAMMER: ASK A31 IF A22_3 = Checked]

A31. How are State-level staff trained on using the [INSERT TOOL NAME FROM A1a or A1b] to flag cases?

In-person (1)

Live virtual session (2)

Online training without a live presenter or facilitator (3)

Written tutorial (4)

Other: ________ (5)

Not applicable; staff are not trained on this (6)

[PROGRAMMER: IF A31_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A32. Once the [INSERT TOOL NAME FROM A1a or A1b] flags a SNAP case as at risk of payment error, what is supposed to happen to that case next? Select all that apply.

It undergoes a second review by an eligibility worker (1)

It undergoes a second review by an eligibility worker supervisor (2)

It undergoes a quality assurance review (3)

Other; specify:________ (4)

No other action is taken (5)

Don’t know (6)

[PROGRAMMER: IF A32_5 | A32_6 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

A33. Is the [INSERT TOOL NAME FROM A1a or A1b] used for any other purpose besides flagging SNAP cases at risk of payment error?

Yes (1)

No (2)

Don’t know (3)

[PROGRAMMER: ASK A33a IF A33 = 1]

A33a. Please describe the other purpose(s) of the [INSERT TOOL NAME FROM A1a or A1b]?

____________________________________________________________________________________________________________________________________________________________________

The next few questions ask about evaluating the performance of the [INSERT TOOL NAME FROM A1a or A1b].

A34. In general, have you been able to ascertain whether the [INSERT TOOL NAME FROM A1a or A1b] has an impact on error rates, either good or bad?

Yes (1)

No (2)

Unsure (3)

A35. How often do you evaluate the effectiveness of the [INSERT TOOL NAME FROM A1a or A1b] to flag cases with payment errors?

Annually (1)

More frequently than annually (2)

Less frequently than annually (3)

We have never evaluated effectiveness (4)

[PROGRAMMER: ASK A35a IF A35 = 1 | A35 = 2 | A35 = 3]

A35a. Please describe the evaluation process.

____________________________________________________________________________________________________________________________________________________________________

A36. Has your State agency assessed whether the [INSERT TOOL NAME FROM A1a or A1b] flags SNAP cases at risk of a payment error in a way that unintentionally affects a particular race, ethnicity, gender, or other protected class more than others?

Note: For example, a tool may disproportionately flag certain races (e.g., Hispanics) if it looks for households with 8+ people..

Yes (1)

No (2)

Unsure (3)

[PROGRAMMER: ASK A36a IF A36 = 1]

A36a. Please describe the assessment of whether the tool unintentionally affects a particular race, ethnicity, gender, or other protected class more than others.

For example, did the agency conduct the assessment before or after tool rollout; is it ongoing? How was unintentional bias assessed?

____________________________________________________________________________________________________________________________________________________________________

A37. Please upload up to five materials that you think may help the study team understand how the [INSERT TOOL NAME FROM A1a or A1b] works to identify SNAP cases at risk of payment error.

If available, please include the following materials:

Tool itself (e.g., a PDF of a checklist tool, an .RDS file with a machine-learning model developed in R)

Documentation on the tool

Procedural manual for using the tool

Evaluations of the tool

Important: Do not include any SNAP participant data in any of your files.

Document 1: [document upload]

Document 2: [document upload]

Document 3: [document upload]

Document 4: [document upload]

Document 5: [document upload]

[PROGRAMMER: Ask SECTION B If SCREENER1_2 = checked]

Section B. State Agencies That Previously Used Case-Profiling Tools

B1. You indicated your State agency previously used a case-profiling tool to identify SNAP cases at risk of payment error but discontinued using that tool. Please provide the name of the tool so that we may refer to it throughout the survey.

If the tool does not have an official name, a two- to four-word description may be used (e.g., Essex County’s Tool). If your State agency has previously used and discontinued multiple tools, please tell us about the most recent tool.

___________________________________________________________________________

B2. Who helped design and/or build the [INSERT TOOL NAME FROM B1] for SNAP? Select all that apply.

SNAP State agency program/policy staff (1)

State IT staff (2)

Other State-level staff (3)

Local SNAP office staff (4)

Vendor/contractor (5)

FNS Regional or National Office (6)

Other; specify: _________ (7)

Don’t know/don’t recall (8)

[PROGRAMMER: IF B2_8 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

B3. Please briefly describe what the [INSERT TOOL NAME FROM B1] was designed to do.

__________________________________________________________________________________________________________________________________________________________________________

B4. What years was the [INSERT TOOL NAME FROM B1] in use? Estimates are fine.

From _______ [year] to _______ [year]

B5. Please briefly describe the reason(s) the [INSERT TOOL NAME FROM B1] is no longer in use.

__________________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: Ask SECTION C If SCREENER1_3 = checked]

Section C. State Agencies That Developed but Never Implemented Case-Profiling Tools

C1. You indicated your State agency developed a case-profiling tool to identify SNAP cases at risk of payment error but never implemented the tool. Please provide the name of the tool so that we may refer to it throughout the survey.

If the tool does not have an official name, a two- to four-word description may be used (e.g., Essex County’s Tool). If your State agency has developed but not implemented multiple tools, please tell us about the most recently developed tool.

_____________________________________________________________________________________

C2. Who helped design and/or build the [INSERT TOOL NAME FROM C1] for SNAP? Select all that apply.

SNAP State agency program/policy staff (1)

State IT staff (2)

Other State-level staff (3)

Local SNAP office staff (4)

Vendor/contractor (5)

FNS Regional or National Office (6)

Other; specify: _________ (7)

Don’t know (8)

[PROGRAMMER: IF C2_8 = CHECKED, NO OTHER OPTION CAN BE SELECTED]

C3. Please briefly describe what the [INSERT TOOL NAME FROM C1] was designed to do.

__________________________________________________________________________________________________________________________________________________________________________

C4. Please briefly describe the reason(s) the [INSERT TOOL NAME FROM C1] was never implemented.

__________________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: Ask SECTION D If SCREENER1_4 = checked]

Section D. State Agencies That Neither Developed nor Implemented Case-Profiling Tools

D1. Are you familiar with case-profiling tools used to identify which SNAP cases are at risk of payment error?

Yes (1)

No (2)

[PROGRAMMER: IF D1_2 = checked, SKIP TO SECTION E]

D2. Has your State agency ever considered developing a case-profiling tool to identify SNAP cases at risk of payment error?

Yes (1)

No (2)

[PROGRAMMER: ASK D2a IF D2 = 2]

D2a. Please share the reasons your State agency has not considered developing a case-profiling tool for SNAP.

____________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: ASK D2b IF D2 = 2]

D2b. Does your State agency plan to develop a case-profiling tool? Why or why not?

____________________________________________________________________________________________________________________________________________________________________

Section E. Context

[PROGRAMMER: ASK E1 IF SCREENER1 = 1]

E1. Do you currently have enough funding to properly administer the [INSERT TOOL NAME FROM A1a or A1b]?

Yes (1)

No (2)

Unsure (3)

E2. Has the COVID-19 pandemic affected your State agency’s ability to administer SNAP QC?

Yes (1)

No (2)

Unsure (3)

[PROGRAMMER: ASK E2a IF E2 = 1]

E2a. Please describe how the COVID-19 pandemic affected your State agency’s ability to administer SNAP QC.

____________________________________________________________________________________________________________________________________________________________________

[PROGRAMMER: ASK E3 IF SCREENER1 = 1]

E3. If the study team has followup questions, who is the best person to contact?

Name: (1) _______________

Title: (2) _______________

Email: (3) _______________

Phone: (4) _______________

You have reached the end of the survey! Thank you for taking the time to respond.

As you know, the final phase of the study involves case studies with six State agencies, which will be selected in collaboration with FNS. We will email you in the coming months if your State agency is selected.

This information is being collected to provide the Food and Nutrition Service (FNS) with key information on case-profiling tools used by SNAP State agencies. This is a voluntary collection, and FNS will use the information to examine risk assessment tools in SNAP. This collection requests personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-####. The time required to complete this information collection is estimated to average # hours (# minutes) per response. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22306 ATTN: PRA (0584-####). Do not return the completed form to this address. If you have any questions, please contact the FNS Project Officer for this project, Eric Williams, at [email protected].