24_X. Interview Guide State-Level IT Systems Staff

X. Interview Guide: State-Level IT Systems Staff

OMB Number: 0584-#### Expiration Date: MM/DD/20YY

Introduction

Good morning/afternoon. Thank you for taking the time to talk with me today. My name is [interviewer’s name], and I work for Westat, a private research company based in Rockville, Maryland. Joining me is my colleague, [name].

Purpose: The U.S. Department of Agriculture’s Food and Nutrition Service, or FNS, is interested in understanding tools used to identify cases likely to have a payment error. These tools may be known by different names, such as case-profiling tools, risk assessment tools, or error-prone profiling. After cases are flagged, they undergo a rigorous process to ensure accurate benefit decisions. FNS hired Westat to conduct a study to learn more about the development and implementation of these tools. The findings from the study will be used to inform the development of case-profiling tools FNS and State agencies use, identify best practices, and develop resources and technical assistance.

How you were selected: We first conducted an online survey of all SNAP State directors. We then worked with FNS to select six State agencies among survey respondents for more indepth case studies on their use of case-profiling tools. After we discussed with the SNAP State Director what we hope to learn from your State agency, the State Director identified you as someone who would have valuable input and should be interviewed.

Risks and privacy: We use all data we collect only for the research purposes we describe. FNS knows which State agencies were asked to participate in each case study but does not know the names or job titles of the individuals interviewed. We will report the results of these interviews for each State agency, but your name will not be linked to your responses. In our reports, we may include direct quotes, but they will be presented without the speaker’s name or title. FNS will receive a redacted copy of the transcript of this interview that has been stripped of identifying information, except for the name of your State agency.

Study costs and compensation: There is no cost to you to participate apart from the time you spend with us for this interview, and there is no compensation. The interview takes 45 minutes.

Voluntary participation: Your participation is entirely voluntary. Refusal to participate will not have any impact on your position, your State agency, or nutrition programs. You may take a break, skip questions, say something off the record, or stop participating at any time.

Questions: If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1.888.920.7631. Please leave a message with your first name; the name of the research study you are calling about, which is the SNAP Risk Assessment study; and a phone number, beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last 45 minutes, until [time]. Is that still okay?

With your permission, I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on our secure server and will be destroyed after the project is complete. FNS will not receive any audio recordings.

Do you have any questions? [Answer all questions]

May I turn on the audiorecorder now? [Turn on audiorecorder if gives consent]

Now that the audiorecorder is on, do you agree to participate? [Pause for response]

And do you consent to be audiorecorded? [Pause for response]

Warmup and Context

1. To start, please tell me how long you have worked at the agency and what your responsibilities are.

2. For the rest of this discussion, we will be talking mostly about the [tool name], which your State agency provided information about in the online survey. What was the nature of your involvement with [tool name]?

[Probe: Designed it, built it, tested it? Is familiar with tool but was not involved in development?]

3. In the survey, the State agency said [tool name] is a [read survey response A18] that flags SNAP cases at risk of payment error using data from [read survey response A20]. Does that description still seem accurate?

1. [If no] How would you revise the description?

4. Has [tool name] been modified in any way since it was first implemented?

[If yes]

1. Please explain how it evolved.

[Probe: When and why it evolved; who initiated those changes?]

2. What were the reasons for those changes?

[Probe: Prompted by staff feedback, policy change, review of data, civil rights complaint, other?]

5. [For electronic tools/algorithms] To what extent does the [tool name] interface with State agency data systems?

[Probe: State eligibility system, statewide online application system, other?]

3. What data are exchanged through the interface?

4. What had to happen to make that interface successful?

5. What suggestions do you have for similar teams trying to integrate a tool like this with existing data systems?

[Probe: Challenges encountered and solutions?]

Developing the Tool [if relevant]

I want to understand how [tool name] was developed and what it was designed to do.

6. The survey indicates the following types of staff were involved in designing the tool: [read survey response A5]. How was it decided who would design the tool?

   1. [If A5 = vendor/contractor] How much input did the State agency have in how the tool was developed?

      1. How much input did the State agency have on the final algorithm for the tool?

7. [If A5 = multiple responses] How did the collaboration go between those different staff?

6. What suggestions do you have for similar teams trying to create a tool like this?

Testing and Reporting [if relevant]

8. [If survey response A7 NA] Tell me about how the [tool name] was tested before going live.

7. What was being tested?

[Probe: Were they looking at the overall accuracy of the tool? Equity of the tool across subgroups? User-friendliness?]

8. Who did the testing?

9. Did those early tests reveal anything that needed to be fixed?

[If yes]

1. Tell me a little about what needed to be fixed.

2. How long did that take to resolve?

1. Was the tool also tested after going live? If yes, how?

[Probe: What were they looking for with those tests—accuracy, equity, user-friendliness?]

10. Did anything need to be fixed when testing the tool after it went live?

[If yes]

1. Tell me a little about what needed to be fixed.

2. How long did that take to resolve?

1. Thinking broadly about the testing phase, what challenges arose during that time?

11. How were those challenges overcome?

11. Were any reporting tools programmed into your systems to provide information on [tool name]?

[Note: We are asking if they track any information on the number or types of cases flagged, the followup steps taken for these cases, such as efforts to find additional documentation on the case or calls to the household to ask questions]

[If yes]

1. What are those reports programmed to track with regard to [tool name]?

2. Who can access those reports?

3. Does the system automatically generate and disseminate reports or is it more that those with access can run the reports anytime they wish?

Wrap-Up

12. If you were to talk to other State agencies considering implementing a similar tool, what advice would you give them?

13. Before we wrap up, are there any key challenges to building case-profiling tools like [tool name] that we haven’t already discussed?

We’ve reached the end of the interview. Thank you so much for taking the time to talk with us and share your experiences. The information you provided gave us valuable insights into how tools like [tool name] work.

[If applicable] I recall you mentioning that you would be willing to share [documents] with me. Those would be really helpful to see, so thank you for offering to send them. You can send them to me at [email address]. I can also set up a secure FTP site to receive the materials if the documents contain identifying information.

This information is being collected to provide the Food and Nutrition Service (FNS) with key information on case-profiling tools used by SNAP State agencies. This is a voluntary collection, and FNS will use the information to examine risk assessment tools in SNAP. This collection requests personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-####. The time required to complete this information collection is estimated to average 0.75 hours (45 minutes) per response. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22306 ATTN: PRA (0584-####). Do not return the completed form to this address. If you have any questions, please contact the FNS Project Officer for this project, Eric Williams, at [email protected].