Child Nutrition Formative Research on Administrative Review and Training Grants - SAs and SFAs

OMB Number: 0584-0524

Expiration Date: 09/30/2019

Appendix A. State Agency Interview Protocol

My name is [name], and I’m a researcher at [Abt Associates/Insight Policy Research]. My colleague, [name], is also present to take notes throughout the interview. As you may know, we are conducting a study of the Administrative Review and Training (ART) Grants for the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture. The primary purpose of this study is to better understand the ART grant interventions and to gather information about perceived results and implementation challenges. Additionally, we are interested in finding out how States sustain the grant-funded activities once the grant has ended. Your candid responses can help identify opportunities for improving the effectiveness of the ART grants and inform FNS about the ways these grants have influenced the administration and delivery of child nutrition programs among grantees.

We are conducting interviews with a total of 20 States for this study. When we complete these interviews, we will summarize your responses with those provided by the other States in a final report for FNS. We will not use any names in that report or identify any individual respondents.

We expect our conversation will take 60 minutes. Do you have any questions for me about the project in general or what we will be discussing today?

With your permission, we would like to record the conversation to ensure our notes accurately reflect your responses. Do I have your permission to record our conversation? (IF NO: No problem. You may hear pauses throughout the interview as I may need to take additional time to record and verify your responses by hand.)

Confirm permission once recording starts. Note the State Agency name and date and time of the call.

• Yes

• No

State Agency
Interview Date
Time Start
Time End

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0524. The time required to complete this information collection is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Services, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0524). Do not return the completed form to this address.

Let’s begin with introductions.

Please provide your name, job title, role in the Administrative Review process generally, and role on ART grant funded activities in particular, including how long you have been involved in that role.

Name	Title	Role in Administrative Review Process	Role with ART Grant	Time in Role (Years/Months)

Planning and Implementation

Extant Data Summary		Note Changes Given by Interviewee
State
Year of Grant Award
Year of Grant Completion
Intervention Strategy(ies)
Goals/Objectives
Performance Measures
Grant Activity Summary

NOTE TO INTERVIEWER: The table above will be populated prior to the interview with data from the application, progress reports, final report, and the ART Grant Summary of Best Practices.

1. As we begin, we are referring to the ART Grant funded in year (verify from table above) and completed in (refer to table above).

Based on my review of your ART grant application, I understand your State Agency applied for this grant for the following reasons (briefly summarize reasons/targeted issues discussed in grant application). Would you say my understanding is correct (as far as you remember)?

1. If not, can you tell me in your own words why your State Agency applied for the ART grant in the first place?

1. Let’s talk about the project that was implemented with ART grant funds and the issues the project was intended to address.

   1. Prior to applying for the ART grant, who was involved in the decision-making to apply for the ART grant? Who was involved in the grant application process? (PROBE: If SFAs were not mentioned, ask:) Were SFAs involved in the grant planning or application process? What role did they play?

2. After the ART grant was awarded what was the process for managing the ART grant project? How did you monitor the project tasks, schedule and budget?

3. We are interested in how States and their SFAs dealt with issues or challenges during implementation.

   1. What issues did you encounter during implementation?

   2. How were these issues addressed?

   3. Were the steps that were taken to address those issues effective? In other words, did they resolve the problems being targeted? Did they make it easier to implement the project afterwards?

   4. What issues did your SFAs encounter during implementation?

   5. How were these issues addressed?

4. Oftentimes, project plans require some tweaking during implementation. How did your project plans or activities change between planning and implementation, if at all?

5. Did the project progress according to the original schedule? If not, what caused the delay?

   1. Were any extensions needed? If so, how many? For what reasons? Did you receive any additional funding or were these no-cost extensions?

6. How did you evaluate the project’s successes? Who was involved in that process and what was their job title and role within the organization?

7. We are also interested in getting a better understanding of how grant funds were utilized and whether you used any other funding sources to implement or sustain grant activities.

   1. Can you provide an overview of how the grant funds were allocated? (NOTE TO INTERVIEWER: If reports include this information, verify that the information in the reports is complete and up to date.)

   2. What additional funding sources, if any, did you use to support grant implementation? (NOTE TO INTERVIEWER: Ask the questions below if additional funding sources were used.)

      1. How were these additional funds used?

      2. How long were these funds used (for example, duration of grant, when grant monies ran out, a specified number of months, etc.)?

      3. Are these funds still available? Are you still eligible to apply for these funds in the future?

Short- and Long-Term Effects

8. Now I want to ask a few questions about the specific procedures or activities your grant targeted, and the outcomes.

   1. (FOR GRANTS PROJECTS RELATED TO ADMINISTRATIVE REVIEW:) Would you please briefly describe the process for Administrative Reviews prior to the intervention? We are interested in processes directly addressed by the Administrative Review such as the off-site and on-site review process, the corrective action process, and any fiscal action process, if applicable.

What tool was used to collect the Administrative Review information from SFAs (for example, Excel spreadsheets provided by FNS, paper forms, etc.)?

1. How did your project change the Administrative Review process at the State level?

2. How did your project change the Administrative Review process at the SFA level?

1. Do you think State staff spent more or less time on the Administrative Review process and other activities related to the Administrative Review process during and after the grant than they did on these activities prior to the grant? What about SFA staff?

1. (PROBE: If the new process requires additional time:) Do you think the new Administrative Review process will go faster for the State staff or SFA as they become more familiar with it?)

3. In general, what impact did the project have on State administrative error rates?

4. How do you think the intervention affected program staff, at the State Agency level and the SFA level?

1. Did it simplify the work of State Agency staff or make it more complicated?

2. How about SFA staff?

3. Do you think State Agency staff spend more or less time on administrative tasks targeted by the project than they did before funding was received?

4. How about SFA staff?

5. (FOR GRANTS INTENDED TO REDUCE ADMINISTRATIVE COSTS:) What savings or increased administrative costs have you seen as a result of the ART grant intervention?

1. Was there a specific area or process you were targeting to reduce costs? (PROBE: If targeted area was mentioned with an observed decrease in costs:) Did you see costs increase in any other area despite decreased costs in the targeted area as a result of ART grant activities?

2. At what point would you say you started to see those savings/increased costs (e.g., how long after implementation of the project)? (NOTE TO INTERVIEWER: Probe from beginning of grant period.)

3. What specific factors contributed to those costs or savings?

4. Do you have any evidence, such as the time necessary to complete the Administrative Reviews or changes in the number staff needed to perform Administrative Reviews that demonstrate these effects or is this based on your estimation?

6. (FOR GRANTS INTENDED TO IMPROVE DIRECT CERTIFICATION PROCESSES:) What impact, if any, did the project funded with the ART grant have on direct certification rates?

1. What about the direct certification procedures?

2. What changes, if any, have you seen since the grant was implemented?

3. What effects, if any, did the grant have on claiming and eligibility under the Community Eligibility Provision (CEP)?

7. (FOR GRANTS USED FOR TRAINING ON PROGRAM REQUIREMENTS AND PROFESSIONAL STANDARDS:) What outcomes were you hoping to see as a result of the training funded by your ART grant?

   1. Which of those outcomes were successfully accomplished by the end of the grant?

   2. How about since the grant ended? (PROBES: What changes on SFA error rates, if any, have you seen; for example, meal counting and claiming, DC matching, verification, etc.)?

   3. What are some other outcomes of the training you think we should know about? These may include positive or negative outcomes that were not specifically targeted.

   4. What do you consider the most important outcomes of the training? What outcomes did you see at the SFA level? At the State level?

9. The ART grants targeted the School Breakfast Program and the National School Lunch Program; however, we are interested in exploring impacts the grants may have had on the administration of other child nutrition programs. Do you administer either the Child and Adult Care Food Program or the Summer Food Service Program?

1. Please describe any impact on your school food service program administration of the Child and Adult Care Food Program or the Summer Food Service Program due to the implementation of the [local project name]. (NOTE TO INTERVIEWER: Probe for unintended consequences.)

Outcomes and Challenges

10. From your perspective, what were the most important outcomes of the ART grant-funded intervention (s), and how do these compare to the anticipated outcomes you discussed earlier? (RQ3) (NOTE TO INTERVIEWER: These may be positive or negative and you may probe accordingly.)

    1. Did you see any unanticipated outcomes?

11. How long after implementation did you first begin to see results from the project funded with the ART grant?

12. From your perspective, what were the top three challenges your State Agency experienced with grant activities before and during the implementation of the ART grant? How did you address these? (NOTE TO INTERVIEWER: Probe for specifics as it relates to what has been discussed about the State Agency’s objectives for the grant activities.)

    1. What are some lingering challenges following implementation of ART grant funding? What plans, if any, are in place to address those challenges? Have you applied or are you currently considering applying for another ART grant in order to address these lingering challenges?

Lessons Learned

13. If you were to apply for another ART grant what would you do differently? (NOTE FOR INTERVIEWER: Probe for during the planning phase, implementation phase and closeout phase.)

Wrap-up

14. Is there anything we did not ask that you think is important for us to know, or you were hoping we would discuss?

15. I realize that SFAs may have multiple ongoing projects at any given time, but we want to make sure the people we speak with focus specifically on ART grant-funded activities when answering our questions. How would you suggest we ask our questions so that respondents understand we are asking specifically about activities funded by the ART grant?

Local Project Name

Thank you for answering my questions.

That completes today’s interview. We may select State Agencies to further discuss details about challenges, successes and lessons learned for future grants. Would you be the best person to identify staff who may have expertise/experience with specific ART grant activities, such as training development, IT solutions, etc. for a future discussion?

• Yes.

• No. Contact_______________________________________________

Thank you again. Have a nice day.

Stop recording and note time interview concluded.