Child Nutrition Formative Research on Administrative Review and Training Grants - SAs and SFAs

OMB Number: 0584-0524

Expiration Date: 09/30/2019

Appendix B. State Agency Follow-up Interview Protocol

My name is [name], and I’m a researcher at [Abt Associates/Insight Policy Research]. My colleague, [name], is also present to take notes throughout the interview. As a reminder, we are conducting a study of the Administrative Review and Training (ART) Grants for the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture. The primary purpose of this study is to better understand the ART grant interventions and to gather information about perceived results and implementation challenges. Additionally, we are interested in finding out how States sustain the grant-funded activities once the grant has ended. Your candid responses can help identify opportunities for improving the effectiveness of ART grants and inform FNS about the ways these grants have influenced the administration and delivery of child nutrition programs among grantees.

We are conducting follow-up interviews with a total of 10 State agencies for this study. When we complete all the interviews, we will summarize your responses with those provided by the other States in a final report for FNS. We will not use any names in that report or identify any individual respondents.

We expect our conversation will take 60 minutes. Do you have any questions for me about the project in general or what we will be discussing today?

With your permission, we would like to record the conversation to ensure our notes accurately reflect your responses. Do I have your permission to record our conversation? (IF NO: No problem. You may hear pauses throughout the interview as I may need to take additional time to record and verify your responses by hand.)

Confirm permission once recording starts. Note the State Agency name and date and time of the call.

• Yes

• No

State Agency
Interview Date
Time Start
Time End

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0524. The time required to complete this information collection is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Services, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0524). Do not return the completed form to this address.

NOTE TO INTERVIEWER: The table below will be pre-populated from data collected during initial interviews with SAs and SFAs. It is for your information only, in case the respondents don’t mention what we learned during the first interviews. They are potential points for probes or to follow-up on if they aren’t mentioned.

Summary of data collection interest area (e.g., intervention challenges, sustainability, outcomes, etc.)	Added areas for inquiry identified by the team from initial interviews (e.g., outcomes that may be of particular interest, innovative interventions etc.)

INTERVIEW AND QUESTIONS START HERE

Let’s begin with introductions.

Please provide your name, job title, and your role (to include how long you have been involved) in the Administrative Review process and ART grant-funded activities.

Name	First Interview Participant?	Title	Role in Administrative Review Process	Role with ART Grant	Duration in Role (Months/Years)

During our first interview, we talked with [refer to list below; welcome back “repeat” interviewees] about ART grant goals and outcomes.

Initial Respondent	Title	Role in Administrative Review Process	Role with ART Grant

Your State has been selected as a subset of former ART grantees so that we may learn more about outcomes, facilitators of success, challenges, and sustainability of grant-funded interventions.

Outcomes and Facilitators of Success

First, we are interested in finding out more about the results of your ART grant and, in particular, the factors that contributed to its success. We plan to refer to SFAs targeted by the grant.

(NOTE TO INTERVIEWER: Ask about areas specifically targeted by the grant.)

1. What do you think were the most important results of your ART grant?

(PROBES: Do you think it affected administrative costs? What about meal claims or compliance? What about the SFA error-rates identified in the Administrative Reviews?)

About how long did it take for you to start seeing those results? (NOTE TO INTERVIEWER: Refer back to each result reported.)

2. How do these results compare with the intended outcomes?

3. Many projects rely on outside vendors to develop and manage software applications related to their ART grant project. Did you contract with a vendor? (If yes:) We would like to know more about the successes and challenges associated with vendor management.

   1. In what areas did the vendor perform well? (NOTE TO INTERVIEWER: If not addressed in the answer ask:) What factors do you think led to their success?

   2. In what areas did you experience challenges with the vendor?

   3. What actions did you take in order to improve their performance?

   4. If you were to do this again, would you contract with a vendor or would you keep the project in-house? Why?

4. Did you design any software applications or other IT solutions in-house? (If yes:) We would like to know more about the successes and challenges associated with in-house development.

   1. In what areas did your in-house development team perform well? (NOTE TO INTERVIEWER: If not addressed in the answer ask:) What factors do you think led to their success?

   2. In what areas did you experience challenges with in-house development?

   3. What actions did you take in order to improve performance?

   4. If you were to do this again, would you still use an in-house developer or would you contract with a vendor? Why?

5. (NOTE TO INTERVIEWER: Ask if grant involved training or professional development.) I want to ask about professional training offered through the grant.

   1. What type of training was developed or improved upon as a result of receiving the ART grant?

2. How was the training offered to SFAs? (PROBES: In what formats; did these include, for example, in-person, online, etc.?)

3. Does the State track how the trainings are being used? (e.g., professional standards tracking, tracking via Administrative Review corrective action)?

4. Have you seen any improvements to program administration as a result of the offered training? (PROBE: Have improvements been identified based on a decrease in administrative error identified in the Administrative Review process?)

5. Have you seen improvement with SFA staff meeting program requirements or professional standards?

6. What other changes or improvements have you seen following the training?

6. State Agencies sometimes receive additional support from other departments within the State Agency or from FNS when implementing ART grants. We would like to know more about that support.

1. What, if any, support or technical assistance did you receive during grant implementation? (PROBE: This might include project management or IT support from another department or assistance from the FNS technical assistance team. Did you request that assistance?)

2. Was assistance available from the beginning or was it made accessible after encountering a specific issue/challenge?

3. Do you feel this support was sufficient?

4. If not, what additional support was needed?

1. What trainings or conferences, if any, helped you implement your project (FNS ART Grant training, Child and Nutrition Access & Accountability through Technology (CNAAT) training, outside professional training, etc.)?

   1. FNS also offered project management assistance to State Agencies. We would like your feedback on their support. Did you take advantage of this support?

   2. Do you feel this support was sufficient?

   3. If not, what additional support would have been helpful?

2. Thinking of your own project team, we would like more details on the process you used to manage your project:

   1. (If interviewee is not a director:) How involved was State Agency leadership (Assistant Director/Director) in the project?

   2. I have some questions about how the ART grant project was managed.

      1. Did you have to hire someone to handle project management responsibilities or did you use existing staff?

      2. Did the selected individual have prior project management experience? If not, did they receive any training on project management?

      3. Did that person fill other roles in addition to managing the project?

9. We also want to learn more about the kinds of support State Agencies provide to SFAs for implementation of ART grants.

a. Can you describe any specific types of assistance your State Agency provided SFAs during the grant – this might include technical assistance, training, manuals, etc.?

b. At what point in the ART grant process was assistance made available to the SFAs?

3. Did the SFA request the assistance or was it offered?

Challenges and Lessons Learned

During the prior interview we discussed some of the implementation issues and challenges encountered during the grant period and we would like to explore those further. Some of the issues we discussed included (review previously identified challenges)

10. What role, if any, did outside organizations or groups (for example, steering committee, State IT agency, State Agency leadership, etc.) play in helping to resolve challenges?

11. Please describe any challenges that may remain unresolved related to the ART grant implementation for either the State Agency or SFAs.

12. If you were to apply for and receive another ART grant, what would you do differently?

13. What advice would you give State Agencies to help ensure successful implementation and outcomes for a similar project?

Sustainability

14. How have the grant outcomes been sustained?

1. What has the State Agency done, if anything, to sustain any improvements from ART grant implementation?

2. What can the State Agency continue to do, or start doing, to sustain improvements?

3. What have the SFAs done, if anything, to sustain the improvements?

4. What can the SFAs continue to do, or start doing, to sustain the improvements?

15. What is the ongoing level of effort associated with continuing the activities implemented under the ART grant? (NOTE FOR INTERVIEWER: Staff level of efforts or description, not actual costs.)

1. From your perspective, do you see the level of effort for staff increasing or decreasing since the ART grant project was implemented?

2. How are/were you able to continue and sustain these activities after the end of your ART grant?

3. Please describe any additional funding sources used for sustaining ART grant activities after the grant period. (NOTE FOR INTERVIEWER: Probe for State Administrative Expense (SAE) funds, other funds.)

Wrap-up

16. Is there anything that we did not ask about that you think is important for us to know?

Thank you for answering my questions. That completes today’s interview. Have a nice day.

Stop recording and note time interview concluded.