RevESTAR Evaluation_Generic ICR_20210112

Date: January 12, 2021

To: Jordan Cohen, OMB Desk Officer, Office of Information and Regulatory Affairs, Office of Management and Budget

Through: Ruth Brown, Desk Officer, U.S. Department of Agriculture, Office of the Chief Information Officer

From: Maureen Lydon, Chief, Planning & Regulatory Affairs Office, Food and Nutrition Service

Re: Under Approved Generic OMB Clearance No. 0584-0524 Request for Approval for Evaluation of the Team Nutrition Training Grant Program: Enhanced Strategies, Training, Action Plans, and Resources (E-STAR)

The Food and Nutrition Service (FNS) of the United States Department of Agriculture (USDA) is requesting approval for formative research under Approved Generic OMB Clearance No. 0584-0524; A2-CN Team Nutrition Training Grants Program.

This request is to acquire clearance to conduct formative research on the USDA Team Nutrition Enhanced Strategies, Training, Action Plans, and Resources (E-STAR) training program. The research will involve a formative process evaluation to describe and understand the E-STAR training program as implemented as well as SNMs’ continued engagement in the program. Findings from this research will inform future iterations of E-STAR.

The following information is provided for your review:

1. Title of the Project: Evaluation of the Team Nutrition Training Grant Program: Enhanced Strategies, Training, Action Plans, and Resources (E-STAR)

2. Control Number: 0584-0524, Expires 12/31/2022

3. Public Affected by this Project:

State and Local/Tribal Government

• School nutrition managers (SNMs) who participate in E-STAR

Individuals:

• Individuals who mentor the SNMs participating in E-STAR (hereafter referred to as “mentors”)

See section 7, Project Purpose, Methodology & Formative Research Design, for a description of the number of participants for each respondent type and by research methodology.

4. Number of Respondents:

Table 4.1 – Research activity by Respondent Type

Respondent category	Respondent type	Research activity	Annual # of respondents (average)*
State, Local, and Tribal Governments	School nutrition managers	SNM Telephone Interviews	80
Individuals	Mentors	Mentor Telephone Interviews	10

*The annual number of respondents includes both responsive and non-responsive individuals. We anticipate an 80% positive response rate for all respondent types.

5. Time Needed Per Response:

Table 5.1 – Time needed for Research Activities by Respondent Type

Respondent category	Respondent type	Research activity	Time (minutes)	Time (hours)
State, Local, and Tribal Governments	School nutrition managers	Appendix B: SNM Interview Protocol Year 1	45	0.75
		Appendix C: SNM Interview Protocol Year 2	45	0.75
		Appendix F: SNM Consent Form	5	0.08
		Appendix G: SNM Mentor Recruitment Materials	5	0.08
		Appendix I: SNM Interview Phone Scripts	5	0.08
Individuals	Mentors	Appendix D: Mentor Interview Protocol Year 1	45	0.75
		Appendix E: Mentor Interview Protocol Year 2	45	0.75
		Appendix H: Recruitment for Mentor Telephone Interview	5	0.08
		Appendix J: Mentor Consent Form	5	0.08

* This also includes any time spent scheduling the interviews.

6. Total Annualized Burden Hours on Public: 126.56 hours (see Appendix A for Microsoft Excel file).

7. Project Purpose, Methodology, and Formative Research Design:

Background

As authorized under Section 6(a)(3) of the Richard B. Russell National School Lunch Act, 42 USC 1755(a)(3), the United States Department of Agriculture (USDA) Food and Nutrition Service (FNS) provides training and technical assistance for school nutrition professionals, nutrition education for children and their caregivers, and encourages school and community support for healthy eating and physical activity. The Team Nutrition Training Grants (TNTGs) are one way that FNS provides technical assistance and training to school nutrition professionals. TNTGs are awarded to State agencies who then implement trainings for school nutrition professionals within the Child Nutrition Programs in their State.

Under the 2019 Team Nutrition Training Grants (TNTGs), FNS awarded the Michigan Department of Education (MDE) with a TNTG to pilot and evaluate a new training program for school nutrition managers (SNMs) (Catalog of Domestic Federal Assistance number 10.574, OMB control number 0584-0512; A2-CN Team Nutrition Training Grants Program). This new program, the Enhanced Strategies, Training, Action Plans, and Resources (E-STAR) program, was developed in 2019 by the Institute of Child Nutrition (ICN). E-STAR builds on a prior program that ICN developed in 2018. While similar to the previous program, the E-STAR curriculum places a heavy emphasis on improving school meal quality and provides a longer, more intensive learning opportunity.

In brief, E-STAR is a two-year training program for SNMs. The program begins with a 2.5-day workshop where SNMs are assigned a mentor and develop a detailed action plan to improve the quality of meals served at their school. Following the workshop, SNMs spend the subsequent two school years participating in periodic Virtual Instructor-Led Trainings (VILTs) and mentoring sessions, and implementing their previously-developed action plans.

The 2019 TNTG that FNS awarded to MDE provides funds for MDE to fully implement all aspects of the E-STAR program. This includes recruiting, managing, and monitoring participation in the E-STAR program by 80 SNMs from participating School Food Authorities (SFAs). Furthermore, schools affiliated with SNMs who participate in the E-STAR training are eligible to receive subgrants to support the implementation of approaches to promote meal quality, such as the procurement of small equipment. As part of the grant requirements, MDE also identifies mentors (e.g., former SFA directors), who will provide mentoring sessions with participating SNMs over the course of the two-year training period. The grant requires a 1:10 mentor-to-SNM ratio to provide individualized technical assistance and training.

During the grant period, MDE will implement the full E-STAR program in coordination with ICN. A total of 80 SNMs and 8 mentors will receive the training. The training will begin in Summer 2021 and finish in Spring 2023. Throughout the duration of the program, MDE will work closely with FNS and the FNS national evaluation contractor, Abt Associates, to meet the objectives of the grant and to assist with the formative research described in this information collection request.

As part of their grant requirements, the State agency grantee (MDE) will recruit, consent, and retain a total of 80 SNMs and 8 mentors to participate in the E-STAR program and its evaluation. Throughout the duration of the grant performance period, MDE will continue to engage these individuals in E-STAR-related activities, and will periodically share data with Abt collected under its own grant requirements to inform the Abt-led process and preliminary outcome evaluations (see Data Sharing section below for more information).

Purpose

The purpose of this formative research is to:

1. Understand how the E-STAR training program was implemented and experienced by the State grantee (MDE), the 80 participating SNMs, and the 8 participating mentors; and
   
   

2. Measure changes (if any) in the knowledge, self-efficacy, and skills among participating SNMs with respect to school meal quality and training their frontline staff.

E-STAR is a new training program that has not previously undergone evaluation. Therefore, FNS has contracted with Abt Associates to conduct formative research (i.e., the process and outcome evaluations) on the program. Results from this research will guide FNS on potential program improvements and inform recommendations for scaling the E-STAR program, if at all, to additional states.

Methodology/Research Design

The study relies on three primary sources of data from two key respondent groups: participating SNMs and mentors. Data collection activities involve SNM surveys, SNM telephone interviews, and mentor telephone interviews. Additionally, the grantee will share data collected under the terms of the cooperative agreement with the study team to include in the analysis. These data describe the recruitment process and participation of the SNMs in E-STAR activities. The outcome evaluation uses a one-group, pre/multiple post-test design where changes in results from the SNM survey will be analyzed over time.

Data collection activities for which FNS is seeking clearance include:

SNM Telephone Interviews (Appendices B and C)

For the purpose of the process evaluation, Abt will conduct two semi-structured telephone interviews with all participating SNMs (up to n=80). The first interview will occur during Spring 2022 (when SNMs are at the end of year 1 of the E-STAR program), and the second interview during Spring 2023 (near the end of year 2 of the program). Each telephone interview will last approximately 45 minutes and (with consent) will be recorded for transcription. We anticipate an 80% response rate for the SNM telephone interviews.

Mentor Telephone Interviews (Appendices D and E)

Another aspect of the process evaluation considers the barriers and facilitators to successful mentoring during the two-year E-STAR program. For this purpose, Abt will conduct two semi-structured telephone interviews with all participating mentors (n=8). Similar to the SNM telephone interviews, the mentor telephone interviews will occur near the ends of years 1 (Spring 2022) and 2 (Spring 2023). Each telephone interview will last approximately 45 minutes and (with consent) will be recorded for transcription. We anticipate an 80% response rate for the mentor telephone interviews.

Data Sharing

Under the requirements described in the FY19 Team Nutrition E-STAR Training Grant Request for Applications (Catalog of Domestic Federal Assistance number 10.574, OMB control number 0584-0512; A2-CN Team Nutrition Training Grants Program), MDE is responsible for recruiting, consenting, and retaining SNMs and mentors to participate in E-STAR’s programmatic and study-related activities. MDE also bears responsibility for tracking implementation of the various program components (e.g., completion of 2.5-day E-STAR workshop, completion of mentoring sessions, and completion of the VILTs, and implementation of SNM action plans). Some of the tracking data will come from ICN, the E-STAR curriculum developer.

Given the collaborative nature of the grant program and this formative research, a data sharing agreement will be established between ICN, MDE/MPHI, and Abt Associates, thus allowing sharing of relevant data between the three entities. Data collected by MDE/MPHI and ICN and shared with Abt (e.g., participant demographics, pre/post test results, E-STAR participation tracking, and E-STAR satisfaction survey data) will be combined with data collected from the SNM and mentor telephone interviews to inform the overall process and preliminary outcome evaluations.

As per IRB and OMB requirements, the sharing of data will only occur using secure systems that meet the specified technical requirements. Specifically, data will be shared using Abt’s secure Huddle server. Huddle is a secure data transfer application for the secure, encrypted transmission of sensitive information such as Personally Identifiable Information (PII) and Protected Health Information (PHI). Huddle is a FedRAMP collaboration cloud service tool that stores data in the cloud and meets U.S. federal security requirements for encryption. The data is encrypted in-transit through the use of TLS, and at rest with 256-bit AES, and is FIPS 140-2 compliant. Only authorized study team members will have access to the relevant data folders in Huddle.

Design/Sampling Procedures

All E-STAR participants and mentors who consent to participate in the E-STAR program are eligible to participate in the data collection activities described in this request. As such, there are no sampling procedures, and each data collection activity will effectively function as a census of the SNMs and mentors recruited to the study.

Recruitment and Consent

As part of their grant requirements, MDE will recruit and consent up to 80 SNMs and 8 mentors to participate in the E-STAR program and all study data collection activities. The SNM and mentor consent forms (Appendix G) describe the planned evaluation data collection activities for the relevant populations, including both those undertaken by MDE and those undertaken by the FNS evaluation contractor, Abt Associates. In addition, the consent form specifies that MDE/MPHI and Abt will share participant data with FNS and with each other.

The study team will contact the participating SNMs and mentors to request their participation in the interviews.

In advance of the telephone interviews, SNMs and mentors will each be contacted by email to first invite their participation, and then, among those who agree to participate, to schedule the interview (Appendices G and H).

Incentives

SNMs and mentors who participate in the study will be eligible for gift cards to compensate for time spent on study activities. SNMs and mentors who complete the Year 1 and Year 2 telephone interviews will receive a $20 electronic gift card for each interview they complete.

In accordance with OMB guidelines, the study team took several factors into consideration when determining whether or not to use gift cards to offset burden.¹ Specifically, the team took into account the key data quality risks posed by the longitudinal study design, its other efforts to reduce non-response bias, the burden on respondents, complexity of the study design, panel retention over the study period, and prior use of incentives for this study population. It is well-established that incentives strongly reduce study attrition (i.e., increase response rates) in longitudinal studies.² In a longitudinal study like E-STAR, respondent retention during the follow-up period is critical, thereby necessitating the use of incentives.

Data Analysis

Process evaluation

Process evaluation analyses will describe the E-STAR program as implemented and summarize how the different components of the training were delivered. Quantitative and qualitative data will describe the following aspects of the program:

1. The E-STAR mentor orientation,

2. The E-STAR SNM workshop,

3. The SNM-developed action plans,

4. The content and amount of support for SNMs, including the VILTs and mentoring, and

5. The implementation of action plans (i.e., delivery of training to frontline staff).

Quantitative data collected via surveys and through MDE’s tracking system will be summarized to provide a descriptive analysis (averages, ranges, and counts) of the amounts and types of training and mentoring delivered to SNMs, as well as SNM plans for which trainings they will provide to their school’s frontline staff in order to improve meal quality.

Qualitative data collected via the phone interviews will be transcribed, coded, and analyzed by qualitative researchers using NVivo. The study team will identify and summarize key themes related to facilitators and barriers to: continued engagement in the E-STAR training program; SNMs’ action plan implementation; and providing effective mentoring. The study team will also examine how well mentoring was aligned with the goals of the training, and how similar or different the mentoring activities were across mentors.

We acknowledge that this evaluation involves a small, non-random sample without a true control group, thus the analyses are exploratory and not generalizable beyond this population and this particular training.

Outcomes/Findings

Study publications will not include information that personally identifies any of the research participants. Research findings will be presented in a final briefing and final report that uses plain language, highlights insights that can inform policy and practice, and includes graphics and figures rather than tables when possible. The presentation of the findings will be designed to inform future decisions related to E-STAR and other efforts to improve school nutrition. Publicly available reports or other documents describing findings will be 508 compliant and will be posted to the FNS website: http://www.fns.usda.gov/ops/research-and-analysis.

8. Confidentiality:

Using the consent forms for SNMs and mentors (Appendices F and J), participants will be informed of privacy act provisions before responding to study activities.

FNS complies with the Privacy Act of 1974 (5 USC 552a). The information gathered in this study will be kept private to the full extent permitted by law. Consent language covering all data collection activities will be provided in consent forms. Study participants will be informed of all planned uses of data, and that their information will be kept private to the extent permitted by law. System of Record FNS-8, FNS Studies and Reports, published in the Federal Register on 4/25/1991 at 56 FR 19078, covers personal information collected under this study and identifies safeguards for the information collected.

Federal Costs: $2,054,013

9. Research Tools/Instruments:

• Appendix A: E-STAR Evaluation Burden Table

• Appendix B: School Nutrition Manager Telephone Interview Protocol for Year 1

• Appendix C: School Nutrition Manager Telephone Interview Protocol for Year 2

• Appendix D: Mentor Telephone Interview Protocol for Year 1

• Appendix E: Mentor Telephone Interview Protocol for Year 2

• Appendix F: School Nutrition Manager Consent Form

• Appendix G: School Nutrition Manager Interview Recruitment Materials

• Appendix H: Mentor Interview Recruitment Materials

• Appendix I: School Nutrition Manager Phone Scripts

• Appendix J: Mentor Consent Form

1 Office of Information and Regulatory Affairs and Office of Management and Budget, 2016, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf. See page 69, questions 75 and 76.

2 Laurie, H., & Lynn, P. (2009). The use of respondent incentives on longitudinal surveys. In P. Lynn (Ed.), Methodology of longitudinal surveys (pp. 205-233). Chichester: John Wiley & Sons.

Section 12.5 reviews the effects of incentives in several prominent panel studies.

7