SMO_Part B Final 1.14.21

Supporting Statement for OMB Clearance for the School Meals Operations Study: State Agency COVID-19 Child Nutrition Waivers Evaluation

Part B

Revision to OMB # 0584-0607, Child Nutrition Program Operations Study II (CN-OPS-II)

January 14, 2021

Holly Figueroa

Social Science Research Analyst

Office of Policy Support

Food and Nutrition Service

United States Department of Agriculture

1320 Braddock Place

Alexandria, Virginia 22314

Phone: 703-305-2105

Email: [email protected]

TABLE OF CONTENTS

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for the Collection of Information 3

B3. Methods to Maximize the Response Rates and to Deal with Nonresponse 5

B4. Test of Procedures or Methods to be Undertaken 8

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 12

TABLES

Table B1.1. Summary of Respondent Universe and Expected and Prior Response Rates 2

Table B4.1. Changes to State administrative data request materials based on respondent feedback and review of data files 9

Table B4.2. Changes to State CN Director Survey based on respondent feedback 10

Table B5.1. Individuals consulted on statistical aspects of study design 12

APPENDICES

A Research Questions

B Section 2202 of the Families First Coronavirus Response Act

C.1 State Child Nutrition Director Survey 2020-2021

C.2 Screenshots of Programmed State CN Director Survey 2020-2021

D.1 FNS-10 Administrative Data Request

D.2 FNS-418 Administrative Data Request

D.3 FNS-44 Administrative Data Request

D.4 Telephone Meeting Agenda

E.1 Study Support Email from FNS Regional Office to State Agencies

E.2 State Agency Child Nutrition Director Advance Email

E.3 Study Brochure

E.4 State Agency Invitation Email

E.5 Reminder Email

E.6 Reminder Call Script

E.7 State Agency Last Chance Post Card

F.1 Public Comment 1

F.2 Public Comment 2

F.3 Public Comment 3

F.4 Public Comment 4

F.5 FNS Response to Public Comment 1

F.6 FNS Response to Public Comment 2

F.7 FNS Response to Public Comment 3

G.1 National Agricultural Statistics Service Comments

G.2 FNS Response to National Agricultural Statistics Service Comments

H Confidentiality Pledge

I Estimated Annualized Burden

B1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The respondent universe for this collection includes the 67 State Agency (SA) Child Nutrition (CN) directors that oversee the National School Lunch Program (NSLP), the School Breakfast Program (SBP), NSLP Seamless Summer Option (SSO), Summer Food Service Program (SFSP), and Child and Adult Care Food Program (CACFP) in the 50 States, District of Columbia, Guam, Puerto Rico, and U.S. Virgin Islands. No sampling or weighting is required as the study will be a census of all 67 SAs. Given the total universe of only 67 eligible States and territories and their relatively distinct characteristics, there is not an efficient sample design that could closely match the comprehensive data on State policies and student meal service that a census will yield to better understand variation and localized concerns.

Expected Response Rates

Achieving high response rates on the survey and the administrative data request is critical to the study’s success. The Child Nutrition Program Operations Study II (CN-OPS II) (OMB number 0584-0607, expiration date 08/31/2022), of which the School Meals Operations Study is a revision, had survey response rates of 100 percent for the State CN survey for three of four data collection years (see Table B-1). Based on this experience, experience with CN administrative data requests on other studies, and the congressionally mandated SA reporting requirements for the COVID-19 CN waivers, a 100 percent response rate is expected for the SA web survey and administrative data request.

Table B1.1. Summary of Respondent Universe and Expected and Prior Response Rates

Respondent	Universe	Target completed cases	Expected Response Rates	CN-OPS II Response Rates
				SY 15-16	SY 16-17	SY 17-18	SY 18-19
State CN Agencies	67	67	100%	100%	100%	100%	96%

* The universe for CN-OPS II only included the 55 State CN agencies that oversee the NSLP and the SBP. Due to the unanticipated school closures related to COVID-19, other CN programs, including the SFSP and the CACFP, were used to provide meals to children beginning in March 2020. Thus, the State agencies that oversee the SFSP and the CACFP have been added to the SY 2020-2021 universe, increasing the universe of State CN agencies to 67.
** The SY 2018-2019 data collection with States was initially planned for spring 2019 but was delayed to fall 2019. The revised timing may have affected the ability of State agencies to participate timely, thus resulting in a slightly lower response rate for SY 2018-2019.

The approach to achieving high response rates builds on prior FNS study experience. The team developed engaging recruiting materials, listed in Appendices E.1-E.7, to describe the study, including a variety of appeals to encourage participation. Upon clearance, all FNS Regional Offices will send the Study Support Email from FNS Regional Office to State Agencies (Appendix E.1) to build support for the study among States. The study team will also provide information to SAs on the various study components prior to data collection using the State Agency Child Nutrition Director Advance Email (Appendix E.2) and the Telephone Meeting Agenda (Appendix D.4). In the event of non-response, State CN directors will receive the Reminder Email (Appendix E.5) every 2-3 weeks (two reminder emails total), a reminder call at 7 weeks using the Reminder Call Script (Appendix E.6), and the State Agency Last Chance Post Card (Appendix E.7) until the target of 67 respondents is reached. States are aware that their participation in this collection (both the survey and administrative data collection component) is mandatory and are already preparing to participate so FNS does not anticipate any issues not reaching 100 percent completion with the 67 target respondents.

A professional survey support specialist will be available to assist respondents by phone or email during business hours. Also, the web survey will allow respondents to save and exit the survey at any point, and then return to access and complete the survey later.

B2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection;

• Estimation procedure;

• Degree of accuracy needed for the purpose described in the justification;

• Unusual problems requiring specialized sampling procedures; and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

For school year 2020-2021, this study will be conducted with a census of State CN directors only. Because this study is a census and a 100 percent response rate is expected, there is no need for sampling, weighting or nonresponse adjustments. As noted in Part A, due to the COVID-19 pandemic, the SFA Director Survey, which usually requires a complicated sampling approach, was cancelled for SY 2020-2021 and thus the sampling methods are not discussed herein. While the state-level data in this study are always collected from a census of SAs that oversee NSLP and SBP, as described in Part A Question 2, the census has been expanded for SY 2020-2021 to include SAs that oversee SFSP and CACFP as well so the state sample size is slightly larger for this collection than in previous iterations (67 rather than 55 State CN Directors).

The respondents will have 10 weeks to complete the web survey, which allows time to plan their approach to the survey. If any respondents need to ask other staff members at the SA to complete different sections of the survey, they may do this by forwarding their unique survey links. Respondents will receive reminder emails and calls from trained survey support personnel. Respondents may also call and/or email professional survey support specialists to request help in completing their survey or with technical issues.

The study team will collect FNS-10 administrative data disaggregated by school food authority or school from State CN agencies for fiscal year 2020. Due to COVID-19 and the unanticipated school closures, we will also collect FNS-418 and FNS-44 administrative data disaggregated either by sponsor or institution, or by site or outlet from State CN agencies for March through September 2020 (the fiscal year 2020 pandemic months).¹ The study team will hold initial telephone meetings with SAs using the Telephone Meeting Agenda (Appendix D.4) to describe the data request and put in place memorandums of understanding to facilitate data sharing. It is not required that SAs sign the memorandum of understanding (included in Appendix D.4) prior to participating in the study but some SAs may prefer to do so. The team expects that SAs will primarily provide the administrative data in Excel files, but this may vary depending on the systems and databases used by the various State agencies. The study team will accept the administrative data in whatever format is most convenient for SAs. While the data will not contain any personally identifiable information, the study team will ensure that data are secure by providing a secure transfer site.

Statistical Methodology for Stratification and Sample Selection. Because the survey and administrative data request will be administered via a census of State CN agencies, there will be no sample stratification or selection.

Estimation Procedure. Because data will be collected from a census of State CN agencies, no estimation procedures are necessary for this collection.

Degree of Accuracy Needed for the Purpose Described in the Justification: Because data will be collected from a census of State CN agencies, this question is not relevant to this submission.

Unusual Problems Requiring Specialized Sampling Procedures: No specialized sampling procedures are planned.

Use of Periodic (Less Frequent than Annual) Data Collection Cycles to Reduce Burden. The data collection procedures will be conducted once. Concern regarding the periodicity of data collection cycles is not applicable.

B3. Methods to Maximize the Response Rates and to Deal with Nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

The study is expected to achieve a 100 percent response rate for both the web survey and the disaggregated administrative data. This means that the data collected will represent the entire universe of State CN Directors and rather than providing estimates to answer the research questions, we will be able to provide actual population totals. Thus, these data will provide reliable answers to the study’s research questions that represent the full population

Achieving the specified response rate involves contacting the States, securing their participation in the study, and then offering support and completion reminders. The study team will use the following methods to maximize participation and reduce nonresponse:

• All FNS Regional Offices will send the Study Support Email from FNS Regional Office to State Agencies (Appendix E.1) to CN directors to encourage participation in the study. Also, the study team will copy the Regional Offices on communications with the SAs to promote participation and response.

• The study team will send all State CN directors the State Agency Child Nutrition Director Advance Email (Appendix E.2) and Brochure (Appendix E.3), which explain the purpose of the study and describe study activities, to provide State CN directors with a positive notice that they will be invited to respond to a survey and provide administrative data in order to fulfill their statutory reporting requirements on the nationwide FFCRA waivers.

• The study team will send all State CN directors the State Agency Invitation Email (Appendix E.4) to invite them to complete the survey.

• Recruiting materials were carefully developed to emphasize the following points, which may resonate with respondents:

• The data collection is designed to gather information on statutory reporting requirements for the COVID-19 CN waivers.

• Full participation in the study will satisfy the statutory reporting requirements.

• Data regarding the use and impact of the COVID-19 CN waivers will be used to inform FNS policy and procedures in future emergency situations.

• Current contact information will be used for initial correspondence and will be updated as needed throughout the data collection period (via the web survey, for example) to facilitate communication with the respondents.

• Designated FNS regional staff will serve as regional study liaisons and will be kept closely informed about the project so that they will be able to answer questions from CN directors and encourage participation.

• A toll-free number and study email address will be provided to all participants so that CN directors can receive assistance with the study. Staff will be readily available to clarify survey questions and work with participants to resolve technical issues, such as difficulty logging on or advancing past pages. Personalized assistance bolsters the perceived legitimacy of the study and will encourage respondents to persist in completing the survey.

• The Reminder Email (Appendix E.5) will be sent to State CN directors every 2-3 weeks for a total of two reminder emails before the reminder phone call (Appendix E.6) at week 7.

• The study team will follow up by telephone with all CN directors who do not complete the survey and urge them to complete the survey. If the respondent prefers to complete the survey or remaining sections of the survey over the telephone, a telephone interviewer will administer the web-based survey or remaining parts of the survey over the telephone using their unique web link and the exact language included in the web survey. However, based on previous experience, we do not expect any States to request to complete by phone.

• The study team will use call scheduling procedures that are designed to call numbers at different times of the day (between 8am and 6pm) and days of the week (Monday through Friday) to improve the chances of finding a respondent at work.

• Toward the end of the field period, all SA non-respondents will be mailed the State Agency Last Chance Post Card (Appendix E.7).

B4. Test of Procedures or Methods to be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The study team pretested the administrative data collection process and materials with three State CN directors in March 2020. Respondents were selected based on their availability to participate within the pre-test timeframe and to ensure variation in States’ participation in CACFP, SFSP, and special provisions, as well as use of data vendors, in order to gather different perspectives on the administrative data collection materials. The team sent a copy of the data request template to each State CN director participating in the pretest, conducted an initial call to discuss the template and proposed data collection process, and asked SAs to submit a sample FNS-10 data file using the instructions provided in the data request. The study team then conducted telephone debriefing interviews with each State CN director participating in the pretest to solicit feedback about the administrative data collection. The interviews focused on asking respondents to identify and share concerns about language or instructions that were unclear or confusing; data elements that were particularly challenging to provide, unavailable, or both; burden and level of effort. The team also reviewed any issues uncovered in the sample data files provided by SAs. A summary of respondent feedback and the changes made to the FNS-10 Administrative Data Request is provided in Table B4.1. The study team incorporated respondent feedback and, in response to the COVID-19 pandemic, developed new materials to collect similar administrative data reported on the FNS-44 and FNS-418 to capture information on meals provided to children during the unanticipated school closures.

Table B4.1. Changes to State administrative data request materials based on respondent feedback and review of data files

Topic	Respondent feedback	Revisions to data request
Availability of school-level data	One respondent expressed uncertainty about their State’s ability to provide school-level data for Part B of the FNS-10.	Since the data request is designed for school-level data, we have created an alternate version of the request for States that are only able to provide SFA-level data. We will only provide SFA-level versions of the data requests if States are unable to provide school-level data.
School level	Respondents did not have information on the school level (elementary/middle/high) for each school; instead their data included grade range for each school.	We have revised the wording on the data request to read “grade range” and will categorize the schools into different school levels during data cleaning.
Use of term “administrative data”	One respondent commented that our use of the terms “administrative data” and “administrative records data” made them think of administrative reviews and made it less clear that the data request pertained to FNS-10 data.	We revised the data request to reference the FNS-10 specifically, where possible, instead of referring to administrative data more generally.
Number of free afterschool snacks served (item 5c-B)	One respondent commented that they were unsure as to whether free afterschool snacks counted in item 5d-B should be included in the count of free afterschool snacks listed in item 5c-B.	A footnote was added to this data element clarifying that the count of free afterschool snacks served should include all free snacks reported in item 5d-B.
Simplifying Part B of the data request	We have removed superfluous data elements from Part B of the school-level data request form.	Based on our review of the pre-test data we determined that if States provide school-level data we will not need to collect Items 10, 11, 16 and 17. For Items 12-14 we will only need to collect enrollment counts for each school, since we will have data on which schools were operating each program in the month of October in the Part A data. Items 10 – 17 will need to be collected for States that provide SFA-level data.

The study team also obtained feedback from these State CN Directors on the State CN Director Survey. Their input and changes made as a result are listed below in Table B4.2.

Table B4.2. Changes to State CN Director Survey based on respondent feedback

Question #	Feedback from State	Survey Change
1.14a	State said some schools scan the children’s meal cards to ensure meals were distributed to parents/guardians of eligible children.	We added a new item, “Scanned children’s meal cards or asked for children’s personal identification numbers used for meal service”
2.5	State said many local program directors are on-site assisting with meal services so are able to conduct monitoring that way.	We added a new item, “On-site while assisting or observing meal service.”
3.2	State said they have done monitoring at a distance, like from their cars.	We added a new response option, “On-site, socially distanced observations”
3.3	State said that they also used state monitoring waivers for safety of staff and because of state travel restrictions.	We added two new response options, “Improve safety of staff” and “State travel restrictions.”
4.1, 4.2a, 4.3, 4.5a	State said that including “in your opinion” made the question difficult to answer. They suggested removing the phrase.	We removed “In your opinion” and added “did local program operators indicate.” 4.1, for example now asks, “What methods did local program operators indicate were the most effective at providing the greatest number of meals to the greatest number of children during the federally declared COVID-19 public health emergency, from March through September 2020?”
4.8b	State said they had resources to hire, but are in a hiring freeze.	We revised item e from, “Resources need to hire new staff” to “Limitations on ability to hire new staff.”

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Table B5.1 lists staff consulted on statistical aspects of the design. The same staff will be responsible for collecting and analyzing the study data.

Table B5.1. Individuals consulted on statistical aspects of study design

Mathematica staff	Title	Phone	Email
Rachel Frisk	Project Director	202-552-6447	[email protected]
Charlotte Cabili	Researcher	202-238-3322	[email protected]
Barbara Carlson	Senior Statistician	617-674-8372	[email protected]
Liz Gearan	Sr Researcher	617-301-8978	[email protected]
Andrew Gothro	Researcher	202-250-3569	[email protected]
Eric Grau	Senior Statistician	609-945-3330	[email protected]
Josh Leftin	Researcher	202-250-3531	[email protected]
Tara Merry	Senior Survey Researcher	609-945-6616	[email protected]
Veronica Severn	Survey Analyst	617-715-6931	[email protected]
Liana Washburn	Research Analyst	202-250-3551	[email protected]
Eric Zeidman	Senior Survey Researcher	609-936-2784	[email protected]
USDA staff	Title	Phone	Email
Holly Figueroa, FNS	Social Science Research Analyst	703-305-2105	[email protected]
Maggie Applebaum, FNS	Branch Chief, Special Nutrition Analysis Branch	703-305-2578	[email protected]
Janis Johnston, FNS	Senior Technical Advisor, Office of Program Integrity	703-305-2106	[email protected]
Doug Kilburg, NASS reviewer	NASS Reviewer	202-690-8640	[email protected]

1 The team will initially request school-, site-, or outlet-level data from SAs. If a SA does not have these data available, the team will request SFA-, sponsor-, or institution-level data from that SA.