Appendix O1. Sample Frame Development and Selection Procedures

APPENDIX O1. SAMPLE FRAME DEVELOPMENT AND SELECTION PROCEDURES

O1.1 Overview

The first stage of sampling for APEC III develops the sample frame for School Food Authorities (SFAs), and the second stage develops the sample frame for schools. The APEC III SFA Sample Selection Memo (Appendix P) and School Sample Selection Memo (Appendix Q) provide details on the selection of SFAs and schools for APEC III. This appendix summarizes the process for developing the sample frame. Throughout the process, Westat made every effort to minimize the burden on States and SFAs, primarily by using public records to obtain school information and simplifying the request to SFAs to a verification of school information (rather than providing new data). SFAs also were provided with a template notification letter to notify their schools. These procedures include six major steps as described in Exhibit O1-1.

Exhibit O1-1. Sample Frame Development and Selection

O1.2 SFA Sample Selection

FNS administrative data¹ were used for SFA sample frame construction. To further minimize burden on SFAs, public records from the National Center for Education Statistics (NCES)² were used as a reference to identify errors or outdated information and to augment FNS administrative data. Using NCES data, updates were applied if necessary. Finally, ineligible SFAs were identified and excluded from the frame. The APEC III SFA Sample Selection Memo (Appendix P) provides a summary of the sampling design, approach, and final selection. SFA sample selection was completed in May 2016.

O1.3 Study Notification to FNS Regional and State Child Nutrition (CN) Directors

FNS sent official study notification E-letters to FNS regional directors (Exhibit 01-2) and FNS regional liaisons (Exhibit O1-3) in May 2016 to notify them that APEC III is being conducted in SY 2017-2018 in 44 States. FNS regional and State child nutrition (CN) directors were notified of the study in June 2016.

Exhibit O1-2. Official Study Notification E-Letter from FNS to FNS Regional Directors

OMB Number: 0584-0530

Expiration Date: XX/XX/XXXX

Dear Regional SNP Directors,

This letter serves as an official study notification of the third Access, Participation, Eligibility and Certification Study Series (APEC III). Food and Nutrition Service (FNS) has contracted with Westat, Inc. to conduct APEC III. Data collection will take place in 44 States during school year (SY) 2017 – 2018, with planning activities currently underway. More information about APEC III is forthcoming.

As you know, APEC III complies with the Healthy, Hunger-Free Kids Act of 2010 and is authorized under the Improper Payments Elimination and Recovery Improvement Act (IPERIA) of 2012, both of which ensure the financial and programmatic integrity of programs such as the National School Lunch Program (NSLP) and the School Breakfast Program (SBP).

At this time, we request that you appoint a regional liaison for APEC III. We recommend that the appointed regional liaison is someone who has experience with the NSLP and SBP, and a working relationship with Child Nutrition (CN) Directors who oversee these programs in their State.

The regional liaisons will receive information about the States and School Food Authorities (SFAs) sampled for study participation. They will also receive a fact sheet with information about APEC III. The regional liaisons will be asked to contact their State CN directors to notify them about the study (a template will be provided). They will also be asked to confirm/identify the primary point of contact at the State level and the School Food Authority (SFA) level. This information will be provided to Westat so they can contact the State and SFA about APEC III.

Please reply to this email by May 27, 2016 with the name, email address, and phone number for the person you appoint as the regional liaison for APEC III.

Please let me know if you have any questions.

Viv Lees

CNP National Office Study Liaison

[email protected]

OMB Number: 0584-0530

Expiration Date: XX/XX/XXXX

Dear Regional Liaison,

The U.S. Department of Agriculture’s (USDA) Food and Nutrition Service (FNS) has contracted with Westat, Inc. to administer the third Access, Participation, Eligibility and Certification Study Series (APEC III).

APEC III complies with the Healthy, Hunger-Free Kids Act of 2010 and is authorized under the Improper Payments Elimination and Recovery Improvement Act (IPERIA) of 2012, both of which ensure the financial and programmatic integrity of programs such as the National School Lunch Program (NSLP) and the School Breakfast Program (SBP).

APEC III data collection will take place during school year (SY) 2017 – 2018. The data collected will be used to:

1. Generate a national estimate of the annual amount of erroneous payments in the NSLP and SBP;

2. Provide a robust examination of the student (household), school, and School Food Authorities (SFAs) characteristics that may contribute to or mitigate error;

3. Conduct a sub-study on the differences in error rates among SFAs using different implementation strategies in their school meals program; and

4. Conduct qualitative analyses to better understand the sources of errors and how to potentially address them.

In addition to replicating the APEC II methodology, a few enhancements will be implemented in APEC III, resulting in a larger effort, which include:

• Increased sample of SFAs, schools, and households size to increase precision;

• Conducting data collection through the entire school year to eliminate concerns regarding seasonality of meal application data;

• Conducting household surveys within six weeks of meal certification to improve recall;

• New procedures to obtain more accurate income data from households;

• Qualitative interviews with SFA directors (or other key staff), cafeteria managers, and parents to better understand the potential sources of error.

Your support in this process will directly contribute to the success of the APEC III study and will impact the future of the NSLP and the SBP. We want to thank you for your continued support, service and commitment to the nutritional well-being of our Nation’s children.

Devin Wallace-Williams, PhD

703-457-6791 (office)

[email protected]

Exhibit O1-3. Official Study Notification E-Letter from FNS to FNS Regional Liaisons

In turn, FNS regional liaisons were asked to send official study notification E-letters to their assigned State child nutrition directors. Appendix O2 provides a sample template of the E-letter for the regional liaison, which the regional liaisons customized for their State child nutrition director as appropriate. In this letter, the FNS regional liaison provided a brief summary of the study and reiterated the congressional mandate for the study. State child nutrition directors were informed of the SFAs in their State selected for the study and encouraged to notify the SFAs about the upcoming study. The FNS regional director provided a sample template E-letter for the State CN director (included in Appendix O2 as an attachment). Each State CN customized the E-letter as deemed appropriate. The FNS notification to State CN directors and the subsequent State CN director notification to SFAs took place in June and July 2016.

Finally, the research team contacted the State CN director. Appendix O3 includes the contact guide for the State CN director. The purpose of this follow-up contact was to confirm participation of the State and its SFAs and obtain any information from the State to help facilitate later participation by SFAs. Any follow-up contact with the SFAs took place between June and August 2016.

O1.4 Study Notification to School Food Authorities (SFAs)

In most cases, the State CN director notified SFAs of their selection for participation in the study. In addition, the study research team independently notified SFAs about the study and requested that they verify information about their schools obtained from public records. Appendix O4 includes the study notification letter and a reference guide sent from the study research team. The SFA study notifications began in July 2016.

01.5 SFA Verification of Public Records

Concurrently with study notification, the SFAs were asked to verify data about their schools obtained from public records. The data were obtained from the National Center for Education Statistics’ Common Core of Data and included school names, addresses, grade levels, number enrolled, number of students certified for free and reduced-price meals, and Provision 1, 2, or 3 status. While the sample frame was developed from the school data obtained from public records, this step provided SFAs with an opportunity to review and update (if needed) the school information the study would use for school sample selection. As needed, study staff utilized an internal resource guide for discussion of the verification of public record process with the SFA (Appendix O5). Finally, SFA directors were sent two follow-up emails: an automatically generated email noting the study’s receipt of their verification of the school data, and a second to inform them of next steps for the study (Appendices O6 and O7, respectively). The SFA verification of public records was completed in November 2016.

O1.6 School Sample Selection

As previously mentioned, the school sample frame was developed from public records from the National Center for Education Statistics (NCES) Common Core of Data (CCD) and verified or updated by SFAs. Appendix Q, APEC III School Sample Selection Memo, provides a summary of the sampling design, approach, and final selection. Upon approval of the school sample, SFAs will be notified of their schools selected for the study (Appendix O8). They will also be asked to notify their schools and provide the study with the appropriate school contact information. The SFA notification of school sample selection is expected to commence in February 2017.

O1.7 Study Notification to Schools

The final step in the sample frame development of SFAs and schools is notifying the sampled schools that they have been selected for participation in the study. The school study notification letter (Appendix O9) notifies the school principal and informs him or her that the research team will provide more details later about study participation. This notification will take place in spring 2017; however, data collection activities will not commence until fall 2017.

1 These included (a) FY15 FNS-742 SFA File (version dated 2-22-2016) and (b) CEP SY15-16 National Elections Data‒September 2015 (version dated 3-25-2016).

2 2013-14 Common Core of Data (CCD) local education agency (LEA) data file.

1