School Nutrition and Meal Cost Study-II (SNMCS-II)
OMB Supporting Statement
Part B
June 28, 2019
Project Officer: Alice Ann Gola
USDA/FNS Office of Policy Support
3101 Park Center Drive, 10th Floor
Alexandria, VA 22302
Phone: (703) 305-4347
Email: [email protected]
PART B. STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods 8
B.2. Procedures for the Collection of Information 14
B.3. Methods to Maximize Response Rates and to Deal with Non-Response 18
B.4. Tests of Procedures or Methods to Be Undertaken 22
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 24
TABLES
B.1.1 Respondent universe for the mainland study 9
B.1.2 SNMCS-II target and SNMCS-I actual response rates 12
B.1.3 Respondent universe and sampling plan for the OACS 13
B.2.1 Respondent universe, sampling plan and expected precision levels for the OACS full data collection 16
APPENDICES
A. richard b. Russell NATIONAL sCHOOL LUNCH ACT, SECTION 28
B. Summary of SNMCS-II Data Collection Plan
C. SFA and School Recruitment Materials
C1. IRB Approval Letter
C2.1. Notification to Regional Offices
C2.2. Study Objectives (Groups 1, 2, & 3)
C3.
State Child Nutrition Director Study Introduction and Data Request
Email
(Groups 1, 2, & 3)
C4. SFA Director Sample Notification Email from State CN Director (Groups 2 & 3)
C5. FSMC Recruitment Letter (Groups 2, 3, & Full Outlying Areas)
C6. SNA Endorsement Letter (Groups 2, 3 & Full and Limited Outlying Areas)
C7. Study Overview (Group 2)
C8. Study Overview (Group 3)
C9. FSMC Recruitment Call Script (Groups 2, 3, & Full Outlying Areas)
C10. SFA Director Recruitment Advance Letter (Groups 2 & 3)
C11. Recruiting Call Script (Groups 2,3, & Full and Limited Outlying Areas)
C12. SFA Director Planning Interview (Groups 2,3, & Full and Limited Outlying Areas)
C13. SFA Post-Planning Email (Groups 2 & 3)
C14. Data Collection Activities and Respondents (Group 3 & Full and Limited Outlying Areas)
C15. Principal Introduction Letter to Schools (Groups 2 & 3)
C16. School Nutrition Manager Introduction Letter (Groups 2 & 3)
C17. Next Steps for Principals Email (Group 2)
C18. School Planning Interview (Groups 2 & 3 and Full Outlying Areas)
C19. Pre-Visit Reminder Email (Groups 2 & 3)
C20. Study Objectives and Overview (Full and Limited Outlying Areas)
C21. State Child Nutrition Director Study Introduction and Data Request Email (Full and Limited Outlying Areas)
C22. SFA Director Sample Notification Email from State CN Director (Full and Limited Outlying Areas)
C23. SFA Director Recruitment Advance Letter (Full and Limited Outlying Areas)
C24. SFA Post-Planning Email (Full and Limited Outlying Areas)
C25. Principal Introduction Letter to Schools (Full Outlying Areas)
C26. School Nutrition Manager Introduction Letter (Full Outlying Areas)
C27. Pre-Target Week Reminder Email (Full and Limited Outlying Areas)
D. Parent and Student Recruitment Materials
D1. School Roster Data Request (Group 2)
D2. School Endorsement Letter - English (Group 2)
D3. School Endorsement Letter - Spanish (Group 2)
D4.
Elementary and Middle/High School Parent (Household) Advance Letter
English
(Group 2)
D5.
Elementary and Middle/High School Parent (Household) Advance Letter
Spanish
(Group 2)
D6. Household Elementary and Middle/High School Brochure - English (Group 2)
D7. Household Elementary and Middle/High School Brochure - Spanish (Group 2)
D8. Parent Passive Consent Response Form - English (Group 2)
D9. Parent Passive Consent Response Form - Spanish (Group 2)
D10. Parent Active Consent Response Form - English (Group 2)
D11. Parent Active Consent Response Form - Spanish (Group 2)
D12. Student Assent Form - English (Group 2)
D13. Student Assent Form - Spanish (Group 2)
E. Menu Surveys
E1. Basic Menu Survey (Group 2)
E1.1 Basic Menu Survey Booklet: Instructions, Example Forms, and Forms (Group 2) (which includes Menu Survey Screener- Basic (Group 2); Daily Meal Counts Form (Group 2); Reimbursable Foods Form-Breakfast (Group 2); Reimbursable Foods Form-Lunch (Group 2); Recipe Form (Group 2); Self-Serve/Made to Order Bar Form (Group 2); NSLP Afterschool Snack Form (Group 2); Menu Survey Enhancements Administered through the Electronic Menu Survey (EMS) (Group 2); and A la Carte Foods Checklist (Group 2))
E2. Expanded Menu Survey (Group 3 & Full and Limited Outlying Areas)
E2.1 Expanded Menu Survey Booklet: Instructions, Example Forms, and Forms (Group 3) (which includes Menu Survey Screener – Expanded (Group 3); Daily Meal Counts Form (Group 3); Reimbursable Foods Form-Breakfast (Group 3); Reimbursable Foods Form-Lunch (Group 3); Recipe Form – Expanded (Group 3); Self-Serve/Made to Order Bar Form – Expanded (Group 3); NSLP Afterschool Snack Form-Expanded (Group 3); CACFP Afterschool Snack and Supper Form (Group 3); Non-Reimbursable Foods Form (Group 3); A la Carte Foods Checklist (Group 3); Non-Reimbursable Foods Inventory (Group 3); and Menu Survey Enhancements Administered through the Electronic Menu Survey (EMS) – Expanded (Group 3)
E2.2 Expanded Menu Survey Booklet: Instructions and Forms (Full and Limited Outlying Areas) (which includes Menu Survey Screener – Expanded (&Full and Limited Outlying Areas); Daily Meal Counts Form (Full Outlying Areas); Reimbursable Foods Form-Breakfast (Full and Limited Outlying Areas); Reimbursable Foods Form-Lunch (Full and Limited Outlying Areas); Recipe Form – Expanded (Full and Limited Outlying Areas); Self-Serve/Made to Order Bar Form – Expanded (Full Outlying Areas); Non-Reimbursable Foods Form ( Full Outlying Areas); and Non-Reimbursable Foods Inventory (Full Outlying Areas)
E3. Fruit and Vegetable Questions & Meal Pattern Crediting Report (Groups 2 & 3)
F. SFA Director and Principal Survey Instruments and Contact Materials
F1. SFA Director Survey Advance Letter (Group 1)
F2. SFA Director Survey Email Invitation (Group 1)
F3. SFA Director Survey (Groups 1, 2, & 3)
F4. SFA Director (Groups 1, 2, & 3) and Principal Surveys (Groups 2 & 3) Follow-Up Email
F5. SFA Director Survey Reminder Call Script (Groups 1, 2, & 3)
F6. SNM Survey (Groups 2 & 3)
F7. Principal Survey (Groups 2 & 3)
F8. SFA Director and Principal Surveys Email Invitation (Groups 2 & 3)
F9. Principal Survey Reminder Call Script (Groups 2 & 3)
G. Cost Study Instruments and Contact Materials
G1. State Agency Indirect Cost Survey (Group 3 & Full Outlying Areas)
G2. State Agency Indirect Cost Survey Invite (Group 3 & Full Outlying Areas)
G3. SFA On-Site Cost Interview (Group 3 & Full and Limited Outlying Areas)
G4. School Nutrition Manager Cost Interview (Group 3 & Full Outlying Areas)
G5. Principal Cost Interview (Group 3 & Full Outlying Areas)
G6. Food Cost Worksheet (Group 3 & Full and Limited Outlying Areas)
G7. On-site Self-serve/Made-to-Order Bar Form (Group 3)
G8. SFA Director Cost Interview Reference Guide (Group 3 & Full and Limited Outlying Areas)
G9. School Nutrition Manager Cost Interview Reference Guide (Group 3 & Full Outlying Areas)
G10. Principal Cost Interview Reference Guide (Group 3 & Full Outlying Areas)
G11. SFA Follow-Up Web Survey (Group 3 & Full and Limited Outlying Areas)
G12. SFA Follow-Up Web Survey and Interview Planning Email (Group 3 & Full and Limited Outlying Areas)
G13. SFA Follow-Up Cost Interview (Group 3 & Full and Limited Outlying Areas)
G14. SFA Follow-Up Cost Interview Reference Guide (Group 3 & Full and Limited Outlying Areas)
H. Competitive Foods Checklists and Cafeteria Observations
H1. Vending Machine and Other Sources of Foods and Beverages Checklists (Groups 2 & 3)
H2. Cafeteria Observation Guide (Groups 2 & 3)
I. Parent and Student Instruments and Contact Materials
I1. Student Interview English (Group 2)
I2. Student Interview Spanish (Group 2)
I3. Parent Interview English (Group 2)
I4. Parent Interview Spanish (Group 2)
I5. Automated Multiple Pass Method (Group 2)
I6. Point Of Sale Form (Group 2)
I7. Milk Form (Group 2)
I8. Food Diary English (Group 2)
I9. Food Diary Spanish (Group 2)
I10. Dietary Recall Texts and Emails English (Group 2)
I11. Dietary Recall Texts and Emails Spanish (Group 2)
I12. Elementary and Middle/High School Student Interview Reminder Flyer English (Group 2)
I13. Elementary and Middle/High School Student Interview Reminder Flyer Spanish (Group 2)
I14. Parent Interview Texts and Emails English (Group 2)
I15. Parent Interview Texts and Emails Spanish (Group 2)
J. Plate Waste Observation and Reimbursable Meal Sale Data Request
J1. Plate Waste Observation Booklet (Group 3)
J2. Reimbursable Meal Sale Data Request Form (Group 2)
K. mainland study data collection plans and example screenshots
K1. Mainland Study Data Collection Plans
K2. School Planning Interview Example Screenshots
K3. Electronic Menu Survey Example Screenshots
K4. SFA Director Survey Example Screenshot
K5. Principal Survey Example Screenshot
K6. State Agency Indirect Cost Survey Example Screenshots
K7. SFA Follow-Up Web Survey Example Screenshot
K8. Parent Interview Example Screenshots
L. Public Comments
L1. Public Comment 1
L2. Public Comment 2
L3. Public Comment 3
L4. Public Comment 4
L5. Public Comment 5
L6. Public Comment 6
L7. Public Comment 7
L8. Public Comment 8
M. RESPONSES TO PUBLIC COMMENTS
M1. Response to Public Comment 1
M2. Response to Public Comment 2
M3. Response to Public Comment 3
n. NASS Comments
O. Confidentiality Agreements
O1. Confidentiality Agreement – Mathematica Policy Research
O2. Confidentiality Agreement – Insight Policy Research
O3. Confidentiality Agreement – DIR
O4. Confidentiality Agreement – Agralytica
P. Sample sizes, estimated burden, and estimated cost of respondent burden for SNMCS-II
Q. MAINLAND STUDY SAMPLING PLANS
The second School Nutrition and Meal Cost Study (SNMCS-II) contains two sub-studies, the “mainland study” and the “outlying areas cost study,” or OACS. Two different methods—the “limited data collection” and the “full data collection”—will be used to collect data and estimate costs in the States and Territories in the OACS, and are described in Section A.2. The design elements that are shared between the mainland study and the two data collection approaches to be used in the OACS are discussed as a whole. Distinct design components for the mainland study or the OACS are discussed separately in each of the sections that follow.
B.1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
For the mainland study, the universe includes public school food authorities (SFAs),1 public schools, students, students’ parents/guardians, and meals served in the SFAs. Although charter schools and the SFAs to which they belong are included, private schools are not. Some data will be collected from all SFAs and schools in the sample, and some data will be collected only from a subsample of SFAs and schools. The estimated size of the respondent universe for the mainland study is presented in Table B.1.1.
Table B.1.1 Respondent universe for the mainland study
Sample group |
Estimated size of respondent universe |
Public SFAs |
17,136 |
Schools (K–12) |
96,220 |
Enrolled students (within schools participating in the National School Lunch Program) |
50,341,948 |
Source: Common Core of Data File 2015-2016 for the school and student counts and FNS-742 Verification Summary Report, school year 2012–2013 for the SFA counts. These counts will be updated at the time of sampling using the latest versions of both the FNS-742 file and the Common Core of Data File.
From the universe of SFAs in the contiguous 48 States and DC, we will select nationally-representative samples that provide unbiased and precise estimates at each level of analysis (SFAs, schools, students (and their parents), and meals) for the population and unbiased and moderately precise estimates for subgroups.2 Key subgroups include SFA and school size (enrollment), poverty level, urbanicity, Food and Nutrition Service (FNS) region,3 school type (elementary, middle, high), and school meal participants/nonparticipants.
The final samples are targeted to yield 500 unique SFAs, 1,000 schools, 2,000 students and their parents, and 3,900 lunch and 2,000 breakfast plate waste observations.4 We will collect some data from all SFAs and schools in the sample, and will collect other data from only a subsample of those units. This approach maximizes statistical precision and data quality while minimizing respondent burden. All sampled SFAs will participate in the SFA Director Survey, providing information on SFA policies and institutional and community characteristics. Principals and school nutrition managers (SNMs) in sampled schools will complete surveys providing information on school-level policies and characteristics, including characteristics of food service operations. In addition, SNMs in all sampled schools will complete a Menu Survey for the target week that will provide data for assessing the nutritional quality of school meals. One sample of SFAs and schools within those SFAs will have students sampled for the student/parent interviews to assess participation in and satisfaction with school meal programs and students’ dietary intakes (Group 2 SFAs). A separate sample of SFAs will provide data for the cost study (Group 3 SFAs).
The SNMCS-II sample design builds on and enhances the proven strategies used in SNMCS-I and is designed to ensure comparability of estimates across the two studies and provide required levels of statistical precision, while minimizing data collection costs and respondent burden. Likewise, the design utilizes random assignment of SFAs (excluding those that serve only charter schools, because the charter-only SFAs are sampled separately) to three groups to manage burden and cost of field operations. Specifically, the sampling approach will first randomly divide a sampling frame of all SFAs (excluding charter-only SFAs) into three separate SFA subframes.5 SFAs will then be sampled from each subframe using methods based on the study objectives particular to that sample, as outlined in the Summary of the Data Collection Plan (Appendix B) and Figure B.2.1. For consistency with SNMCS-I, the largest SFAs6 in the SFA frame will be included in both Groups 2 and 3, but all other SFAs will be sampled in only one of the three groups:
Group 1 includes 125 SFAs but no schools.
Group 2 includes 125 SFAs and 250 schools (2 per SFA).
Group 3 includes 250 SFAs and 750 schools (3 per SFA). In a subset of 130 schools in 65 SFAs, the sample also includes 3,900 National School Lunch Program (NSLP) lunches and 2,000 School Breakfast Program (SBP) breakfasts.
The majority of target response rates for SFAs, schools, and parents and students are consistent with actual response rates from SNMCS-I (OMB Control Number 0584-0596, Discontinued 07/31/2017). Target response rates range from 90 to 100 percent across the instruments associated with SFAs and schools, compared to 83 to 96 percent from SNMCS-I.7 The lowest response rates for school-level instruments from SNMCS-I were for the Vending Machines and Other Sources of Foods and Beverages Checklists, which were completed by school staff. For SNMCS-II, we expect 100 percent response on these observational instruments because they will instead be completed by field interviewers.
Target response rates for the instruments associated with parents and students are shown in Table B.1.2 as compared to the response rates from SNMCS-I. The lowest response rates for student-level instruments from SNMCS-I were for the Student Interview and 24-hour dietary recall, whose targets have been raised to 70 percent for SNMCS-II. This is due to a design change in the survey mode that is expected to increase response from elementary school students. For SNMCS-I, dietary recalls for elementary school students were completed in person with both the student and parent/guardian, whereas for SNMCS-II the parent-assisted portion of the dietary recall will be completed by phone, allowing for increased flexibility in scheduling appointments with the parents.
Table B.1.2. SNMCS-II target and SNMCS-I actual response rates
Instrument |
SNMCS-II target response rate (%) |
SNMCS-I weighted response ratea (%) |
Student Interview, including height and weight measurements |
70 |
63.6 |
Parent Interview |
90 |
88.5 |
24-hour dietary recall |
|
|
First recall |
70 |
63.6 |
Second recall |
70 |
68.7 |
Reimbursable meal sales administrative data |
90 |
89.5 |
Plate waste observations |
|
|
Lunch trays |
85 |
82.7 |
Breakfast trays |
85 |
88.9 |
Source for SNMCS-I: Zeidman et al. 2019.
aThe response rates are weighted using raw sampling weights—that is, weights that correct for unequal probability of selection before any nonresponse adjustments. The response rates for individual instruments reflect the percentage of eligible students or parents that completed each instrument. Students and parents were eligible if the student was present at school on the target day and the case was pursued (that is, not part of unattempted backup student sample).
Taking into account data collection from SFAs, schools, and students and their parents, the overall response rate is estimated to be 64%.8 Efforts to ensure a high response rate across data collection activities are discussed in Section B.3.
The States and Territories to be included in the OACS were identified through a feasibility assessment of remote data collection to estimate meal costs (OMB Control Number 0584-0606 FNS Generic Clearance for Pre-Testing, Pilot, and Field Test Studies Expired 4/30/2019). The OACS respondent universe includes public SFAs and schools in Alaska, Guam, Hawaii, Puerto Rico, and the United States Virgin Islands (USVI), and similar to the mainland study, the universe will be based on the combined FNS-742 and the Common Core of Data. The estimated size of the respondent universe for the OACS is presented in Table B.1.3, along with the target completed sample sizes.
Table B.1.3 Respondent universe and sampling plan for the OACS
|
Population |
Completed sample |
||
State/territory |
SFAs |
Schools |
SFAs |
Schools |
Alaska |
54 |
513 |
28 |
51 |
Guam |
1 |
41 |
1 |
24 |
Hawaii |
1 |
288 |
1 |
63 |
Puerto Rico |
7 |
1,435 |
7 |
55 |
USVI |
2 |
31 |
2 |
23 |
Source: Mathematica internal simulations based on the 2012–2013 FNS Form 742 file and 2015–2016 Common Core of Data file.
We will use a stratified sampling plan to select a representative sample of schools in each of the outlying areas. The sampling design is structured to obtain an equal level of statistical precision in each outlying area and, similarly, to produce an equal level of minimum detectible differences (MDD) for comparing the meal costs in each area to the cost of meals in the 48 contiguous States and DC.
In contrast to the sample design for the mainland study, we have eliminated the first stage of selection of SFAs in all of the outlying areas to improve the precision in the cost estimates for each of the outlying areas and respond to the fact that, except for Alaska and Puerto Rico, there are only one or two SFAs in these areas. In each of the areas, we will select a random sample of schools stratified by school type (elementary, middle, and high school). Schools in Alaska, Guam, and Puerto Rico will be stratified by additional characteristics. In Alaska, the majority of the school meals are served in a small number of large SFAs; the majority of SFAs in the State are small and rural. Schools will therefore be stratified by SFA size to ensure representation of small SFAs; the school selection will determine which SFAs are included in the study. Schools in Guam will be stratified by whether they are managed by the food service management company (FSMC) so that both FSMC-managed schools and non-FSMC-managed schools are included. For the full data collection in Puerto Rico, we will stratify the sample of schools by SFA to ensure schools are selected from all seven SFAs. Sample selection will yield a total of 51 schools for Alaska, 24 for Guam, 63 for Hawaii, 55 for Puerto Rico, and 23 for USVI.
B.2. Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
The sampling plan for the mainland study is designed to support national and subgroup-level estimates for SFAs, schools, students, and meals, using carefully designed stratification and probability-proportionate-to-size (PPS) sampling methods. The sampling plan for the mainland study mirrors the sampling plan used in SNMCS-I, which was reviewed and approved by OMB in 2014 (OMB Control Number 0584-0596, Discontinued 07/31/2017). For this reason, details about the mainland sampling plan are presented in Appendix Q. Details about the mainland data collection procedures are presented in Appendix K, and Section B.3.4 summarizes changes from SNMCS-I designed to maximize response rates. In the sections that follow, we describe the sampling plan for the OACS, which is new to SNMCS-II.
Sections A.2.1.2 and A.2.1.3 describe the data collection procedures for the limited and full data collection approaches for the OACS. The limited data collection for the OACS is restricted to only SFA-level data collection. The limited approach will be used for data collection from all seven of the SFAs in Puerto Rico. In addition, if the full data collection is not feasible in USVI (as discussed in Section A.2), the limited data collection will collect data from both SFAs in USVI. Because the limited data collection includes only SFA-level data, the sample for the limited data collection is a census.
The full data collection for the OACS will use a stratified sampling plan to select a representative sample of schools in each of the outlying areas. The sampling design is structured to obtain an equal level of statistical precision in each outlying area and, similarly, to produce an equal level of MDDs for comparing the meal costs in each area to the cost of meals in the 48 contiguous States and DC.
In each of the outlying areas, we plan to select a random sample of schools9 stratified by school type (elementary, middle, and high school). Schools in Alaska will also be stratified by SFA size, schools in Guam will be stratified by whether they are managed by the FSMC, and schools in Puerto Rico will be stratified by SFA to ensure schools are selected from all seven SFAs. The sampling will yield a total 51 schools for Alaska, 24 for Guam, 63 for Hawaii, 55 for Puerto Rico, and 23 for USVI as shown in Table B.2.1. For estimation of the 95 percent confidence intervals, we assumed the meal costs on average in each of the areas would be $3.25 and that the standard deviation of the costs would be $1.35 (about 40 percent higher than for the mainland based on data from SNMCS-I). In developing this design, we set the sample sizes to achieve a precision level equal to one-third of the standard deviation value associated with the U.S. meal costs (one-third of 98 cents) for each area10 and about two-thirds of a standard deviation for comparison with the mainland. Accordingly, Table B.2.1 shows a precision level of +/- 32 to 33 cents in each area, and for comparison of each area with the mainland +/- 66 to 69 cents at 80 percent power. For the analysis, we will prepare survey weights to account for the sample design and survey nonresponse as discussed in Section B.2.4. We will conduct all comparisons using a prepared set of replicate weights to support the use of jackknife variance estimation methods.
Table B.2.1. Respondent universe, sampling plan and expected precision levels for the OACS full data collection
|
Population |
Proposed sample |
Expected
confidence half interval |
|||||
State |
SFAsa |
Population of schools |
SFAs |
Schools |
Area estimate |
MDD to mainland (50 percent power) |
MDD to mainland (80 percent power) |
|
Alaska |
54 |
513 |
28 |
51 |
0.32 |
0.46 |
0.66 |
|
Guam |
1 |
41 |
1 |
24 |
0.33 |
0.48 |
0.68 |
|
Hawaii |
1 |
288 |
1 |
63 |
0.33 |
0.48 |
0.69 |
|
Puerto Rico |
7 |
1,435 |
7 |
55 |
0.32 |
0.46 |
0.66 |
|
United States Virgin Islands |
2 |
31 |
2 |
23 |
0.33 |
0.47 |
0.67 |
|
Source: Mathematica internal simulations based on the 2012–2013 FNS Form 742 file and 2015–2016 Common Core of Data File. These counts will be updated at the time of sampling using the latest versions of both the FNS-742 file and the Common Core of Data File.
aEach area is compared to the mainland separately; the outlying areas as a whole will not be compared to the mainland.
For the full data collection in the OACS, we will select a large enough sample of schools (and, in Alaska, SFAs) to achieve the target completed sample sizes, considering nonresponse and ineligibility.11 Sampling is not required for the limited data collection in the OACS, which will use a census of SFAs. We will order the samples of SFAs (Alaska only) and schools within SFAs randomly so that, in response to refusals, recruiters may recruit the next unit on the list until the desired number of cooperating SFAs/schools is obtained.
We will construct school-level weights to be used in estimating meal costs for each outlying area. The weights will account for the probabilities of selection and observed differential response rates across various subgroups. We will also post-stratify weights so that they total to benchmarks obtained from the most recent CCD and VCR-742 data by school -level characteristics.
To conduct the nonresponse adjustments, we plan to use a propensity modeling procedure to predict the probability of responding, given the available data collected on the sampling frame. We will weight the responding cases by the inverse of the predicted probability of response using a weighting class methodology that divides the propensity scores into classes and assigns the average score within the class to each case. This approach, outlined by Wun et al.12, helps eliminate the need to make large adjustments to the survey weights to increase the precision of the estimates.
As a final process in preparing weights, we will adjust weights using calibration or post-stratification methods13 to ensure that weighted totals or proportions mimic those for which we have comparable data from the sampling frame or other published sources. To complete this task, we plan to use the SUDAAN WTADJX procedure to conduct the nonresponse adjustments, as well as subsequent trimming of weights as needed, and to implement the post-stratification procedures in a single step. This process will ensure timely and consistent production of the final sampling weights that incorporates a scientific strategy based on the distribution of the nonresponse adjusted weights. In situations in which the response rate drops below 80 percent, we will conduct a nonresponse bias study during the creation of the nonresponse adjustments. In particular, for Puerto Rico, we will examine any available data we have at the SFA level to ensure the weighted values match SFA estimates obtained.
There are no sampling methods that we consider to be unusual in nature, and the sampling methods are described in detail in the prior sections and Appendix Q.
This is a one-time data collection for the mainland study and four of the five outlying areas participating in the full data collection for the OACS. The proposed data collection involves two rounds of data collection in Puerto Rico, as described in Section A.6. The data collection procedures will differ in the two rounds, and are not periodic data collection cycles to reduce burden.
B.3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
Target response rates are 70 percent for instruments associated with parents and students and range from 90 to 100 percent across the instruments associated with SFAs and schools (see Section B.1.1). A wide range of methods will be used to maximize participation and reduce nonresponse in all aspects of data collection. We will undertake several activities to lay the groundwork for our intensive recruitment campaign, including securing endorsements and training the recruitment team. A comprehensive set of recruitment materials, discussed in depth in Section A.2, describe the purpose of the study in a straightforward way that stresses the important role each participating SFA, school, and individual plays in the study’s success.
Gaining national, regional, and State/Territory support for SNMCS-II is critical to our success in securing participation. We have obtained a letter of endorsement from the School Nutrition Association, and USDA will also provide a letter of support for recruiting SFAs for the study. Such letters will provide critical study support and recruitment leverage when reaching out to SFA directors.
Recruiters will take part in a comprehensive training that will cover project details, anticipated challenges, and expectations. With a full understanding of the project and its goals within the current environment of school food service, recruiters will impart a level of aptitude and professionalism in all communications with study participants.
We will begin the first outreach steps of our recruitment strategy working with FNS to gain support at the regional and State/Territory levels. We will contact each FNS Regional Office (RO) regarding FNS’s contract with Mathematica and explain the importance of participation at all levels to the success of the study and ask for their support by sharing this information with the Child Nutrition (CN) directors of their States and Territories.
The study team will locate any sample overlap with other projects and use existing relationships to help make recruiting more efficient. Many SFAs, districts, and their staff have worked with our team on recent and ongoing studies. For example, because we anticipate that some SFAs using FSMCs will require additional encouragement to participate, we will dedicate Agralytica staff to recruiting those SFAs, as they have a long history of working with FSMCs.
Following outreach to ROs and State/Territory CN directors, recruiters will begin sending communications to all sampled SFA directors in Groups 2 and 3 and the OACS. The initial mailing will introduce the study and include a list of sampled schools (except for the limited data collection in the OACS, which includes only SFA-level data collection). Recruiters will call to confirm receipt of the mailing, assess eligibility, describe study objectives, address any SFA concerns, explain the study timeline and participation requirements, and discuss incentives. Incentives will help us overcome the competing demands and time constraints that study participants face. Except for the limited data collection in the OACS, recruiters will also confirm contact information for study schools, and inquire about the basic characteristics of the school’s food service operations.
Based on findings from the outlying areas feasibility assessment approved by OMB on March 19, 2018 under OMB Control Number 0584-0606 FNS Generic Clearance for Pre-Testing, Pilot, and Field Test Studies , we will take further steps to promote a high response in the OACS. This includes providing ample time for SFAs and schools to complete study instruments, and including workbooks of data to be collected ahead of the request, so respondents can see specifically what information will be asked of them. In the absence of in-person visits to SFAs and schools in the OACS, we will also provide extra support through phone technical assistance.
Student and parent recruitment will be coordinated through each Group 2 SFA’s sampled schools. We will maximize parental consent rates by advocating for passive consent wherever possible; if active consent is required, we will obtain verbal consent if districts allow it. We will also enlist the help of school liaisons to maximize return rates for active consent forms by offering liaisons a larger incentive payment. We will also train field interviewers to build rapport with students to maximize assent rates. Participation incentives will be communicated during student and parent recruitment to attract attention to the study and encourage participation among sample members who might otherwise ignore the request.
We will implement several strategies to minimize nonresponse during data collection. First, after fielding has begun, we will conduct quality assurance visits to SFAs in the mainland study to ensure that interviewers are following study procedures and engaging effectively with district and school staff and students. We will use these visits to adjust any interviewer behaviors or procedures that appear to be adversely affecting response rates. Second, we will accommodate the schedules of SFA and school staff when scheduling cost interviews, and we will strive to minimize disruptions to staff and students when we are working in schools. Third, we plan to offer incentives to respondents for the Menu Survey and Parent and Student Interviews as well as school liaisons’ assistance with coordinating student data collection (described in Section A.9). Finally, we will monitor response rates across instruments and activities closely and adapt our design to optimize response rates. We may target subgroups that have lower response rates with more frequent or intensive follow-up (for example, by calling select sample members instead of sending email reminders) to avoid potential bias resulting from differential response.
Several adjustments to the procedures that were used in SNMCS-I are designed to minimize item and unit nonresponse:
Collecting SFAs’ Provision 1, Provision 2, Provision 3, or Community Eligibility Provision status from State CN agencies at the start of recruiting to avoid SFA director nonresponse to these critical questions.
Having field interviewers, rather than school liaisons, complete the Vending Machine and Other Sources of Foods and Beverages Checklists to minimize unit nonresponse. Field interviewers will also collect meal price data in the Cafeteria Observation Guide.
Enhancing the user interface of the Electronic Menu Survey and increasing technical assistant outreach to promote more timely and complete data collection.
Requesting help from other staff to encourage or facilitate participation. For example, SFA directors can urge SNMs to complete the Menu Survey.
Conducting parent-assisted child dietary recalls on the telephone, which provides additional flexibility for interviewing that was not possible with in-person, in-school interviews.
Offering parents the option to complete the Parent Interview on the web or by telephone.
We anticipate that the overall response rate for the study will not exceed 80 percent. In addition, despite the efforts described in the preceding section, it is possible that response rates for individual instruments may fall below this threshold. In such cases, we will need to account for the potential that respondents to the various instruments may differ from nonrespondents in important ways. As described in Section B.2.4, we will calculate adjustments to the sampling weights to account for these differences as much as possible, using covariates available on the frame that are associated with (1) the propensity to respond and (2) the outcome variables of interest. To assess how well these nonresponse adjustments account for differences between respondents and nonrespondents, we will conduct nonresponse bias analyses for each set of weights that are used to meet the objectives of the study. The nonresponse bias analyses will summarize the response rates corresponding to each set of weights, assess the differences between respondents and nonrespondents overall and within subgroups of interest, and evaluate how much estimates using nonresponse-adjusted weights differ from the frame.
B.4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
There are three different sets of pre-testing activities associated with this study. The first one is described in this section. The other two pre-testing activities associated with the OACS were submitted under OMB Control Number 0584-0606 FNS Generic Clearance for Pre-Testing, Pilot, and Field Test Studies. The SNMCS-II instruments are based on the instruments used in SNMCS-I. Therefore, there was no need to conduct a comprehensive pretest of all the instruments. Instead, the pre-test was focused on (1) staff surveys that had been substantially revised, (2) users’ experiences completing the Electronic Menu Survey (Appendices E1 and E2), (3) cost interviews, and (4) selected on-site observation forms. Instruments were revised if pre-test participants expressed difficulty answering questions, or whether we observed difficulties through the administration process. Results of the pre-test allowed us to update burden estimates and refine on-site procedures for data collection. We recruited three SFAs in two States to participate in pre-testing SFA- and school-level instruments. Following notification from the ROs, we contacted State CN directors for assistance identifying potentially cooperative SFAs with specific characteristics needed for pre-testing (for example, meal service in a cafeteria so that plate waste observations could be conducted). We then contacted candidate SFAs to screen them using questions from the SFA Director and School Planning Interviews, gain cooperation, and identify one school per SFA for the on-site visit. Each pre-test included an on-site visit to the school and the SFA. In schools, we pretested the Vending Machine and Other Sources of Foods and Beverages Checklists (Appendix H1), Cafeteria Observation Guide (COG; Appendix H2), Plate Waste Observation Booklet (Appendix J1), On-site Self-Serve/Made-to-Order Bar Form (Appendix G7), SNM Cost Interview (Appendix G4), and Principal Cost Interview (Appendix G5). We pretested the Food Cost Worksheet (Appendix G6), SFA On-Site Cost Interview (Appendix G3), and the revenue statement portion of the SFA Follow-Up Cost Interview (Appendix G13) with SFA directors. We also asked SFA directors and SNMs to complete forms in the Electronic Menu Survey that included new questions or were updated to simplify users’ experience and improve quality of responses on the web. In addition, we asked SFA directors to complete the Fruit and Vegetable Questions and Meal Pattern Crediting Report. Finally, we sent SFA directors, SNMs, and principals copies of the respective staff surveys to complete. Three SFA directors, three SNMs, and three principals participated in the pre-test. We used feedback from the pre-test to clarify the wording of questions and response options, streamline instructions, improve question flow, and adjust estimated burden for the cost instruments. Below describes changes made to the instruments as a result of pre-testing:
School Planning Interview (Appendix C18)- Clarified questions about food bars and increased flexibility in how respondents provide information about their schedule.
Menu Survey (Appendices E1 and E2)- Clarified questions about a la carte and non-reimbursable foods, as well as questions about foods paired/offered together. Changed question order and clarified instructional text.
SFA Director Survey (Appendix F3)- Expanded response options an added examples to question text.
SNM Survey (Appendix F6)- Expanded response options.
Principal Survey (Appendix F7)- Clarified question text and expanded response options.
SFA On-Site Cost Interview (Appendix G3)- Restructured questions about salary and fringe, reorganized questions and simplified instructional text, clarified question text, and added follow-up questions pertaining to USDA Foods.
SNM Cost Interview (Appendix G4)- Added a screener question about production kitchens, updated list of food service staff tasks, and expanded response options.
Principal Cost Interview (Appendix G5)- Removed redundant questions, clarified question text, and expanded response options.
Competitive Foods Checklists (Appendix H1)- Added screener questions to be asked of the SNM to determine the competitive foods available at the school, expanded response options, and updated programming instructions for better data quality.
Cafeteria Observation Guide (Appendix H2)- Changed question order and updated programming instructions for better data quality.
Plate Waste Observation (Appendix J1)- Clarified instructions and refined procedures for ease of use by field observers.
Statistical methods were not changed as a result of pre-testing. Burden associated with pre-testing the instruments and procedures to be fielded in the OACS was reviewed and approved by the Office of Management and Budget on March 19, 2018 and March 6, 2019 (OMB Control Number 0584-0606, FNS Generic Clearance for Pre-Testing, Pilot, and Field Test Studies, Expires 3/31/2019). This burden is therefore not included in the burden estimates for the SNMCS-II collection; findings from the pre-test were used to refine the procedures and study materials described in this information collection request. Although the 2019 assessment has not yet been completed, we anticipate the findings will not affect estimated burden or instrumentation.
B.5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The information will be collected and analyzed by Mathematica Policy Research, Insight Policy Research, Decision Information Resources, Agralytica, and Relyon Solutions. The sampling and weighting procedures were developed by Dr. Michael Sinclair (telephone: 202-552-6439), building on previous work on SNMCS-I (OMB Control Number 0584-0596, Discontinued 7/31/2017), School Nutrition Dietary Assessment Study-III and -IV (SNDA-III and SNDA-IV; OMB Control Number 0584-0527, Discontinued 09/30/2012), and School Lunch and Breakfast Cost Study-II (SLBCS-II; OMB Control Number 0584-0533, Discontinued 04/30/2008) projects. The sampling plans were reviewed internally by Barbara Carlson (telephone: 617-674-8372), senior statistician at Mathematica. The study plan, including the sampling and weighting procedures, was reviewed by Trent D. Buskirk, Director, Center for Survey Research, Professor of Management Science and Information Systems (Data Science), University of Massachusetts–Boston (telephone: 781-964-4997). The statistical procedures included in this information request have also been reviewed by Hua Fan with the USDA National Agricultual Statistics Service (NASS) (telephone: 202-720-0830). Comments from NASS are included in Appendix N, and responses are incorporated in this supporting statement.
1 SFAs serving only institutionalized populations or SFAs operated by States or the Federal government will be excluded from the sampling universe. SFAs that serve charter schools only will be included for SFA-level analyses, but charter schools will not be included in the sampling universe for school-level, student-level or meal-level analyses. Private schools and SFAs serving private schools only will be excluded from all sampling frames.
2 National estimates of characteristics (percentages) will have a 95 percent confidence interval of no more than plus or minus 5 percentage points and estimates of means will have a 95 percent confidence interval of no more than plus or minus 5 percent of the mean. The confidence intervals for key subgroups will not exceed plus or minus 10 percentage points or 10 percent of mean values. The samples we propose will meet the subgroup precision requirements for subgroups comprising roughly 25 percent of the population of SFAs, schools, or students.
3 These are the seven regions that administer USDA’s food and nutrition programs.
4 Sample sizes described in this section are stated in terms of numbers of participating SFAs, schools, students, and parents (that is, the target completed sample sizes). The sizes of the samples selected will be expanded to allow for nonparticipation due to ineligibility or noncooperation.
5 One subframe (for Group 1) will include all the charter-only SFAs. The other two subframes (for Groups 2 and 3) will include only SFAs operating for public schools (and not those operating for charter schools only). More details about the approach for dividing the SFA sampling frame into three subframes is provided in the next section.
6 The largest SFAs and their schools will participate in the SFA Director, SNM, and Principal Surveys and will be asked to both provide data for the cost study and participate in the student/parent interview data collection. Based on preliminary analyses, we expect there to be up to 15 SFAs that will be designated as large in this step. Once we have a sampling frame constructed for this study, we can confirm the exact number of large SFAs.
7 Zeidman, E., Beyler, N., Gearan, E., Morrison, N., Niland, K., Washburn, L., Carlson, B., Judkins, D., LeClair, L., Mendelson, M., Wommack, T., Carnagey, J., Murphy, M., and Williamson, A. School Nutrition and Meal Cost Study: Study Design, Sampling, and Data Collection. Prepared for the U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, Project Officer: John Endahl. Alexandria, VA.
8 The overall response rate was estimated by multiplying the expected response rate for the SFA Director Survey (95.7%), Menu Survey (96.2%), and Student Interview (70%).
9 This process omits the first step of SFA selection that is used in the mainland study because, except for Alaska and Puerto Rico, each outlying area has only one or two SFAs. Moreover, because we are not collecting data on-site, the added level of clustering of the schools by SFA is not needed to reduce cost. Therefore, we can use stratified simple random sampling to select the schools to improve the statistical precision in the OACS.
10 Furthermore, this level of precision for each area translates to +/- 10 percent of the estimated meal cost at $3.25.
11 Because of the small number of schools in the USVI, we will select all 31 schools into the sample to obtain 23 completes.
12 Wun, L., T.M. Ezzati-Rice, R. Baskin, J. Greenblatt, M. Zodet, F. Potter, N. Diaz-Tena, and M. Touzani. “Using Propensity Scores to Adjust Weights to Compensate for Dwelling Unit Level Nonresponse in the Medical Expenditure Panel Survey.” Agency for Healthcare Research and Quality Working Paper No. 04004, October 2004. Available at http://www.ahrq.gov.
13 Deville, Jean-Claude, and Carl-Erik Särndal. “Calibration Estimators in Survey Sampling.” Journal of the American statistical Association, vol. 87, no. 418, 1992, pp. 376–382.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SNMCS II DRAFT OMB PART B |
| Subject | OMB |
| Author | MATHEMATICA |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |