Supporting Statement Part B
Farm to School Census and Comprehensive Review
Ashley Chaifetz
Social Science Research Analyst
Office of Policy Support
Food and Nutrition Service (FNS)
United States Department of Agriculture (USDA)
3101 Park Center Drive, Alexandria, VA 22302
703-457-7741
October 24, 2018
1. Respondent Universe and Sampling Methods. 4
2. Procedures for the Collection of Information 7
3. Methods to Maximize the Response Rates and to Deal with Non-response.. 12
4. Test of Procedures or Methods to be Undertaken. 14
5. Individuals Consulted on Statistical Aspects & Individuals Collecting and/or Analyzing Data 15
Tables
A. Section 18 of the Richard B. Russell National School Lunch Act
B. Section 305 of the 2010 Healthy Hunger Free Kids Act
C. Distributor Interview Telephone Script
D. Census Survey
E. Census Survey - Screenshots
F. Distributor Interview Recruitment Email
G. Distributor Interview Frequently Asked Questions (FAQs)
H. Distributor Interview Scheduling Phone Script
I. Distributor Interview Confirmation Email
J. Distributor Interview Thank You Email
K. Census Recruitment Email from FNS to State Child Nutrition Directors
L. Census Recruitment Email from Study Team to State Child Nutrition Directors
M.1. Pre-Census Recruitment Email from State Child Nutrition Directors to SFAs (Email from Study Team to State Child Nutrition Directors)
M.2. Pre-Census Recruitment Email from State Child Nutrition Directors to SFAs (Template for State Child Nutrition Directors to send to SFAs)
N.1. Census Reminder Emails from State Child Nutrition Directors to SFAs (Email from Study Team to State Child Nutrition Directors)
N.2. Census Reminder Emails from State Child Nutrition Directors to SFAs (Template for State Child Nutrition Directors to send to SFAs)
O. Census Recruitment Email from FNS to State Department of Agriculture Directors
P. Census Invitation and Reminder Emails from Study Team to SFAs
Q. Census Frequently Asked Questions (FAQs)
R. Census Preparation Worksheet
S. Census Follow-up Telephone Script
T. Census Thank You Email
U1. 60-Day Federal Register Notice Comments Received Hanna
U2. 60-Day Federal Register Notice Comments Received Frinzell
U3. 60-Day Federal Register Notice Comments Received Herritt
U4. 60-Day Federal Register Notice Comments Received Jean Public
U5. 60-Day Federal Register Notice Comments Received Helget
U6. 60-Day Federal Register Notice Comments Received Ragalie-Carr
U7. 60-Day Federal Register Notice Comments Received Fogel
U8. 60-Day Federal Register Notice Comments Received Harvey
U9. 60-Day Federal Register Notice Comments Received Herald
V1. 60-Day Federal Register Notice Responses to Comments Hanna
V2. 60-Day Federal Register Notice Responses to Comments Frinzell
V3. 60-Day Federal Register Notice Responses to Comments Herritt
V4. 60-Day Federal Register Notice Responses to Comments Jean Public
V5. 60-Day Federal Register Notice Responses to Comments Helget
V6. 60-Day Federal Register Notice Responses to Comments Ragalie-Carr
V7. 60-Day Federal Register Notice Responses to Comments Fogel
V8. 60-Day Federal Register Notice Responses to Comments Harvey
V9. 60-Day Federal Register Notice Responses to Comments Herald
W. NASS Comments
X. Responses to NASS Comments
Y. Summary of Pretest Findings and Changes
Z. Federal-Wide Assurance (FWA) Certificate
AA. Confidentiality Agreement
AB. Estimates of Respondent Burden
AC. IRB Exemption Certificate
AD. Summary of Data Collection Activities
JUSTIFICATION
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.
This collection includes two primary data collection components: (1) interviews to be conducted by telephone with distributors of school food (the “Distributor Interview” – see Appendix C) and (2) a structured web survey (the “Census” – see Appendices D and E) with School Food Authority (SFA)a Directors.
The Distributor Interview: The Distributor Interview (Appendix C) will obtain the perspectives of large-scale broadline and produce food distributors in various regions of the country on the processes and challenges to local food purchasing and procurement. Twenty-five respondents for this interview will be purposively sampled based on their expected substantive contributions and likelihood of participating. The expected response rate for the distributor interview is 80 percent. The study team expects to receive a high response rate given that the sample will be selected based on anticipated likelihood of participation, and because this approach will rely on existing good relationships with FNS in an effort to secure distributors’ agreement to participate in this study.
The Census: The 2019 Farm to School Census (“Census”) of School Food Authorities (SFAs) (Appendices D and E) will collect data on local food purchasing for school meals, school gardens, other farm to school activities and policies, and evidence of the economic and nutritional impacts of farm to school activities.
As part of the Census, the universe of 55 State Child Nutrition Directors will be asked to provide a list of public and private SFAs that administer the NSLP in the State or territory for the purpose of constructing the most up-to-date list frame possible (Appendices K and L). The same State Child Nutrition Directors will send a pre-Census notification email (Appendix M) and two email reminders (Appendix N) to the SFAs in their State throughout the data collection period to all 20,000 potential respondents (public and private). The expected response rate for the request to provide contact information and forward emails is 100 percent (see Table B-1).
FNS will also send an informative email to the universe of 55 State Department of Agriculture Directors (Appendix O), alerting them to upcoming study activities, introduce them to the members of the study team, and asking for their help publicizing the Census. The expected response rate is 100 percent (see Table B-1).
The respondent universe for the Census includes all (publicb and private) SFA Directors in all 50 states, Guam, Puerto Rico, the Virgin Islands, American Samoa, and Washington, D.C. that participate in the National School Lunch Program (NSLP). The study team will survey all SFAs as part of the Census. The total estimated respondent universe is the full universe of 20,000 SFAs. FNS plans to publish Census data at multiple levels: national, regional, state, and SFA. These estimates will serve different audiences and stakeholders. The national and regional estimates will be most useful to federal policy makers and analysts. For these audiences, it would be adequate to survey a sample of SFAs, rather than a Census. Samples might also be adequate for use by state authorities responsible for policy on farm to school issues. However, at the SFA level, only a complete Census will meet the needs of SFAs and schools interested in knowing what their local system is doing to bring fresh local farm goods into local school menus. FNS currently has a web site that allows the public to look up responses for their SFA.c FNS plans to update this site with data from the 2019 Census. Giving this information to SFAs and schools in a manner that allows them to compare their SFA with other SFAs will help them to better understand barriers and successes related to implementing farm to school activities. As a result of this plan, no sampling of SFAs will be conducted.
The expected response rate for SFAs, defined as the proportion of the sample that completes a questionnaire, is 80 percent (see Table B-1), which will achieve approximately 16,000 completed surveys (20,000 x .80 = 16,000). This expectation is based on requirements from the Office of Budget and Management and the high response rate of the previous Census, which was 70 percent.d The study team expects to receive a higher response rate on the current Census effort due to new procedures such as offering the option for SFAs to complete the web survey over the phone, which was not done in the previous Census, and more intensive follow-up with a larger number of non-respondents (described in greater detail in Part B, Question 3). Table B-1 shows the respondent universe, sample sizes, expected response rates, and target number of completed cases for each respondent type.
Respondent category/ Instrument |
Universe |
Sample |
Expected response rate |
Target completed cases |
2015 Census Response Rates |
Distributor Interview |
|||||
Food Distributors |
* |
25 |
80% |
20 |
N/A |
Census |
|||||
School districts (SFAs)1
|
20,000 |
20,000 15,000 |
80% 80% |
16,000 |
70% |
States2
|
55
55
55 |
55
55
55 |
100%
100%
100% |
55
55
55 |
100%
100%
N/A |
TOTAL |
20,0553 |
20,080 |
80% |
16,075 |
70% |
Note: The 2015 Census OMB Control No. is 0584-0593, Farm to School Census Survey. The expiration date was 2/29/2016.
*There is no known universe of distributors that sell food to SFAs and schools.
1 Estimates are based on data from the FNS-742 School Food Authority (SFA) Verification Collection Report form (OMB Control Number 0584-0594 Food Program Reporting System (FPRS), expiration date September 30, 2019).
2 States do not respond to the Census, but provide information on the sample frame and encourage SFAs to participate.
3 This estimate only includes States and SFAs (public and private). It does not include food distributors (for which there is no known universe).
2. Describe the procedures for the collection of information including:
• statistical methodology for stratification and sample selection,
• estimation procedure,
• degree of accuracy needed for the purpose described in the justification,
• unusual problems requiring specialized sampling procedures
any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Overview of Data Collection Procedures: The Distributor Interview will consist of 25 representatives of school food distributors who will be purposively sampled and approached based on a list developed by FNS. Beginning one week after OMB approval, distributors will receive an initial request to participate, via email, from the study team (Appendix F). A document answering frequently asked questions (FAQs) will be provided at this time (Appendix G). Next, the study team will contact participants by telephone (Appendix H) to schedule a time to complete the interview when it is convenient for the respondent. A maximum of eight phone attempts will be made per distributor to schedule the interview. Once the interview is scheduled, the study team will send distributors a confirmation email that summarizes logistical information for the interview (Appendix I). Following the interview (Appendix C), all participating distributors will receive a thank you email (Appendix J). Recruitment and non-response follow-up for the Distributor Interview will occur up to 24 weeks following OMB approval.
As part of the Census, the universe of 55 State Child Nutrition Directors will be asked to provide a list of public and private SFAs that administer the NSLP in the State or territory for the purpose of constructing the most up-to-date list frame possible. The sample frame will be constructed one week following OMB approval. State Child Nutrition Directors will send three emails to SFAs (including one pre-Census notification [Appendix M] approximately one week after OMB approval, and two reminder emails [Appendix N] sometime during data collection – approximately 6 to 24 weeks following OMB approval).
FNS will also send an introductory email to the universe of 55 State Department of Agriculture Directors (Appendix O). This email will alert them to upcoming study activities, introduce them to the members of the study team, and ask for their help publicizing the Census by announcing the Census in any upcoming newsletters or at an upcoming events.
The Census will solicit responses from the universe of 20,000 SFAs. First, SFAs will receive a pre-Census recruitment email from State Child Nutrition Directors (Appendix M), approximately 1 week after OMB approval, introducing SFAs to the study team. About 6 weeks following OMB approval, SFAs will receive an email invitation from the study team that contains a unique link to participate in the online Census survey (Appendix P, Section 1.1). A document answering frequently asked questions (FAQs) will be provided to the SFAs (Appendix Q), along with an optional worksheet to help SFAs prepare for the Census (Appendix R). Over the course of the study (up to 24 weeks following OMB approval), SFAs will receive up to ten reminder emails to complete the survey (eight from the study team [Appendix P, Sections 1.1 – 1.9] and two from State Child Nutrition Directors [Appendix N]). Up to two reminder call attempts will be made by the study team to a subsample of non-responding SFAs (Appendix S), during which time the respondent will be encouraged to complete the survey over the phone. All respondents who complete the survey will be thanked by email for their participation in the Census (Appendix T).
Statistical Methodology for Stratification and Sample Selection: The Distributor Interview is exploratory and thus not intended to produce generalizable quantitative estimates. Its purpose is to obtain the perspectives of large-scale food distributors on the processes and challenges to local food purchasing and procurement. The Distributor Interview will include a sample of 25 potential respondents, representing at least four FNS regions, including two broadline and two produce distributors from each region. Respondents will be purposively sampled based on their expected contributions and likelihood of participating. Potential respondents will be identified through FNS’s Farm to School regional coordinators.
Next, as part of the Census, the universe of 55 State Child Nutrition Directors will be asked to provide a list of public and private SFAs that administer the NSLP in the State or territory for the purpose of constructing the most up-to-date list frame possible. To ensure state lists are complete, the study team will compare the list of SFAs received from state agencies to information from two lists: (1) A list of public and private SFAs that submitted the FNS-742 School Food Authority (SFA) Verification Collection Report form (OMB Control Number 0584-0594 Food Programs Reporting System (FPRS), expiration date September 30, 2019) for school year 2017-2018, and (2) a list of public school districts from the U.S. Department of Education’s National Center for Education Statistics Common Core of Data (CCD) for school year 2017-18 (the most recently available data). The Census is a one-time collection; it will be completed once in school year 2018-2019. Concern regarding the periodicity of data collection cycles is not applicable.
There is no sampling in the Census except for phone-based nonresponse follow-up. The Census will be a survey of the complete universe – including all (public and private) SFA Directors in all 50 states, Guam, Puerto Rico, the Virgin Islands, American Samoa, and Washington, D.C. that participate in the National School Lunch Program (NSLP). After all eight Census reminder e-mails (Appendix P) are sent by the study team, if the response rate remains lower than 80%, a subsample of up to 2,681 non-respondents will be followed up by telephone (Appendix S). Their responses will be collected via their unique web survey links, and will be weighted and pooled with web survey responses in main analyses and also used to assess any potential non-response bias through comparison to SFAs that completed the web survey earlier. In technical descriptions of the survey, we will report the weighted response rate. For example, if the response rate at the end of the email phase is 67 percent and 36 percent of sampled nonrespondents are successfully interviewed by phone, the weighted response rate will be 78.9 percent.e This subsample will be stratified by SFA size, as follows:
Large SFAs: SFAs with 25,000 or more students.
Mid-Size SFAs: SFAs with an average enrollment of 2,500 to 24,999.
Small SFAs: SFAs with an average enrollment under 2,500.
Within each stratum nonresponding SFAs will be selected with simple random sampling. However, because of the greater impact of Farm to School policies at large school districts, the study team would not use subsampling on the largest SFAs. That is, all web non-respondents in the large SFAs stratum (approximately 300 SFAs) will be selected for the non-response follow-up. For mid-size SFAs with an average enrollment of 2,500 to 24,999, the study team will attempt telephone follow-up for 50 percent of web non-respondents among the approximately 3,700 SFAs of this size. Finally, for those SFAs with an average enrollment under 2,500, the study team will attempt telephone follow-up of 25 percent of web non-respondents. The study team will prepare suitable nonresponse adjustment weights to be used for all Census estimates, and will prepare suitable estimates of standard errors on critical outcomes.
Estimation Procedure: SFAs in the sample selected for phone-based nonresponse follow-up will be given Horvitz-Thompson weights of 4 for the small SFA stratum, 2 for the medium SFA stratum, and 1 for the large SFA stratum.f On the weighted sample, we will model response propensity with a logistic regression in terms of variables on the frame we will have constructed. We will then invert these estimated response propensities to obtain nonresponse-adjustment factors, which we will then apply to the Horvitz-Thompson weights for respondents, while zeroing out the weights for nonrespondents.
Degree of Accuracy Required: In as much as the data collection is a census with plans to publish every individual response, this question is not relevant to this submission.
Unusual Problems Requiring Specialized Sampling Procedures: No specialized sampling procedures are planned.
Use of Periodic Data Collection Cycles to Reduce Burden: The Farm-to-School Census is a periodic data collection. This will be only the third time it is conducted. The prior such Census iterations were in 2013 and 2015.
3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
In 2015, USDA surveyed 18,104 school districts as part of the 2015 USDA Farm to School Census; 12,585 completed the survey for an overall response rate of 70%.g To ensure an 80 percent response rate on the 2019 Census, the study team will follow a multistep process including:
recruitment of SFAs by trained, experienced research staff
utilizing a user-friendly web interface for the Census [Confirmit Horizons]
email and telephone support provided by the study team
email reminders (up to 10 total per SFA) provided by the study team and State Child Nutrition Directors
follow-up phone calls (up to two per SFA)
offering nonresponding SFAs the opportunity to complete the survey over the phone through a member of the study team
In addition, all SFAs will receive a document answering frequently asked questions (FAQs) (Appendix Q).
If the response rate is lower than expected after all reminder emails have been sent, the study team will sample up to 2,681 non-respondents to receive up to two follow-up phone calls (2,011 public school SFAs and 670 private school SFAs).h This is a cost-effective approach to nonresponse follow-up that is also considered a well-accepted practice in many government data collection efforts, including the largest run by the federal government, the American Community Survey. By focusing intense efforts on a subsample, it is possible to simultaneously reduce burden, as well as reduce costs and the risks of nonresponse biases.
The expectation is that from the up to 2,681 non-respondents (2,011 public SFAs + 670 private SFAs) that will be followed up, about 965 SFAs (724 public + 241 private) will respond after two reminder phone call in order to reach the expected response rate of 80%. Their responses will then be compared to the SFAs who responded to the web survey without phone follow-up to assess any potential non-response bias. With this sample size, assuming a 5% significance level and 80% power, the most conservative minimum detectable difference of a proportion test is about 4.5%. This estimate can change depending on the variable analyzed and if the number of non-respondents being successfully followed up changes.
The study team will prepare suitable nonresponse adjustment weights to be used for all Census estimates, and will prepare suitable estimates of standard errors on critical outcomes.
The Distributor Interview is exploratory and thus not intended to produce generalizable quantitative estimates. The results from this interview will be used to determine whether a larger survey is feasible or desirable. To maximize response rates we will rely on existing relationships that the Advisory Panel members and FNS staff have with distributors to assist with survey completion. These relationships proved to be useful during the pre-test period, and are expected to be useful again in the full study.
4. Describe any tests of procedures or methods to be undertaken.
The Census was cognitively pretested in May 2018 with six SFAs and three distributors prior to OMB clearance (see the Summary of Pretest Findings and Changes, Appendix Y). Feedback from pre-test respondents (both for the Census and Distributor Interview) was summarized and evaluated by the study team (see Appendix Y). A final determination about re-wording or eliminating the questions was reached after extensive discussions about instrument length, scope of the project, and maintaining consistency with previous rounds of the Census.
The Census survey pretest included six public SFA directors from the list of 13 potential respondents provided by FNS. The study team called and recruited potential respondents to be part of the pretest. As the online version was not yet programmed, the study team mailed a paper version of the Census survey to respondents along with a business reply envelope and cover letter. The study team asked recruited respondents to complete the paper questionnaire independently, then engaged in a phone follow-up with participants to answer questions of burden, clarity, and administration. Data were analyzed to assess whether the correct questions were skipped, the correct survey paths were taken, and that no questions were incorrectly left blank. The study team then incorporated respondent feedback into final drafts of the Census, particularly around clarity and ways to streamline administration. As a result of the pretest, 1 of the 53 questions was re-worded for clarity, the response lists for 6 questions were edited to add clarity to options or to add options, and zero questions were eliminated. The average response time for the Census pretest was one hour. Specific changes are further discussed in the Census Interview Pretest Memorandum in Appendix Y, including a question by question table of respondent suggestions, and subsequent revisions, from the pretest process.
The Distributor Interview pretest included three distributors from the list of 14 potential respondents provided by FNS. As a result of the pretest, 17 of the 36 questions were re-worded for clarity, four questions were re-ordered to allow for better flow with the interview topics and one question was eliminated. For recruitment, the study team found during the pretest interviews it would be best to: (1) Introduce the study with a formal email from FNS and the study team project director; (2) Follow the email with a direct call from the study team interviewer; (3) Send a confirmation email with an Outlook meeting invitation and an abbreviated set of topic questions; and (4) Send a short thank you email after the interview was completed. Specific changes are further discussed in the Distributor Interview Pretest Memorandum in Appendix Y, including a question by question table of respondent outcomes, and subsequent revisions, from the pretest process.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The study team is led by Ashley Chaifetz (FNS Project Officer) and Maria Boyle (Abt Project Director). Abt Associates will collect and analyze data for this project. The study team has consulted with David Judkins, a principal scientist at Abt on statistical issues. A team of eight advisory members have also provided input on the study design. (See Table B-2 below.)
Name |
Title |
Affiliation |
Telephone |
Ashley Chaifetz |
Social Science Research Analyst |
Food and Nutrition Service (FNS), U.S. Department of Agriculture (USDA) |
703-457-7741 |
Maria Boyle |
Senior Associate |
Abt Associates |
617-520-2331 |
David Judkins |
Principal Scientist |
Abt Associates |
301-347-5952 |
Sean Cash |
Economist |
Friedman School of Nutrition Science and Policy at Tufts University |
617-636-3737 |
Jeff O'Hara |
Agricultural Marketing Specialist |
Agricultural Marketing Service, U.S. Department of Agriculture (USDA) |
202-756-2575 |
Katherine Ralston |
Senior Agricultural Economist |
Economic Research Service, U.S. Department of Agriculture (USDA) |
202-694-5463 |
Anupama Joshi |
Executive Director |
National Farm to School Network |
847-917-7292 |
Colleen Matts |
Farm to Institution Specialist |
Michigan State University |
517-432-0310 |
Sandy Curwood |
Director |
Virginia Department of Education, Office of School Nutrition Programs |
804-225-2074 |
Nessa Richman |
Network Director |
Farm-to-Institution New England (FINE) |
802-369-3090 |
Eva Ringstrom |
Senior Director of Impact |
FoodCorps |
212-596-7045 |
Becca Jablonski |
Assistant Professor |
Colorado State University |
970-491-6133 |
Lydia Oberholtzer |
Senior Research Assistant |
Penn State University |
301-891-0470 |
Ricardo Salvador |
Senior Scientist |
Union of Concerned Scientists |
202-331-6956 |
Mingshan Zheng |
Mathematical Statistician |
National Agricultural Statistics Service (NASS), U.S. Department of Agriculture (USDA) |
202-720-0830 |
a SFAs are defined as the governing body responsible for the administration of one or more schools and which has the legal authority to operate a nonprofit school food service approved by FNS to operate the National School Lunch Program (NSLP). For the purposes of this study, SFAs may refer to the governing body that has this authority for a school district, several school districts, or an individual school – either public or private.
b Public includes charter schools that operate NSLP.
c For more information please see: https://farmtoschoolcensus.fns.usda.gov/data-explorer
d For more information please see: https://farmtoschoolcensus.fns.usda.gov/about
e 36 percent of 33 percent provides another 11.9 percentage points to the weighted response rate.
f Given a sampling rate of 0.25 for small SFAs, the Horvitz-Thompson weight for them is 1/0.25=4. Similarly, 1/.5=2 for medium SFAs.
g For more information please see: https://farmtoschoolcensus.fns.usda.gov/about
h The study team estimates that after each study team reminder email approximately 20% of non-respondents will complete the Census. After sending eight email reminders it is expected that 13,319 SFAs will respond and 2,681 will not have responded, out of the total 16,000 total estimated responses.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | HancDa |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |