NASS Comments - Summer Food Service Program - Participant Characteristics - 04-08-2014

REVIEW OF EVALUATION OF THE SUMMER FOOD SERVICE PROGRAM (SFSP) PARTICIPANT CHARACTERISTICS

Reviewer:

John Richards, Mathematical Statistician

Summary, Estimation, and Disclosure Methodology Branch

National Agricultural Statistics Service

United States Department of Agriculture

T: 202-720-3388

After careful review, all aspects of this study are statistically valid with the following exceptions.

1. There are two types of respondents included in this study that have smaller than recommended sample size: the parents/caregivers of participants in the SFSP and the parents/caregivers of eligible nonparticipants in the SFSP. A sample size of n=25 for each of these respondents is too low to draw any kind of applicable inference about the population of these types of people. In Exhibit 5, Sampling Universe, Sample, and Response Rates, it is shown that an 80% response rate is expected. Given this assumption, 20 respondents will complete the questionnaire for each type of respondent. Because most of the data being collected is categorical, and if 20 people complete, the maximum proportion margin of error given the desired 95% confidence level is . This margin of error, of course, takes advantage of the normal approximation, but for variables such that a small proportion answers one of the possible values of that variable, this formula for margin of error cannot be used and nonparametric statistics will have to be implemented.

An 80% response rate among parents and caregivers is pretty lofty, especially given the type of population. Although the incentive of a $25 gift card for completing the survey will possibly boost response rates, a target of 80% is ambitious especially when the last response rate was only 45% in 2003.

Not only is the parent/caregiver sample size small, but those particular samples are drawn as a convenience sample from only two possible school districts in the nation. Although the sampling procedure done for the rest of the respondent types (states, sites, and sponsors) are following sound sampling guidelines, the sample for the parents/caregivers come from only the two school districts providing the lists so cohort effects will be present.

Extreme caution must be used in choosing the estimators for the variables that will be analyzed from the parents and caregivers.

2. Another small concern is the 100% expected response rate from state administrators, especially when outlying American territories are included in the survey. Most will probably respond since state-level governments tend to be cooperative with federal agencies, especially with federally funded programs such as the SFSP. Given the long data collection time period, 100% response is definitely possible, but other circumstances beyond the interviewers’ control may hinder them from completing the questionnaire. These may include illness, retirement, language barriers, and noncompliance. Also, the table in Exhibit 5 shows an expected response rate of 100%, 96%, and 95% for the state agencies, sponsor organizations, and sites, respectively. The response rates for the sponsor organizations and sites are lofty, as well. However, imputation calculations assume “at least an 80% response rate among sponsors and sites.” This is fairly reasonable, but no mention is made about the possibility of the necessary calculations if all state agencies do not complete the questionnaire. It would be slightly inappropriate to include the state agencies calculation procedures in the second paragraph of Item 2 in Part B , which is where the aforementioned quote appears, because all state agencies will be contacted whereas only sample of the sponsor organizations and sites will be contacted. However, some information should be included on the methodology for accounting for the possibility that a 100% response rate may not be achieved from the state agencies.

In conclusion, the methodology of surveying the state agencies, sponsor organizations, sites, and parents/caregivers follow a sound statistical procedure to accurately assess the SFSP. The only troublesome aspects of this survey are the small convenience sample of the parents/caregivers, and the expected 100% response rate among state agencies. Because of funding or other resources, if it’s impossible to increase the sample size of parents/caregivers, nonparametric statistics may need to be used to make any statistical inference based on these data. Otherwise, the information collected from these parents/caregivers can only be used for anecdotal purposes. Also, an imputation plan needs to be in place to account for possible nonresponse among the state agencies. The implementation of that plan may not need to be executed given the long data collection period, though. Thank you for providing service to the SFSP.

Response to comments from:

Optimal Solutions Group, LLC

M Square Research Park

5825 University Research Court

Suite 2800

College Park, MD 20740

Phone: 301-918-7301

E-mail: [email protected]

With respect to the small sample size of parents/caregivers of participants in the SFSP and the parents/caregivers of eligible nonparticipants in the SFSP, it is based on the RFP. The sample of 25 parents/caregivers of participants and 25 parents/caregivers of non-participants is a sample of convenience without national representativeness. Therefore, the small sample size is not an issue, because the results of the qualitative interviews will be descriptive, without any inferences to the population and without calculating any estimates or confidence intervals.

The RFP stated:

Because SFSP sites are typically open to all children without requiring registration or enrollment, randomly selecting a nationally representative sample of participants and their parents/caregivers is difficult. The Offeror will augment data collected for Objectives 1-6 with qualitative data from interviews with 25 parents/caregivers of participants in order to address Question #13 under Objective #5. The purpose of this qualitative research is to collect more specific descriptive information on the reasons parents/caregivers cite for having their children participate in SFSP programs, how often they attend SFSP sites, what could improve their attendance, and how they heard about the program.

With respect to the 100% expected response rate from state administrators, it was achieved by the 2003 study and is likely to be replicated by the current study. The data will be collected from state agencies from 54 states and territories¹, including Washington, D.C., Guam, Puerto Rico, and the U.S. Virgin Islands. Given the data collection protocol with official FNS introduction, numerous contacts, different data collection modes, and multiple points of contact, the 100% response rate is likely to be achieved.

If during the data collection, such factors as illness, retirement, language barriers, and noncompliance are encountered, the data collection protocol will inquire about a different point of contact and will use multiple contacts and different modes of data collection to collect the data. In the highly unlikely possibility that a state will fail to respond to the survey, FNS administrative data will be used to estimate SFSP participation rates and other characteristics of the state.

With respect to the high expected response rates for sponsors and sites surveys, these numbers are based on the 2003 study and the current study will strive to achieve at least an 80% response rates with sponsors and sites. The data collection protocol will use official FNS introductions, numerous contacts, different data collection modes, and multiple points of contact, to achieve response rate of higher than 80%.

1 http://www.fns.usda.gov/cnd/Contacts/StateDirectory.htm