Supporting Justification for OMB Clearance for the Community Eligibility Provision (CEP) Characteristics Study
Part B
March 28, 2016
Office of Policy Support
Food and Nutrition Service
United States Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302
Project Officer: John Endahl
Telephone: 703-305-2127
Email:[email protected]
Part B. Collections of Information Employing Statistical Methods B-3
B.1 Respondent Universe and Sampling Methods B-3
B.2 Procedures for the Collection of Information B-8
B.3 Methods to Maximize the Response Rates and to Deal with Nonresponse B-11
B.4 Test of Procedures or Methods to be Undertaken B-13
B.5 Individuals
Consulted on Statistical Aspects and Individuals Collecting
and/or
Analyzing Data B-14
Tables
Table B1. Distribution of participating LEAs into four strata
Table B2. Distribution of non-participating LEAs into two strata
Appendices
A. Research Questions
B1 Email Notification to Regional Offices
B2. Invitation Letter to State Agency Child Nutrition Directors/Frequently Asked Questions (FAQs)/Web Survey Information Sheet
B3. Follow-Up E-Mail to State CN Directors
B4.1. CN Director Telephone Interviewer Script
B4.2. LEA Director Telephone Interviewer Script
B5 Thank You Letter to State CN Director for Completing the Survey
B6 Email Notification to State Child Nutrition Directors List of Sampled LEAs and Frequently Asked Questions (FAQs)
B7. Invitation Letter to LEA Directors, Frequently Asked Questions (FAQs) and Web Survey Information Sheet
B8. Follow-Up E-Mail to LEA Directors
B9. Thank You Letter to LEA Director for Completing the Survey
C1. State Child Nutrition Director Survey
C2 State Administrative Data Request
D1. LEA Participating Web Survey
D2. LEA Non-Participating Web Survey
E.1. Section 28 of the Richard B. Russell National School Lunch Act Amended Through February 2014
E.2 Healthy Hunger-Free Kids Act of 2010
F Excel Table of Estimated Burdens (excel spreadsheet)
G. Comments from NASS Reviewer
B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
Respondent universes. There are three respondent universes. The first is all State Agencies (SAs) with at least one school eligible for CEP during school year (SY) 2016-17. This universe is estimated to include all 50 States, the District of Columbia and Guam (N = 52). The second universe is all Local Education Agencies (LEA) with at least 1 school participating in CEP during SY 2016-17 (participating LEAs), while the third is defined by all LEAs with at least 1 school eligible for participation in CEP but without any schools participating during SY 2016-17 (non-participating LEAs). The estimates of the sizes of the two LEA respondent universes are based on administrative data from the U.S. Department of Agriculture’s Food and Nutrition Service (FNS) for SY 2015-161 and data from the Center on Budget and Policy Priorities (CBPP) for SY 2014-15 and SY 2015-16.2 The actual universes in SY 2016-17 will be updated before the start of data collection. In general, all LEAs that are eligible and/or participated in the CEP are included in the respondent universe except LEAs that only serve residential students.
There are 6,798 eligible LEAs in the SY 2015-16 CBPP database and 2,937 participating LEAs in the FNS database. Therefore, the universes include 2,937 participating and (6,798-2,937=) 3,861 non-participating LEAs in SY 2015-16. Based on trends in CEP participation, we estimate that for SY 2016-17, the universe for participating LEAs will be 3,818 and the universe for non-participating LEAs will be 2,980.
Sampling. All (a census) State Agency (SA) Child Nutrition (CN) directors will be asked to complete the SA CN Director Survey. A purposeful sample of 12 SAs will be selected to transfer administrative data on all of the LEAs in their State for SY 2013-14 through SY 2016-17. The States will be selected to maximize the number of pairs of similar participating and non-participating LEAs so as to conduct an impact analysis of participation in CEP. The impacts will be assessed in terms of food service revenues and participation in school meals.
Nationally representative samples will be selected for the participating and non-participating LEAs. The precision requirements are defined by +/- .05 percent from an estimated proportion value, which is assumed to have a population value of 0.5, with 95 percent level of confidence. The research questions require representation of the LEAs with participating schools by length of time participating in CEP. As illustrated in Table B1, the universe of participating LEAs will be allocated to four mutually exclusive strata. Then within the strata, the LEAs will be sorted into groups defined by their identified student percentage (ISP). Systematic sampling will then be used to select the sample of participating LEAs. It is expected that the sampling rates will be similar within stratum so that a responding sample of size 386 will satisfy the precision requirements of the study. Therefore, assuming an 80 percent response rate, the participating LEA sample size is 483 (Table B1).
Stratum
|
Stratum
|
Universe
|
Expected number of LEAs to be sampled |
0 Years (New in SY 2016-17) |
1 |
881 |
111 |
1 Year (New in SY 2015-16) |
2 |
719 |
91 |
2 Years (New in SY 2014-15) |
3 |
1,580 |
200 |
3 or more Years (New in SY 2013-14) |
4 |
638 |
81 |
All LEAs |
Total |
3,818 |
483 |
Using the trends in the FNS and CBPP databases, we estimate that in SY 2016-17 there will be 749 non-participating LEAs in the CEP pilot States and 2,231 non-participating LEAs in the non-pilot States by SY 2016-17. As shown in Table B2, these will form two strata for sampling. Once again, a sample of approximately 483, assuming an 80 percent response rate, will satisfy the precision requirements for the study. In this case, with two strata (pilot versus non-pilot states), the allocation would be 121 and 362, respectively (Table B2). Sampling within the stratum will be with simple random sampling.
Stratum
|
Stratum
|
Universe
|
Expected number of LEAs to be sampled |
LEA in Pilot State |
1 |
749 |
121 |
LEA not in Pilot State |
2 |
2,231 |
362 |
All LEAs |
Total |
2,980 |
483 |
Expected Response Rates
The SA CN Director Survey will be conducted as a census of all 52 SA CN directors (including all States and the District of Columbia) and will not involve sampling. We expect a 100 percent response rate for the SA CN Director Survey. Additionally, we expect that all 12 selected States for the administrative data request will respond. Given that the SAs are significantly different from each other owing to their salient differences, any sampling procedures for the SAs would have to introduce complex samples, which would increase the sample size. The burden savings from sampling are not justified by the loss of information on the heterogeneity of the States. Based on our experience with other LEA surveys, we expect the response rate to be 80 percent from the participating and non-participating LEAs. Thus, the overall response rate will be approximately 80.5 percent.
The CEP Characteristics Study will include both a cross-sectional/longitudinal analysis as well as an impact analysis. The impact analysis will use a difference-in-differences (DD) comparing pre-program outcomes for both treatment and a matched comparison group. The matched comparison group will be comprised of an equal number of eligible and nonparticipating LEAs. Matching will be restricted of each treatment LEA to a single comparison LEA within the same State to improve the strength of the match. Private school LEAs and those operated by State or tribal agencies will be excluded in the DD analysis. LEAs will be matched using propensity score matching (PSM) to match participating (treatment) and non-participating (potential comparison) LEAs exclusively within state matches. A purposive selection of 12 States will be selected to be involved in the impact analyses based on LEA sample size required to achieve statistical power, be geographically diverse, and include several of the six “early implementing” as well as “late implementing” States.
B.2 Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
The methodology for stratification and sample selection was presented in Question 1.
Estimation procedure. The survey data from the participating and non-participating surveys will be weighted to reflect the probability of selection into the sample. Estimates of proportions, means, and other summary statistics will be performed with the weighted data and presented as national estimates. To estimate the variances of the estimates, both the Taylor series linearization method and replicate weights methods will be utilized. Post-survey, unit nonresponse will be assessed by estimating a latent variable model of the propensity to respond to a survey. This model will (a) determine if systematic (predictable from observed characteristics of the units) nonresponses appear in the data and, if so, (b) predict the probability of responding to a survey based on the observed characteristics. Then, the responding cases will be weighted by, approximately, the inverse of the predicted probability of response.3
To measure the impact of participation in CEP on student participation and Federal revenue, we will use DD estimation. In DD estimation, differences in pre-program outcomes for both the treatment (participating in CEP) and a matched comparison group are compared to post-program outcomes. Matching treatment LEAs to comparison LEAs controls for persistent observable differences between the two groups, and the DD framework nets out additional sources of potential unobservable bias from the impact estimates, including historical/economic, maturation/changing demographic, and policy influences. As a result, we are less likely to falsely conclude that the policy had an impact, by virtue of the DD’s ability to deal with plausible rival explanations of program impacts.
We will also use comparative interrupted time series (CITS) methods, which use a projection of the pre-program trend for the two groups, to examine the robustness of findings from the DD approach. CITS, accounts for differences in the baseline slopes between the two groups. However, it requires that we have at least two pre-CEP data-points for all LEAs, and some literature strongly recommend at least four pre-CEP data points to capture an unbiased trend.4 Therefore, we will use DD as our primary analytic approach, but will conduct sensitivity analyses by implementing CITS on a restricted sample that only includes LEAs for which we can collect at least two years of baseline data.
We will use the posttest-only comparison group design to assess the impact of CEP on the type of school breakfast program, non-Federal revenues and the ability of the LEA to cover costs (breakeven), and the components of change in program participation and student attendance. In this design, we compare the treatment group with a comparison group after the treatment has been implemented. We will use regression adjustment to control for differences in district baseline (pre-CEP) characteristics using available administrative data from the National Center for Educational Statistics and FNS.
Degree of accuracy needed for the purpose described in the justification. The study has been designed to meet FNS’ expectations for detection of differences in outcomes between
participating and non-participating LEAs at widely accepted levels of statistical significance and power. For example, with 80 percent power (95 percent confidence level), we can detect a difference in proportions between stratum of approximately 0.15 and a difference in proportions between participating and non-participating LEAs of 0.09.
Unusual Problems Requiring Specialized Sampling Procedures
We do not anticipate any unusual problems requiring any specialized sampling procedures
Any Use of Periodic (Less Frequent Than Annual) Data Collection Cycles to Reduce Burden.
The data collection procedures will be conducted once during SY 2016-17.
B.3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
Overall response rate projections were presented earlier. Achieving the specified response rate to the web surveys involves locating the sample members to secure participation using procedures described below. We estimate 80 percent of the sampled LEA directors will complete the web-administered survey. We expect 100 percent of SA CN directors to complete their survey.
Below we describe procedures to be followed to maximize the number of sample members who complete the survey:
The letters inviting LEA directors and SA CN directors to participate will be very carefully developed to emphasize the importance of this study and how the information will help the Food and Nutrition Service (FNS) to better understand the characteristics of LEAs that participate in CEP and LEAs are eligible but do not participate in CEP.
The current contact information will be used for all initial correspondence and be updated as needed throughout the data collection period to facilitate communication with the study team.
Designated FNS regional staff will serve as regional study liaisons and be kept closely informed of the project so that they will be able to answer questions from LEAs and States and encourage participation.
A toll free number and study email address will be provided so that LEAs and States can receive assistance with the study.
Sampled LEA directors and SACN directors will have the option of completing the web-based survey as a telephone survey.
Periodic email reminders will be sent to sample members who have not yet completed the survey.
We will follow up by telephone with all sampled LEA and SA CN directors who do not complete the survey within a specified period and urge them to complete the survey. At that point, if the directors prefer to complete the survey or remaining sections of the survey over the telephone, a telephone interviewer will administer the survey or remaining parts over the telephone.
The following procedures will be used to maximize the effectiveness of telephone reminders:
Use a core of interviewers with experience working on telephone surveys, particularly interviewers who have proven their ability to obtain cooperation from a high proportion of sample members.
Conduct a telephone interviewer training session specific to this study.
Use call scheduling procedures that are designed to call numbers at different times of the day (between 8am and 6pm) and days of the week (Monday through Friday), to improve the chances of finding a respondent at work.
Provide a toll-free number and email help address for respondents to verify the study’s legitimacy or to ask other questions about the study.
B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Draft SA CN and LEA director surveys were pre-tested in late February 2016 through March 2016. The pre-test instruments included newly developed questions and questions that were heavily edited from the previous FNS surveys. These were evaluated in terms of understandability (confusing wording or layout, failure to grasp what we were looking for, etc.) and length of time to answer. Two SA CN directors and 8 LEA directors from the two States participated in the survey pretest phase. After reviewing written responses, selected pretest participants were debriefed by phone and provided opportunities for general comments about the instruments.
Child Nutrition Director Survey Pre-Test Findings
2M received survey comments from both two SA CN directors for the SA survey and the Administrative Data Request form. Both indicated that the Administrative Data Request could be completed by their respective information technology departments although one SA CN director indicated that the burden would be relatively high for their State.
The average time required to complete the SA survey averaged to approximately 1 hour.
Feedback for the SA survey primarily focused on the need for more clarification for response options (e.g., definitions, introduction, etc.) and additional instruction for optional selections (e.g., “select all that apply”). SA comments highlighted the similarity between questions 2.7 and 2.9, as well as the open-ended questions connected to both questions (2.8 and 2.10). After further review, both questions (2.9 and 2.10) were deleted from the instrument.
LEA Director Survey Pre-Test Findings
All participants indicated that the surveys were clear and generally easy to navigate. The majority of critical feedback provided by LEAs was directed toward the Participating Survey; LEA comments focused on improving instructions and providing additional clarification on certain terms or acronyms.5 Additionally, LEAs suggested additional response categories to a few items, which were incorporated into the Participating and, where relevant, the non-participating survey. The average time required to complete the participating LEA survey was approximately 2 hours, while completing the non-participating survey took approximately 1 hour.
B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The Contractor, 2M Research Services, and their Subcontractor, Abt Associates, will conduct this study.
|
1 “Summary of National CEP Election Data as of September 1, 2015.”
2 http://www.cbpp.org/research/food-assistance/community-eligibility-database-schools-that-can-adopt-community-eligibility.
3 See Wun L, Ezzati-Rice TM, Baskin R, Greenblatt J, Zodet M, Potter F, Diaz-TenaN, Touzani M. Using Propensity Scores to Adjust Weights to Compensate for Dwelling Unit Level Nonresponse in the Medical Expenditure Panel Survey. Agency for Healthcare Research and Quality Working Paper No. 04004, October 2004, http://www.ahrq.gov. This approach divides the propensity scores into classes based on the observable data (for example, size) and assigns the average score within the class to each case, eliminating large adjustments to the survey weights and increasing the precision in the estimates.
4 Somers, M., Zhu, P., Jacob, R., & Bloom, H. (2013). The validity and precision of the comparative interrupted time series design and the difference-in-differences design in educational evaluation.
5 The web administration of the survey will have highlights for terms that will allow the respondent to “hover” over the term and see the definition.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Freeland_s |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |