Supporting Statement Part B Final 10.7.16

SUPPORTING STATEMENT - PART B for

OMB Control Number 0584-NEW:

Study of Nutrition and Activity in Child Care Settings

Alice Ann Gola

Research Analyst

Office of Policy Support

Food and Nutrition Service

U.S. Department of Agriculture

3101 Park Center Drive

Alexandria, VA 22302

Phone: 703-305-4347

Email: [email protected]

Table of Contents

B.1 Respondent universe and sampling methods 1

B.2 Procedures for the collection of information 8

B.3 Methods to maximize the response rates and to deal with nonresponse 23

B.4 Test of procedures or methods to be undertaken 27

B.5 Individuals consulted on statistical aspects & individuals collecting and/or analyzing data 29

References 29

List of Exhibits

Table B.1. SNACS Respondent Universe 2

Table B.2: Expected Completed Sample Sizes by Target Population and Data Collection Effort 4

Table B.3: State Stratification, Allocation of SSUs, and Population Counts 11

Table B.4: Expected Sample Size and Precision Level for Primary Subgroup Estimates 18

Appendices

Appendix A. Authorizing Legislation

Appendix B. Research Questions, Outcome Measures, and Data Sources

Appendix C. Recruitment Materials

C.1. State Agency Recruitment Letters and Call Scripts

C.2. Study Fact Sheet

C.3. Study Endorsement Letter from USDA

C.4. Study Endorsement Letter from the President of the CACFP Forum

C.5. Sponsor Recruitment Letter and Call Script

C.6. Study Brochure

C.6a. Study Brochure (English)

C.6b. Study Brochure (Spanish)

C.7. Provider Recruitment Letter and Call Script

C.8. Parent Consent Talking Points

C.8a. Parent Consent Talking Points (English)

C.8b. Parent Consent Talking Points (Spanish)

C.9. Parent Consent and Permission Form

C.9a. Parent Consent and Permission Form (English)

C.9b. Parent Consent and Permission Form (Spanish)

C.10. Brief Description of Study for Provider Newsletter

Appendix D. Nutrition and Wellness Data Collection Instruments

D.1. Provider Web Survey and Related Communications

D.1a. Provider Web Survey

D.1b. Provider Web Survey Related Communications

D.2. Menu Survey and Related Communications

D.2a. Menu Survey

D.2b. Infant Menu Survey

D.2c. Food Description Guide

D.2d. Menu Survey Related Communications

D.2e. Menu Survey Training Video

D.3. Reference Portion Measurement Form

D.4. Meal Observation Form

D.4a Meal Observation Form – Meals Prepared by Provider

D.4b Meal Observation Form – Meals Brought From Home

D.5. Classroom Waste Observation Form

D.6. Environmental Observation Form

Appendix E. Parent and Child Data Collection Instruments

E.1. Child Food Diary

E.1a. Child Food Diary Non-Child Care Day (English)

E.1b. Child Food Diary Child Care Day (English)

E.1c. Guidelines for Reviewing Food Diary with Parent (English)

E.1d. Child Food Diary Non-Child Care Day (Spanish)

E.1e. Child Food Diary Child Care Day (Spanish)

E.1f. Guidelines for Reviewing Food Diary with Parent (Spanish)

E.2. Height and Weight Form

E.2a. Standing Height and Weight Form

E.2b. Standing and Holding Weight Form

E.3. Infant Food Intake Form

E.4. Parent Interview

E.4a. Parent Interview (English)

E.4b. Parent Interview (Spanish)

Appendix F. Meal Cost Instruments

F.1. Pre-Visit Cost Telephone Interview

F.2. Pre-Visit Cost Form

F.3. Sponsor Cost Interview

F.4. Center Director Cost Interview

F.5. Center Foodservice Cost Interview

Appendix G. Federal Register Notice

Appendix H. Public Comments

H.1. Public Comment 1

H.2. Public Comment 2

H.3. Public Comment 3

H.4. Public Comment 4

H.5. Public Comment 5

H.6. Public Comment 6

H.7. Public Comment 7

H.8. Public Comment 8

H.9. Public Comment 9

H.10. Public Comment 10

H.11. Public Comment 11

H.12. Public Comment 12

H.13. Public Comments All Other

Appendix I. Response to Public Comments

I.1 Response to Comments 1

I.2 Response to Comments 2

I.3 Response to Comments 3

I.4 Response to Comments 4

I.5 Response to Comments 5

I.6 Response to Comments 6

I.7 Response to Comments 7

I.8 Response to Comments 8

I.9 Response to Comments 9

Appendix J. National Agricultural Statistics Service (NASS) Comments

Appendix K. IRB Approval and Confidentiality Agreement

K.1. IRB Approval

K.2. Confidentiality Agreement

Appendix L. Respondent Burden

Appendix M. Research Questions, Key Measures, Analysis Methods, and Subgroups and Covariates by Objective

B.1 Respondent universe and sampling methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The Study of Nutrition and Activity in Child Care Settings (SNACS)¹ will encompass several samples and data collection procedures to answer the research questions for the study. One of these will be a nationally representative sample of child care centers and family day care homes that receive meal reimbursements from the Child and Adult Care Food Program (CACFP). Congress required this study (HHFKA of 2010, Pub. L. No. 111-296, Section 223). Additionally, there will be associated samples of children cared for by the sampled CACFP providers.

The sample of child care providers will be selected using a multi-stage cluster sample design. Within the sampled child care centers (but not family day care homes), there will be a nationally representative sample of children for whom meals will be observed. Nested within this child sample, will be smaller nationally representative samples of children in CACFP centers for which dietary intake data will be collected with several levels of intensiveness. Also within sampled CACFP child care centers (but not family day care homes), there will be a nationally representative sample of infants for whom dietary intake will be measured. Finally, there will be a small sample of convenience of CACFP family day care homes at which there will be a feasibility study of meal observation and measurement of dietary intake.

Except for the last mentioned sample (convenience sample of CACFP family day care homes), all samples will provide unbiased and precise estimates at each level of analysis (providers, children, infants, and meals) for the population and unbiased and moderately-precise estimates for subgroups. Key provider-level subgroups include provider type (family day care home, center), urbanicity, sponsored/independent center, number of provider sites (single or multi-site), and center size (number enrolled). Please see Section B.2.C for more detail regarding the precision of estimates.

The universe for the SNACS, shown in Table B.1, includes CACFP child care centers, CACFP family day care homes, Head Start centers,² CACFP afterschool programs, CACFP at-risk programs, and the children and infants enrolled with the providers. The sponsors of providers and the parents of the children and infants enrolled with the providers, and meals served at the providers will also be included in the data collection. The target population of children in this study consists of infants and children up to age 12 who are in child care where the provider receives CACFP meal reimbursements. Detailed information about the sample design is included in Section B.2.A.

Table B.1. SNACS Respondent Universe

Sample Group	Estimated Size of Respondent Universe
	Number of Providers	Number of Enrolled Children
CACFP Child Care Centers (excluding Head Start)	25,990	947,249
CACFP Family Day Care Homes	116,813	714,662
Head Start Centers	13,178	480,294
CACFP Afterschool Programs	3,969	90,493
CACFP At-Risk Programs	15,797	360,172

Source: Child Care in America: 2014 State Fact Sheets and FNS National Data Bank

As shown in Table B.2, meal observations and dietary intake records will be obtained in 372 providers for 340 infants and 2,172 older children. An additional 947 meal observations will be obtained to supplement the sample for the purpose of estimating portion sizes. Some (797) of the additional meal observations will be obtained from children who are selected, but fail to complete dietary intake data collection. The remaining 150 meal observations will be conducted in 30 randomly selected CACFP family day care homes (FDCHs). The proposed final samples of completed responses will include 1,326 child care provider organizations and 3,459 children/ infants³ and their parents. Table B.2 presents the target numbers of responding providers and children/infants. For each data collection, the table provides the final number of completed responses that will be used in analysis (accounting for refusals, non-attendance on observation days, and other non-response) for CACFP child care centers, Head Start centers, CACFP FDCHs, CACFP afterschool programs, and CACFP at-risk programs.

Table B.2: Expected Completed Sample Sizes by Target Population and Data Collection Effort

Panel A: CACFP Child Care Providers

Type of Data Collection	CACFP Child Care Centers	Head Start Centers	CACFP FDCHs	CACFP Afterschool	CACFP At-Risk	Total Providers
Provider web surveys & menu survey	300	300	300	213	213	1,326
Cost data collection	150	150		150	150	600
Meal observations & dietary intake recordsa,b	101	127	30	57	57	372
Meal observations onlyb			30			30
Meal observations & dietary intake recordsa,b (subsample: 2nd child care day)	10	13		6	6	35
Dietary intake records (subsample: 2nd non-child care day)	10	13		6	6	35

Panel B: Children and Infants

Type of Data Collection	CACFP Child Care Centers		Head Start Centers	CACFP FDCH	CACFP Afterschool	CACFP At-Risk	Total Children	Total Infants
	Children	Infants
Meal observations & dietary intake recordsa,b	608	340c	764	120	340	340	2,172d	340
Meal observations onlyb	236		297	150	132	132	947
Meal observations & dietary intake recordsa,b (subsample: 2nd child care day)	60		76		36	36	208
Dietary intake records (subsample: non-child care day)	60		76		36	36	208
Average number of intakes per provider	6	5	6	4	6	6

^a Meal observations provide information on dietary intake during child care day. Dietary intake records will be collected outside the child care day using food diaries.

^b Meal observations in all types of sites will be used to determine portion sizes for meals and snacks. Additional CACFP FDCHs providers will be selected for meal observations. Additional meal observations will also be obtained in all other providers for children who are sampled, but do not complete dietary intakes.

^c Dietary intake only. No data collection outside of child care for infants.

^d Children in centers with 2 days of dietary intake (child care day and non-child care day). Children in FDCHs with one day of dietary intake (96 children in FDCHs expected to have 2 days of dietary intake).

The full provider sample of respondents used for analysis will comprise 1,326 provider organizations completing the provider web survey and menu survey, allocated among provider types as indicated in the table. Within the provider survey sample of respondents, the sub-sample for cost data collection will comprise 600 completed provider interviews, evenly allocated across CACFP child care centers, Head Start centers, CACFP afterschool programs, and CACFP at-risk programs. CACFP FDCHs will not participate in cost data collection.

Within the subsample of providers in which meal observations and dietary intake records will be collected, a second day of meal observations and dietary intake records for a child care day will be obtained for 208 children at 35 providers, and dietary intake records for a second non-child care day will be obtained in a different subsample of 208 children at 35 providers (bottom 2 rows of Panels A and B in Exhibit B.2). This additional data collection (which will be used to estimate usual dietary intake) will occur in child care centers, at-risk and afterschool.

In order to have an efficient sample of providers we are not sampling sponsors directly, though sponsors will provide some of the data needed for the study. The provider web surveys will collect information reported by sponsors for characteristics, policies, or practices at the sponsor level. All CACFP sponsors with at least one provider in the survey sample will be eligible to respond to one or more survey modules. Sponsors will also provide some information for the cost data collection. No other data collection will involve sponsors as respondents.

We anticipate a response rate of 81% at the sponsor and provider levels for the surveys, cost interviews, and observations. Our expectations are based on recent child nutrition program data collection experience under the mandate of  the Healthy, Hunger-Free Kids Act of 2010 (HHFKA) for cooperation with evaluations, and on the recruitment and incentive plans described in Section A.9 of Part A of this submission. Recent child nutrition program studies, including the School Nutrition Dietary Assessment Study (SNDA)-IV, the CACFP Assessment of Sponsor Tiering Determinations 2008-2010, and the Community Eligibility Provision Evaluation, have included a mandate for respondents receiving child nutrition program funds to cooperate. Similar to the proposed SNACS, SNDA-IV and the Community Eligibility Provision Evaluation collected descriptive data about child nutrition program operations through web surveys at the organization and site level, and conducted detailed menu surveys on foods served to children. SNDA-IV had weighted response rates of 94.0 percent for the SFA director survey, 97.7 percent for the school-level menu survey, and 96.7 percent for the school foodservice manager survey (among recruited SFAs and schools). School foodservice managers received a $50 incentive for completing the menu survey and foodservice manager survey. The Community Eligibility Provision Evaluation had a response rate of 92.8 percent for the web survey of SFAs and 100 percent for the menu survey among recruited SFAs and schools. The CACFP Assessment of Sponsor Tiering Determinations collected three rounds of data from CACFP sponsors: (1) rosters of family day care homes, (2) meal claims data for sampled homes, and (3) sponsor tiering documents for a subset of sampled homes. In 2010, the first year when the HHFKA requirement was in effect, the sponsor cooperation rate with the initial request to participate was 93.3 percent and the response rate for the subsequent data collection was 100 percent. CACFP sponsors received a $100 incentive to offset the cost of providing data and documentation. The anticipated response rate of 81 percent for the SNACS at the sponsor and provider levels reflects the conservative expectation that, while the response rates for similar studies have generally been above 90 percent, the burden of the study may lead to a lower response rate, particularly among smaller sponsors and providers.

For the parent and child data collection, after allowing for an 80% parental consent rate and an 80% attendance rate on the day of observation, we anticipate the following response rates: 100% for observations of consented children present on the day of observation; 90% for the food diary for the child care day; and 80% for the non-child care day (conditional on completing the first food diary). Thus, we anticipate a cumulative response rate of (80% x 80% x 100% x 90% x 80% =) 46% for the combination of the meal observations, child care day dietary intake, and non-child care day dietary intake in child care centers, at-risk programs, and afterschool programs. For the dietary intake data collection in FDCHs, we anticipate a lower response rate given the demands on the time of the proprietors of FDCHs (who often have no adult assistants) but this component of SNACS is a feasibility study and the response rate will be an important indicator of future feasibility of data collection among this group of child care providers. We anticipate a response rate of 90% for infants in CACFP centers, since the only non-response will occur when infant feeding records in the child care center are not completed.

A wide range of methods, beginning at the recruitment stage with the provision of comprehensive recruiting materials and continuing through data collection in which highly skilled and trained data collectors will be employed, will help maximize participation in the study and reduce nonresponse in all aspects of data collection. These efforts are discussed in detail in B.3. In addition, we will draw a backup sample at the provider level in case provider recruiting is more challenging than expected. However, once we are in the field collecting data, the schedule is too compressed to permit any additional sample to be fielded or any other adjustments in recruiting to be made.

This study is being conducted for the first time, so there are no historical response rates to report.

B.2 Procedures for the collection of information

Describe the procedures for the collection of information including:

• (A) Statistical methodology for stratification and sample selection,

• (B) Estimation procedure,

• (C) Degree of accuracy needed for the purpose described in the justification,

• (D) Unusual problems requiring specialized sampling procedures, and

• (E) Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

B.2.A. Statistical methodology for stratification and sample selection

To achieve the nationally-representative samples of providers, children, and meals required to achieve the SNACS research objectives, we will use a multi-stage cluster sampling design. First, we will select a national sample of 20 States. States are the most efficient choice for first-stage sampling units because of the high costs of obtaining and processing lists of childcare providers. (There is no national list of CACFP providers.) Next, we will select a sample of metropolitan areas and clusters of nonmetropolitan counties from the selected States as secondary sampling units (SSUs). The third stage is the sampling of providers (and their sponsors) within SSUs. Classrooms and children within classrooms will be sampled from the selected providers. Given the small sample size targets for some provider domains, provider sampling will not be independent across sample states and SSUs as is more typical in multi-stage surveys. The coordination of provider sampling across sample states and SSUs will make it possible to select a very efficient provider sample for each targeted provider domain. Probabilities have been designed at all stages so that it would be possible to have a nearly self-weighting sample of providers within each targeted provider domain. A slight drawback of this design is that variance estimation will be challenging. This is discussed in Section B.2.B.

B.2.A.1 Selection of States

At the first stage, we will select a national sample of 20 States. The sampling frame will comprise the 48 contiguous States and the District of Columbia.⁴ The largest six States will be selected with certainty. The remaining 14 States will be selected using a stratified design with probabilities proportional to size (PPS) within strata. The State strata will be mostly defined by FNS region as show in in Table B.3. However, the Mid-Atlantic and Northeast regions will be collapsed and then re-split by the percentage of low-income children who live in rural areas. These strata were chosen because of the importance of assuring regional directors of the relevance of study findings to each region and because of the importance of supporting an oversample of nonmetropolitan SSUs.

Within each stratum, two States will be selected with probability proportional to a weighted measure of size (MOS). In order to support the aforementioned desired oversample of rural providers, the MOS will be designed to result in a state sample where the population of low-income children is more rural than the nation. Specifically, the MOS will be calculated as:

We will define “low-income children” as those under age 18 living in a household with annual income less than 100 percent of the federal poverty level. ⁵ The selection will use the Durbin-Brewer method as implemented in SAS Proc SurveySelect. Let and , where h indexes strata and i indexes States within strata. Then the first State is selected from a stratum with probability , and conditional on State i being selected on the first draw, the probabilities of the remaining States on the second draw are calculated as . We have verified that Z_hi < 1 / 2 for all States in our stratification, so there will be no issues with negative probabilities of selection.

B.2.A.2 Selection of secondary sampling units (SSUs)

At the next stage, we will select a PPS systematic sample of geographically-defined SSUs from within the selected States. The SSUs will be classified into one of four types: 1) a Metropolitan Statistical Area (MSA); 2) a large Metropolitan Division (a compact grouping of interconnected counties within a larger MSA); 3) a large non-MSA county; or 4) a grouping of adjacent smaller non-MSA counties. Grouping of smaller non-MSE will only be done where analysis of the American Community Survey (ACS) indicates that there are not enough low-income children to support at least 20 child care providers. For these small counties (with roughly less than 5,000 low income children), collapsing will be done by professional judgment with the aid of maps. Within each of the 20 selected States, the SSUs will be stratified by MSA/non-MSA status and sorted geographically within those strata. The measure of size will be the number of children under age 18 living in a household with annual income less than 100 percent of the federal poverty level according to the most up-to-date version of the ACS available at the time of sampling.

In most of the non-certainty States, we will select two metropolitan SSUs and one nonmetropolitan SSU. However, in the first stratum, there are so few poor children living in nonmetropolitan areas that we will not select any nonmetropolitan SSUs. We will offset this by selecting two nonmetropolitan SSUs in each of the States selected in the Southeast, the region with highest numbers of poor children in both metropolitan and nonmetropolitan areas (not including the certainty States). We will also select additional metropolitan SSUs in the Southeast, as indicated in Table B.3. In both California and Texas, we will select six SSUs from the MSA stratum and one SSU from the non-MSA stratum. In each of the other four certainty States, we will select four SSUs from the MSA stratum and one SSU from the non-MSA stratum. This approach will yield the target total of 60 urban SSUs and 20 rural SSUs. See Section B.2.B for a discussion of variance estimation.

Table B.3: State Stratification, Allocation of SSUs, and Population Counts

FNS Region	States	Sample SSUs			Total CACFP Providers	Total Daily Attendance at CACFP Providers
		Metro	Non-Metro	Total
Non-Certainty States
Mid-Atlantic and Northeast 1	DE, DC, MD, NJ, MA, CT, RI	4	0	4	11,852	229,939
Mid-Atlantic and Northeast 2	WV, VA, PA, ME, VT, NH	4	2	6	8,398	206,012
Midwest	MI, MN, WI, IN	4	2	6	18,196	361,250
Mountain Plains	CO, KS, UT, NE, MO, ND, SD, IA, MT, WY	4	2	6	16,316	320,027
Southeast	NC, SC, TN, GA, AL, KY, MS	8	4	12	15,690	563,340
Southwest	LA, NM, AR, OK	4	2	6	15,347	262,159
West	AZ, NV, WA, ID, OR	4	2	6	9,453	180,907
Certainty States
California		6	1	7	20,029	587,713
Florida		4	1	5	5,826	323,759
Illinois		4	1	5	9,386	159,160
New York		4	1	5	12,385	317,483
Ohio		4	1	5	4,526	127,791
Texas		6	1	7	11,300	515,061
Total		60	20	80	158,704	4,154,601

B.2.A.3 Selection of providers and sponsors

For each sampled SSU, we will create a comprehensive sampling frames for CACFP providers. We will obtain lists of CACFP sponsors and providers from State agencies. The samples for all targeted provider domains will be selected from among all 80 SSUs, though not all SSUs will necessarily contain sample providers from every targeted domain. Given the small national target sample sizes for some of the targeted provider domains, the uneven distribution of some provider domains (particularly the CACFP at-risk programs) across states and SSUs, and a lack of advance knowledge about this distribution, we will not allocate a fixed provider sample size for each targeted provider domain within each SSU. Rather, we will select a systematic PPS sample with carryover to select the provider sample for each targeted domain. This involves sorting all listed providers in a targeted domain by state and SSU and then drawing a systematic PPS sample with a MOS=(1/State_Prob)(1/SSU_Prob). Any listed providers with a MOS greater than the national sampling interval (number of listed providers / desired sample size for the domain) will be selected with certainty and the national sampling interval will be recalculated to still yield the desired sample size. This procedure guarantees the achievement of the desired national sample size for each targeted provider domain while also maintaining a nearly self-weighting provider sample. It does, however, pose some challenges for variance estimation that are discussed in Section B.2.B.

As noted in Section B.1, we are not sampling sponsors directly. However, sponsors with at least one provider in the sample may be asked to participate in data collection activities, depending upon their organizational structure or assignment of responsibilities.

B.2.A.4 Selection of classrooms

In order to make on-site data collection manageable, we will sample a single classroom at each provider. As part of the sampling process, we will obtain a roster of all eligible classrooms along with the number of children currently enrolled in each. We will select one classroom from each provider. This will be done over the phone. A study team member will construct a roster of classrooms based on information from the provider and enter them into a desktop application that will randomly select one.

Given the restriction on the number of sample classrooms per provider, we need to complete 8 to 9 child observations in the sampled classroom in order to achieve the desired precision levels (see section B.2.C below). We must also account for the need to obtain parental consent (expected rate of 80%) and the fact that some children will not be in attendance on the observation day (expected attendance rate of 80%). This means that the sampled classroom must have a class size of at least 14 children [9 / (80% x 80%)], on average. If, as a rule, we sample classrooms with fewer than 14 children, we will not achieve the required overall sample size. Based on these considerations, we plan to only sample classrooms that have at least 14 students enrolled, where possible. For sites with no classrooms of that size, the threshold will be reduced as needed to facilitate selection. The number of sampled providers where this threshold will need to be reduced cannot be accurately predicted prior to the recruitment. If necessary, we will select additional providers to make up the difference so that the target sample sizes for child-level data collection activities are achieved. To be ready for the eventuality, we will predesignate a reserve sample for each targeted provider domain. The size of this reserve sample will be 30 percent of the primary sample for each targeted provider domain.

B.2.A.5 Selection of children

The sampling frame of children for each provider will comprise the children in the selected class whose parents consent to participate in the study. In child care centers, at-risk programs, and afterschool programs, we plan to observe 8-9 children per site. We will select approximately equal numbers of boys and girls for the child and infant samples.

B.2.A.6 Selection of meals

For CACFP and Head Start child care centers (CCC), the meal observations for child intake data will provide a sample comprising meals served to 1,905 children in 228 locations. Similarly, we will use the sample of 472 children observed in 57 at-risk programs and the same number in afterschool programs to provide data on meals served for these programs. For CACFP homes, the meal observations for child intakes will provide a sample of meals served to approximately 120 children in 30 FDCHs. In addition, we will observe approximately 150 children’s meals in 30 more FDCHs to assure a sufficient sample of meals for portion size estimates. (Numbers cited in this paragraph calculated from Table B.2, Part A, rows 3 & 4, Part B, rows 1 & 2).

B.2.B Estimation procedure

The four main research objectives of SNACS are to:

1. Assess nutrition and wellness policies/practices and meal quality for infants and children in CACFP child care settings

2. Describe food and nutrient intakes of infants and children in CACFP child care centers and outside of child care, and assess the feasibility of collecting these data in FDCHs

3. Determine the meal costs and revenues in CACFP child care centers

4. Describe and assess plate waste in CACFP child care centers

The primary method of analysis will be descriptive tabulations, including means and distributions of continuous measures (e.g., calories per lunch) and proportions or distributions for binary or categorical variables (e.g., proportion of providers serving whole-grain products for snacks). We will estimate these values and their standard errors for the relevant universe of child care providers and for subgroups using weights and variance calculations that take into account the complex sampling design. As noted in Section A.16, the table presented in Appendix M summarizes the key outcomes, sample, subgroups, and estimation methods for each of the research objectives.

For producing population-based estimates and for all statistical analyses, each responding provider, child and meals will be assigned a sampling weight. This weight will be a combination of the base weight and an adjustment for nonresponse. For example, the base weight for a provider is the inverse of the probability of selection of the provider and will be the product of the State weight, SSU weight within the State and the provider weight within the SSU. The base provider weights will be adjusted for nonresponse within each SSU. After data collection is complete, we will examine the data and determine the appropriate method of adjustment (e.g., use of nonresponse adjustment cells based on sample frame data).

For variance estimation, we will set up 24 variance strata and 48 variance units. For the noncertainty States, the variance strata will be the same as the sampling strata and the variance units will be the selected States. These will thus contribute 7 variance strata and 14 variance units. The six nonmetropolitan SSUs from the certainty States will be arranged into three pairs. We will pair California with Texas, Florida with New York, and Ohio with Illinois. The three pairs will be variance strata and the six nonmetropolitan SSUs will be variance units. The 28 metropolitan SSUs within the certainty States will be arranged into 14 pairs, strictly within each State. The pairing will be with respect to the sort used to select the sample, so that SSUs in each pair will be geographically close. The 14 pairs will be variance strata and the 28 metropolitan SSUs will be variance units. All analyses will then be conducted with appropriate survey procedures in commercial statistical analysis packages (e.g. SAS survey procedures or the Stata “svy” commands). These procedures calculate the squared difference in outcome rates between the two variance units in each variance stratum and then sum the squared differences across variance strata. We expect that this variance estimation will be slightly conservative (yield confidence intervals that cover the truth more than 95 percent of the time) for several reasons. First, they do not capture the finite population correction factor that comes from the Durbin-Brewer selection procedure. We could apply this reduction, but the survey research literature suggests that the resulting variance estimates can be very unstable. We think it better to tolerate a little conservatism than the possibility of very noisy variance estimates. Second, the cross-state pairing of the nonmetropolitan SSUs in certainty States will induce some positive bias in variance estimates. This is unavoidable given that we will only select one SSU in each State. We could pair them with metropolitan SSUs in the same State, but we think the bias may be less from cross-state pairing than from pairing metropolitan with nonmetropolitan. Third, the systematic selection of metropolitan SSUs in the certainty States reduces the true variance in a way that cannot be captured with any unbiased variance estimation procedure. Finally, the carryover of systematic sampling of providers across states and SSUs means that paired variance units will be more dissimilar to each other than they would be if the provider samples were selected independently across states and SSUs. This will also create some positive bias in variance estimates. We will be using the work of Judkins and Hidiroglou (2004) to guide us in reducing this last bias.

B.2.C Degree of accuracy needed for the purpose described in the justification

In order to satisfy the research objectives, estimates of overall national population percentages should have a margin of error no larger than plus or minus 5 percentage points at the 95% confidence level. Estimates of population percentages for subgroups like urban and rural, center type, etc. should have a margin of error no larger than plus or minus 7 to 10 percentage points at the 95% confidence level, the precision level typically specified by FNS for subgroup comparisons. Finally, the estimates for meal costs should have a margin of error no larger than plus or minus 10% at the 95% confidence level.

The complex nature of the SNACS design requires a thorough estimation of the design effects in order to determine the sample sizes needed to fulfill these requirements. The estimated design effects used in this study plan were obtained from the Early Childhood and Childcare Study (ECCS; Glantz, et al., 1997), as shown in Table B.4. The ECCS design has a number of key commonalities with SNACS, particularly with respect to the national sampling of CACFP providers and children within providers and the outcomes measured. ⁶ Both studies selected 20 sample states at the first stage of selection. This is critical because we suspect that state is a major source of variation given the role of state agencies in promulgating detailed regulations for child care providers. One major difference, however, between ECCS and SNACS is the precision requirements for subgroups, particularly the urban and rural centers, which require substantial oversampling of rural areas; this requirement with no other design changes would increase the design effect by approximately 20%. However, other aspects of the SNACS design are expected to yield smaller design effect components relative to the ECCS because of the decision to sample providers directly (as opposed to first sampling sponsors and then selecting providers within the sampled sponsors). The intraclass correlation coefficient within a SSU is expected to be smaller than within a sponsor. Taking all of these considerations into account, we assume that the design effects from the ECCS are good proxies for the design effects that will be observed in SNACS. For cost estimation, we also note that federally mandated reimbursement policies are likely to reduce between-state and between-SSU variance on this class of estimators.

Table B.4 Design Effects Assumed for SNACS, by Target Population

Population	Design Effect
Family Day Care Home (FDCH) Providers	2.8
Head Start centers	2.3
CACFP centers	1.6
FDCH Children	2.9
Head Start Center Children	3.9
CACFP Center Children	3.1
CACFP at-risk and afterschool programs (provider level)	2.2
Children in CACFP at-risk and afterschool programs	3.5

Source: Early Childhood and Childcare Study (Glantz et al., 1997).

The sample sizes discussed above in section B.2.A are chosen to achieve the aforementioned precision. Except for the cost sample, a population percentage of 50% was assumed for the calculations of sample sizes required to achieve precision targets for an estimated proportion. For example, Menu Survey estimates based on CACFP FDCHs must have no more than a 10 percentage point margin of error. Under a simple random sample, this would require about 100 completions. When we account for the estimated design effect of 2.8 for FDCHs, we estimate that approximately 100 x 2.8 = 280 completions with this population are needed. This target sample size was rounded up to 300 in our design to provide some protection against uncertainty in the actual design effect that will be observed in SNACS. For the cost sample, the overall and subgroup sample sizes of centers are based on the samples of School Food Authorities (SFAs) for the cost component of the School Nutrition and Meal Cost Study (Ponza et al., 2014), which provides the closest available model for the expected variances and design effects for measures of CACFP meal costs.

Table B.5 provides the expected precision of estimates for each study component (provider surveys, child intake and plate waste, and meal costs and revenues) and for subgroups within these study domains. Provider subgroups include breakdowns based on CACFP participation and provider characteristics (center vs. FDCH vs. at-risk/afterschool, rural/urban, sponsored/independent, single/multi-site, and enrollment size). Child subgroups are based on provider characteristics; infants are treated as a separate subgroup. Other subgroups of interest may be defined once more information about the sample is known.

Table B.5: Expected Sample Size and Precision Level for Primary Subgroup Estimates

Domain/Subgroup	Sampling Unit	Proposed Sample Size	Expected Precision (95% Confidence Level), by Proportion Estimatedb
			p=10%	p=30%	p=50%
Provider Surveys
All CACFP providersa	Provider	900	2.9	4.5	4.9
Child care centers (Head Start + other CACFP CCC)	Providers	600	3.4	5.1	5.6
CACFP family day care homes	Providers	300	5.7	8.7	9.5
Urban/rural status (CACFP providers) a
Rurala	Providers	210	6.0	9.2	10.0
Urbana	Providers	690	3.2	4.9	5.3
Sponsor status (CACFP centers)
Sponsored	Centers	420	4.0	6.1	6.7
Independent	Centers	180	6.1	9.3	10.2
Number of sites (CACFP centers)
Multi-site	Centers	336	4.5	6.8	7.5
Single-site	Centers	264	5.1	7.7	8.4
Enrollment (CACFP centers)
Size: Enrollment total in lowest third of sample (e.g., <20)	Centers	200	5.8	8.9	9.7
Size: Enrollment total in middle third of sample (e.g., 20-49)	Centers	200	5.8	8.9	9.7
Size: Enrollment total in highest third of sample (e.g., 50+)	Centers	200	5.8	8.9	9.7
CACFP at-risk	Programs	213	6.0	9.1	10.0
CACFP afterschool	Providers	213	6.0	9.1	10.0
Child Intakes and Plate Waste
All CACFP child care centers (Head Start + other CACFP CCC)	Children	1,372	3.0	4.5	4.9
Urban/rural status (CACFP centers)
Rural CCC	Children	340	6.0	9.1	9.9
Urban CCC	Children	1,032	3.4	5.2	5.7
Sponsor status (CACFP centers)
Sponsored CCC	Children	960	3.5	5.4	5.9
Independent CCC	Children	412	5.4	8.3	9.0
Number of sites (CACFP centers)
Multi-site CCC a	Children	768	4.0	6.1	6.6
Single-site CCC a	Children	604	4.5	6.8	7.5
Enrollment (CACFP centers)
CCC Size: Enrollment total in lowest third of sample (e.g., <20)	Children	457	5.1	7.9	8.6
CCC Size: Enrollment total in middle third of sample (e.g., 20-49)	Children	457	5.1	7.9	8.6
CCC Size: Enrollment total in highest third of sample (e.g., 50+)	Children	457	5.1	7.9	8.6
Infants in child care centers	Infants	340	6.0	9.1	9.9
CACFP at-risk	Children	340	6.0	9.1	10.0
CACFP afterschool	Children	340	6.0	9.1	10.0
Meal Costs and Revenues
All CACFP child care centers (Head Start + Other CACFP CCC) a	Centers	300	5.0% of mean
Urban/rural status (CACFP centers)
Rural	Centers	68	10.0% of mean
Urban	Centers	232	5.4% of mean
Sponsor status (CACFP centers)
Sponsored	Centers	210	5.7% of mean
Independent	Centers	90	8.7% of mean
Number of sites (CACFP centers)
Multi-site	Centers	168	6.4% of mean
Single-site	Centers	132	7.2% of mean
Enrollment (CACFP centers)
Size: Enrollment total in lowest third of sample (e.g., <20)	Centers	100	8.2% of mean
Size: Enrollment total in middle third of sample (e.g., 20-49)	Centers	100	8.2% of mean
Size: Enrollment total in highest third of sample (e.g., 50+)	Centers	100	8.2% of mean
CACFP At-risk	Centers	150	6.7% of mean
CACFP Afterschool	Centers	150	6.7% of mean

^a Excludes at-risk and afterschool CACFP programs.

^b Expected precision represents half-width of 95% confidence intervals for proportions of 10%, 30%, and 50%. For example an estimated proportion of 50% for survey data for all providers (n=900) will have a 95% confidence interval of +/- 4.9 percentage points.

For analyses of CACFP meal costs, we provide the precision levels as a percentage of the estimated mean: 5.0% for all child care centers and 5.4% to 10.0% for the listed primary subgroups, depending on sample size. As an example, suppose that the mean cost per CACFP lunch is $2. The 95% confidence interval would be plus or minus $0.10 for all child care centers and between $0.11 and $0.20 for subgroups.

B.2.D Unusual problems requiring specialized sampling procedures

We will need to exclude toddlers (age 12 to 24 months) enrolled in infant rooms from data collection due to the differences in the food intake data that will be collected for infant versus toddler classrooms. For children aged 1 year and older in toddler classrooms, food intake will be collected using meal observations during the daycare day and food diaries for time outside child care. By contrast, infant data collection is restricted to the child care day. Data will be collected using the infant intake form. Given these differences, it will not be possible to combine data for 12–24 month old children from the two different classroom types, and the infant intake form may not be appropriate for this age group in any case. For this reason, it is more analytically appropriate to exclude toddlers enrolled in infant rooms from our sample entirely. Note that it is generally younger toddlers (age 12–18 months) who may be included in infant classrooms.

B.2.E Any use of periodic data collection cycles to reduce burden.

SNACS is a one-time study, so concern regarding the periodicity of data collection cycles is not applicable.

B.2.F. General data collection procedures

Prior to data collection, sampled States, child care providers, and parents will be recruited to participate in the study. As described in A.2.A and presented in Appendix C, extensive materials have been developed to facilitate recruitment activities. Staff who are involved in recruitment activities will be trained prior to beginning recruitment. Training will include extensive study background, objectives, importance of the study and what is expected of respondents participating in the study.

Child care providers that agree to participate in the study will be contacted to schedule data collection activities. Data collection will occur during a period convenient for the child care providers. Highly trained and skilled field interviewers will be responsible for conducting the on-site data collection activities. Field interviewers with extensive experience in school settings will be part of the data collection team. These interviewers have experience recruiting and implementing data collection with each of the different types of respondents, parents, administrators, teachers. A cohesive team is assigned to each site, allowing for consistency in communication establishing and maintaining rapport among the parents, teachers and staff. During data collection, the team closely monitors the progress of data collection to ensure that each component of the protocol has been completed accurately and thoroughly. Field managers and interviewers have weekly meetings to discuss work in progress, plan future visits to new centers, and carry lessons forward learned on site. This facilitates smooth and efficient data collection and minimizes burden for centers and families.

B.3 Methods to maximize the response rates and to deal with nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

A wide range of methods will be used to maximize participation in the study and reduce nonresponse in all aspects of data collection. First, we will undertake several activities to lay the groundwork for our intensive recruitment campaign: developing recruitment materials, securing an endorsement from the National CACFP Forum, creating a data management system to support recruiting, obtaining institutional review board (IRB) approval (Appendix K.1), and training the recruitment team. Our recruitment plans are tailored for each study population to ensure the highest rate of cooperation. Second, data will be collected by skilled, highly trained staff who are experienced in strategies to maximize response rates. Finally, section 305 HHFKA mandated that entities, including child care providers, that receive USDA funds must cooperate with USDA evaluations (see Appendix A). This mandate will help ensure the cooperation of State agencies, sponsors, and providers.

In eliciting cooperation from sponsors, providers and provider staff, we have found that the following strategies prove successful:

• Comprehensive training for all recruiters that will cover project details, anticipated challenges, and expectations. With a full understanding of the project and its goals within the current provider foodservice environment, our recruiters will impart a level of aptitude and professionalism in all communications with study participants. We will use senior-level staff for sponsor and provider refusal conversion.

• Obtain endorsement and support from relevant organizations. Such endorsements, which will come in the form of letters of support from USDA and a national CACFP association, will be part of the recruitment materials.

• Provide sufficient information about the study purposes, objectives, and methodology so that potential participants have an informed basis for their decision.

• Provide a realistic appraisal of total respondent participation in the study effort in terms of contributions of time, information, space, and human resources. This will also include a statement of anticipated benefits to them. Our use of center liaisons and pre-test procedures (see B.4 for description of pre-test procedures) are designed to build the most realistic and efficient study procedures possible.

• Identify the most appropriate respondent. The best respondent for each data collection component will most likely vary across provider type, and may include FDCH providers, center directors, nutrition staff, nutrition/kitchen managers and/or directors of sponsoring agencies. For example, in family day care homes, the provider is most likely responsible for all the activities associated with the provision of meals and snacks, and is thus the obvious respondent for the food preparer module. In centers, the director or a staff nutritionist may participate in activities such as menu planning or food purchasing. Early and accurate identification of the most appropriate respondent will assure targeted and efficient recruitment and data collection activities, and ultimately higher response rates.

• Demonstrate knowledge of local and site-specific conditions. We adapt our procedures and materials as needed for both recruitment and data collection. Some of these programs are sponsored by school districts or independent schools, while others are operated by community-based organizations. For programs operated in schools, we will insure that the school district, school principals, and program sponsors are all aware of the study. Where necessary, we will follow school district procedures for formal clearance of research.

• Use of highly trained data collectors. Our staff of skilled and highly trained data collectors will help respondents with any comprehension or technical issues that arise during data collection. These efforts will help overall participation rates as well as help minimize item nonresponse.

Several additional strategies will be used for parent/child recruitment and data collection. These strategies have proven to yield an honest, collaborative relationship between the research team and participants in the study:

• Use of friendly, informative, encouraging and easy to understand study materials.

• Use of an on-site study liaison. One of the challenges to recruitment and response will be clear and reliable communication with parents. Having a consistent presence on-site will be vital to recruitment and response since much of the interaction with parents can occur at hurried drop-off or pick-up times and we expect that parents and caregivers may alternate who is on site on any given day. An on-site study liaison will be appointed in each sampled provider to facilitate the data collection by visiting the selected classroom to distribute the informational packets, describe the study, and encourage return of consent materials and food diaries. We expect this person to be a volunteer or part time employee at the center who has established ties to the center and the community, thereby facilitating communication between the parents and the research team. The liaison will be encouraged to add the content to newsletter articles, blurbs in lunch menus, or information on the center website to raise awareness about the study. The liaisons will be provided with suggested content and materials to use in communications with parents.

• In an effort to mitigate non-response, interviewers are trained to identify potential sources of respondent reluctance or refusals as early in the field period as possible. Recruiting and other materials, including the parental phone scripts and talking points included in this Information Collection Request, can then be tailored to address these concerns. In addition, Abt Associates study staff will be available through a toll-free number to answer questions and provide guidance to interviewers as any issues arise.

The Provider Web Survey is designed to be self-administered and email invitations will be sent to respondents. Follow-up activities to encourage survey completion will include email reminders and telephone follow-up calls. The Menu Survey is also designed to be self-administered using a paper instrument. Respondents will be offered training via a training video and a toll-free “help” number will be provided for respondents to call with any questions or problems. As described in A.9, respondents will be offered an incentive to encourage cooperation.

All other survey instruments will be completed in-person during the on-site data collection period. As described above, data will be collected from child care providers, parents, and children by skilled and highly trained data collectors. Because they will be on-site, they will be able to follow-up as needed to encourage participation and obtain completed survey instruments. As discussed in A.9, incentives will be offered for completion of some of the on-site data collection activities.

If the cumulative provider response rate falls below 80 percent, we will conduct nonresponse analysis as required by standard Office of Management & Budget (OMB) guidance. At a minimum, we will compare responding vs. non-responding centers using data from the sampling frame (e.g., on poverty level, MSA/non-MSA status, CACFP/Head Start status/at-risk programs/afterschool programs), and compute nonresponse weights to adjust responses to reflect nationally representative proportions consistent with the American Community Survey and Childcare Aware of America (2012).

B.4 Test of procedures or methods to be undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

We conducted pre-tests of all data collection instruments and procedures with 9 or fewer respondents. While the instruments and procedures were adapted from instruments used in prior research studies, many that were validated previously, all were modified to some extent for use in the current study. We tested the instruments in a variety of types of child care providers, reflecting the diversity expected in the study. We revised all instruments to incorporate pre-test findings.

B.4.A Provider web survey

Eight child care providers participated in the pre-test of the web survey. They included three child care centers, one Head Start center, two FDCHs, and two afterschool programs. We e-mailed the survey to respondents and asked them to return it when completed. We conducted telephone debriefings to determine any unclear or difficult questions and to solicit their recommendations on other changes to improve the instrument.

B.4.B Nutrition and wellness on-site data collection

The nutrition and wellness on-site data collection involves seven instruments: Menu Survey; Meal Observation Booklet; Infant Food Intake Form; Environmental Observation Form; Standing Height and Weight Form; Child Food Diaries; and Parent Survey. We conducted in-person pre-tests of all relevant forms in one CACFP center and one Head Start center. This allowed a complete test of all instruments and procedures, permitting us to assess the on-site data collection activities as a whole. In addition, we pre-tested a subset of the instruments in one child care center, two FDCHs and one Afterschool program. No one question was tested on more than nine respondents. We conducted in-person debriefings with all respondents to assess difficulty and clarity of the questions.

B.4.C Cost on-site data collection

We pre-tested all cost data collection instruments and procedures in three child care settings, one each in a CACFP center, Head Start Center, and Afterschool program. Prior to the on-site portion of the pre-test, we conducted the pre-visit survey by telephone and asked respondents to complete the Pre-Visit Forms and return them via mail. We tested all other instruments in-person over the course of one day. We conducted debriefings with all respondents at the completion of each interview.

B.5 Individuals consulted on statistical aspects & individuals collecting and/or analyzing data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Name	Affiliation	Telephone Number	e-mail
Susan Bartlett	Project Director, Abt Associates Inc.	617-349-2799	[email protected]
Kristen Copeland	Principal Investigator, Pediatrician, Cincinnati Children’s Hospital Medical Center	513-636-1687	[email protected]
David Judkins	Sampling Statistician, Abt Associates Inc.	301-347-5952	[email protected]
K. P. Srinath	Sampling Statistician, (retired)	na	na
Lauren Olsho	Director of Analysis, Abt Associates Inc.	617-520-2326	[email protected]
Evan Schulz	National Agricultural Statistics Service (NASS)	202-690-8640	[email protected]

References

Child Care Aware of America. (2012). Parents and the high cost of child care: 2012 report. Arlington, VA: Author.

Glantz, F.B., Rhoda, D.T., Cutler, M.J., Rhodes, W., & Wrobel, M. (1997). Early Childhood and Child Care Study: Profile of participants in the CACFP. Final Report submitted to the U.S. Department of Agriculture, Food and Consumer Service. Cambridge, MA: Abt Associates.

Judkins, D., and Hidiroglou, M. (2004). Double sampling in a multi-stage design. Proceedings of the Section on Survey Research Methods of the American Statistical Association, pp. 3743-3749.

Ponza, M., et al. (2014). School nutrition and meal cost study plan. Unpublished deliverable submitted to Food and Nutrition Service, U.S. Department of Agriculture, John Endahl, project officer.

U.S. Department of Agriculture and U.S. Department of Health and Human Services. 2010. Dietary guidelines for Americans, 2010, 7th ed. Washington, DC: U.S. Government Printing Office.

1 The title of the study was published in the 60-day notice as the “Study on Nutrition and Wellness Quality in Childcare Settings (SNAQCS)”; however, for participant recruitment materials and surveys, the study will be referred to as the Study of Nutrition and Activity in Child Care Settings (SNACS) since this title is more consumer friendly.

2 A small fraction of Head Start Centers does not participate in CACFP. Based on frame building activities in 20 sampled states, this fraction is about 3.4% of all Head Start Centers. Only those that participate in CACFP will be selected for SNACS.

3 As shown in Table B.2, Panel B: total meal observations & dietary intake records for children (2,172) and infants (340) and meal observations only (947).

4 Alaska and Hawaii are excluded because they account for a very small percentage of providers and participating children and the cost of collecting on-site data in these States is prohibitive.

5 We would have preferred defining the MOS with respect to the number of children age 12 to align with the study population, but the Census Bureau does not release estimates for the number of children age 12 and under in households in poverty broken out by metro versus non-metro status. We therefore use children 18 and under as a proxy for those 12 and under.

6 The ECCS sampled at 4 stages: States, sponsors, providers, and children and families.