OMB Part B_Revised 3-19-15 Clean

[Type text]

Contract Number: AG-3198-C-13-0012

Child and Adult Care Food Program Sponsor and Provider Characteristics Study

Project Officer:

Laura Zatz

Office of Policy Support

Food and Nutrition Service, USDA

Telephone: 703-305-2131

OMB Supporting Statement – Part B

Date Submitted: October 31, 2014

Table of Contents

Part Page

A. Justification SSA-1

Table of Contents

B. Statistical Methods SSB-1

List of Exhibits

A.1 Estimates of Respondent Burden Attached

A.2 Data Collection and Reporting Schedule SSA-11

B.1 Allocation of Sponsor Sample and the Level of Precision SSB-3

B.2 Allocation of Provider Samples and Anticipated Level of Precision SSB-4

B.3 Allocation of At-Risk Sponsor Sample and Anticipated Level of Precision SSB-4

B.4 Allocation of At-Risk Center Sample and Level of Precision SSB-5

List of Appendices

A Instruments A-1

A1 Sponsor Instruments A1-1

A1.1 Child Care Center Only Sponsors A1.1-1

A1.2 FDCH Only Sponsors A1.2-1

A1.3 Head Start Only Sponsors A1.3-1

A1.4 Mixed Sponsors A1.4-1

A1.5 At-Risk Sponsors A1.5-1

A2 Provider Instruments A2-1

A2.1 Child Care Centers A2.1-1

A2.2 Independent Child Care Centers A2.2-1

A2.3 Family Day Care Homes A2.3-1

A2.4 Head Start A2.4-1

A2.5 At-Risk Centers A2.5-1

A2.6 Child Care Centers – Spanish A2.6-1

A2.7 Independent Child Care Centers – Spanish A2.7-1

A2.8 Family Day Care Homes – Spanish A2.8-1

A2.9 Head Start – Spanish A2.9-1

A2.10 At-Risk Centers – Spanish A2.10-1

Table of Contents (continued)

Part Page

B Recruitment Materials B-1

B1 State CACFP Agency Letter B1-1

B2 Provider Sample - Sponsor Letter B2-1

B3 Sponsor and Provider Recruitment Letters B3-1

B4 Sponsor and Provider Brochures B4-1

B5 Provider Recruitment Letter and Brochure (Spanish) B5-1

C Federal Register Notice C-1

D Comments on the 60-Day Federal Register Notice D-1

E Response to Comments from the National Agricultural Statistics Service (NASS) E-1

F CACFP Sponsor and Provider Characteristics Study: Summary of Findings and
Recommendations from Pretest Cognitive Interviews F-1

Part B: Statistical Methods

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The CACFP Sponsor and Provider Characteristics Study includes two major components. The objective of the first component is to produce national estimates of the characteristics of all CACFP sponsors and providers that serve children. The objective of the second component is to produce national estimates of the characteristics of one important CACFP sub-group: sponsors and centers that participate in the CACFP At-Risk Afterschool Center Program.

Universe for CACFP

The study focuses on the child care component of the program, which in FY 2014 included 20,975 sponsors and 175,828 child care providers. Child care providers participate in the CACFP under the umbrella of a sponsoring organization that assumes fiscal responsibility and provides training and monitoring to ensure that its providers comply with all of the CACFP regulations.

The at-risk component of the CACFP allows centers serving children in low-income areas to receive CACFP reimbursements at the free reimbursement rate for a meal or snack for school-age children in care after school hours. This includes traditional child care centers providing care to preschool children during school hours and afterschool meals/snacks to school-age children, and non-traditional outside-school-hours centers. The survey will include a separate nationally representative sample of at-risk centers and their sponsors to provide information about the at-risk component of the CACFP. The universe for this component is a sub-group of the population of the main study of all CACFP Sponsors and Providers.

Sampling Frame for the Study

There is no list frame for the universe described above. To avoid building a list frame that covers the whole universe, which would be extremely expensive, we will use a cluster sample design, using States and DC (hereafter, DC will be treated as a State in reference) as the primary sampling unit (PSU). Using States as PSUs eliminates the need to construct a separate PSU sample frame. Cluster sampling designs using States as PSUs have been used in the four previous national studies of the CACFP. This study will select probability sample of States, and then a complete enumeration of CACFP sponsors and providers will be obtained for each sampled State by contacting the State agency that administers the program within the State. To select a sample of states more efficiently, this study will select States using sampling with probability proportional to size (PPS). The measure of size is discussed below. From the selected states, we will obtain list frames of sponsors and providers for October 2014¹ (the start of the program year). These data will be used as the second and third stages of sampling

Sampling Plan

The study will use a three-stage sampling design. The first stage will include a nationally representative sample of States. In the second stage we will select a representative sample of sponsors within each of the States included in the sample. The third stage will include a representative sample of providers from the sampled sponsors. However, for self-sponsoring independent centers we will use a two-stage sampling design where independent centers are selected from sampled States without going through the sponsor-stage. The first-stage sample of States will be used for both components of the study. However, separate second- and third-stage samples will be selected for the main study component and for the at-risk component.

First-Stage Sampling of States

Since States are highly variable in terms of their size relevant for the study, we will use probability proportional to size (PPS) sampling. We need an appropriate measure of size (MOS), which has important implications for the sampling efficiency. One important consideration is that the sponsor types, as defined by the types of providers the sponsors administer, are very important subgroups. There are three sponsor types: child care center, Head Start center, and family daycare home (FDCH) sponsors. We need to ensure that the precision requirement is met not only for the entire sample but also for each subgroup.

Data from the FNS National Data Bank (March 2014) indicate that about 81 percent of sponsors are for child care centers, about 13 percent for Head Start centers, and about 7 percent for FDCHs. However, the distribution of each type of CACFP provider is very different from that of their sponsors. FDCHs account for 66 percent of all CACFP providers, and child care center and Head Start center sponsors account for 27 and 7 percent of providers respectively. Because of these uneven distributions for both sponsors and providers, we have chosen the number of meals served by CACFP providers as the MOS for selecting States as a compromise of conflicting sampling efficiency concerns. This information is available in the FNS National Databank for 50 States and DC. With this MOS we will need 23 States in the first-stage sample to meet the precision requirement. No MOS will be perfect for all sponsor and provider types because one MOS has to be used for different types with different distributions. The compromise MOS is more stable over time than MOS’s of individual types and highly correlated with them. The PPS method with some MOS was used in all of the previous national studies of the CACFP.

Main Study Component

Second-Stage Sampling of Sponsors. We will select a stratified sample of sponsors within each of the 23 States in the first-stage sample. The strata include: a) sponsors of child care centers; b) sponsors of Head Start centers; and c) sponsors of FDCHs.² In defining these strata, we will use FNS’ classifications for sponsors with more than one type of provider under their aegis. If a sponsor has at least one Head Start center, it will be classified as a Head Start sponsor. Similarly, if a sponsor has at least one FDCH, it will be classified as a FDCH sponsor. Note that independent child care centers (ICCCs) will be included in the provider sample discussed below.

FNS has established an upper bound of ±10% for 95 percent confidence intervals for estimating a population proportion of 50 percent. The sample sizes for this study will provide confidence intervals of ±8.5%. Exhibit B.1 shows the allocation of the second-stage sample of sponsors for the main study component. These sample sizes assume design effects (Deffs) similar to those of the 1997 study (Glantz, et al., 1997) and 90 percent sponsor response rate.³ We have assumed a high response rate of 90 percent because the survey is mandatory. However, if the response rates are only 85 percent, the expected margin of error will still be comfortably below the maximum of ±10% set by FNS.

Exhibit B.1 Allocation of Sponsor Sample and the Level of Precision
Sponsor Stratum	Initial Sample Size	Response Rate	Number of Respondents	Design Effect	Margin of Error
Child Care Center	220	0.9	200	1.5	±8.5%
Head Start Center	300	0.9	270	2.0	±8.5%
FDCH	530	0.9	480	3.6	±8.5%
Total All Sponsors	1,050	0.9	950	2.3	±4.9%

The initial sample size for each sponsor type will be equally allocated to the 23 sampled States. Then, each sampled State will be stratified by sponsor type, and a sample of the allocated number of sponsors will be selected by the PPS method using the square root of the number of providers sponsored as the MOS. The use of PPS sampling for the second stage represents a trade-off between the ideal approaches for CACFP sponsor and provider samples. The ideal sampling approach for the sponsors is an equal probability sampling method of sponsors within each state, whereas the ideal one for the providers (in the third stage) is the PPS method for selecting sponsors with the MOS defined by the number of providers for each sponsor and selecting a fixed number of providers from each selected sponsor. Using the square root MOS in the PPS method for selecting sponsors provides a compromise between the two conflicting ideals.

Third-Stage Sampling of Providers. We will select providers from within the second-stage sample of sponsors. Nesting the provider sample within the sponsor sample will allow us to conduct linked analyses of sponsor and provider characteristics.

As for the sponsor samples, the sample sizes for the provider samples will provide 95 percent confidence intervals of ±8.5%. The sample sizes assume Deffs of 1.5 and response rates of 80 percent (similar to those for the 1997 study). The allocation of the provider samples is shown in Exhibit B.2. We will obtain one provider from each of the respondent center sponsors. To select the same number of providers as the sponsor sample size, we need to select only one from each sampled sponsor. However, we expect that we will need more than one provider for some sponsors to reach the target center respondent sample sizes.

To obtain the desired level of precision for FDCHs we need a total of 400 completed surveys. However, with an anticipated 80 percent response rate, we need an initial sample of 500 FDCHs selected from 480 respondent FDCH sponsors. Therefore, we need to select more than one FDCH from some sponsors. We decided to select 2 FDCHs from the largest 20 sponsors and 1 FDCH from each of the other 460 sponsors in the sample. This would help stabilize the overall sampling probabilities, which is desirable, because the distribution of FDCH sponsors in terms of the number of FDCHs is highly skewed and the squared root MOS would tend to produce the sampling probability unstable for large sponsors.

Exhibit B.2 Allocation of Provider Samples and Anticipated Level of Precision
Sponsor Type Stratum	Sponsor Sample for Selecting Providers	Initial Provider Sample Size	Number of Respondents	Design Effect	Margin of Error
Child Care Center
Sponsored	200	250	200	1.5	±8.5%
Independent	NA	250	200	1.5	±8.5%
Head Start Center	270	340	270	2.0	±8.5%
FDCH	480	500	400	3.0	±8.5%
Total	950	1,340	1,070	2.5	±4.7%

At-Risk Study Component

We will obtain a complete enumeration of the at-risk centers (ARCs) and their sponsors from each of the 23 State CACFP agencies included in the study. Some child care centers serve meals/snacks only as part of the afterschool at-risk component of the CACFP, while others serve meals/snacks in both the regular and at-risk components of the CACFP. To simplify the terminologies, a traditional child care center with an AR component will be referred to as a Mixed-AR center. A sponsor associated with any combination of traditional centers, Mixed-AR centers, and AR-only centers will be referred to as a Mixed-AR sponsor; whereas a sponsor who is only associated with at-risk-only centers will be referred to as an Only-AR sponsor.

Exhibit B.3 Allocation of At-Risk Sponsor Sample and Anticipated Level of Precision
At-Risk Sponsor Stratum	Initial Sample Size	Response Rate	Number of Respondents	Design Effect	Margin of Error
Only At-Risk	340	0.9	306	2.3	±8.5%
Mixed At-Risk	340	0.9	306	2.3	±8.5%
Combined	680	0.9	612	2.3	±6.0%

Second-Stage Sample of AR Sponsors. We will select a stratified sample of AR sponsors in each of the 23 first-stage sample of States. Two strata will be used: Mixed-AR sponsors and Only-AR sponsors. Within each State AR sponsors will be selected using probability proportional to size (PPS) sampling, with the positive square root of the number of AR centers as the MOS. As for the main study component, the AR sponsor sample sizes will provide 95 percent confidence intervals of ±8.5%. Because there are no prior studies of AR sponsors or centers on which to base estimates of the Deffs, to be conservative we assumed a Deff of 2.3 (somewhat higher than the Deff for Head Start centers) for each type of AR. As for the main study component, we have assumed a 90 percent response rate for AR sponsors. The allocation of the AR sponsor sample is shown in Exhibit B.3.

Third-Stage Sampling of AR Centers. Assuming the same Deff of 2.3 as for the AR sponsor samples and a response rate of 80 percent, we determine the required sample sizes to achieve the same level of precision for each stratum of sponsored AR centers. We take into consideration in the sample size determination that some Mixed AR sponsors will sponsor traditional centers without an AR component and/or at-risk-only centers in addition to mixed-AR centers. Therefore, the mixed-AR center stratum needs a larger initial sample to make up for the loss. We inflate the initial sample size by 10 percent for the Mixed-AR sponsor stratum.

For the sample of AR ICCCs, we will select directly from the sample frame of AR ICCCs provided by each sampled State. Because the AR-ICCC sample will be selected directly from States, we optimize the design by allocating the AR-ICCC sample across sampled States proportionally to the ratio of the number of AR ICCCs to the MOS used for State selection in each sampled State. This helps reduce the effect of variable weights on the Deff, so we use a moderate Deff of 1.5 for ICCCs. The sample allocation for the AR provider strata is summarized in Exhibit B.4.

Exhibit B.4 Allocation of At-Risk Center Sample and Level of Precision
Sponsor Stratum	Sponsor Sample for Selecting Providers	Initial Provider Sample Size	Number of Respondents	Design Effect	Margin of Error
At-Risk-Only SCCCs	306	383	306	2.3	±8.5%
Mixed At-Risk SCCCs	306	425	306	2.3	±8.5%
At-Risk ICCCs	NA	250	200	1.5	±8.5%
Combined AR-Sample	612	1,058	812	2.5	±5.4%

B.2 Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This is a one-time data collection effort with no unusual problems that require specialized sampling procedures. It has been 18 years since similar data were collected from sponsors and providers. Procedures for the collection of information addressed below include:

• statistical methodology for stratification and sample selection;

• estimation procedure; and

• degree of accuracy needed.

Statistical Methodology for Stratification and Sample Selection

The statistical methodology for stratification and sample selection was discussed in Section B.1.

Estimation Procedures

National estimates of the prevalence of CACFP sponsor and provider characteristics will be derived from the study sample. Confidence intervals around each estimate will account for sampling variation.

Degree of Accuracy Needed: Precision, Statistical Power, and Minimum Detectable Differences

Levels of precision were discussed above in Section B.1. FNS is also interested in estimates of the difference between key subgroups of CACFP sponsors and providers. In this context, accuracy is defined in terms of minimum detectable difference (MDDs). The sample size for this study have minimum detectable differences (MDDs) of two standard deviations (about 10 percentage points) in prevalence rates between key subgroups with 80% statistical power and α₂ = .10.

B.3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Based on the 1997 CACFP Sponsor and Provider Characteristics Study, we estimate a 90 percent overall response rate for the Sponsor Survey and an 80 percent overall response rate for the Provider Survey. We do not anticipate a problem obtaining these response rates. The major factor ensuring high response rates is that participation in the survey is not voluntary. HHFKA stipulates that “States, State educational agencies, local educational agencies, schools, institutions, facilities, and contractors participating in programs authorized under this Act and the Child Nutrition Act of 1966 (42 U.S.C. 1771 et seq.) shall cooperate with officials and contractors acting on behalf of the Secretary, in the conduct of evaluations and studies under those Acts” as a condition of receiving funding. We plan to reference HHFKA in the invitation to respondents. In addition, the Sponsors Forum and the National CACFP Sponsors Association (the two national associations of CACFP sponsors) have endorsed the study and are encouraging the members to participate in the study if they are selected into the sample. We have also made every effort to minimize the burden placed on web-survey respondents. The web survey allows respondents to work within their schedule by starting and stopping completion of the survey as often as they need.

Key respondent refusal factors include adverse reaction to “introductory” materials and contacts; lack of interest in study aims; lack of adequate incentive for participation; cultural barriers; inadequate training of data collectors; and task demands of study participation (e.g., length of instrument and record gathering time). To reduce the potential for nonresponse bias, a wide array of strategies will be utilized and are presented below.

Outreach

The contractor has introduced the study at several national conferences whose primary audiences are CACFP sponsors and providers. The presentation included information on study background, role of sponsors and providers, benefits to sponsors and providers, and a study timeline. The feedback from conference attendees has been extremely positive. In addition, the two major national CACFP sponsor associations, The CACFP Sponsors Forum and The National CACFP Sponsors’ Association, support the study and are encouraging their members to participate if selected into the sample and to encourage their providers to participate if selected.

Once the sponsor sample is drawn, the relevant CACFP State Agency will be provided with the names of the sponsors who have been selected for the survey. They will be asked to notify and encourage their sponsors to complete the survey. Similarly, sponsors will be provided with the names of their providers that have been selected for the survey. They will be asked to encourage their providers to participate in the study.

Instrument Design and Data Collection Mode

Each instrument has been designed in a user-friendly manner that minimizes complicated skip patterns and encourages participation and survey completion. Instruments were pretested and revised based on comments from the pre-test sponsors/providers. All provider instruments will be available in English or Spanish. Respondents will also have the option of completing a paper instrument, web version, or completing the survey via telephone with a data collector who has received extensive project-specific training, including training on refusal avoidance techniques. Providing different modes of data collection allows the respondent to select the approach with which they are most comfortable, thus increasing the likelihood that they will participate. The contractor is also offering a toll-free help line and dedicated email account, providing an opportunity for respondents to immediately reach out for assistance, when desired.

Recruitment

In addition to the questionnaire, each data collection package will include an introductory letter, customized brochure, and endorsement letters (sponsors only). Sampled sponsors and providers will receive a customized brochure and cover letter introducing their respective component (sponsor or provider) of the study. The brochure (Appendix B) will explain the study objectives and the importance of their participation; provide instructions for completing the survey, confidentiality assurances, and information on how to seek assistance. Sponsors will also receive study endorsement letters from The National CACFP Sponsors Association and The CACFP Sponsors Forum.

Follow-up with Non-Respondents

Approximately two weeks after the initial mailing, a second data collection package will be sent to non-respondents. For sponsors and providers with an email address on the sample file, we will also send a reminder notice with a link to the URL for their survey. Two weeks after the follow-up mailing, bilingual telephone data collectors will begin calling non-respondents. At that time, they will attempt to complete the survey over the phone. To maximize the likelihood of making contact, up to 20 call attempts to respondents with incomplete questionnaires or unresolved issues will be made. These call attempts will be spread across a variety of days and times. The exact start date of the telephone follow-up may be delayed slightly if the volume of web and mail responses remains relatively high in order to minimize calling respondents who are in the process of completing the survey or who have imminent plans to respond using another mode. To ensure that multiple calls can be made to each sample case, and to maximize the chance of reaching respondents, the telephone field period will last approximately four weeks.

To maximize contact likelihood, the call scheduler will be configured to make up to 20 call attempts to unresolved sample cases and ensure that these attempts are spread across a variety of days and times.

Nonresponse Adjustments

In spite of the use of extensive refusal avoidance procedures, participant refusal is unavoidable. In order to ensure that the data are reliable and study estimates are nationally generalizable, the initial sampling weights will be adjusted for non-response. These weighting procedures will minimize the effects of nonresponse.

If response rates do fall below 80 percent we will conduct a nonresponse bias analysis. This will include comparing selected characteristics of responding sponsors and providers to the characteristics of non-responding sponsors and providers. Additional information such as total meals claimed, proportion of meals claimed at the free and reduced-price rates, and composition of race/ethnicity of children enrolled will be requested from the non-respondents.⁴ If there are significant differences for these variables between responding and non-responding sponsors and/or providers, we will report on any potential biases not adjusted for by the weighting adjustments.

B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

All instruments have been pre-tested using paper versions of the instruments. Each instrument was pre-tested with no more than 9 respondents, who were chosen from CACFP sponsors and providers in California and New York. Participants in the pre-test will not be included in the study sample.

The web implementation of the instruments was not pretested due to time constraints for development and the need to complete the Provider Survey before the end of the 2014-2015 school year.⁵ Findings from the pre-test of the paper instruments are summarized in Appendix F. However, the web versions of the instruments will be extensively tested in-house to ensure skip patterns are correctly set.

B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Name	Affiliation	Telephone Number	e-mail
Frederic Glantz	Project Director, Kokopelli Associates LLC	505.983.0785	[email protected]
Hyunshik Lee	Sampling Statistician, Westat	301.610.5112	[email protected]
David Marker	Sampling Statistician, Westat	301.251.4398	[email protected]
Theodore Macaluso	Senior Technical Advisor, Kokopelli Associates, LLC	703.620.0868	[email protected]
Andrew Dau	National Agricultural Statistics Service	202.720.6482	[email protected]

1 Because participation in the CACFP varies over the course of the program year, USDA considers October and March to be the most representative months for reporting purposes.

2 Note that at-risk centers are included in the study of all CACFP sponsor and providers but are not part of this stratification.

3 Surveys often include partial completes in the survey respondent data instead of discarding them because they contain useful information for analysis, which would otherwise have been wasted. For both the sponsor and provider surveys we count any survey in which the respondent has completed at least 2/3 of the items as a completed survey in our response rate calculations. The cutoff of 2/3 is arbitrary but a number between 0.5 and 1 is usually used to determine the cutoff, and 2/3 represents a value reasonably higher than 0.5.

4 The burden estimates shown in Exhibit A.1 include the burden of obtaining this additional information from non-respondents.

5 Due to the additional cost of re-programming the web versions following each revision of the instruments, web versions of instruments are typically not programed until the instruments have been finalized. Had this been done after FNS gave final approval before pretesting, it would have been impossible to obtain OMB clearance in time to complete the Provider Survey before the end of the 2014-2015 school year as planned.