Appendix C1 Data Collection Summary

APPENDIX C1. DATA COLLECTION SUMMARY

C1.1 Overview

To measure certification, aggregation, and meal claiming errors, the data collection approach requires data collection from multiple sources in multiple phases. To ensure that household survey data are conducted within four months and the sample includes applications submitted over the course of the study year, field data collection tasks will be time sequenced in phases. Figure C1-1 presents a summary of the data collection activities that will take place during each phase for field data collection. The abstractions of data from the income eligibility applications, sampling of households and the conduct of the household survey will be conducted at all three phases, with a targeted focus during phase 1 since we anticipate that the majority of applications are submitted by October.

Center visits will commence in Phase 1 when feasible to improve efficiency. However, it is expected that center visits will continue through Phase 2. Center visits will focus on collecting data to estimate aggregation and meal claiming errors. Starting in Phase 2 and continuing through Phase 3 we will request that the sponsors complete the sponsor survey by mail. Phase 3 will begin in June 2017 and will include a request for additional administrative data from sponsors and centers.

Figure C1-1. Phases of EPICCS Data Collection

Phase 1					Phase 2					Phase 3
Oct	Nov	Dec	Jan		Feb	Mar	Apr	May		Jun	Jul
Sponsor/Center Visit & Household Wave 1					Sponsor/Center Visit & Household Survey Wave 2					Sponsor/Center Visit & Household Survey Wave 3
Pre-visit Interview					Application Abstraction					Application Abstraction
Application Abstraction					Sampling for Household Survey					Sampling for Household Survey
Sampling for Household Survey					Recruitment for Household Survey					Recruitment for Household Survey
Recruitment for Household Survey					Household Surveys					Household Surveys
Household Surveys										Sponsor Administrative Records
										Center Attendance Records
Center Visits for Meal Claiming and Aggregation										Center Administrative Records
Meal Observations
Center Meal Counts and Claims
							Sponsor Survey
							Mailed Sponsor Survey

Table C1-1 presents an overview of the approach for each type of error. Table A2-1 in Part A of the

Supporting Statement provides a complete list of the data collection instruments and forms.

Table C1-1. Overview of EPICCS data collection approach by type of error

Type of error	Sources of error	Key Data Collection
Certification	Administrative Error Household Reporting Error	Abstract Income Eligibility Application Data (Appendix C4 and Appendix C4a) In-person NAMES Survey (Appendix C25)
Aggregation	Center Meal Counting Sum of meals at classroom/table level differs from aggregated count submitted to Sponsor Center to Sponsor Center aggregated count is misreported to Sponsor (i.e., sponsor records for center differ from center records) Sponsor to State Sponsor aggregated count for the center is misreported from Sponsor to State (i.e. State records differ from sponsor records) State to USDA State aggregated count is misreported State to USDA (i.e. USDA records differ from State records)	Abstract center enrollment (Appendix C3) and center attendance for the target month (Appendix C8) for comparison against center meal counts Abstract center meal count records for target month (Appendix C9) Abstract center meal claims to sponsor for target month (Appendix C10) Abstract sponsor meal claim for center for target month submitted to State (Appendix C11 and Appendix C12) Obtain State meal claim records for sampled centers for target month (Extant data from FNS, States, and other sources) (Appendix C18)
Meal Claiming	Meals prepared do not meet the meal pattern requirements Meal content Meal amount (for number and age of children) Meals served do not meet the meal pattern requirements Meal content Meal amount (for number and age of children) Meals served to an adult or child who is not enrolled is claimed for reimbursement	Review Meal Production Records and Conduct Meal Observations (Appendix C5) or (Appendix C6)

Data collection for EPICCS will address the three types of erroneous payments: certification errors (administrative and household reporting), aggregation errors, and meal claiming errors. The data collected for certification errors will take place in three rounds (October 2016, February 2017, and June 2017) to ensure more complete coverage of households who submitted income eligibility applications over the course of the year. The following is a summary of the data collection activities and how the information will be used.

C1.2 Sponsor and Center Data Collection

C1.2.1 Pre-Visit Interview (Appendix C2)

Each sponsor-center pair varies in its operational characteristics and responsibility designations such as: licensed capacity, staffing ratio, daily schedule, records maintenance, and meal preparation and service. This information is relevant to ensure the center’s data collection visits can be customized and conducted in a time efficient manner. The pre-visit interviews with the sponsor and/or center will be used to: (1) identify the best location (sponsor or center) for collection of income eligibility applications; (2) schedule the data collection site visits at the convenience of the sponsoring organization and/or center; and (3) collect center’s characteristics for the data collectors to adequately prepare for on-site visits. Further, it will save time and reduce burden on the staff on the days of the data collection visits. Appendix C2 provides a copy of the pre-visit interview.

Whenever possible, information will be pre-populated in the subsequent data collection forms based on information obtained during pre-data collection activities and the pre-visit interviews. The pre-visit interviews will be conducted by telephone.

C1.2.2 Application Abstraction (Appendix C4)

Data from income eligibility applications will be used to assess certification errors due to administrative error and household reporting error. Categorical eligibility information may be included on the application. This will include indication of receipt of SNAP, FDPIR, or TANF benefits with a case number documented.¹ If CACFP eligibility is established through information sharing between State agencies, similar to direct certification, this documentation will also be abstracted.

While on site at the sponsor or center, the data collector will access children’s applications and categorical eligibility documentation from sampled centers. Data collectors will work with staff to access application or other eligibility documentation records and/or extract reports from electronic records (Appendix C4). Data collectors will scan records for later data entry. If time permits, data collectors may begin data entry into the data entry forms on the laptop computer while on-site. However, to minimize burden and time at the sponsor/center, data collectors will enter the much of the data off-site. Data collectors will abstract identification information, income eligibility information, and eligibility determination as follows.

• Identification Information: Child’s full name, date of application, household size, number of children covered by application, and whether the child is a foster child.

• Income Eligibility Information: Income data by source for all household members, whether household receives SNAP, FDPIR or TANF benefits and corresponding case number.

• Eligibility Determination: Sponsor/Center’s determination of household size, income, and certification status (free, reduced price, or paid), and reason for denial if applicable (income too high, incomplete application, etc.).

Data from the master list of enrolled children will also be abstracted to document the center’s official record of certification status for all children. This will be compared to the certification status recorded on the application. In addition, any changes in enrollment or certification status since the application date will also be abstracted. The application abstraction procedures will be repeated at each phase.

C1.2.3 Meal Claiming Data (Meal Observations) (Appendices C5 - C7)

During the child care center site visits, the data collectors will request the food production records (for scanning and later data entry). In addition, the data collector will conduct observations of breakfast and lunch meal service. The observation protocol will vary based on whether the meal is served restaurant/cafeteria style or family style (Appendix C5 and C6). Together, meal production records (plans for meals prepared) and observations (meals served) will be used to measure meal claiming accuracy (meal classification as reimbursable or non-reimbursable meals).

Meal claiming errors occur when meals prepared and/or served does not meet the meal pattern requirements and they are claimed as a reimbursable meal. Child care centers serve meals in one of two formats, as follows:

1. Restaurant/Cafeteria Style in which children are provided an individual tray or lunch box with their meal.

2. Family Style in which the meal for all children is placed on the table in serving dishes and children serve themselves from the main dish, with assistance from the supervising adult, as necessary.

The majority of centers use family style meal serving, which is encouraged because it can increase children’s acceptance of offered foods and their willingness to try new foods. For family style, meals served to children (i.e., made available on the table) are considered to be reimbursable if they meet the meal pattern requirements outlined in Federal Regulation §226.20. According to Food and Nutrition Services (FNS) Instruction 783-9, revision 2, for family-style meals to qualify as reimbursable, centers must place a sufficient amount of prepared food on each table to provide the full portions of each of the food components for all children at the table based on their age. The regulations also require that: “The family style meal service allows children to make choices in selecting foods and the size of the initial servings. Children should initially be offered the full required portion of each meal component. During the course of the meal, it is the responsibility of the supervising adults to actively encourage each child to accept service of the full required portion for each food component of the meal pattern. For example, if a child initially refuses a food component, or initially does not accept the full required portion of a meal component, the supervising adult should offer the food component to the child again”. For centers that serve meals family style, we will work with FNS to develop an algorithm to determine if the family style serving meets the reimbursable meal requirements.

The meals prepared must contain the appropriate meal components and amounts based on the number and ages of the children served. CACFP requires child care centers to complete daily food production records for each meal prepared to document fulfillment of food preparation requirements. The food production records are also often used to organize the way the cook plans and prepares meals. The records include the required meal pattern and food components and can be used to record the menu, serving sizes, types and amounts of foods prepared, and the number of children served. The required meal pattern includes an age appropriate serving of milk; fruits/vegetables; grains/bread; and meat/meat alternative (for lunch meals). Data collectors will scan the food production records for the target day (observation day) for later data entry of abstracted information. Using data from master list of enrolled children and attendance records, we will independently determine if the food production records reflect that the meal is reimbursable according to USDA meal requirements.

Meal observations will be conducted as a supplement to confirm that the appropriate food items and quantity was actually served at the table or child level. For family style, the data collector will observe and record the food items and quantity served at the table, as well as the number and ages of the children served. This will occur for a random sample of tables (if there are more than one). For cafeteria style, the data collector will observe and record the serving of each food item served on a random sample of children’s trays or meal boxes.

Some child care centers may have more than one location or table for serving meals (e.g., in each classroom or separate rooms) and/or multiple meal serving periods in each location. While the data collector will conduct observations during each meal period for breakfast and lunch, he/she will observe only one location or table during each meal period. In instances where there is more than one serving location or table in a meal period, the data collector will randomly select which serving location or table to observe using the sampling algorithm (Appendix C7). For family style meals, the serving location will be a single table.

We are measuring two components of the requirements for reimbursable meals: meals prepared and meals served. The distinction between meals prepared versus served is made because the meals are prepared at the center level, whereas the meals are served at the table level (for family style) or child level (for cafeteria style). If the meal is found non-reimbursable at the center level, then all meals served on that day are non-reimbursable. If the meal is found non-reimbursable at the table level, all meals served to children at the table are non-reimbursable. If the meal is found non-reimbursable at the child level, then the single meal served to the child is non-reimbursable. As an illustration, Table C1-2 and Table C1-3 present a depiction of various scenarios for meal claiming errors at centers that serve family style or cafeteria style.

Table C1-2 Hypothetical Center A (30 children, family style, two meal periods, one table per meal period with 15 children)

Scenario	Center meal production records meet requirement	Table 1 meets requirement	Table 2 meets requirement	Number of reimbursable meals	Number of non-reimbursable meals
1	Yes	Yes	Yes	30	0
2	No	Yes	No	0	30
3	Yes	No	Yes	15	15
4	Yes	No	No	0	30

Table C1-3. Hypothetical Center B (30 children, cafeteria style, two meal periods, 15 children at each meal period)

Scenario	Center meal production records meet requirement	Number of children served meals that meet requirement	Number of reimbursable meals	Number of non-reimbursable meals
1	Yes	30	30	0
2	No	30	0	30
3	Yes	25	25	5

When sampling of serving location is necessary for breakfast and/or lunch, the data collector will enter the following information about each serving location into an electronic meal transaction sampling algorithm on the laptop computer (Appendix C7): 1) number of meal periods at the serving location; and 2) approximate number of children served at the serving location. The sampling program will provide the data collector with the randomly selected serving location to observe for each meal period.

Our findings on reimbursable meals will be compared to the meal claim data for the target day (day of observation). In most cases, these data will not be available until several months later when the meal claims are submitted. We will follow-up with the center to request these data for comparison.

C1.2.4 Aggregation Data (Meal Count and Claiming) (Appendix C3, Appendices C8 – C12, and Appendices C16 – C18)

During data collection visits to the center and sponsor, the data collector staff will request meal counting and claims records for abstraction and entry in the electronic standardized data abstraction forms on the laptop computer. Errors can occur during the process of counting, consolidating, and claiming for meal reimbursements. These types of errors occur after the meal status is recorded as reimbursable, and are referred to as “aggregation errors”. They can occur at the center and/or the sponsor level. At the center level, errors can occur in adding up the number of meals served for a group at the center; adding up the daily count at the center; and/or adding the daily counts at the center to weekly or monthly levels. At the sponsor level, errors can occur in entering the incorrect amount for a center or totaling counts across centers and filling out and submitting the appropriate claims forms. There can be significant variations in this process across centers and sponsors, including the level of automation. This data will be used to measure aggregation error at the various stages of meal counting and claiming by comparing data from each stage/source.

We will collect data for each sampled center for a target month² for breakfast (if applicable) and lunch. Data will be collected from both the sampled centers and the sponsors for the same target month. The meal count and claim data will include reimbursable meal counts, number of children at each certification level (free, reduced-price, paid), enrollment, daily attendance, and number of serving days, to help assess the accuracy of the meal counts. Specifically, we will obtain records for the target month to abstract the following items (scanned for later data entry).

From Centers:

• Enrollment (Appendix C3)

• Attendance (Appendix C8)

• Center Meal Count records (Appendix C9)

• Center Meal Claims submitted to sponsor (Appendix C10)

From Sponsors³:

• Sponsor Meal Claims submitted to State (for sponsored centers) (Appendix C11);

• Center Meal Claims submitted to State (for independent centers) (Appendix C12)

From Extant FNS Data/States:

• State Meal Claim records for sampled centers. (Appendix C18)

The data will be collected from each source for the same target month. This data will be used to measure aggregation error at the various stages of meal counting and claiming by comparing data from each stage/source.

Child care centers will be asked to provide electronic data files or hardcopy reports with the following administrative information:

• Extended attendance records (Appendix C16) for children from the households who completed the survey. These data will be used to generate an estimate of certification error attributable to each household.

• Final meal count records (Appendix C17) for the month in which the meal observations were conducted.

C1.2.5 Sponsor Survey (Appendices C13 – C15)

The sponsor director, or other designated person, at the sponsoring organizations will be asked to complete a survey about their characteristics and the approaches taken in providing technical training, and monitoring of the child care centers under their purview. Appendices C14 and C15 provide copies of the sponsor surveys. In addition to including sponsor characteristics in the analyses, these data will be used to determine what impacts, if any, that specific organizational type, training, records management, or monitoring activities might have upon the rates of erroneous payments at the center level. The sponsor survey will be a mail survey, with a cover letter (Appendix C13).

C1.3 NAMES Survey Data Collection

C1.3.1 Recruitment for Household Survey

While on site at the sponsor or center, the data collector will obtain a list of children who, at the time of visit, were certified (as free, reduced-price, or paid) either by application or categorical eligibility. The data collector will process the list to exclude ineligible children and enter the child IDs and certification status into a sampling program on their laptop computer. They will also enter the number of children certified for free, reduced-price, and paid meals. The program will then select the sample of certified children, ensuring appropriate distribution of students certified for each eligibility status. After sampling, the household contact information will be collected for the sampled children, and entered in the sampled household contact information data entry form on the laptop computer. Following transmission of the sampling data from the field, the home office team will mail a recruitment packet to the sampled households within one business day. All recruitment and data collection materials will refer to the household survey component of the Study as the National Assessment of Meal Eligibility and Services Study (NAMES). The packet will include an introductory letter and study brochure (Appendices B16 and B18). Using the Recruitment Call Script (Appendix B20), the recruitment call will take place five to seven business days after the mailing. During the call, the data collector will provide information about the study, its importance and the incentive payment, explain how and why they were selected, answer questions, and schedule the in person household survey.

The data collector will also inform the respondent that income documentation will be requested during the in person interview, and provide examples of acceptable forms of documentation. The staff will also explain that an additional incentive will be provided if they collect and provide the income documentation. Further, a NAMES income worksheet (Appendices C23 and C24) will be sent via mail or email (if email address is provided) to help the respondent prepare for the income portion of the survey.

C1.3.2 NAMES Survey (Appendix C25)

The data collector will travel to the respondent’s home or other location if requested by the respondent to conduct the interview. Prior to the start of the survey, informed consent will be discussed and obtained. Appendices C19 and C20 provide a copy of the NAMES Survey Consent Form and Appendices C25 and C26 provide a copy of the NAMES Survey. During the interview, the data collector will refer to the household income worksheet (Appendices C23 and C24), if available, to move more efficiently through the interview. The appropriate incentive payment will be provided immediately upon completion of the survey. The survey will be conducted via CAPI, and transmitted to Westat within a few hours of completion. This information will be used to assess the accuracy of household reporting on the income eligibility application, measuring certification error due to household reporting error.

The interview is estimated to take 45 minutes to complete. During the in-person interview, the data collector will ensure that the respondent understands the definition of income, and request the documentation of income from paid work and benefits payment. During the interview, the data collector will refer to the household income worksheet, if available, to move more efficiently through the interview. The data collector and respondent will review the available documentation together during the interview. The appropriate incentive amount will be provided immediately upon completion of the survey. Lastly, the respondent will sign two copies of the appropriate incentive receipt (Appendices C29 and C30). One copy will be given to the respondent to keep for their records and the second copy will be retained by the study.

Collecting Eligibility Information

A key data element from the household interview is household size and income. We will follow a similar process used for APEC II for obtaining this information. The respondent will be asked to report on household size and income during the month in which the income eligibility application was submitted, which will be 1 to 4 months prior to the survey. The following is a summary of the process:

• Ask the respondent to list all persons in the household.

• Ask the respondent to specify the relationships among the household members and sharing of resources. This will yield information that will be used during the analysis to identify the economic unit for assessing the child’s eligibility.

• Ask the respondent to identify all the different sources of current income, by household members. Asking for the sources of income first, without asking for amounts or documentation, will encourage the disclosure of more sources.

• Ask for the specific amount of current income per person and source, including review of documentation for each amount and source.

• Ask for documentation for each income source. If the documentation is not for the application month, the data collector will document if it is for

• Current month;

• Between application month and current month;

• 1 to 3 three months prior to the application; or

• More than 3 months since application month

• Ask if the income amount (if not from application month) is about the same, less, or more than the application month.

• Ask questions about participation in SNAP, FDPIR, and TANF to determine categorical eligibility. Respondents will be asked to provide a current benefits statement to substantiate the benefits amount reported and data collectors will record case numbers from provided documentation.

• At the end of the sequence, income sources across all adults and sources will be summed to come up with a total monthly amount.

The survey data will be transmitted to Westat within a few hours of completion. This information will be used to assess the accuracy of household reporting on the income eligibility application, and the measuring certification error due to household reporting error. The household survey procedures will be repeated at each phase.

1 The documentation of case number on the income eligibility application is not required by all States. If it is not included on the application, the data collector will ask if other documentation is available. The household survey does request documentation (e.g., benefits statement).

2 The target month will be the most recent calendar month in which the center has submitted to the Sponsor/State. This can be one to three months prior to the data collection visit.

3 This will require a separate (remote) request from the sponsors after completion of the center data collection visit.

7