APPENDIX S. DATA COLLECTION SUMMARY
To successfully measure certification, aggregation, and meal claiming errors (see Table S1-1 for definitions), the data collection approach requires collecting data from four primary sources: (1) School Food Authorities (SFAs); (2) schools; (3) households (parents/guardians); and (4) States. Both quantitative and qualitative information will be collected through data abstraction, observations, web surveys, and participant interviews. Figure S1-1 outlines the timeline for data collection. Table S1-2 presents an overview of each data collection activity organized by data source, and the following text describes the data collection approach for each data collection task. Sections S.2 through S.5 summarize the data collections. Section S.6 summarizes the study management system (SMS) that tracks and manages the data. Appendices B1‒B14 include SFA and State data collection forms. Appendices C1‒C6 include the school data collection forms. Appendices D1‒D16 include the household survey data collection forms. Appendices I1 and I2 consist of the household survey consent forms.
Table S1-1. Definition of errors
Type of error |
Summary / Definition |
Certification errors |
|
Meal claiming errors |
|
Aggregation errors |
|
Data collectors will travel to each SFA, school, and household to collect study data. Data collectors will record data abstracted from hard-copy records directly to a web-based data entry form on a study laptop. At schools, data collectors will also conduct meal observations and in-person interviews with a sample of cafeteria managers. Using a MiFi internet connection, the data will automatically be saved and securely transmitted. The in-person household surveys will be conducted as a computer assisted personal interview (CAPI).2 Data collectors will securely transmit the CAPI data immediately after the completion of the survey. The SFA director interview and household in-depth interviews will be conducted by phone by home office staff. The interviews will be recorded and later transcribed, and the responses will be stored in the secure database. SFA directors will complete a web-based survey on their own. Administrative data from the SFAs and States will be requested electronically and submitted via a secure web portal.
To ensure that household survey data are both representative of applications submitted throughout the school year and conducted within 6 weeks of certification,3 field data collection tasks will be time sequenced in phases (see Figure S1-1). During phase 1, data collection efforts will focus on the household survey. This is because 66 - 80% of applications are certified in the fall, and thus the majority of the household surveys will be conducted in phase 1. The process of sampling for the household survey will take place at three intervals during the school year: fall (August 2017‒November 2017), winter (December 2017‒February 2018), and spring (March 2018‒June 2018). Based on APEC II findings on when applications were submitted, approximately 66 percent of the students will be sampled in the fall, 32 percent in the winter, and 2-3 percent in the spring.
Figure S1-1. APEC III Data Collection Schedule
Phase 1 |
|
Phase 2 |
|
Phase 3 |
August 2017 to November 2017 |
December 2017 to February 2018 |
March 2018 to June 2018 |
||
Household Wave 1 |
SFA/School Visit & Household Survey Wave 2 |
SFA/School Visit (continued) & Household Survey Wave 3 |
||
Sampling for Household Survey |
Sampling for Household Survey |
Sampling for Household Survey |
||
Household Surveys |
Household Surveys |
Household Surveys |
||
Household In-Depth Interviews |
Household In-Depth Interviews |
Household In-Depth Interviews |
||
|
SFA Pre-Visit Interview |
SFA Pre-Visit Interview |
||
|
Application Abstractions |
Application Abstractions |
||
|
CEP Records Reviews |
SFA Meal Participation Data Request |
||
|
|
|
||
|
SFA Director Web Survey and Phone Interview |
SFA Director Web Survey and Phone Interview (continued) |
||
|
|
|
||
|
School Visits for Meal Claiming and Aggregation Data Collection |
School Visits for Meal Claiming and Aggregation Data Collection (continued) |
||
|
School Pre-Visit Interview |
School Pre-Visit Interview |
||
|
Meal Observations |
Meal Observations |
||
|
Cafeteria Manager Interview |
Cafeteria Manager Interview |
||
|
|
School Meal Counts and Claims |
School Meal Counts and Claims |
|
|
|
|
|
|
|
|
|
|
State Meal Claim Data Request |
As shown in Figure S1-1, SFA data collection will commence in phase 2, and continue in phase 3 as needed. However, sampling for the household survey will take place during each phase to correspond with the household survey. The SFA director survey and SFA director in-depth interviews will occur during phases 2 and 3. In addition, SFA administrative records will be obtained in phase 3.
A brief pre-visit interview will be conducted with the SFAs to plan for data collection, schedule the visits, and address any questions or concerns they may have. The interview will identify operational characteristics and schedules for SFAs. This information is relevant for ensuring that data collection is customized as needed and conducted efficiently. The SFA pre-visit interview (Appendix B5) will be conducted by telephone with email follow-up as needed.
The sampling at the student level for the household survey (for non-CEP schools) will be conducted in all three phases. For non-CEP schools, the sample will be drawn from the administrative list of applicants or directly certified students for the current school year from the sampled schools. An electronic file with the administrative data will be requested from the SFA in order to conduct the sampling (Appendices B1 and B2).
Table S1-2. Data collection activity/Instrument by Source
Source |
Activity/Instrument |
Mode of Data Collection |
Key Data Elements |
SFA |
Pre-Visit Interview |
|
|
Sampling for Household Survey Protocol |
|
|
|
Application Abstraction Form |
|
|
|
CEP Records Review Form |
|
|
|
SFA Meal Claim Abstraction Form |
|
|
|
SFA Director Survey |
|
|
|
SFA Director Interview Guide |
|
|
|
Meal Participation Data Abstraction |
|
|
Table S1-2. Data collection activity/instrument by Source
Source |
Activity/Instrument |
Mode of Data Collection |
Key Data Elements |
School |
Pre-Visit Interview |
|
|
Meal Observation Forms (Breakfast and/or Lunch) |
|
|
|
Cafeteria Manager Interview Guide |
|
|
|
School Meal Count Abstraction Form |
|
|
|
Parents / Guardians |
Household Survey |
|
|
Household In-Depth Interview Guide |
|
|
|
State |
State Meal Claims Abstraction Form |
|
|
While we expect to receive the files electronically from all SFAs, a manual backup form has been developed for the rare instances in which the data collector must conduct the sampling in person (Appendices B1 and B3).
As part of the household sample selection, the sampling program will sort the students by certification status to ensure proportional representation on this measure within a school. The sampling algorithm will sample approximately 16 students per school4. For only the students who are sampled, the data collector will record contact information for that student’s household (obtained from school records or the application) into the laptop computer, and that information will be securely transmitted electronically to the contractor daily. The household will be contacted via mail and phone to be recruited for the household survey.
Household application data abstraction will take place at the SFA in phases 2 and 3 (non-CEP schools). Application abstraction data will be used to determine certification error due to administrative error and household reporting error. After sampling, the data collector will abstract key data elements from either the hard-copy application or the electronic record for each sampled household and enter the data directly onto a web-based data entry form on the laptop computer. Using a MiFi internet connection on their computer, data collectors will enter data into the Application Data Abstraction Form (Appendix B7) that will be automatically saved and securely transmitted. In the event that internet connectivity is not available, data will be entered on hard copy and later entered into the web-based data entry form.
For CEP schools, we will review administrative data to draw the sample of students.5 In sampling student records within participating CEP schools, the targeted number of students is 24 per SFA. From the SFA, we will request the electronic file with the list of students for the year on which the claiming percentages are based (i.e., the base year) (Appendices B3 and B4). From this data file, the analytic team will identify and sample students who are classified as “identified students” and “non-identified” students.
For CEP schools, a records review will be used in phase 2 to determine certification error due to incorrect identified student percentage (ISP) or documentation error. Using a matching technique similar to the one used in APEC II,6 we will attempt to match a sample of identified students to SNAP, TANF, or other program data sources (depending on the sources the SFA used) to calculate an independent estimate of the ISP.
To conduct the analysis of ISP, we will request electronic data sets from the SFA (or State) with student-level data for all schools represented in the CEP groups in the SFAs with CEP schools. These data sets will include enrollment information for both identified and non-identified students from the base year in which the CEP ISP was determined. We will also request the SNAP, TANF, or other program data files that were used to identify students eligible for meal benefits in the base year. The analysis team will use these data sets to estimate certification error in CEP schools.
We expect the student-level data set to include the following variables (where available) to facilitate the matching:
Administrative identifiers such as SNAP or TANF case ID
Names: Student and parent first, middle, and last name
Student date of birth
Address: Street address, apartment or unit number, city, state, zip code
Phone numbers: Home, work, and/or mobile
Demographics: Gender, race, ethnicity
If the data are not available in electronic format from the SFA (or State) the data collector will attempt to obtain hard-copy records, if available, from the SFA during the site visit (Appendix B3). The data collector will make a copy and ship the records to the Westat home office via FedEx, or scan the records and upload the file to a secure FTP site maintained at Westat.
SFA meal claim data will be used to determine aggregation errors at the SFA level. Aggregation errors occur in the process of tallying the number of meals served each month (by claiming category), transferring it from school to SFA, from SFA to the State, and from the State to USDA for reimbursement. Aggregation error may occur at one of four points:
Point-of-sale aggregation error;
School reports of meal counts to the SFA;
SFA reports of meal reimbursement claims to the State agency; and,
State claims submitted to USDA.
Data collectors will abstract information from administrative data to identify any aggregation errors (Appendix B9 or B10). For SFA reports of meal reimbursement claims to the State agency, we will obtain meal claim records for sampled schools for the target month to identify if the SFA aggregated count for the school is misreported from the SFA to the State. Data will be entered into the SFA Meal Claim Abstraction Form on their computer, saved, and securely transmitted. These data will be compared electronically to aggregation data collected from other sources to identify discrepancies.
Data will be collected from the SFA directors using a web survey during phases 2 and 3 (Appendix B11). The SFA director survey provides contextual information on SFAs and their processes. The survey will include questions on the SFA’s policies, procedures (e.g., training), and characteristics (e.g., technology level) that may be related to error rates or to understanding the errors. In addition, the survey will focus on questions about the SFA’s characteristics, participation in meal programs, certification processes, training, school information, CEP schools, and claiming percentages information. SFAs will be provided with an access code to complete the survey on a user-friendly website. Survey responses will be stored in a study management tracking system.
In addition to the survey, semi-structured interviews will be conducted (Appendix B12) with a random sample of 60 SFA directors on how SFA policies, procedures, and characteristics may affect errors, as well as what actions would be most effective in reducing errors. The interviews will be conducted by phone, audio recorded (with the SFA director’s permission) and later transcribed. The transcriptions will be stored in the secure database.
The SFA Director Interview will be conducted by researchers with experience in qualitative research methods. In addition, the interviewers will be trained on project objectives and the interview protocols. The [home office] interviewer will contact the SFA Director by email and/or phone to schedule the SFA Director Interview at a time that is convenient for him/her.
In addition to application data, meal participation data will be requested via electronic data file transmissions during phase 3 (Appendix B13). The data request will include meal participation data and any changes in certification status or enrollment. Meal participation data will provide information on how many meals the sampled students received during the school year. This will be used to estimate payment errors. Also, since student certification status may change during the school year, the SFA will be asked to provide an updated list to identify students with a change in certification status. Data will be submitted via a secure web portal.
School data collection includes a school pre-visit interview (Appendix C1), meal observation (Appendix C3), a cafeteria manager interview (Appendix C5), and school meal count abstraction (Appendix C6).
A brief pre-visit interview will be conducted with the schools to plan for data collection, schedule the visits, and address any questions or concerns they may have. The interview will identify operational characteristics and schedules for schools. This information is relevant for ensuring that data collection is customized as needed and conducted efficiently. The school pre-visit interview (Appendix C1) will be conducted by telephone or email, based on the preference and availability of the school principal.
Meal observation data will be used to determine meal claiming errors by identifying meals incorrectly claimed as reimbursable based on meal components and/or meal recipient. To identify meal claiming errors, data collectors will use school meal schedules to randomly sample and observe meals during each breakfast and lunch period. The purpose of meal observations is to record meal components on the student’s tray and whether the cashier recorded the meal as reimbursable. There are a number of different systems used by schools to offer their students reimbursable meals. For example, a school may have cafeteria-style service or meals in the classroom. At the senior high school level, “offer versus serve” (OVS) is required at lunch unless the school or SFA demonstrates to the State agency that their system does not accommodate OVS. Schools are strongly encouraged to modify their meal service system to incorporate OVS to the extent possible.
Data collectors will randomly select breakfast and lunch meal periods and serving lines for observation using school meal schedules and a sampling algorithm. The sampling algorithm will be on the data collector’s laptop and determined in real time at the school. The sampling algorithm will ensure that a random sample of cashiers and meal periods are sampled for observation (see sample in Appendix C4). At the meal level, data collectors will observe and record (1) items on each tray and the number of servings of each item; (2) whether the meal was served to a student or a non-student/adult; and (3) whether the cashier recorded the tray as a reimbursable meal.7 Data collectors will also record whether the school is offer versus serve, which informs criteria of reimbursable meals. OVS criteria requires at least three food items are selected (one serving of fruit or vegetable and two other food items for breakfast and one serving of fruit or vegetable and two other food items for lunch). Therefore, the criteria for reimbursable meals will be adjusted accordingly for OVS schools. Following the same protocol from APEC II, data collectors will record meal observations in an easy-to-use hard-copy booklet (see Appendix C3) and then transfer the data electronically on the laptop computer to upload after each day of observation.
During the school visit, the data collector will conduct an in-depth interview with a sample of 60 school cafeteria managers to understand what contributes to sources of meal claiming errors. The interview includes questions about the cafeteria design and layout, cashier practices, training cashiers, meal counting and claiming processes, and understanding challenges for ensuring all meals served are reimbursable (Appendix C5). Cafeteria managers will also be asked for any suggestions on how to improve the process of creating and recording reimbursable meals as well as the meal counting and claiming process. The interviews will be audio recorded, with the cafeteria manager’s permission, and then transcribed. The transcriptions will be stored on the secure database.
Cafeteria managers will be selected for the interview to be conducted as part of their school data collection visit. All schools that agree to participate in the study will also agree to the cafeteria manager interview with the understanding that their cafeteria manager may or may not be selected.
School meal count abstraction will be completed during the visit. School meal count data will be used to determine aggregation errors at the school level. As previously mentioned, aggregation errors occur in the process of tallying the number of meals served each month (by claiming category), transferring it from school to SFA, from SFA to the State, and from the State to USDA for reimbursement. Data collectors will abstract meal counts by cashier/school and by the meal counts submitted to the SFA (Appendix C6). These will be compared electronically to identify discrepancies.
Parents and guardians in sampled households will complete an in-person household survey. The survey data will be used to determine any certification error due to household reporting error. A random sample of 60 households will also complete an in-depth qualitative interview by phone that will provide further insight into the experience of completing an application. Gathering data in a qualitative format will allow for greater understanding on application sections that may be confusing to the household as well as how the household defines members of the household and reportable income.
Once a household has been sampled into the study, a recruitment packet will be mailed to the sampled household from the home office. The study management system (SMS) will provide automatic alerts upon receipt of transmitted sampling data,8 and the household contact information will be obtained from the data entered in the SMS.
Recruitment packet. Recruitment letters packets will be mailed within a few business days after a household has been sampled. The recruitment packet will introduce the household survey component of the study and encourage participation in the in-person household survey. The recruitment letter and brochure provide a summary of the study, explain benefits of participation, and provide information about the incentive for participation. The letter alerts parents/guardians that they will be contacted by phone by a data collector within the next few days. The letter will include the study’s contact email and toll-free number, encouraging parents/guardians to contact the study team if they have questions or concerns. Household recruitment materials are included in Appendices T1‒T8.
Recruitment phone call. Data collectors will conduct a follow-up recruitment phone call within 3 to 5 days after the mailing to recruit the selected households to participate in the study. They will provide information about the study and its importance, how and why the students were sampled, why parent/guardian participation is important, and will answer any questions about the study and their participation. The discussion will also include information about how the data will be collected, what type of documentation they will be asked to provide, and data privacy. Also, during the recruitment call, the data collector will inform the respondent that income documentation will be requested during the in-person interview and provide examples of acceptable forms of documentation.
Maximizing response rates by gaining the cooperation of APEC III household respondents is critically important to the study’s success. The majority of the APEC III data collectors will be experienced data collectors, particularly in recruiting for and conducting surveys. In addition, data collectors will be trained on gaining respondent cooperation based on five key principles:
Knowledge About the Study. Become an expert about APEC III.
Active Listening. Pay attention to what the respondent is saying; listen carefully.
Identifying Respondent Concerns. Diagnose the concerns of the respondent.
Immediately Selecting a Response. Have a large and ready supply of responses “in stock” to match various respondent concerns.
Delivery of Response. Tailor what is said. Use words appropriate to that respondent; deliver information relevant to the concern.
Data collectors will also employ refusal conversion techniques with reluctant individuals. Data collectors will often not reach a household interview respondent on the first, or even second, try. Data collectors will be trained to utilize the following approaches to reaching the parent or guardian:
If there is no answer nor voicemail, data collectors should then try again at a different day and time of day.
If there are multiple telephone numbers listed for a respondent, they can try a different number.
For subsequent contact attempts, allow a few days between calls and vary the day of week and time of day of the calls.
Avoiding leaving a voicemail message at every attempt. A general guideline is to leave a message on the first contact and then again on the fifth contact.
If unsuccessful after several contacts, data collectors may consider an alternative method such as texting or emailing the respondent, if contact details are available.
If unsuccessful after five contacts, discuss with the field supervisor. Supervisors can draw on their experience to suggest some new strategies, including the possibility of trying some in-person contact attempts, and updating contact information.
For those households agreeing to participate, the data collector will schedule an appointment and email or mail the appointment reminder (Appendix D1/D2 (Spanish version)) and household income worksheet (Appendix D3/D4 (Spanish version)) to help the respondent prepare for the income portion of the survey. The data collector will explain that the respondent will receive a $30 Visa gift card for completing the household survey and an additional $20 for providing income documentation. The data collectors will schedule an appointment to conduct the in-person household survey at a time that is convenient for the parent/guardian, which will include evening and weekend appointments. Data collectors will contact participants 1 to 2 days prior to remind them about the scheduled appointment.
Household Survey. The household survey will be conducted in person on a laptop computer using a computer assisted personal interview (CAPI) (D5/D6 (Spanish version)). The data collector will review and complete the consent form (Appendix I1/I2 (Spanish version)). The household interview takes 45 minutes to complete. During the household interview, the data collector will ask questions covering:
Enrollment status. Whether the student is still at the school.
Participation in the school lunch and breakfast programs. To what extent during the last full week of school, the student participated in the school lunch and breakfast program.
Perceptions of school meals. Child and parent satisfaction with the school meals.
Household composition. Names, age, gender, and relationships of the household members.
Income and earning sources. Household income from all sources (children and adults). In addition, the data collector will document the source of documentation.
Public assistance. Benefits received by household members through TANF, SNAP, or other benefits, such as FDPIR.
Total monthly income. The computer program computes the total income for the application month and the interview asks if the amount seems accurate. If no, the interviewer has the ability to change the income for various income sources.
Demographics. Demographics used for categorization purposes.
Obtaining income documentation for verification was a challenge in APEC II. This shortcoming, and methods for addressing it, was identified as a special study consideration by FNS. APEC III protocol includes an enhancement to income verification that will improve the quality and completeness of the data. The following procedures to improve the response rate of income documentation with the household survey will be implemented:
1. Inform parents in advance that income documentation will be requested during the in-person survey.
2. Provide examples (in advance) of the appropriate types of documentation that would be requested during the survey.
3. Emphasize that the documentation would be viewed and not removed from the home.
4. Provide respondents with a short worksheet to help them prepare for the income questions.
5. Provide a base incentive for completing the survey ($30), as well as an additional incentive ($20) for taking the time to gather and provide the income documentation. This results in a total a Visa gift card in the amount of $50.9
These steps will improve the rates of households providing income documentation during the survey.
After households complete the household survey, respondents will be asked if they are the person who completed the application for their household. If so, they will be asked if they are willing to participate in a telephone interview should they be randomly selected. Among those agreeing to be contacted, we will analyze household survey responses and application abstraction data to identify two groups of households—with and without reporting errors. Over the course of the 9-month data collection period, we will select an equal mix of households for a total of 60 households, and conduct in-depth interviews on a rolling basis.
In advance of the interview, the respondent will be sent a letter explaining the purpose of the interview, including the main points of consent, as well as contact information to discuss their rights as a participant and answer any study questions. The participant will also receive a blank copy of the meal application for the respondent’s district to refer to as a guide during the interview. Documents will either be emailed or mailed, depending on the respondent’s preference. Interviews will take place over the phone and will be conducted by home office staff with the household member who completed the application. Verbal consent will be obtained.
The interview will include questions on general experience with the application process to identify any areas that were confusing, if the respondent used a paper application or web-based version, and if any school staff or other knowledgeable individuals were available to answer questions (Appendix D13/D14 (Spanish version)). Interview questions will also include more specific items on identifying household members, listing income, defining child income, income frequency, language accessibility, and any suggestions for how to improve the application. Respondents will receive a separate incentive of $20 for participating in the household in-depth interview.
The interviews will be recorded, with the respondent’s permission, and then transcribed. The transcriptions will be stored in the secure database.
State data will be collected in phase 3 of the data collection period. State meal claim data will be used to determine aggregation errors at the State level. This claim data will be for the same month in which the data collector abstracted school and SFA meal claim data (i.e., the target month). As previously mentioned, aggregation errors occur in the process of tallying the number of meals served each month (by claiming category), transferring it from school to SFA, from SFA to the State, and from the State to USDA for reimbursement. Home office staff will collect the State meal claims submitted to USDA via an electronic data request (Appendix B14). The States will submit the meal claim data through a secure web portal. These data will be compared electronically to aggregation data collected from the school and SFA to identify discrepancies.
All recruitment and data collection activities will be tracked and managed in the customized study management system (SMS). The SMS is a secure, web-based system used to maintain and track all study activities. SMS security is enforced by strict user access hierarchies with password protection to enforce basic information security rules that customize each person’s access. A person’s access to information in the SMS will be limited to what is needed to accomplish his or her assigned responsibilities.
The SMS will allow study staff to view all sample information; conduct and monitor the recruitment of SFA’s to manage the release of the sample; schedule and manage data collection activities; generate study mailings; document the status of data collection activities for all respondents; and generate status reports. The SMS will be used to record data collection visits and household surveys. The SMS will provide the platform for tracking study ID numbers, launching study instruments and data entry forms, facilitating transmission of data, and housing all study materials to be accessed by study staff. Additionally, the SMS will facilitate information sharing between the home office team and field data collectors. This communication will be supplemented with email notifications and regular meetings conducted by field supervisors.
In summary, the SMS will serve as the primary management tool for tracking study progress and statuses in close to real-time over the course of recruitment and data collection. This will allow for rapid identification and resolution of any data collection and/or operations-related issues.
1 At the time of data collection, verification will most likely not have taken place. At the end of data collection, the study team will request participation data from SFAs for sampled students. At that time, we will check for any changes in certification status that may have taken place as a result of verification.
2 The CAPI does not require an internet connection to conduct the survey. However, an internet connection is needed to securely transmit the survey shortly after completion.
3 A fixed date will be established for each district.
4 On average 11 households per school are expected to complete the household survey.
5 The sample may include all students from the base year, or a sample of students, depending on the availability of CEP records.
6 The analysis team will establish the matching steps, and rules for combinations of matching variables sufficient for an acceptable match.
7 The data collector will be positioned close enough to the cashier to observe his/her transactions for each sampled student.
8 Data collectors will be required to transmit data at least once per day.
9 If any documentation is provided, even if it is not for all income sources, the respondent will receive the additional incentive.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Megan Collins |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |