Evaluation of the Pilot Project for Canned, Frozen, or Dried Fruits and Vegetables in the Fresh Fruit and Vegetable Program
Contract
Number:
AG-3198-S-14-0028
OMB Supporting Statement
Part
A: Justification
July 10, 2014
Project Officer: Allison Magness
Office of Policy Support
Food and Nutrition Service/USDA
Telephone 703-305-2098
This page has been left blank for double-sided copying.
CONTENTS
Part A: Justification 1
A.1 Explanation of circumstances that make collection of data necessary 1
A.2 How the information will be used, by whom, and for what purpose 1
1. Study objectives 2
2. Study design 3
3. State- and SFA-level data collection 9
4. School-level data collection 9
5. Interviewer-administered forms 12
6. Parent- and student-level data collection 13
A.3 Uses of improved information technology to reduce burden 17
A.4 Efforts to identify and avoid duplication 18
A.5 Efforts to minimize burden on small businesses or other entities 18
A.6 Consequences of less frequent data collection 19
A.7 Special circumstances requiring collection of information in a manner inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations 19
A.8 Federal Register comments and efforts to consult with persons outside the agency 20
1. Consultations outside the agency 20
A.9 Payments to respondents 21
A.10 Assurance of confidentiality 21
A.11 Questions of a sensitive nature 22
A.12 Estimates of respondent burden 23
A.13 Estimates of other annual costs to respondents 23
A.14 Estimates of annualized government costs 24
A.15 Changes in hour burden 24
A.16 Time schedule, publication, and analysis plans 24
1. Project time schedule 24
2. Publication plans 25
3. Plans for analysis and tabulations 25
A.17 Display of expiration date for OMB approval 31
A.18 Exceptions to certification statement 31
APPENDIX A. Study Objectives, Research Questions, Outcome Measures,
Data Sources, and Planned Analysis
APPENDIX B. Sample Respondent Burden Table
APPENDIX C. Data Collection Material for State CN and SFA Directors,
School Staff, Parents and Students (Study Introduction,
Survey Invitation, Surveys, Interviews and Forms)
APPENDIX C.1a. Email - Study Introduction, Survey Invitation –
State Child Nutrition (CN) Directors
APPENDIX C.1b. Survey – State Child Nutrition (CN) Directors
APPENDIX C.2a. Email – Study Introduction, Survey Invitation –
School Food Authority (SFA) Directors
APPENDIX C.2b. Survey – School Food Authority (SFA) Directors
APPENDIX C.3a. Email – Study Introduction, Survey Invitation –
Food Service Managers (FSM)
APPENDIX C.3b. Survey – Food Service Managers (FSM)
APPENDIX C.3c. Daily Meals Count Form
APPENDIX C.3d. School-Level Reimbursable Fruit and Vegetable Form
APPENDIX C.4a. Letter – Study Introduction – Principals
APPENDIX C.4a1. Frequently Asked Questions by Schools
APPENDIX C.4b. Telephone Script – Principals / School Staff
APPENDIX C.4c. Email – Survey Invitation – Principals
APPENDIX C.4d. Survey – Principals
APPENDIX C.5a. Letter – Study Introduction, Survey Invitation –
Classroom Teachers
APPENDIX C.5b. Survey – Classroom Teachers
APPENDIX C.6a. Letter/Email – Study Introduction, Schedule Interview –
Stakeholders
APPENDIX C.6b. Master Interview Protocol – Stakeholders
APPENDIX D. Other Data Collection Forms
APPENDIX D.1. Nutrition Education, Promotional Material and
Menu Collection Form
APPENDIX D.2. Classroom-Student Level Fresh Fruit and Vegetable Observation Form for Snack and Recall and Plate Waste
APPENDIX D.3. Prototype - FFVP/Monthly Reimbursement Claims Form
APPENDIX E. Consent, Surveys and Interviews
APPENDIX E.1a1. Letter – Study Introduction, Study Notification “implied consent” (English)
APPENDIX E.1a2. Letter – Study Introduction, Study Notification “implied consent” (Spanish)
APPENDIX E.1b1. Letter – Study Introduction, Instructions for “active consent” Form
(English)
APPENDIX E.1b2. Letter – Study Introduction, Instructions for “active consent” Form
(Spanish)
APPENDIX E.1c1. Telephone Script – Verbal Consent
(English)
APPENDIX E.1c2. Telephone Script – Verbal Consent
(Spanish)
APPENDIX E.2a1. Letter – Study Introduction, Survey Invitation – Parent
(English)
APPENDIX E.2a2. Letter – Study Introduction, Survey Invitation – Parent
(Spanish)
APPENDIX E.2b1. Survey – Parent
(English)
APPENDIX E.2b2. Survey – Parent
(Spanish)
APPENDIX E.3a. Administration Script – Assent and Survey – Student
APPENDIX E.3b1. Assent – Student
(English)
APPENDIX E.3b2. Assent – Student
(Spanish)
APPENDIX E.3c. Survey – Student
APPENDIX E.4a. Foods Eaten at School Today (FEST) Diary – Training Script
APPENDIX E.4b. Foods Eaten at School Today (FEST) Diary
APPENDIX E.4c. In-School Diary-assisted (FEST) Dietary Recall for Consumption in School only Using Electronic Multiple Pass Approach
APPENDIX F. Federal Register Notice
APPENDIX G. Public Comments
APPENDIX H. Response to Public Comments
APPENDIX I. Comments from the National Agricultural Statistics Service (NASS)
APPENDIX J. Confidentiality Pledge
This page has been left blank for double-sided copying.
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
Section 4214(c) of the Agriculture Act of 2014 calls for the Secretary of Agriculture to evaluate the Canned, Frozen, or Dried (CFD) Fruits and Vegetables pilot project in the Fresh Fruit and Vegetable Program (FFVP) (referred to hereafter as FFVP-CFD) to address the impact of giving high-need, low-income elementary schools greater flexibility to offer CFD fruits and vegetables in addition to fresh produce. The congressionally mandated evaluation is slated for school year (SY) 2014–2015, with schools to be selected by Food and Nutrition Service (FNS) in summer 2014. Schools selected for the pilot—elementary schools in low-income areas that have difficulty accessing, preparing, or storing fresh produce—will design and implement their pilots to best suit their needs. The Section 4214(c) legislation specifically states the pilot should evaluate:
“1. The impacts on fruit and vegetable consumption at the schools participating in the pilot project;
2. The impacts of the pilot project on school participation in the Program and operation of the Program;
3. The implementation strategies used by the schools participating in the pilot project;
4. The acceptance of the pilot project by key stakeholders; and
5. Such other outcomes as are determined by the Secretary.”
Indicate how, by whom, how frequently, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
The Agriculture Act of 2014, Section 4214, calls for the Secretary of Agriculture to carry out an evaluation of a pilot project in elementary schools participating in the FFVP that can show issues with access to fresh fruits and vegetables and that may need flexibility in offering CFD varieties to supplement fresh produce in FFVP snacks in SY 2014–2015. Because of the time constraints and funding specified by Congress, the study design has been carefully constructed to estimate impacts of the FFVP-CFD pilot on school- and student-level outcomes using the most exacting methods possible. To best achieve the study objectives, the study has two components: (1) impact (schools and students), and (2) implementation (schools and School Food Authorities (SFAs) (see Exhibit A.2.a). The evaluation of the CFD pilot project has four broad study objectives:
To examine the impacts on students’ fruit and vegetable consumption in pilot schools
To describe the impacts of the pilot on school participation in FFVP
To describe the implementation strategies used by schools
To describe the acceptance of the pilot by key stakeholders
The evaluation will provide FNS with the data and essential information needed to answer the study’s objectives and research questions (see Appendix A, “Study Objectives, Research Questions, Outcomes Measures, Data Sources, and Planned Analysis”). Focusing on an important food group (fruits and vegetables) and the types and forms in which they may be offered to, and consumed (or wasted) by, children provides a unique opportunity to identify successful strategies to increase consumption of fruits and vegetables in school, contributing to children’s longer-term dietary habits and health and wellness.
Exhibit A.2.a. Overview of evaluation approach
Study objectives |
Study component |
Sample |
Data collection procedures |
Main outcomes |
1. To examine the impacts on students’ fruit and vegetable consumption (and plate waste)a in pilot schools |
Impact |
Students |
Diary-assisted in-school dietary recall |
Total quantity of FVs consumed in school by type and form
|
|
Implementation |
Schools |
Snack observations; school and FFVP menus |
Frequency, variety, and nutritional value of FVs offered (and amount selected, consumed, and wasted by students) |
2. To describe the impacts of the pilot on school participation in FFVP |
Impact |
Schools |
Snack observations; diary-assisted in-school dietary recall; student and parent surveys |
School participation rate Frequency of program participation Reasons for not participating |
3. To describe the implementation strategies used by schools |
Implementation |
State Child Nutrition Directors, Schools, Food Service Managers (FSMs) |
Observations; surveys; school and FFVP menus; review of pilot applications |
Percentage of daily FFVP and CFD snacks offering FVs |
4. To describe the acceptance of the pilot by key stakeholders |
Implementation |
Schools,
|
Surveys; semi-structured qualitative interviews |
Student attitudes toward FVs and pilot Stakeholder opinions on pilot and costs |
a Plate waste is defined as the difference between the amount of food served to a child and the amount a child consumes (that is, the amount left “on the plate”).
FV = fruits and vegetables.
The key elements of the design are:
A pre-post design for all major study components, comparing data for fall 2014 and spring 2015 in SY 2014–2015. The design allows for standard FFVP and FFVP-CFD pilot comparisons both within and between the fall and spring seasons.
Same students selected for both fall and spring data collections to control for differences in food preferences, body size, and activity levels.
Sampling of schools across weeks (and SFAs, as needed) with collection for all days of the week to broaden the types of fruits and vegetables offered to students to strengthen the comparisons.
Collection of fruit and vegetable intakes in school from students, in both fall and spring, on program days and nonprogram days, to measure impacts of pilot for both types of days and for the week as a whole.
Collection of data on the student’s activity level before or after the snack, food type and format (fruit versus vegetable, savory versus sweet), and portion served, from which the study will use a regression analysis to control for differences in the fall to spring snack service and student consumption levels that are not directly related to changes resulting from the implementation of the pilot program.
Two study components: (1) impact (schools and students), and (2) implementation (schools and possibly SFAs), to best achieve the study objectives.
The student sample will include (1) in-person interviewer-administered dietary recalls with students in grades 4 through 6, and (2) a brief survey on frequency of participation in the standard FFVP and FFVP-CFD pilot and fruit and vegetable preferences and attitudes.
The school sample will include a representative sample of up to 100 schools (or if fewer than 100 schools, a census of schools in the CFD pilot). Data from these schools will be collected through mail surveys and administrative records in fall 2014 and spring 2015. Data collection will include (1) information from pilot applications and school selections, (2) menu survey of school meal and FFVP/CFD offerings in a target week, (3) observations and student dietary recall throughout the target week of fruits and vegetables offered in classrooms/kiosks and cafeteria during the full week with student dietary recalls on program and nonprogram days, and (4) an FSM survey on the FFVP, pilot, local partners, and local nutrition education and promotion.
The implementation study will use descriptive data from a (1) fall survey conducted with all State/Indian Tribal Organization (ITO) Child Nutrition Directors to describe their experience in the FFVP and interest in the pilot, and (2) a review of applications for the pilot. An SFA Director survey and information from school-based surveys and stakeholders will be conducted in fall 2014 and spring 2016. In spring 2015, up to 50 semi-structured qualitative interviews will be conducted to examine pilot strategies that schools implement.
To best meet the study objectives and reduce respondent burden on students and parents, the study will focus on students’ in-school consumption of fruits and vegetables and other foods and beverages in the school day. In addition, to enhance the evaluation, data collection for up to 100 schools will take place over a 10 to 12 week period in both fall and spring, and will include all five days of the school week. Both of these methods will result in a greater variety of the types of fruits and vegetables offered between the two periods to eliminate some of the seasonal and timing affects associated with the pre-post design. Furthermore, collecting data over the entire week results in data for both program and nonprogram days (if used), allowing for an in-depth analysis of student consumption on both standard FFVP and FFVP-CFD pilot program and nonprogram days. The advantage of collecting student consumption on nonprogram days is that it allows the contractor (Mathematica Policy Research, Inc.) to measure the impact of the pilot on student weekly intake, and on program and nonprogram days separately, and to measure any spillover effects of the FFVP to nonprogram days (an important issue identified in the prior FFVP evaluation). Moreover, it provides a greater level of observation in the spring, when the schools may be using a mix of fresh fruits and vegetables and CFD products, to account for any changes in the number of FFVP service days and portion size changes due to the pilot.
The evaluation of the FFVP-CFD pilot will yield important information in four primary domains: (1) student-level impacts, (2) school-level impacts, (3) program implementation, and (4) acceptability to key stakeholders. Questions on student-level impacts will seek to determine whether the pilot affects children’s consumption of fruits and vegetables in school, including whether fruits and vegetables consumed through the FFVP supplement children’s in-school intakes (leading to increased consumption overall) or substitute for some portion of in-school intake (leading to no change or a decrease in overall consumption). Questions on school-level impacts will provide important information on impacts of the FFVP-CFD pilot on the school environment, including participation in FFVP, characteristics and nutrient content of the fruits and vegetables offered to and taken by students in FFVP snacks, and relationships between the fruits and vegetables offered, fruits and vegetables in school meals, and the school nutrition environment. Questions on pilot implementation will provide a wealth of information on the characteristics of schools that are selected for the pilot; how schools implemented the pilot, and how that affected fruit and vegetable offerings in school meals across a typical pilot week; and nutrition promotion and education activities. The questions on the acceptability of the program to State Child Nutrition Directors, school administrators and staff, students and their parents, and community partners will provide information to address the fourth study objective.
An overview of the data collection activities, including data collection instruments, targeted sample, and mode, is provided in Exhibit A2.b, followed by a more detailed presentation of the data collection efforts. The instrumentation plan builds on instruments used in prior school nutrition studies sponsored by FNS, including the School Nutrition Dietary Assessment (SNDA) Studies III and IV, the prior evaluation of the FFVP, and the upcoming School Nutrition and Meals Cost Study (as appropriate).
The data collection design makes the field staff labor as efficient as possible by balancing the workload over the week, and it provides collection on all days of the week so they can observe all types of fruits and vegetables being offered. This is especially important in the spring, when schools will use different mixes of CFD and fresh fruits and vegetables. The 10- to 12-week periods of observation in September through November 2014 and February through May 2015 also result in nearly continuous data collection and maximize the array of foods observed in each period.
On the program days, the contractor will observe the fruits and vegetables offered and the number taken by all students in the selected classroom (or at the point of service) for standard FFVP (fall) and FFVP-CFD (spring) snacks offered during the school day. For the selected students on a program day, they will observe fruit and vegetable portions served (and wasted) from each snack. Students will complete an abbreviated diary (as a memory aid for the in-school dietary recall) of the fruits and vegetables consumed during the school day (including the observed snack(s) and the lunch and breakfast meals consumed in school and excluding consumption before and after school hours).
Exhibit A.2.b. Data collection activities
Instrument |
Initial Sample |
Mode |
Appendix |
|
State Child Nutrition Director Survey |
54 States (fall only) |
Self-administered, editable PDF |
C1 |
|
SFA Director Survey |
50 SFAs |
Self-administered, editable PDF |
C2 |
|
FSM Survey |
100 schools |
Self-administered, editable PDF |
C3 |
|
Daily Meal Count Form |
|
Self-administered |
||
School-Level Reimbursable Fruits and Vegetable Form |
|
Self-administered, (partially completed by field staff, remainder completed by FSM) |
||
Principal Survey |
100 schools |
Self-administered, editable PDF |
C4 |
|
Teacher Survey |
250 classrooms |
Self-administered |
C5 |
|
Stakeholder qualitative interview |
50 Stakeholders (spring only) |
Semi-structured qualitative telephone interviews |
C6 |
|
School-Level Reimbursable Fruit and Vegetables Form, Nutrition Education, Promotional Material and Menu Collection Form |
100 schools |
On-site observation – field staff check off FVs offered at breakfast and lunch in cafeteria and confers with FSM for the remainder of the form, (observe meals all five days). Staff will also collect information on nutrition education promotional material and menus (target week). |
C3d
D1
D1 |
|
Classroom and Student- Level FFVP Snack Form |
250 classrooms |
Interviewer observes and indicates when FVs are offered and when program snacks are served up to two times a day. |
D2 |
|
3,300 students |
Field staff observes 12 sampled children in each sampled classroom over a target week during the time program snacks are served. Form will assist in the dietary recall and plate waste data collection |
|||
FFVP Monthly Reimbursement Claim Forms (one quarter) |
100 schools |
SFA Director (or FSM) will provide per FFVP reporting requirements |
D3 |
|
Parent consent (fall only) |
5,000 students |
By mail self-administered, with follow-up by telephone if district allows |
E1 |
|
Parent Survey |
5,000 parents |
Self-administered; mailed with consent materials |
E2 |
|
Student Survey |
5,000 students |
Self-administered in school |
E3 |
|
“Foods Eaten at School Today” booklet |
3,300 (3,000 students) |
Self-administered hard-copy record of foods consumed at school |
E4 |
|
In-school dietary recall (students)a |
3,300 (3,000 students) |
In-person, diary-assisted (“Foods Eaten in School Today”) dietary recall for consumption in school only; starts and ends in classroom |
E4 |
|
Note: All data collection activities will be repeated in the fall and spring, unless otherwise noted. Appendix B includes the expected response rates and sample sizes.
a The same students will be interviewed in the fall and spring, to the extent possible. We will ask 10 percent of the program-day student sample to repeat recall on the second day.
FV = fruit and vegetable; n.a. = not applicable.
At the end of the school day or the next morning, the field staff will complete the dietary recall with the student and their diary. On nonprogram days, field interviewers will collect the in-school dietary recall data from selected students to provide data for the entire target week, so that it will be possible to gauge the impact of the pilot on the program days, on the nonprogram days, and for the week as a whole.
During one of the weeks, field staff will administer a survey to all students in selected classrooms (for whom parental consent will have been obtained), as well as a teacher survey, to produce approximately 3,562 student surveys (14.25 completed per class, assuming 21 students per classroom, of which 68 percent consent and are in school on that day) and 200 teacher surveys. The contractor will interview the same students in the same classrooms in the fall and spring, to help eliminate the effects of taste differences (and other differences in unobserved characteristics that do not change) that could bias the results if an independent sample of students were selected for each period.
This will create, on average, a sample of 30 students (12 per classroom) among a selected group of two to three classrooms with an in-school food diary, recall, and plate waste observations per school, or a total of 3,000 students sampled—3,300 recalls (with the extra day of recall for 10 percent of the students collected, to estimate students’ usual energy and nutrient intake).
This next section describes each data collection instrument in detail, including the type of data collected, the data collection procedures, and required training. For quick reference, a road map is included in the appendices, showing a copy of the instruments, materials for respondents, and other data collection materials.
State Child Nutrition Directors, SFA Directors, and FSMs will provide data to characterize the reasons that States did or did not apply for the pilot; food service operating policies and practices; and other characteristics of SFAs, schools, and students. (See Appendix C for State- and SFA-level data collection letters and instruments.) Data will be collected using self-administered surveys, sent as editable PDF documents. No more than 10 States, but at least five States, may be selected for the evaluation. However, all 54 State Child Nutrition Directors will receive a survey in the fall.
The State Child Nutrition Agency/Director Survey (Appendix C1) will address program costs at the State level, the application and selection process, and State oversight. The SFA Director Survey, to be conducted in the fall and the spring, will explore implementation strategies and challenges at the SFA and school levels, acceptance measures, and perceived barriers to implementing the FFVP (Appendix C2). The contractor will email these surveys in an editable PDF format to respondents, with brief instructions on the methods and timeline for completion. A combination of email and telephone reminders to ensure a high response rate within a timely data collection window will be used for the Child Nutrition Director Survey and the SFA Director Survey.
The contractor will collect school-level data from public and private schools selected for the pilot, using multiple instruments to gather information on fruits and vegetables offered in reimbursable meals, nutrition education and promotion activities, standard FFVP and FFVP-CFD implementation, and perceptions of the program. All instruments will be fielded in the fall and spring during a target week, with modifications specific to the pilot in the spring versions.
FSM data collection. See Appendix C3 for FSM letters and instruments. FSMs will complete a brief FSM Survey to provide information on the standard FFVP in the fall (see Appendix C3b). The spring survey will provide information on FFVP-CFD implementation in their school, the perception of students and staff of the pilot, and any ongoing nutrition education and promotion activities. The FSM will also complete the Daily Meal Counts Form in both fall and spring, to report the number of USDA-reimbursable breakfasts and lunches served each day of the target week through the National School Lunch Program (NSLP) and School Breakfast Program (SBP), as well as FFVP snacks. The form will also be used to report total enrollment (needed to measure participation rates) during the target week (see Appendix C3c). The field staff will ask the FSM to complete the last three columns of the School–Level Reimbursable Fruits and Vegetable Form (see Appendix C3d). The field staff will complete the first two columns of the form. This form will be used to identify the specific fruits and vegetables offered in USDA-reimbursable meals each day of the target week; it includes fruit bars, side salad bars, entrée salad bars, and potato bars. For each fruit and vegetable offered, the form will capture whether it is fresh, frozen, canned, or dried.
Principal Survey. Principals will provide data needed to characterize the standard FFVP and the FFVP-CFD and dimensions of the schools’ implementation of the FFVP, including cost. Data on outside partners for the FFVP and what role these partners serve, will also be collected to provide greater contextual information on the schools’ strategies for implementing the FFVP-CFD pilot. Survey items also include principals’ attitudes toward the perceived challenges in implementing the program and their satisfaction with the program and the pilot. See Appendix C4 for letters and instrument. The contractor will collect these data in both the fall and spring, using self-administered surveys sent as editable PDF documents, which will allow staff the flexibility to complete the survey electronically or return it by mail.
Teacher Survey. The survey will measure teachers’ opinions about the program and their perceptions of student attitudes toward it, which will provide important information on acceptance of the pilot project by key stakeholders. Teachers will also report on how many of the fruits and vegetables provided to the students in their classroom as free snacks are eaten and which types of fruits and vegetables students like the most and least. To provide data on classroom implementation strategies, teacher surveys will collect data on the extent to which teachers encourage students to eat fruits and vegetables (for example, provide nutrition education or food-related activities; verbally encourage students). The contractor will collect these data using self-administered hard-copy surveys, which we will encourage teachers to complete when students are taking their survey in the classroom in both the fall and spring (see Appendix C5 for letters and instrument).
Stakeholder qualitative interviews. In addition to the surveys, qualitative interviews will be conducted by telephone in the spring as part of the implementation study, to address study objectives 3 and 4 (see Appendix C6 for letters and instrument). A total of up to 50 semi-structured interviews (approximately five per school (site) in up to 10 sites) will be conducted with stakeholders. Potential respondents include the State Child Nutrition Director, SFA Director, Parent Teacher Association (PTA) president, a local supermarket partner liaison, a local purchasing cooperative, and/or another community partner. Examples of topics covered in the interviews include (1) factors related to administering the pilot (such as planning and preparations, level of effort and staffing, promotion strategies, and partnerships established); (2) perceived acceptance levels and experiences; (3) barriers to implementing and sustaining efforts, and (4) successes and lessons learned for future implementers.
School meals and FFVP menu collection. Contractor field staff will obtain a copy of the school menu for the target week from the district or school website; the school menu should reflect foods that will be offered to students through the SBP, NSLP, or FFVP (fall) or FFVP-CFD (spring). Staff will review the menu before their visit. When they arrive on-site, they will obtain a revised menu for the target week, if any changes were made, and attach the weekly menu to an observation form (see Appendix D1, Nutrition Education, Promotional Material, and Menu Collection Form). Direct observations of fruits and vegetables served at school meals will provide objective measures of fruits and vegetables offered, to be used in analysis along with the menu data and to support the in-school recall data collection (see Appendix D1 and D2 forms).
School-Level Reimbursable Fruits and Vegetable Form. As described above, (see Appendix C3), this form will be used to identify the specific fruits and vegetables offered in USDA-reimbursable meals each day of the target week. Interviewers will complete separate forms for breakfast and lunch, filling out the first two columns on each form, then they will ask the FSM to complete the remaining columns on each form.
School-Level Nutrition Education Promotional Material and Menu Collection Form. Observers will complete the forms and attach a copy of the school menu to the recording sheet (see Appendix D1).
Classroom Student-Level FFVP Observation Form for Snacks, and Recall and Plate Waste. This form (used in both fall and spring) will capture snack information on the days of the week fruits and vegetables were offered to students, the times of day they were offered, and the distribution location. Field staff will also record information about the presence or lack of nutrition education and promotional material in the classroom and about staff encouraging students to eat or taste the FFVP snacks. This instrument will also be used to document the specific fruits and vegetables offered to students; the form and sizes of the items (including fresh, frozen, canned, or dried); the portion size(s) offered; and, if feasible, the number of portions actually served to (taken by) sampled students (that is, students sampled for the in-school dietary recall) (see Appendix D2). This form will be completed for each sampled classroom on program days during the target week.
Administrative data on FFVP (Reimbursement Claim Forms). The contractor will collect available administrative data consisting of one-quarter of monthly claims/performance reports from FSMs in each school (in the fall and again in the spring). The contractor staff will obtain a copy of the monthly claims forms that are submitted by the school/SFA to the State for the FFVP. These forms will provide information on the implementation of the program, including the number of operating days, the types of fruits and vegetables purchased, and expenditures on labor, supplies, and other administrative costs (such as equipment, storage, and record keeping). See Appendix D3 for a sample template of these administrative forms.
Parental consent for selected students. The contractor may encounter two types of parental consent when contacting the schools for participation in the study. The type of consent required by districts and schools will either be “implied” consent, in which a letter is sent to parents telling them that data collection activities are planned and giving them a chance to object to their child’s participation; or (2) “active” consent, in which parents must sign forms to allow their child to participate in the study. Where regulations are ambiguous, the evaluation team will attempt to negotiate implied consent as an option, emphasizing that it is less burdensome for the school and that the study poses minimal risk to respondents, given the nonsensitive nature of most data collected and the privacy protections in place. See Appendix E1 for letters and consent forms.
In districts that require implied consent, parents will receive a study notification letter informing the parent of how to contact the study team (toll-free number, study email, or mail) if they choose not to allow their child to participate in the study (see Appendix E1a). In districts with active consent, the parents will receive a consent form and instructions on how to return the signed form indicating if their child’s participation is allowed or not allowed (see Appendix E1b and E1c). The contractor will prepare consent packets (principal letter, study brochure, study notification letter [implied]) or consent form [active] with a postage paid return envelope), and discuss the distribution of the consent material with a designated person at the school (we refer to this person as the “school liaison”). Some schools prefer to provide the evaluation team with parents’ contact information so they can mail the consent packet directly to a parent’s home. Other districts/schools prefer that the classroom teacher or school liaison distribute the consent packets to the students. Despite clear return instructions, parents often will return the active consent forms directly to the school instead of mailing them to the study team; therefore, school liaisons will be provided with extra postage-paid envelopes so they can easily mail any forms that parents return to the school directly to the study team. As necessary, and when allowed to do so by school districts, the study team will contact parents directly by telephone to follow up on unreturned consent forms, address any concerns, and record verbal consents (see Appendix E1c).
Parent Survey. The contractor will ask parents of all students in sampled classrooms to complete a survey in the fall and in the spring of the school year: fall and spring. This short survey will collect information on parent and student attitudes toward the standard FFVP and the FFVP-CFD, as well as the frequency with which students eat school meals (see Appendix E2 for letter and instrument). In the fall, the contractor will send this survey home to parents with a parental consent notification/consent form (Appendix E1) to encourage an efficient return. In the spring, they will mail this survey to parents. Telephone follow-up with nonresponders in the fall will take place in accordance with the school’s consent procedures. The contractor will follow up by telephone in all schools in the spring. Parents will be mailed a $5 gift card for completing a parent survey.
Student Survey. Students in selected classrooms will read and sign a student assent form (see Appendix E3b) prior to completing the student survey. The contractor will administer a short paper-and-pencil survey to all students with parent consent in the sampled classrooms (grades 4 through 6). This survey will gather information on the frequency of fruit and vegetable consumption, as well as attitudes and behaviors related to their consumption. The contractor will administer this survey during the target week in both the fall and spring to the same students (see Appendix E3 for the introductory script, student assent form, and student survey). The student will receive a $5 gift for participation.
In-school dietary recalls for students in grades 4 through 6. The contractor will collect dietary recalls to measure in-school intake during the school day. This is roughly equivalent to the data collected with elementary students in SNDA-III, where we successfully gathered reliable food intake information, without parental assistance, within several hours of consumption. The contractor’s field staff will meet with students in the morning of their recall day to provide them with the abbreviated diary, “Foods Eaten in School Today” (see Appendix E4) (a simplified version of My Food Diary used in the prior FFVP evaluation) and show them how to use it. Students will record the food they eat during the school day as a memory aid for the interviewer-administered recall interview. The contractor will use an electronic multiple pass method computer-assisted personal interview (CAPI), to collect data on students’ dietary intakes in a one-on-one interviewer-administered setting at school (see Appendix E4c for screen shots of the dietary instrument). The multiple-pass method obtains information from respondents in five standardized steps designed to efficiently collect complete and accurate food intake data while minimizing respondent burden. Of the students selected for the in-school dietary recall, 10 percent will have a second day’s observation for the recall and plate waste for the estimation of usual in-school intake of energy and key nutrients.
The 24-hour dietary recall is the preferred method for obtaining quantitative data on foods consumed, portion sizes, and the extent to which fresh and CFD fruits and vegetables replace or add other foods in the diets of children. Although dietary recalls are costly to train staff on and administer, the research questions related to student food intakes cannot be adequately answered with other dietary data collection methods. Therefore, a dietary recall that captures total food consumption in school is planned; this reduces the burden of contacting parents and students outside of school to collect information for the rest of the 24-hour period, and reduces data collection burden while meeting study objectives.
The Classroom and Student-Level Form described above and shown in Appendix D2 is also used to support the dietary recall and plate waste components. Field staff will observe standard FFVP participation (in the fall) and FFVP-CFD participation (in the spring) for the sample of students selected for the in-school dietary recall (3,000 students).
The study is designed to minimize burden on the students, improve accuracy, and prevent conditioning effects. The data collection will yield up to 3,300 recalls per period (6,600 collections for fall and spring combined) to provide FNS with an extensive sample to conduct reliable analysis overall and for the subgroups of interest (for example, grade level, gender, FRP status).
Ultimately, all the data collected for the study and resulting analyses will be published and available to the public. All results will be presented in aggregated form in the final report made available in the research section of the USDA FNS website http://www.fns.usda.gov/ops/research-and-analysis.
Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.
FNS is committed to complying with the E-Government Act of 2002 to promote the use of technology. In compliance with the act, information technology has been incorporated into the data collection to reduce respondent burden. Electronic mail will be used, whenever possible, to communicate with respondents (see, for example, Appendix C). The State Child Nutrition Director, SFA Director, and principal surveys will be self-administered, sent as PDF documents, which will allow staff the flexibility to complete the survey electronically and return it by email or return their handwritten responses by mail. We will contact the State Child Nutrition Directors, SFA Directors, and principals by email to remind and encourage them to complete the survey and to try to get a commitment of a date by which they will complete the survey. We will also follow up by telephone with those who do not complete the survey, and we will collect data from them in an interviewer-administered telephone survey.
Field interviewers will use an electronic multiple-pass CAPI to collect data on students’ food consumption in school. The multiple-pass method obtains information from respondents in five standardized steps designed to efficiently collect complete and accurate food intake data, while minimizing respondent burden. This study will increase accuracy by having sampled students complete an abbreviated food diary to record the foods and beverages they consumed on the target day of the recall. Interviewers will have predetermined information on the foods and portion sizes served in school meals in the target week, minimizing the need to ask students details on the foods consumed in school meals. The student dietary interview will focus on recording the foods and amounts consumed during the school day (including lunch brought from home), supported by the available information on the school meal and snack items offered, the student food diary, and the field (school) observations of FFVP snacks selected and consumed by sampled students. Of the 38,709 estimated responses, 6,270 (or 16%) will be collected using CAPI.
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purpose described in item 2 above.
We have made every effort to avoid duplication. FNS has reviewed USDA reporting requirements, State administrative agency reporting requirements, and special studies by other government and private agencies. To our knowledge, there is no similar information available or being collected for the current timeframe.
If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.
Information being requested or required has been held to the minimum required for the intended use. Although smaller States and SFAs may be involved in this data collection effort, they deliver the same FFVP program benefits and perform the same function as any other SFA. Therefore, they maintain the same kinds of administrative information on file. FNS estimates that one percent of our Local and Tribal respondents are small entities, approximately 8 respondents.
Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
The proposed study is a one-time data collection effort to evaluate a pilot of offering CFD fruits and vegetables in the FFVP in SY 2014–2015. The proposed data collection plans call for collecting data from schools, students, and stakeholders. Without this information, FNS will not be able to meet Congress’ requirements for an interim report on the pilot program in January 2015, or an evaluation report on the impacts of the pilot program one year following the end of data collection (June 2016).
Explain any special circumstances that would cause an information collection to be conducted in a manner
requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.”
There are no special circumstances. This collection of information is conducted in a manner consistent with the guidelines in the Code of Federal Regulations, 5 CFR 1320.5.
If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.
Notice of this study was published in the Federal Register on Thursday, March 27, 2014 (Vol. 79, No. 59, pp. 17127–17130). A copy of the notice is included as Appendix F. FNS could receive comments to the Federal Register Notice until May 27, 2014. Public comments are in Appendix G. FNS responded to the said comments on June 27, 2014 (see Appendix H).
Staff members of FNS’s Child Nutrition Programs and the Office of Policy Support who have in-depth knowledge of the topic areas studied the data collection and the study and analysis plan and provided feedback on this information collection. One senior researcher at the National Cancer Institute (NCI), Dr. Amy Subar (240-276-6695), was contacted about the use of NCI’s publicly available dietary data collection software and processing for the collection of students’ dietary intake and reported use with students.
In addition to soliciting comments from the public, FNS consulted with Brent Farley (phone number: 202-720-3489) from the National Agricultural Statistics Service for expert consultation about the availability of data, the design, level of burden, and clarity of instructions for this collection (Appendix I).
Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
The contractor will provide incentives to students and parents for completion of their fall and spring surveys ($5 gift card per completed survey). Non school respondents for the stakeholder qualitative interviews (for example, community partner, parent representative of a PTA) will receive a $50 gift card for their time participating in a one-hour telephone interview. No incentives will be provided to any State, SFA, or school district respondent.
Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
All respondents’ information will be kept private and not disclosed to anyone but the analysts conducting this research, except as otherwise required by law. Section 9(b) of the National School Lunch Act (Public Law 103-448) restricts the use or disclosure of any eligibility information to persons directly connected with the administration or enforcement of the program.
The individuals at the SFA or school district level participating in this study will be assured that the information they provide will not be released in a form that identifies them. No identifying information will be attached to any reports or data supplied to USDA or any other researchers. For data collected through the State-level surveys, the State Child Nutrition Directors are publicly known, but individual respondents will not be identified by name. A system of record notice (SORN) titled FNS-8 USDA/FNS Studies and Reports in the Federal Register on April 25, 1991, Volume 56, Pages 19078-19080 discusses the terms of protections that will be provided to respondents.
During the project, hard-copy documents will be stored in secured file cabinets and rooms, and electronic data will be maintained on secured, password-protected computer servers. Both sources of data will be accessible only by approved contractor staff. At the close of the study, all hard-copy documents will be shredded.
All contractor staff are required to sign a confidentiality agreement. In this agreement, staff pledge to maintain the privacy of all information collected from the respondents and not to disclose it to anyone other than authorized representatives of the study. Issues of privacy will be discussed during training sessions with staff working on the project. The text of the confidentiality agreement is shown in Appendix J.
Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
The surveys and interviews with State Child Nutrition Directors, SFA Directors, FSMs, stakeholders, principals, teachers, and students do not involve questions of a sensitive nature. All respondents will be informed that they can decline to answer any question they do not wish to answer and that there are no negative consequences for not participating. Respondents will also be assured of privacy at the outset of the interview, and, if required by the district, receive a copy of their parental consent form, which addresses the issue. All survey responses will be held in a secured manner; respondents’ answers will not be reported to school officials or any other program or agency, and they will be combined with the responses of others so that individuals cannot be identified. FNS and the contractor will comply with the requirements of the Privacy Act. Most questions have been used in the previous FFVP evaluation and other school nutrition surveys with no evidence of harm.
Questions in the parent survey contain the following topics that may be considered sensitive items: student’s participation in FRP school meals and race and cultural origin.
Provide estimates of the hour burden of the collection of information. The statement should:
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.
The public affected by this study are State, local, and tribal governments, including State agencies and local education agencies; private schools; and individuals, including elementary students and parents. Appendix B shows sample sizes, estimated burden, and estimated annualized cost of respondent burden for each part of the data collection and for all data collection. Estimated response times are based on response times for similar instruments completed by the same types of respondents in FNS’s SNDA studies and informed by pretesting of select instruments and protocols for FNS’s current School Nutrition and Meals Cost Study. Annualized cost of respondent burden is the product of each type of respondent’s annual burden and average hourly wage rate. As shown in Appendix B, the total estimated burden across all data collection components is 10,125 hours. The total cost of respondent burden is $60,781.
Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14). The cost estimates should be split into two components: (a) a total capital and start-up cost component annualized over its expected useful life; and (b) a total operation and maintenance and purchase of services component.
There are no capital and start-up, or ongoing operation and maintenance, costs associated with this information collection.
Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.
The total cost to the Federal government is $4,998,735. The largest cost to the Federal government is to pay a contractor $4,998,041 over a 30-month period to conduct the study and deliver data files. This is based on an estimate of 48,549 hours, with a salary range of $34.61 – $368.11/ hour. This contract cost includes overhead costs, as well as the cost for computing, copying, supplies, postage, shipping, and other miscellaneous items. This information collection also assumes a total of 20 hours of Federal employee time: for a GS-13, step 1 at $34.69 per hour for a total of $693.80 on an annual basis. Federal employee pay rates are based on the General Schedule of the Office of Personnel Management for 2014.
Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.
This is a new information collection that will add 10,125 burden hours and 44,790 annual responses to the OMB inventory as a result of program changes due to a new statute, Section 4214(c) of the Agriculture Act of 2014.
For collections of information whose results are planned to be published, outline plans for tabulation and publication.
This schedule assumes OMB clearance will be received no later than September 15, 2014. Exhibit A16.a shows the planned schedule for FFVP-CFD.
Exhibit A.16.a. Project time schedule
Activity |
Schedule |
Sample schools |
08/16/14–09/14/14 |
Conduct data collection |
09/15/15–05/29/16 |
Interim Report to Congress Analyze data and prepare evaluation reports |
11/15/14-12/15/14 06/01/15–06/30/16 |
Prepare data files and documentation |
01/01/16–06/30/16 |
Briefing |
05/31/16–07/29/16 |
An interim report due to Congress in January 2015 will describe the selection of schools for the pilot program and progress in data collection in fall of SY 2014–2015. The contractor will analyze the information collected using descriptive tabular, cross-tabular, and multivariate (regression) analysis. The study will release and prepare a detailed evaluation report. The report will address the major research questions encompassed in the study’s objectives to provide the most reliable findings on the CFD pilot program. The report will be posted on the USDA FNS website http://www.fns.usda.gov/ops/research-and-analysis. Some of the tabulations will be similar to those completed for the previous national FNS evaluation of the FFVP and national FNS school meal program studies, to provide relevant information on how policy and program changes affect the nutritional quality of school meals and FFVP snacks.
Create study database and analysis files. The contractor will prepare labeled SAS files with variable names keyed to instrument question numbers and then clean data files by checking for consistency, missing values, outliers, and other problem values. Next, it will create and add constructed variables and merge sampling weights. Complete documentation, including the file structure, codebook, variable definitions and formulation, descriptions of editing and imputation procedures, and SAS code, will accompany each set of data files, to allow full replication of all analyses presented in the evaluation report.
Data from the fruits and vegetables served in school menus and FFVP snacks, dietary intake interviews, and plate waste observations will need to be coded to reflect the foods identified and the nutrients they contain, which involves use of highly technical software, specialized databases, and skilled coders. The contractor will create an in-school day dietary recall SAS data set that will contain one record for each food consumed by a student, along with the corresponding food groups, MyPlate equivalents, and nutrients for the amount consumed. The variables for the plate waste analysis are drawn from the FFVP observation database created from the FFVP observations for student recalls. Variables will include child-level food and nutrient amounts wasted per 100 grams; the child ID (which links to age and gender); school name and ID; snack (morning, afternoon) time of day; and snack duration.
Prepare sampling weights and unit nonresponse adjustments. The data will be weighted to take into account the probabilities of selection associated with each unit of data. For example, for the student diary/recall detailed dietary assessments, the data will be weighted to account for the SFA, school within SFA, classroom, and student within classroom selection procedures. Weights will be adjusted for survey nonresponse using a propensity modeling procedure to predict the probability of responding to the component based on the available data collected on the sampling frame. As a final process in preparing the weights, weights will be adjusted using calibration or poststratification methods to ensure the weighted totals or proportions mimic those for which comparable data from the sampling frame or other published sources are available.
Specify tabulations. For each research question for the FFVP-CFD pilot evaluation (see Appendix A), researchers will specify tabulations of the data for SFAs, schools, students and their parents, stakeholders, or FFVP snacks, as well as for subgroups of interest. Key subgroups are (1) demographic student categories, such as age (grade), race/ethnicity, social-economic status, gender, and free and reduced-price school meals status; and (2) SFA or school characteristics, such as number of years participating in the FFVP, urbanicity, FRP certification rate, enrollment, race and ethnicity, and percentage of students with limited English proficiency (LEP) status.
Estimate descriptive statistics, including cross-tabulations, using appropriate statistical methods. As an initial analysis, the contractor will create descriptive statistics (frequencies, rates, means, and totals) for each period and for the differences across the periods, along with their associated 95 percent confidence intervals, using the weighted recall data to account for the sample design and any nonresponse adjustment procedures. Statistical tests for differences between student and school subgroups, and overall differences in fruit and vegetable outcomes between fall and spring, will be conducted. Analyses will be conducted using statistical software such as SUDAAN or STATA to compute standard errors that adjust for the complex sample design.
Nutrient data will be analyzed using special statistical methods to estimate the distribution of usual energy and nutrient intake in school, using two days of intake for a subsample of students. The primary outcome for plate waste—the amount of plate waste—will be presented as means and percentages of fruits and vegetables wasted relative to the amounts selected/served. Plate waste estimates will be in terms of amounts of MyPlate Equivalents and key nutrients (energy, micronutrients). Other outcomes include school-level descriptions of how leftover foods are disposed of and the FSM’s estimate of the extent of overall fruit and vegetable wastage.
To address participation, the contractor will first examine how the FFVP student participation rate (that is, the percentage of students who took a fruit and/or vegetable snack when it was offered, based on student/classroom observations and student recalls) changed between the fall and the spring. The contractor will report the change in the participation rate overall across schools, and by school characteristics as a descriptive statistic. They will also examine differences in participation rates at the student level, overall and by student subgroups, such as grade (or age), gender, race/ethnicity, and FRP certification status. To better understand participation, the contractor will describe the reasons given for not participating, based on data from the student and parent surveys.
Estimate multivariate regression models. The contractor will augment the descriptive tabulations by creating regression-adjusted values for the fall-to-spring differences. The benefit of using regression is that it will control for the differences in the types of fruits and vegetables (for example, apple slices versus a banana) offered at each meal/snack between the two periods and the number of days of the week they were offered, as well as when the measurements were collected (which week in the fall or spring to assess seasonal effects), and improve the estimates of the pilot impacts. In these regressions, the contractor will measure the change in the amount consumed overall and as a function of the percentage of fresh fruits and vegetables replaced by CFD foods in the pilot. Separate regression analyses will be conducted by major food type (fruit, vegetable, cooked versus fresh, and whether the pilot offering was CFD (in additional to and/or in place of fresh). The data will also be weighted to account for the selection of classrooms and students and their random assignment to daily time slots.
Analysis plan for implementation and stakeholder acceptance. Implementation study data collection activities include:
Surveys of all State Child Nutrition Directors, to describe their experience with the FFVP and learn whether they applied for the pilot and their reasons for doing so (or not doing so).
A document review of pilot applications, to describe the reasons or need stated in the applications for the pilot.
Surveys with school officials, parents, and students.
Up to 50 semistructured qualitative stakeholder interviews (up to one hour long) in up to 10 randomly selected schools (sites). Potential respondents include the State Child Nutrition Director, SFA Director, PTA president, a local supermarket partner liaison, a local purchasing cooperative, and/or another community partner. See the protocol in Appendix C6 for a list of topics and questions.
The implementation study component will draw on several data sources to address how the pilot was planned and implemented, and how well it was accepted by a range of stakeholders (see Exhibit A16.b).
The use of many data sources has two benefits. First, it allows for integration of both quantitative and qualitative methods. On-site observations and open-ended inquiry from the qualitative interviews then build on survey responses to understand rationale and how contextual and operational factors may influence implementation at any given school district or school. Second, multiple data sources provide a rich set of information about the views of many stakeholders on the same issues, allowing researchers to report findings in a more meaningful narrative than using quantitative survey data only.
Exhibit A.16.b. Overview of proposed data sources and their contribution to the implementation study
Data source and purpose |
Program/pilot planning and context |
FFVP program operations and delivery |
School and participant characteristics, participation, and stakeholder experiences |
State Child Nutrition Director survey (census of States) |
X |
X |
X |
State/ITO applications for the pilot and other program documents |
X |
X |
X |
Surveys of principals, FSMs, teachers, students, and parents |
X |
X |
X |
School on-site observations |
|
|
|
FFVP and pilot activities |
|
X |
|
School meals |
|
X |
X |
Nutrition education and promotion (environment) |
X |
|
X |
Qualitative interviews in selected SFAs/schools |
|
|
|
Community partners |
X |
X |
X |
State Child Nutrition Director, FSM, school officials |
X |
X |
X |
PTA president |
|
|
X |
Other key stakeholders |
X |
|
X |
The first step in analyzing data collected through documents, interviews, and observations will be to summarize the information in a systematic and structured way, including the results from State/ITO applications and other materials and semi-structured interviews using standardized reporting templates. Data collectors will enter finalized information into a common database to be used for cross-site analysis. The second step in the analysis process is to develop theme tables (or key topic findings) using the information in the database. Theme tables will be developed for each major research question. Whereas some themes will be straightforward in describing frequencies and types of responses to protocol questions (for example, commonalities related to operating contexts), other themes may result from higher-level synthesis (for example, factors related to reported challenges in implementing the pilot). Finally, themes will help to answer research questions and allow for large volumes of data to be reduced to a manageable number of topics and facilitate cross-study theme comparisons. Findings from the implementation analysis will be synthesized and included in the evaluation report.
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
The agency plans to display the expiration date for OMB approval of the information collection on all instruments.
Explain each exception to the certification statement identified in Item 19 "Certification for Paperwork Reduction Act.
This study does not require any exceptions to the Certification for Paperwork Reduction Act (5 CFR 1320.9).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Ksonnenfeld |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |