APPENDIX F. Data Collection Methods for Impact Evaluation

Appendix F.

Data Collection Methods for the Impact Evaluation

This appendix provides a detailed description of the data collection methods that will be used by the contractor, RTI International, to collect data for the impact evaluations of the three demonstration projects.

INN Impact Evaluation

Table F-1 summarizes the data collection methods for the evaluation of the BASICS and Pick a better snack™ interventions. As described below, we will make multiple attempts by mail and telephone to maximize the response rate for the survey.

Table F-1.—Data Collection Methods for the Impact Evaluation of the BASICS and Pick a better snack™ Interventions

	Pre-survey	Post-survey
Mode	Mail/telephone	Mail/telephone
Data collection period	2-month period prior to intervention	Begin conducting survey 1 week after conclusion of intervention
Incentive amount	$10	$15

We will work with INN and the selected schools to coordinate the data collection for the pre- intervention surveys at the intervention and control schools. INN will make the initial contact with treatment and control schools to encourage their participation in the evaluation study. RTI field interviewers will also meet in person with the school principal and teachers, as appropriate, to discuss the study procedures and obtain their buy-in.

So that we can match the responses to the FNS parent survey and the INN student survey, we will obtain informed consent from the parent/caregiver for their participation and their child’s participation in the study. For parents that agree to participate in the study, we will assign a unique identification number that will allow us to match the parent and student data.

We will send home with the child a packet with a cover letter, an information sheet that provides all the elements of informed consent, a Contact Card, an informational brochure describing the study, the questionnaire booklet for the parent survey, and a postage-paid envelope for returning the questionnaire to RTI. The cover letter will provide an invitation for parents/caregivers and their child to participate in the study and request that the parent/caregiver complete the Contact Card and provide their contact information if they agree to participate. Children will receive a token incentive for returning the Contact Card to their teacher regardless of whether their parent agrees to participate in the study. INN will administer the student survey in the classroom to children whose parents consent for them to participate in the study We will provide incentives to the school and participating teachers to encourage parent/caregiver response to the invitation to participate in the evaluation study (e.g., provide a classroom incentive if 75 percent or more of parents/caregivers return the Contact Card).

For caregivers from whom we receive contact information, we will send a thank-you/reminder postcard. Nonrespondents will be contacted by telephone to complete the survey; a minimum of 10 call attempts will be made to each working phone number. For the post-intervention survey, we will mail a pre-notification letter during the last week of the intervention, mail the survey, mail a reminder/thank-you postcard, mail a second survey to nonrespondents, and then contact nonrespondents by telephone to complete the survey (a minimum of 10 call attempts will be made to each working phone number).

UKCES Impact Evaluation

Table F.2 summarizes the data collection methods for the LEAP2 intervention evaluation. As described below, we will make multiple attempts by mail and telephone to maximize the response rate for the survey.

Table F.2.—Data Collection Methods for the Impact Evaluation of the LEAP2 Intervention

	Pre-survey	Post-survey
Mode	Mail/telephone	Mail/telephone
Data collection period	2-month period prior to intervention	Begin conducting survey 1 week after conclusion of intervention
Incentive amount	$10	$15

We will work with UKCES and the selected schools to coordinate the data collection for the pre-intervention surveys at the intervention and control schools. UKCES will make the initial contact with treatment and control schools to encourage their participation in the evaluation study. RTI field interviewers will also meet in person with the school principal and teachers, as appropriate, to discuss the study procedures and obtain their buy-in. We will send home with the child an invitation for parents/caregivers to participate in the study, which will request their caregiver’s contact information. Children will receive a token incentive for returning the parent reply form. Furthermore, we will provide incentives to the school and participating teachers to encourage parent/caregiver response to the invitation to participate in the evaluation study (e.g., provide a classroom incentive if 75 percent or more of parents/caregivers return the parent reply form).

For caregivers from whom we receive contact information, we will then mail a hard copy of the pre-intervention survey with a cover letter offering a monetary incentive for completing the pre-intervention survey and a reminder/thank-you postcard. Nonrespondents will be contacted by telephone to complete the survey; a minimum of 10 call attempts will be made to each working phone number. For the post-intervention survey, we will mail a pre-notification letter the last week of the intervention, mail the survey, mail a reminder/thank-you postcard, and then contact nonrespondents by telephone and attempt to complete the survey (a minimum of 10 call attempts will be made to each working phone number).

MSUE Impact Evaluation

Table F.3 summarizes the data collection methods for the Eat Smart, Live Strong evaluation. Our data collection procedures will be tailored to meet the needs of older adults; for example, the questionnaire will be printed in 16-point type.

Table F.3.—Data Collection Methods for the Impact Evaluation of the Eat Smart, Live Strong Intervention

	Pre-survey	Post-survey
Mode	Participants independently complete self-administered questionnaire in group setting	Mail/telephone
Data collection period	1 week before beginning of intervention	Begin conducting survey 1 week after conclusion of intervention
Incentive amount	$10	$15

As described below, we will work with MSUE to implement the pre-survey data collection and make multiple attempts by mail and telephone to maximize the response rate for the post-survey. We do not plan to conduct the post-intervention survey in person because only those individuals who complete all four sessions will be asked by MSUE to complete their post-intervention data collection, whereas the contractor/FNS evaluation will include surveys of all study participants, including individuals who drop out of the intervention (i.e., do not attend all four sessions). Additionally, using a mail/telephone survey approach for the Wave I study yielded high response rates and is more cost-effective than conducting in-person interviews.

We will work with MSUE to coordinate the data collection for the pre-intervention surveys at the intervention and control centers. MSUE will make the initial contact with treatment and control centers to encourage their participation in the study.

One week before the first intervention session, study participants in the treatment and control groups will enroll and consent to be in the study and complete the FNS impact evaluation questionnaire and the data collection for the MSUE evaluation. Half of the participants will complete the FNS impact instrument first, and half will complete the MSUE impact instruments first. The first session will be 90 minutes, with 50 minutes allocated to MSUE data collection and 25 minutes allocated to FNS data collection. We plan to provide participants with a 15-minute break (with snacks courtesy of MSUE) in between completing the FNS and MSUE instruments to give participants an opportunity to rest.

Based on the pretest findings, we determined that it would be most effective to have participants individually read and complete the questionnaire instead of having the field interviewer read the questions and the participant record their answers. An RTI field interviewer will be available to answer any questions participants may have as they complete the survey. The RTI field interviewer will also collect contact information so that we can contact study participants by mail and telephone for the post-survey.

For the post-survey, we will mail a pre-notification letter the last week of the intervention to all study participants, mail the survey, mail a reminder/thank-you postcard, and then contact nonrespondents by telephone to complete the survey (a minimum of 10 call attempts will be made to each working phone number).