Ssb_11-16-11

1

HHS/CDC/NCIPC

SUPPORTING STATEMENT FOR

OMB INFORMATION COLLECTION REQUEST

Part B

12 December 2011

Evaluation of Dating Matters: Strategies to Promote Healthy Teen Relationships

Supported by:

Department of Health and Human Services

Centers for Disease Control and Prevention

National Center for Injury Prevention and Control

Division of Violence Prevention

Government Project Officers:

Point of Contact for OMB

Andra Tharp, PhD (Project Lead)

[email protected]/770-488-3936

Principal Investigators

Craig Bryant, BS

Tessa Burton, MPH

Dawn Fowler, PhD

Phyllis Holditch Niolon, PhD

Co-Investigators

Kimberley Freire, PhD

Frank Luo, PhD

Kevin Vagi, PhD

Table of Contents

B. Collections of Information Employing statistical procedures

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with Non-Response

4. Test of Procedures or Methods to be Undertaken

5. Individuals consulted on Statistical Aspects and Individuals collecting and/or analyzing information

B. Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Outcome Evaluation

These estimates are based on the following information:

• Number of communities/sites: 4

• Number of schools across 4 communities/sites: 44 (12 in 3 communities, 8 in 1 community)

• Number of students in each middle school: 600 (200 per grade)

• Number of school staff in each school: 40

• Number of schools implementing the standard model of TDV prevention: 22 (across 4 sites/communities)

• Number of schools implementing the comprehensive model of TDV prevention: 22 (across 4 sites/communities)

Design. Four implementation sites have been funded, and they have identified 44 schools in total to implement the two models of teen dating violence prevention. In consultation with each of the sites, the evaluation contractor and CDC have determined that a within-site simple random assignment of schools to one of the two prevention models is appropriate for three of the four funded communities: Alameda County, Broward County, and Chicago. The advantage of simple random assignment is that is easy to implement, can be easily understood by stakeholders, and yields data that can be easily analyzed. If there are a small number of units, however, there is a concern that the groups might not end up as equal to each other as in designs with a large number of units being randomized. This is of particular concern when the schools being randomized start out with large differences on important factors.

In the fourth funded community—Baltimore—two of the study schools cover 6^th -12^th grades, thus forming a separate block from the remaining 6^th-8^th grade middle schools. In Baltimore, therefore, blocking by school type is the most effective approach. For the Baltimore site, we will have two blocks, one with two schools (that are 6-12 schools ) and one with ten schools (all the other schools that are 6-8 grade schools), and conduct random assignment within each of these blocks.

Population. The study population includes students in 6^th, 7^th and 8^th grades at 44 schools in the four participating sites. At most, schools are expected to have 6 classrooms per grade, with an average of 30 students per classroom yielding a population of 23,760 students (44 schools * 3 grades * 6 classrooms per grade * 30 students per classroom).

The sampling frame for parents, given that we would only include one parent per student, is also 23,760 for the three years of data collection covered by this package. Based on our research and consultation with middle schools, most schools with 600 students have approximately 40 staff. If we assume 40 educators per school, the sampling frame for the educator sample is 1,760.

Students: . The study will survey samples of classrooms from all three middle school grade levels in the 44 schools, annually over a 4 year data collection period (see Figure 2). (Please note that we recognize that our OMB approval will expire after 3 years and we will submit a new package at that time so that the life of the project is approved.) In each year of data collection, we will recruit 30 students per classroom * a sample of 4 classrooms per grade * 3 grades * 44 schools, resulting in a student sample of 15,840. We assume a 95% participation rate (n = 15,048) for the baseline student survey (due to students being absent and parents not providing consent for student participation). Because this is a longitudinal data collection, the mid-term and follow-up surveys will lose some students due to attrition (e.g., students absent; students move out of district; parents withdraw permission). At mid-term, we assume a retention rate of 92.5% of the 15,840 students (n = 14,652), and at follow-up (at the end of the school year), we assume a retention rate of 90% of the 15,840 students (n = 14,256).

Figure 2: Population and Samples

ANNUAL STUDENT SAMPLE
Total Number of Participating Schools	44
Number of Grades Participating	3
Number of Classrooms Surveyed per Grade	4
Number of Students per Classroom	30
Annual Student Sample (=44*3*4*30)	15,840
Baseline Sample assuming the 95% Participation Rate	15,048
Mid-Term Sample assuming 92.5% Retention Rate	14,652
Follow-up Sample assuming 90% Retention Rate	14,256
ANNUAL PARENT SAMPLE
Initial Sampling Frame (based on student sample)	15,840
Random Selection of 17% of parents of student sample	2,693
Baseline Sample assuming 90% Participation Rate	2,424
Follow-up Sample assuming the 90% Retention Rate	2,181
ANNUAL EDUCATOR SAMPLE
Total Number of Participating Schools	44
Number of Educators per School	40
Initial Sampling Frame	1,760
Baseline & Follow-Up samples assuming 90% Participation Rate	1,584

Parents: We will recruit parents of 17% of the student sample (15,840) inclusive of parents participating in the parent curricula, and those who choose not to participate in the parent curricula, from both the Dating Matters schools and the standard-of-care schools. We will recruit a sample of 17% of eligible parents per grade per school for a total of 2,693 parents. Assuming 90% of the 2,693 parents agree to participate at baseline (n=2,424) and we retain 90% of participating parents from baseline, we will have a final follow-up sample of 2,181 parents.

Educators: We will attempt to recruit all educators in each school (44 schools * 40 educators per school = 1,760), who are assumed to stay in their positions over the study period (in contrast to the cohorts of students moving through the school). We expect a 90% participation rate for an estimated sample of 1,584 educators.

School data extractors: We will attempt to recruit one data extractor per 44 schools to extract school data to be used in conjunction with the outcome data for the students. Individual level school data will only be collected for students participating in the evaluation, so this data will reflect the same sampling frame as the student survey data. As a result, the data extractors in each school will access individual school-level data for those students in their school who consented and participated in the baseline student survey (3*4*30*95% = 342).

Implementation Evaluation

Focus Groups

The evaluation contractor will conduct both student and implementer focus groups.

For the student focus groups, the contractor will work with teachers and principals to construct how students are selected and grouped together, resulting in groups of 10 students per group. Two groups will be held per each of the 4 sites (10x2x4 = 80 total student participants) moderated in a uniform manner according to the student focus group guide (Attachment ZZ).

Student implementer focus groups will be organized by site (moderated according to guidance in Attachments AAA and BBB), with two annual focus groups per site with 10 implementers in each group (10x2x4 = 80 total student program implementer participants).

Parent program implementer focus groups will be organized by site (moderated according to guidance in Attachments AAA and BBB), with two annual focus groups per site with 10 implementers in each group (10x2x4 = 80 total parent program implementer participants).

All other instruments

Due to the exploratory nature of the implementation evaluation, sampling techniques will not be employed for the Brand Ambassador survey, Communications Coordinator Tracking Form, Student Program Master Trainer TA form, or Fidelity Session Logs. Key health department and school leadership will participate in the capacity and readiness assessments.

Because the implementation evaluation is a crucial aspect of the program implementation and potential respondents represent a relatively small number of individuals, we will implement data collection methods to obtain a high rate of participation (described below). These respondents will comprise the target population, and entire sampling frame, for the purpose of the implementation evaluation. The specific number of respondents is outlined in detail below:

School leadership: based on the predicted number of one school leadership (e.g., principal, vice principal) per comprehensive school (22 schools), the number of respondents will be 22.

Local Health Department representative: based on the predicted number of four communities/sites and four local health department representatives working on Dating Matters per community, the number of respondents will be 16.

Parent Program Manager: With a maximum of one parent program manager per community/site, the number of program manager respondents will be 4.

Community Representative: based on the predicted number of 10 community representatives per 4 communities/sites, the number of respondents will be 40.

Parent Curricula Implementers: it is expected that each school implementing the comprehensive approach (n = 22) will have one male and one female parent implementing the parent programs respondents will be (2 parents x 22 schools) 44 implementers. Please note that on the burden table the number of respondents is multiplied by the number of sessions in each parent program.

For example, the 6^th grade program has 6 sessions and 264 (44 x 5) are listed.

The 7^th grade program has three sessions and 132 (44 x 3) are listed.

The 8^th grade parent curriculum is mailed to parents and, as such, does not involve implementers or session logs.

Student Curricula Implementers: based on the predicted number of seven student curricula implementers per grade per school (n = 22) that will be completing fidelity instruments, the total number of respondents will be 154 per grade. Please note that on the burden table, the number of respondents is multiplied by the number of sessions in each student curricula program.

For example, the 6^th grade curriculum has 6 sessions, so a total of 924 total respondents are listed (154 x 6).

The 7^th grade program has 7 sessions, so a total of 1078 total respondents are listed.

The 8^th grade comprehensive program has 10 sessions and 1540 respondents are listed.

The 8^th grade standard program has 10 sessions and 1540 total respondents are listed.

Brand Ambassadors: The Brand Ambassador Implementation Survey will be provided to each brand ambassador in each community. With a maximum of 20 brand ambassadors per community, the feedback form will be collected from a total of 80 brand ambassadors.

Communications Implementers (“Brand Ambassador Coordinators”): The Communications Campaign Tracking form will be provided to each brand ambassador coordinator in each community. With a maximum of one brand ambassador coordinator per community (n = 4), the feedback form will be collected from a total of 4 brand ambassador coordinators.

Student Program Master Trainer TA Form: With a maximum of 3 master trainers per community. There will be 12 master trainers. It is anticipated that they will receive up to 50 TA requests per year and complete the form 50 times.

The following table provides our exclusion criteria and rationale:

Exclusion Criteria	Rationale
Students and educators who cannot complete surveys in English will be excluded from the study	We do not have the resources to conduct data collection in languages other than English. We assume that most students and educators who are attending U.S. schools will be able to complete the survey in English.
Parents who cannot complete the surveys in English or Spanish will be excluded in the survey.	As we anticipated that there might be a number of parents who would not be able to participate in the curriculum or the surveys unless it was offered in Spanish, we are translating our materials and planning to be ready to collect data in Spanish. However, we do not have the resources to translate into/collect data in other languages.
Brand Ambassadors who cannot complete the survey in English will be excluded.	We do not have the resources to conduct data collection in languages other than English. We assume that most students who are attending U.S. schools will be able to complete the survey in English.
Parents participating in the parent curricula groups whose children are not in the student curricula groups will be excluded in the survey.	Although any parent can participate in the parent curricula, for the purpose of evaluating the impact of Dating Matters, it is unnecessary to collect data for parents whose child is not receiving the Dating Matters program.
All other respondents who cannot complete the implementation focus groups in English will be excluded.	We do not have the resources to conduct data collection in languages other than English. We assume that most individuals working in the type of agencies that will be eligible to participate in the implementer focus groups will be able to participate in English.

B.2. Procedures for the Collection of Information

Outcome Evaluation:

Students: At the beginning of the school year, a random sample of classrooms will be selected from each grade at each school to participate in the evaluation. All students in the participating classrooms will be approached to complete the survey following IRB approved consenting/assenting guidelines. The CDC IRB has approved passive parental consent procedures; each site and the contractor also will be submitting separate local IRB applications. If one or more sites require parents to actively consent to their children participating in the study, we will amend our CDC IRB approval to reflect this. Parents will be informed about the survey and the potential release of school data, including the topics covered by the survey, in at least one of the following ways. First, the contractor will work with each school to include an announcement about the survey in the school newsletter or other school publication. Second, a letter about the survey will be given to all students to take home. The exact notification mechanism and content of the letters/announcement will be determined by the contractor in consultation with each school in which either the comprehensive initiative or standard practice will be implemented. A notification letter may include the following elements: parents will be encouraged to discuss the survey with their children, parents will be given the name, phone number, and email address of a school staff member to contact to request a copy of the survey, ask questions about the survey, and/or to refuse permission for their child to take part in the survey. Permission will also be sought to obtain the student’s school data. Notices will be provided at least several weeks before the administration of the first survey. Parents will be able to refuse their child’s participation until the last business day before the start of data collection. If no such reply/refusal is received, consent will be implied by non-response. Figure 3 (below) represents the student follow-up paradigm over the course of the study.

Student assent will be obtained the day of the baseline survey administration. The assent form (see Attachment TT), will be read to students and they will have the chance to ask any questions before signing or deciding not to participate. Non-participating students will be directed to complete other work as assigned by the teacher to occupy him/herself while assenting peers complete the surveys.

The proctoring of the student surveys will be coordinated by NORC field personnel. These staff will be trained regarding the survey instruments, the purpose and goals of the project, legal requirements, human subject guidelines, survey administration protocols, data security protocols, and the coordination of school personnel involved in the administration. NORC will develop data collection training materials and conduct training for the data collection staff (at each site at a central location, or if it is more cost-efficient, all data collection staff for each site will gather for a single centralized training session). NORC staff will employ quality control measures to ensure appropriate school survey administration. These measures will include supervisory review of checklists, random calls (by the contractor’s project manager) to site personnel to confirm adherence to protocols, the availability of refresher training for survey staff, helpline (email/telephone) support for timely support of field personnel, scheduling standards for completion of classroom surveys in each school within a window, and central review of survey packages delivered by field personnel.

The baseline and follow-up assessments will be in paper-and-pencil format on scannable forms. Surveys will come with a pre-attached unique identifier number generated through a sequential process based on the students’ school and classroom. In addition, each survey will have a removable sticker with the student’s name affixed. This will allow staff to distribute surveys easily in classrooms. Students will be instructed to remove the name portion of the label before returning completed surveys to the staff. This process will occur at pre-test and at all post-tests. The ID-to-name code matrix will only be available to the research team and will be secured in the (contractor) project director’s office. The assessments will be administered during a class period or other school designated time during the school day. Students will read the items silently to themselves and answer the questions. When they are finished, they will insert the survey into a secure envelope being monitored by the data collector. Procedures for having students complete unfinished surveys or complete surveys if they were absent on the day of survey administration will be negotiated between the schools and the evaluation contractor. Follow-up surveys will follow similar procedures to the baseline survey except that students will not need to be re-assented.

To assess the possible attrition bias from participating students who drop out of the study, we will employ multiple contact methods and conduct non-response analyses. Additional efforts will be used to retain students in the sample who are lost (e.g., absent for one or more survey), and their survey responses will be compared to students regularly surveyed in their classroom settings. The main goal is to assure that subsequent outcome analyses are adjusted for any identified biases in student participation.

Parents: The following material covers parent sample description, recruitment, survey administration, and response rates. The study involves recruiting three parent samples as follows:

1. Parents participating in the parenting program (likely to be a small sample in each school).

2. Parents in treatment schools but not participating in the parenting program; and

3. Parents in comparison schools and thus not offered the parenting program.

NORC will select a random sample of 17% of the parents from the 15,840 students in the study (44 schools * 3 grades * 4 classrooms/grade * 30 students per class= 15,840 students * 17% = 2,693 parents). Half of this parent sample will be drawn from the standard-of-care schools, and half will be recruited from the comprehensive program schools. Notably, in the comprehensive schools, some of the parent sample will be parents who participated in the parenting program (Group I) and some will not.

Participants of the parent curricula (Group I) will be consented and complete the baseline survey, where possible, before the first parent group and the receipt of any parts of the curriculum. In cases where some advance notice of the timing of the parent session is available, NORC staff will coordinate planned site visits around these points to allow for in-person distribution and collection of the parent surveys. These surveys are designed to be self-administered and respondents will be provided options to complete the survey through a paper-and-pencil modality through the U.S. Mail or online. In some cases, it might not be possible to administer the surveys prior to the first parent session. However, these surveys would then be administered as soon after the first parenting session as possible. Follow-up surveys for parents participating in the Dating Matters parenting programs will delivered in the spring via the same methods (options of paper copies in the mail and web surveys will be provided) offered to parents in Groups II and III of the full parent sample. Because the scheduling of the Dating Matters parenting programs in each site is not yet known, exact plans for delivering the baseline and follow-up surveys will be finalized in coordination with sites and schools. In the case of parent intervention (Families for Safe Dates), implemented with 8^th grade parents (this intervention will occur through the delivery of material via the mail), parents will be mailed consent forms and the survey before the other materials and asked to complete and mail back the survey before beginning the curriculum. Parents in in Groups II and III will be mailed an introductory letter with the opportunity to complete the survey online or to return a postage-paid postcard to obtain a hard copy of the survey, which they may complete on paper and return in a stamped, addressed envelope. Parents will be asked to participate in a follow-up survey at the end of the school year as well.

The administration of the parent surveys will be coordinated both by NORC field personnel and central staff, reflecting the different parent samples and survey modalities. For parents in Group I, who physically participate in the Dating Matters parenting programs, survey data will be collected to the extent possible onsite during one of the sessions (depending on the survey round/timing). As necessary in a given site, alternative means of surveying parents in Group I will be parallel to the aforementioned protocols for Groups II and III. As indicated above, for survey administration of the parent survey during Dating Matters parenting program sessions, NORC staff (where possible) will coordinate planned site visits around these points to allow for in-person distribution and collection of the parent surveys.

To assess the possible non-response bias from recruited parents refusing participation in the study, we will employ multiple contact methods and conduct non-response analyses. The main goal is to assure that subsequent outcome analyses are adjusted for any identified biases in parent participation. The eventual analyses of the parent data will be strengthened by propensity score matching of parents receiving the parenting program to comparison school parents. Propensity score matching assesses the predicted probability of membership in the treatment versus control group based on observed predictors obtained from a regression to create a counterfactual group. Propensity score matching would involve calculating the distance (e.g., Mahalanobis distance) between the parent participants who attended the parenting program to each comparison school parent who completed a survey. The nonparticipant with the minimum distance is chosen as the match for the treatment parent participant and both cases are put into a separate analytic comparison pool. This approach will strengthen the evaluation of the delivery of the Dating Matters parenting programs.

Because some parents (and all educators, see below) will be contacted by email, NORC will train staff to ensure consistent and effective communications.

Educators: Educators will be recruited via email to participate in an online survey; as they will not be providing any personally identifiable information. NORC will collaborate with each school to obtain a list of Educators’ school email addresses. The Educator surveys will be administered through a web-link distributed to all Educators in the participating schools via their individual school email addresses. NORC will collaborate with site and school representatives to send to Educators an email prompt and links from a centralized local source (e.g., someone in each school’s administration, or one email from the Site Coordinator). Follow-up to non-respondent Educators will be conducted directly through email, with up to ten email follow-up requests for participation sent directly to each non-responding Educator. Each educator will be emailed a distinct survey link, so that we will be able to determine whether the educator has participated in the survey but will not know which answers are his/hers. The Web mechanism will include a disclosure statement and informed consent before Educators can complete the respective surveys. The consent “form” will appear first on the screen before the actual survey, and the educator will imply consent by clicking to continue on to the survey. The web-based survey will request Educators to enter a unique password code which will allow Educators to (1) save their entries and return to the survey if interrupted, and (2) will allow NORC to ensure only one survey response per Educator. Password identifiers will be stripped for confidentiality as the Educator survey component is a repeated cross-sectional design. Educators will be surveyed at the beginning of the school year before any implementation has commenced and at the end of each school year thereafter. Some participating educators will also potentially be implementers of the curriculum, and this dual role will be documented in the survey.

School data extractors: School data extractors will be recruited by the evaluation contractor, in collaboration with onsite evaluation study coordinators. Depending on the preference and regulations regarding student data at each site, extractors will either be representatives of the contracting agency, local health department, or school personnel. One extractor per school will be identified.

School leadership: School leaders will be identified by grantees as representatives of schools participating in the Dating Matters initiative. Grantees will identify one school administrator or staff member who has sufficient knowledge of the school environment to be able to complete the school component of the capacity/readiness assessment. A representative of school leadership will likely be identified as the main point of contact at each school for the school implementation of the Dating Matters curricula or that point of contact’s designee. Completion of the capacity/readiness assessment is a program activity within the Dating Matters initiative and within the duties defined for staff working on the initiative.

Local Health Department representatives: The Local Health Department representative will be the staff member responsible for overseeing the Dating Matters initiative (i.e., project coordinator) and up to three other staff members identified by the project coordinator. Staff member participants will be individuals with sufficient experience at the Local Health Department to be able to complete questions about the organization. Completion of the capacity/readiness assessment is a program activity within the Dating Matters initiative and within the duties defined for one or more staff working on the initiative.

Parent Curricula Implementers: Parent curricula implementers will be recruited by the program manager and will be trained and certified to implement the curricula. Program managers will be given detailed instructions about the type of parents in the community who will be suitable as implementers. These instructions have been used across the world in implementations of Parents Matter! CDC and the TA contractor will work during the planning year to develop a detailed plan to recruit the implementers and administer the surveys.

Student Curricula Implementers (6^th, 7^th, 8^th grade comprehensive and 8^th grade standard): Student curricula implementers will be recruited by the local health department and schools. It is expected that in most sites, teachers in the schools implementing Dating Matters will implement the student curricula, but it is possible that some sites may opt to use community organizations to deliver the curricula. The TA contractor and program developers/CDC and will work with the sites to assist them in recruiting implementers. CDC and the TA contractor will work during the planning year to develop a detailed plan to recruit the implementers and administer the surveys.

Brand Ambassadors: Brand ambassadors will be recruited exclusively from the comprehensive school communities. Brand ambassadors will be competed through a selective application process. The paper and pencil application will be made available at the comprehensive schools at the beginning of each academic year. Selections will be made by the Brand Ambassador Coordinators in each community. Once a brand ambassador has been selected, they will be required to obtain parental permission (form provided) and also sign a student assent form. Both forms outline guidelines for participation in the brand ambassador program as well as any evaluation activities.

Communications Coordinators: Communications implementers will be selected by the Health Department (or its communications designee) in each community. This staff position was identified in the Dating Matters initiative program funding opportunity and all sites have identified an individual who will serve in this role.

Student Program Master Trainers: Student program master trainers will be selected by the Health Department in each community and will receive training to train student program implementers in Safe Dates and the CDC-developed curricula. Master trainers will be the first line of TA for the program implementers.

Statistical Concerns and Power Estimations:

Outcome Evaluation:

The design of this study makes a great deal of practical sense. The comprehensive Dating Matters initiative employs interventions targeting multiple levels of the social ecology (student, parent, school, community) and is therefore more “expensive” from a resource perspective than the existing, widely disseminated evidence based student curriculum, Safe Dates. Therefore comparing Dating Matters to Safe Dates will allow us to see whether the extra investment of resources is “worth it” in terms of effectiveness. However, it raises one major statistical concern; as we already know that Safe Dates is likely to have an effect (it did in the original trial, but has never been tested in this population), then our study must be sufficiently powered to detect an improved effect of Dating Matters. As we are randomizing to condition at the school level, the school is the unit of analysis used for our power analysis. We determined that we would need to randomize at least 40 schools in order to have sufficient power to detect effects on our main outcomes. The sites have collectively identified 44 schools. Because there is the potential for attrition at the school level, power analyses have been conducted on the minimum anticipated sample of 40 schools. Of course, if all 44 schools maintain participation, the outcome analysis will have greater power to detect an effect of the intervention. The principle assumptions informing the outcome evaluation power analyses are as follows:

1. Using prior knowledge that dating violence is around 20%, we assumed proportions from the control to vary from 10% to 20%.

2. We assumed that the intervention lowers dating violence; thus, we assumed that the proportion of violence from the treatment group would be lower than that of the control group. We assumed treatment group proportions vary from 4% to 25%.

3. We assumed a type I error of 5%. This is the significance level (alpha).

4. We assumed that the plausible values for true control group rate of dating violence lie between 8% and 50%. Using constraints control proportions helps improve accuracy in measuring the required sample size. Lowering the width of the range of plausible values of the control group proportions helps increases statistical power.

5. We assumed that there is grade effect (J=3). Thus 6^th, 7^th and 8^th grades vary. We however assumed no class effect. Thus the different classroom groups within the same grade are assumed not to be significantly different.

6. We assumed that there is a school effect (conservatively, K=40).

Based on a power analysis for an RCT design, at the individual student level the study will have >90% power to detect differences as small as 8 percentage points between the intervention and comparison conditions.

Based on a power analysis for an RCT design, the parent analyses (assuming n= 2,161) will have acceptable statistical power (>80%) to detect differences as small as 8 percentage points between the intervention and comparison conditions (e.g., 20% compared to 12% for the prevalence of dating violence). As with the student power analysis, this power analysis was based on a three-level Cluster Randomized Trial (3-level CRT) where students are nested within classes and classes are nested within schools. In fact, if the parent sample drops as low as 1,680 parents, it will still provide an 80% power level. The issue of greater concern is obtaining a representative sample of parents and avoiding any bias being introduced by non-response.

Implementation Evaluation:

A critical component of an independent evaluation is the collection of multiple forms of data that can serve both as inputs to an ongoing process evaluation and to the eventual outcomes evaluation. The implementation evaluation will track and monitor what activities are being implemented, barriers to implementation, implementers’ level of fidelity to program curricula, and the integrity of the random assignment process in addition to capturing the cost of implementation, nature of TA provided, and the capacity and readiness of agencies implementing Dating Matters (local health department and schools). In general the implementation evaluation data will be used to describe implementation and enhance training and technical assistance for program improvement, and no statistical analyses will be performed. However, fidelity instruments will be taken into account in the analyses assessing the impact of Dating Matters.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

The response rates expected for the student and parent samples are detailed in Figure 2: Population and Samples, above. In section B2, Procedures for Collection of Information, the methods that will be applied to achieve these response rates are described. These methods are based upon prior work carried out by the evaluation contractor.

The approach to ensuring the highest possible retention of the student sample begins with a survey instrument designed for a middle school population, and field protocols which have been tested in prior work with the target population. Proctor training will support informed responses to student concerns about participating. Administrative support for students absent on the day of surveys will be coordinated by school and site to ensure maximum participation of assenting students whose parents have waived consent.

The exact procedures that will be used to maximize cooperation and to achieve the desired high response rates will be determined in collaboration with the sites and the contractors. Because schools have committed to participate through Memorandums of Understanding (schools listed in Attachment CCCC) in their health department’s application process, we are optimistic that they will be willing to work closely with us to maximize response rates. For students, procedures will be put in place to allow students who do not finish during the allotted data collection time to complete their surveys and to allow students who are absent to complete their surveys. If parent surveys are mailed, a pre-survey letter will be mailed (see Attachment WWW). The respondents will be provided with a stamped, pre-addressed envelope in which to return their surveys. If parent surveys are conducted online, the consent “form” and survey data will be received through secure connections on the web. The contractor will utilize either email or postcards to send the sampled parents several follow-up reminders encouraging participation (see Attachment YYY). Educators will be sent reminder emails (see Attachment ZZZ) and encouraged by school administrators to complete the surveys.

As noted above, school commitment was obtained prior to sites applying for Dating Matters funding. Moreover, in their applications, grantees agreed to complete the capacity and readiness assessments and participate in the implementation evaluation as part of their programmatic activities. Response rates to the implementation evaluation instruments including the focus group guided discussions will be maximized by integrating the instruments into grantee activities and trainings and by employing both on-site program oversight (e.g., master trainers, parent program managers, communications coordinator) and off-site oversight (training and technical assistance provider) and support. The importance of participating in the implementation evaluation in order to improve program quality will be emphasized to implementers and strategies to increase efficiencies of information collection and decrease burden will be used, such as administering the capacity and readiness assessments electronically. Furthermore, information from the implementation evaluation will be used in a feedback loop to implementers. This information will help the TA provider determine whether program components are being delivered adequately, and which areas of program implementation may require subsequent or specific technical assistance. Feeding information back to the local health department and schools (e.g., report from capacity and readiness assessments) and to implementers from the implementation evaluation will increase the relevance of the evaluation for the grantees and increase participation. Finally, information about the integrity of the random assignment process is important to subsequent outcome analyses.

B.4. Test of Procedures or Methods to be Undertaken

All surveys and procedures will be reviewed and “walked through” by the evaluation contractor and the liaison at the health department. If possible, we will pilot test the student survey with a maximum of 9 middle school students, the parent survey with a maximum of 9 parents, and a maximum of 9 educators who would not be eligible to be in our sampling frame to test survey length. We will use this process to assure that the student survey is of a management length for students to complete in a class period, and to test and fine-tune recruitment procedures (email, mail) for parents.

All capacity/readiness assessments have been piloted with 4 local health departments by the contractor who developed the instrument.

The fidelity instruments were adapted from instruments that have been used in past evaluations of the programs we utilize in Dating Matters (e.g., Safe Dates and Parents Matter!).

Survey instruments have been developed with the extensive input of expert consultants prior to awarding the evaluation contract, and through a secondary process of expert consultation under the evaluation contract. When possible, and in most cases, we have utilized pre-existing surveys that have been used in samples similar to ours. Sources of items are documented; all items used in the outcome instruments have been tested for reliability and validity. We worked with consultants within and outside of CDC to determine the appropriate length and reading level for the respondent population of the three surveys (middle school student survey; parent survey; educator survey), noting in particular the high-risk urban settings in which this study is being conducted by design. The student instruments are designed to be completed within one 45-minute classroom period.

For the focus groups, we relied upon protocols used by other researchers studying primary prevention programs in the area of youth dating violence (Taylor, Stein, Wood and Mumford, 2011). Also, given the organic nature of focus groups the questions set forth in our submitted focus group should be considered exemplary. Focus groups are used as an exploratory research tool (Greenbaum 1993; Vaughn, Schumm & Sinagub 1996), and involve the explicit use of group interaction and exploration to produce data and insights that would be less accessible without the interaction found in a group and exploratory probing (Morgan 1988:12). Given the exploratory approach inherent in focus groups, there could be some deviation from the focus protocol included in this OMB submission (Greenbaum, 1993).

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

All instruments and procedures have been reviewed extensively by CDC. The following individuals have worked closely in developing the instrument and procedures that will be used, and will be responsible for data analysis: Phyllis Holditch Niolon, Natasha Elkovitch Latzman, Linda Anne Valle, Andra Tharp, Dawn Fowler, Tessa Burton, Kimberly Friere, Craig Bryant, Frank Luo, and Kevin Vagi, and from the evaluation contractor (NORC at the University of Chicago) team, Bruce Taylor, Elizabeth Mumford, Deborah Gorman-Smith, Michael Schoeny, Dorothy Espelage, Elizabeth Hair, and Pamela Loose. The expert panel on evaluation design (see Supporting Statement A) also consulted on statistical concerns with various evaluation designs. The expert panel on capacity and readiness (see Supporting Statement A) consulted on the development and use of the capacity and readiness assessments. The evaluation contractors are in the process of being identified, but offerors have extensive expertise in methodological design and data collection. CDC staff listed above will be responsible for analyzing the data.

References.

Greenbaum, T. L. (1993). The handbook for focus group research. New York: Macmillan.

Krueger, R. A. (1994). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage.

Morgan, D. L. (1988). Focus groups as qualitative research. Newbury Park, CA: Sage.

Taylor, B.G, Stein, N., Woods, D. and Mumford, E.A. (2011). Dating violence prevention programs in

New York City public middle schools: A multi-level experimental evaluation. Final Report.

National Institute of Justice, Washington, DC: Government Printing Office.

Vaughn, S., Schumm, J. S., & Sinagub, J. (1996). Focus group interviews in education and psychology. Thousand Oaks, CA: Sage.