October 2007
HHS/CDC/NCIPC
SUPPORTING STATEMENT FOR OMB INFORMATION COLLECTION REQUEST
EVALUATION OF THE SAFE DATES PROJECT
Supported by:
National Center for Injury Prevention and Control
Centers for Disease Control and Prevention
Rita Noonan
Centers for Disease Control and Prevention
Division of Violence Prevention
4770 Buford Highway NE
Mailstop K-60
Atlanta, GA 30341
Table of Contents
Section Page
A. JUSTIFICATION 4
1. Circumstances Making the Collection of Information Necessary 4
2. Purpose and Use of the Information Collection 6
3. Use of Improved Information Technology and Burden Reduction 8
4. Efforts to Identify Duplication and Use of Similar Information 9
5. Impact on Small Businesses or Other Small Entities 10
6. Consequences of Collecting the Information Less Frequently 10
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 11
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 11
9. Explanation of Any Payment or Gift to Respondents 11
10. Assurance of Confidentiality Provided to Respondents 15
11. Justification for Sensitive Questions 17
12. Estimates of Annualized Burden Hours and Costs 17
13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 21
14. Annualized Cost to the Federal Government 2
15. Explanation for Program Changes or Adjustments 22
16. Plans for Tabulation and Publication and Project Time Schedule 2
17. Reason(s) Display of OMB Expiration Date is Inappropriate 2
18. Exceptions to Certification for Paperwork Reduction Act Submissions 2
B. STATISTICAL METHODS (Used for Collection of Information Employing Statistical Methods) 2
1. Respondent Universe and Sampling Methods 2
2. Procedures for the Collection of Information 28
3. Methods to Maximize Response Rates and Deal with Nonresponse 30
4. Test of Procedures or Methods to be Undertaken 32
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 32
References 33
Attachments
Implementation Evaluation Data Collection Instruments
Effectiveness Evaluation Data Collection Instrument
Cost Data Collection Instruments
Relevant Portion of Public Health Service Act, Title III, General Powers and Duties of Public Health Service [42 U.S.C. § 301 (241.)a.]
Federal Register Notice to the Public
RTI Institutional Review Board Approval Notice
Summary of Pilot Test Results
Assurances of Confidentiality and Study Descriptions Provided to Respondents
Lead Letter to Parents to Recruit Students for Effectiveness Evaluation
Lead Letter to School Staff for Implementation/Cost Evaluation
Telephone and Email Follow-Up for Implementation Survey Non-Response
List of Tables
Number Page
1 Implementation Evaluation Research Questions 6
2 Effectiveness Evaluation Research Questions 7
3 Cost Evaluation Research Questions 7
4 Persons Consulted Outside the Agency 12
5 Study Participation Incentives 13
6 RTI Studies Involving School Incentives and Corresponding Response Rates 14
7 RTI Studies Involving Consent Tracking Incentives and Corresponding Response Rates 14
8 RTI Studies Involving Student Consent Return Incentives and Corresponding Response
Rates 14
9 RTI Studies Involving Adult Respondent Incentives and Corresponding Response Rates 15
10 Description of Sensitive Questions, Justification for Inclusion, and Use of Data 18
11 Annualized Burden for Respondents 20
12 Annualized Cost for Respondents 21
13 Time Schedule for the Entire Project 25
14 Numbers of Student and Adult Respondents 28
1. Circumstances Making the Collection of Information Necessary
The Centers for Disease Control (CDC) requests an information collection request for the Evaluation of the Safe Dates Project. Safe Dates is a research-based adolescent dating violence prevention program. The Safe Dates program includes a nine-session dating abuse curriculum, a play about dating abuse, and a poster contest.
Teen dating violence is a major public health problem that affects all strata of society and all major social institutions, including the medical and mental health care, social services, economic, and criminal justice systems. Recent Centers for Disease Control and Prevention (CDC) data indicate that one in eleven high school students reports being physically hurt by their dating partner in the last year, which translates into nearly 1.5 million teens per year. Students who reported dating violence victimization were significantly more likely to suffer from poor grades and engage in other risky health behaviors such as binge drinking, fighting, current sexual activity, and suicide attempts. These threats to adolescent health can be minimized by implementing effective prevention programs, such as Safe Dates, in schools and communities. Early research showed that intimate partner violence often begins during adolescence, with the first episode typically occurring by age 15 (Henton, Cate, Koval, Lloyd, & Christopher., 1983). Recent surveys report that as many as 25 percent of adolescents experience dating violence victimization and/or perpetration, resulting in severe physical, psychological, and developmental consequences (Bergman, 1992; Coker, McKeown, Sanderson, Davis, Valois, & Huebner, 2000; Foshee, Linder, Bauman, Langwick, Arriaga, Heath, et al., 1996; Jezl, Molidor, & Wright, 1996; Malik, Sorenson, & Aneshensel, 1997; Silverman, Raj, Mucci, & Hathaway, 2001).
In contrast to interventions directed towards adults, which are almost always targeted programs focusing on victims, perpetrators, or criminal justice responses to intimate partner violence, adolescent dating violence prevention programs are usually school-based and oriented much more toward prevention and universal populations. However, relatively few evaluations of adolescent dating violence prevention programs have been evaluated, and most did not use experimental evaluation designs (Avery-Leaf, Cascardi, O’Leary, & Cano, 1997; Jaffe, Sudermann, Reitzel, & Killip, 1992; Jones, 1991; Krajewski, Rybarik, Dosch, & Gilmore, 1996; Lavoie, Vezina, Piche, & Boivin, 1995) and did not measure actual violence outcomes (Avery-Leaf et al., 1997; Jaffe et al., 1992; Jones, 1991; Krajewski et al., 1996; Lavoie et al., 1995; Macgowan, 1997; Weisz & Black, 2001). In contrast, the Safe Dates Project employed a randomized experimental design and measured actual psychological, physical, and sexual perpetration and victimization outcomes, in addition to moderators and mediators of the effects of the program on such outcomes (Foshee et al., 1996). Five waves of data spanning 4 years were analyzed among a large sample of adolescents from the general school population in a rural North Carolina county (Foshee, Bauman, Ennett, Suchindran, Benefield, & Linder, 2005). Significant program effects were found at all four follow-up periods on psychological, moderate physical, and sexual dating violence perpetration and moderate physical dating violence victimization (Foshee et al., 2005). Positive effects were also found on sexual victimization. However, that effort was limited to one rural county in North Carolina. As a result, it is unclear whether the results would generalize to more diverse student settings. Moreover, the original study did not collect information on implementation drivers (and how changes in those factors impact program effectiveness) or on program costs. The proposed evaluation presents a unique opportunity to simultaneously evaluate the effectiveness, cost, and implementation of an adolescent dating violence prevention program among a universal population of diverse school settings, using a rigorous experimental evaluation design with measurement of actual violence.
The Evaluation of the Safe Dates Project is designed to assess three aspects of the Safe Dates project:
Implementation. The evaluation will examine how program fit and implementation climate (bolstered by the implementation drivers of teacher training and observation in one of the experimental conditions) affect implementation fidelity.
Effectiveness. The evaluation will assess the program’s impact on desired outcomes, including prevention of and reductions in dating violence victimization and perpetration (including psychological abuse, stalking, physical violence, and sexual violence) among high school students). The influence of program implementation on effectiveness will also be evaluated.
Cost. The evaluation will determine the time and monetary costs of delivering Safe Dates in a school setting to assess cost-effectiveness and cost-utility of the program.
The evaluation will require participation from staff and students at 54 schools (18 treatment schools receiving the ‘enhanced’ Safe Dates program with teacher training and observation, 18 treatment schools receiving the Safe Dates program without teacher training and without classroom observation, and a control group of 18 control schools not receiving the Safe Dates program).
The research will include 17 data collections: 4 targeted at students, 3 targeted at school principals, 4 targeted at school prevention coordinators, and 6 targeted at teachers. The 17 data collections include: (1) a baseline survey of students at treatment schools receiving the Safe Dates program (with or without teacher training and observation) and students at control schools, (2) a baseline survey of principals at all schools, (3) a baseline survey of prevention coordinators at all schools, (4) a baseline survey of teachers delivering the Safe Dates program, (5) a time use and cost survey among teachers delivering the Safe Dates program with training and observation, (6) a survey after the first class’ fifth program session of teachers at treatment schools, (7) a survey after the last class’ fifth program session of teachers at treatment schools, (8) a survey after the first class’ ninth program session of teachers at treatment schools, (9) A survey after the last class’ ninth program session of teachers at treatment schools, (10) a survey after the fifth program session of a subset of students at treatment schools, (11) a survey after the ninth program session of a subset of students at treatment schools, (12) a mid-implementation survey of principals at treatment schools, (13) a mid-implementation survey of prevention coordinators at treatment schools, (14) an end-of-school-year survey of principals at treatment and control schools, (15) an end-of-school-year survey of prevention coordinators at treatment and control schools, (16) a follow-up survey of students at treatment and control schools after 13 months, (17) a survey of prevention coordinators at treatment and control schools after 13 months. This submission requests approval for all 17 surveys. No other data (e.g., school records, etc.) will be gathered. Copies of these surveys are included in Attachments A, B, and C.
Authority for CDC’s National Center for Injury Prevention and Control to collect this data is granted by Section 301 of the Public Health Service Act (42 U.S.C. 241) (Attachment D).
2. Purpose and Use of the Information Collection
Implementation. The purpose of the implementation evaluation is to provide a detailed description of the relationships between the Safe Dates program’s fit (including school motives for implementing, potency of the school’s violence prevention initiatives, and competing or complementary programs), implementation climate (including leadership support, goal clarity, dedicated resources, teacher training and observation, reinforcement, and removal of barriers to implementation), and implementation fidelity (whether or not the program is implemented as intended). CDC will use the implementation evaluation data to (1) understand how well the Safe Dates program fits into a variety of school settings; (2) assess how implementation climate affects implementation effectiveness (e.g., fidelity), (3) evaluate the extent to which the experimental interventions (including teacher training and observation and other support and resources available to teachers delivering the program) enhance implementation fidelity; (4) explore the extent to which activities were similar or different across teachers, schools, states, treatment conditions, and varying levels of program fit and implementation climate; (5) guide CDC and other federal staff to develop policy and strategies that will effectively improve implementation, dissemination, and effectiveness of adolescent dating violence prevention programs; (6) help school staff better support teachers in delivering adolescent violence prevention programs; and (7) guide future implementation and dissemination evaluation studies. Implementation fidelity is the primary proximal outcome that will be examined. Distal outcomes also will be assessed at the school level (e.g., post – study plans to persist with the program), teacher level (e.g., perceived importance of program content; satisfaction with program) and the student level (e.g., expected future benefits from program involvement). Key research questions for the implementation evaluation are presented in Table 1. Copies of the implementation evaluation data collection instruments are included in Attachment A. Implementation evaluation instruments administered to staff at control schools will be revised to avoid referring specifically to the Safe Dates program and will instead ask about general violence prevention activities.
Table 1. Implementation Evaluation Research Questions
1. Did the provision of two implementation drivers, teacher training and observation, significantly impact the extent to which the program was implemented with fidelity by teachers in school settings above and beyond the level of fidelity achieved using a curriculum guide only? 2. What additional factors (e.g., program fit, other aspects of implementation climate) explained the extent to which the program was implemented with fidelity and stakeholder reactions to the program? |
Effectiveness. The purpose of the effectiveness evaluation is to evaluate the effect of the Safe Dates program on desired outcomes, including main effects of the program on physical and sexual dating violence perpetration and victimization; mediator variables targeted by the Safe Dates program (and found to be significant mediators by Foshee et al., 1998), and variance through potential moderator variables. Key research questions for the effectiveness evaluation are presented in Table 2. Changes in the following outcomes will be examined: dating violence perpetration and victimization, dating violence norms, gender stereotyping, conflict management skills, belief in the need for help when violence occurs, and awareness of services. Information will be collected from adolescent respondents to determine whether the effects of the program varied based on variables including previous history of dating violence and demographic characteristics. A copy of the effectiveness evaluation data collection instrument is in Attachment B.
Table 2. Effectiveness Evaluation Research Questions
a. Previous history of dating violence b. Demographics (e.g., race/ethnicity, gender, socioeconomic status)
a. Dating violence norms b. Gender stereotyping c. Conflict management skills d. Belief in the need for help when violence occurs e. Awareness of services
|
Cost. The purpose of the cost data collection is to describe the economic (monetary) and time (opportunity) costs associated with program delivery. These costs will then be combined with the effectiveness results to conduct a cost-effectiveness and cost-utility analysis. Cost-effectiveness of Safe Dates will be estimated on eight outcomes: (1) any perpetration of sexual violence, (2) any victimization of sexual violence, (3) any perpetration of physical violence, (4) any victimization of physical violence, (5) any perpetration of psychological violence, (6) any victimization of psychological violence, (7) any perpetration of stalking, or (8) any victimization of stalking. Results will be expressed using summary measures, such as the average cost per unit of average outcome change (e.g., cost per reduced prevalence of sexual violence victimization). Key research questions for the cost data collection are presented in Table 3. Copies of the cost data collection instruments are included in Attachment C.
Table 3. Cost Evaluation Research Questions
1. What are the total costs of implementation? 2. How are these costs split between monetary and time (opportunity) costs? 3. How are the costs allocated across primary Safe Dates activities? 4. How are the costs distributed between labor and non-labor costs? 5. What is the cost-effectiveness of Safe Dates? |
The information obtained from the proposed data collection activities will be used to inform CDC, University of South Florida (USF, contracted by CDC to assist with implementation evaluation), schools, policy makers, parents, prevention practitioners, and researchers about the implementation of the Safe Dates program and its effectiveness and cost-effectiveness in preventing dating violence among adolescents. This information will enable CDC to more effectively address dating violence among adolescents. Finally, the data provided from the proposed evaluation will enable CDC to determine how and whether to implement, disseminate, and/or improve the Safe Dates program.
CDC will disseminate results to peer-reviewed journal readers and professional conference participants, as well as through an executive summary and a full report. The executive summary will be written in clear language to be understandable by a wide range of audiences (parents, schools, policy makers, researchers). The full report will include an overview of background literature to provide contextual information about the purpose of the Safe Dates program and evaluation approach, a detailed summary of evaluation methods and activities, the evaluation results, discussion of findings in comparison with those of other relevant program evaluations, strengths and limitations of the evaluation, and recommendations for future evaluations of this scope for schools, evaluators, and policy makers.
3. Use of Improved Information Technology and Burden Reduction
Three types of data will be collected: implementation data from students, teachers, school prevention coordinators, principals; effectiveness (or outcome) data among students; and cost data from teachers delivering the Safe Dates program.
Implementation and effectiveness data will be collected from students using classroom-administered scannable paper and pencil questionnaires. Although we considered alternate modes of administration of these questionnaires, conducting student questionnaires by telephone would be extremely time-consuming and costly, given the number of students (n=10,000) expected to participate. Further, we believe there would be serious issues of privacy and confidentiality if students were asked to disclose sensitive information regarding violence over the telephone. We considered having students complete Web questionnaires but determined that not every student has access to the hard/software needed. Even if each classroom had a computer, there would be no privacy for the students and little availability for all students to use the computer to complete the survey in a timely manner. Finally, we considered collecting student data using mail questionnaires, but we felt that students in the ninth grade are not reliable enough to complete and mail the questionnaire without parental supervision. This option also raised questions of confidentiality and privacy regarding mailing data about violent experiences. We anticipate a higher response rate using classroom survey administration relative to mail survey, particularly because many students will have no knowledge of the Safe Dates program at baseline (nor are control group students at follow-up likely to have knowledge of the program). In sum, we determined machine-scannable, paper and pencil questionnaires would be the best methodology for collecting student effectiveness data, while minimizing potential biases that might jeopardize our ability to address the evaluation research questions.
Implementation and cost data will be gathered from school staff via on-line Web questionnaires. Copies of all questionnaires are located in Attachments A and C. During the planning phases of this project, we considered alternate modes of gathering the implementation and cost data, each of which had important disadvantages. One method considered was to collect the implementation and cost information via telephone questionnaires. However, because school staff are in class most of their work day, they usually are not available by telephone during the school day except for very limited periods of time. Thus, telephone interviews would be time-consuming and would likely result in a high rate of incomplete data. We also considered administering paper and pencil questionnaires and having respondents fax the completed questionnaires. The main disadvantage of that mode is that after completing each questionnaire, it would take approximately 5 minutes to fax each of the approximately 1,240 questionnaires, and it is likely that there would be transmission problems and difficulties reading the pages of the fax that would require retransmission. As a result, we anticipated poorer response rates and missing data if paper implementation and cost questionnaires were faxed. Further, once the data are received, they must be edited, coded, keyed, and verified to avoid errors in the database.
Utilization of the Worldwide Web has the advantages of being able to allow respondents to complete as much of each questionnaire as desired in one sitting and to continue at another time, minimizing the possibility of respondent error by electronically skipping questions that are not applicable to a particular respondent, and creating the least burden to the respondent. Current technology requires minimal programming expense for Web-based questionnaires, and Web questionnaires eliminate the need for printing paper questionnaires and postage for mailing. Further, as each questionnaire is completed, the data are automatically placed into a database maintained by RTI without the need for additional time to fax or mail the questionnaire or code, key, and verify it. Finally, since data are automatically entered into the database, they can be analyzed in real-time, and a variety of reports can be generated quickly. In addition, it is highly likely that school teachers and administrators will have adequate computer hardware/software at home or at school to complete a Web-based questionnaire. We anticipate higher response rates to the on-line Web implementation and cost questionnaires than we could have anticipated for a mailed questionnaire. In sum, we have determined Web surveys to be the best methodology for data collection from school staff respondents. However, we will offer teachers the option to complete surveys by paper and pencil or electronically and fax or e-mail them if they prefer. Teachers will be allowed to complete these surveys online or through fax or email because data are non-sensitive, unlike the effectiveness data.
4. Efforts to Identify Duplication and Use of Similar Information
We have carefully reviewed existing data sets to determine whether any of them are sufficiently similar or could be modified to address the need for information on the implementation and cost-effectiveness of the Safe Dates program on adolescent dating violence. Efforts to avoid duplication include a review of existing studies involving intimate partner violence in various populations. We first explored the idea of utilizing existing data from the original Safe Dates evaluation (Foshee, Bauman, Arriaga, Helms, Koch, & Linder, 1998). However, it would be difficult for us to generalize these findings from one rural county in North Carolina to a national population of adolescents. Furthermore, the original Safe Dates evaluation did not collect data on implementation drivers and cost-effectiveness that would be vital to the purpose of our study.
We then examined the idea of using existing data from the 2005 Youth Risk Behavior Survey (YRBS), a national study (OMB ID 0920-0493) that asked students in grades 9-12 about dating violence victimization and experiences with forced sexual intercourse (Eaton, Kann, Kinchen, Ross, Hawkins, Harris, et al., 2006). Dating violence victimization questions included measures of being hit, slapped, or physically hurt on purpose by a boyfriend or girlfriend during the 12 months preceding the survey. However, questions about forced sexual intercourse did not ask respondents to specify whether the experience involved a dating partner. Furthermore, the survey utilized a cross-sectional design, which would not be useful for the purposes of determining implementation drivers, effectiveness, and cost-effectiveness in our study. Moreover, the data did not include implementation or cost measures of any prevention programs for students, which would not be sufficient for our study aims. The National Longitudinal Study of Adolescent Health (AddHealth) examined determinants of health and health-related behaviors of adolescents in grades 7-12 (Resnick, Bearman, Blum, Bauman, Harris, Jones, et al., 1997). Second wave analysis assessed the prevalence of psychological and physical violence victimization in adolescent dating relationships (Halpern, Oslak, Young, Martin, & Kupper, 2001). However, the study did not include measures of perpetration, stalking, or sexual violence among dating partners and did not collect exposure, implementation, or cost data of any dating violence prevention programs.
Finally, we investigated the possibility of using existing data from the National Violence against Women Survey (NVAWS), a telephone survey that interviewed 8,000 adult women and 8,005 adult men to provide data on women’s and men’s experiences with violent victimization (Tjaden & Thoennes, 2000). However, the NVAWS excluded individuals younger than age 18, which would prevent examination of our study’s targeted population of ninth graders. In addition, the NVAWS was cross-sectional in nature and similarly did not evaluate any exposure, implementation, or cost-effectiveness measures of prevention programs. We also reviewed program announcements, requests for applications (RFAs), and requests for proposals (RFPs) from other federal agencies. To date, no duplication of effort has been identified.
5. Impact on Small Businesses or Other Small Entities
No small businesses will be involved in this study.
6. Consequences of Collecting the Information Less Frequently
The present study will provide the primary data needed for federal policy makers to assess the implementation, effectiveness, and cost of the Safe Dates program on dating violence among adolescents.
Implementation. If this evaluation were not conducted, it would not be possible to determine what predicts strong implementation of the program or its value and impact on the lives of the people it is intended to serve. Failure to collect these data could preclude effective use of school resources to benefit students. Less frequent implementation evaluation data collection would not allow for measurement of dynamic implementation characteristics and predictors as they change and interact with program effects.
Effectiveness. Less frequent effectiveness evaluation data collection would not allow for measurement of pre- and post-implementation characteristics of students.
Cost. Detailed information about time and money spent on the program will need to be captured quickly after each activity is conducted. Of note, we observed during our pilot test of the cost data collection forms that forms that were completed less frequently (with longer time elapsed between program activities and recording of time and money spent) contained data that was less accurate and more an overall estimation of time spent in general, rather than detailed accountings of variation between class periods and program lessons. Collection of these data will be limited to teachers at schools receiving the Safe Dates program and teacher training and observation; from these data, we can estimate costs among teachers at schools receiving the Safe Dates program without teacher training and observation.
There are no legal obstacles to reduce the burden.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances that require the data collection to be conducted in a manner inconsistent with 5 CRF 1320.5 (d)(2).
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A. The Federal Register 60-day notice was published on January 22, 2007, in Volume 72, Number 13, Pages 2697-2698 (see Attachment E). There were no public comments.
B. A list of consultants on this project is provided in Table 4. There were no unresolved issues. It is important to note that the implementation evaluation scales and surveys used in this study have been employed in previous organizational behavior or psychology studies (e.g., Klein, Conn, & Sorra, 2001; Panzano & Roth, 2006; Panzano, Roth, Crane-Ross, et al., 2004; Vaidyanathan, 2004; Yeo, 2006), as well as ongoing research (e.g., Crane-Ross & Maychack, 2005; Holahan, Aronson, Jurkat, & Schoorman, 2004; Slep & Heyman, 2006).
9. Explanation of Any Payment or Gift to Respondents
Numerous empirical studies have shown that incentives can significantly increase response rates (Abreu & Winters, 1999; Shettle & Mooney, 1999; Singer, Van Hoewyk, Gebler, Raghunathan, & McGonagle, 1999). Table 5 lists the incentives that will be offered to schools, staff, and students for their participation in the study.
Table 4. Persons Consulted Outside the Agency
Dean Fixsen, Ph.D., Co-Director National Implementation Research Network University of South Florida 13301
Bruce B. Downs Blvd., MHC 2312 |
(813) 974-4446
(phone) |
Vangie Foshee, Ph.D., Associate Professor University of North Carolina School of Public Health Department
of Health Behavior and Health Education |
(919) 966-6616
(phone) |
Sandra
Naoom, M.S.P.H., Associate Director University of South Florida 13301
Bruce B. Downs Blvd., MHC 2312 |
(813)
974-2312 (phone) |
Phyllis
Panzano, Ph.D., President |
(614)
221-1474 (phone) |
School incentives. Experience has taught us that schools can be reluctant to burden their staff or to decrease classroom time devoted to core academic education activities. To encourage cooperation, each school will receive a cash incentive of $1,500 for the first year of participation and $1,000 for participation in the second year. A school-level incentive in this amount is needed each year in order to ensure cooperation with data collection, particularly since school staff and administration are subject to change each year. These incentives also are offered as a small “thank you” to the participating schools in all three study conditions and are intended to recognize the time burden placed on teachers and school administrators and to convey appreciation for their contribution to this important study. We expect that one of every 10 schools we contact will agree to participate in the study, partly as a result of the incentive offered; thus, we expect to contact approximately 540 schools in order to successfully recruit 54 schools for this study. Table 6 shows several studies where we have employed equivalent school incentives and response rates achieved that are equal to or better than the participation rate we expect to achieve. Although these studies differ in other respects that could account for some variability in response rates, overall, school incentives of $1,000 to $1,500 per year were generally associated with higher participation rates compared with no incentive.
Consent tracking incentives. The RTI Institutional Review Board (IRB) requires active parental consent for participation of students younger than age 18 in a research study. Because obtaining active parent consent in schools is a very difficult task, we will provide a $25 gift card to each teacher for each classroom in which at least 90 percent of the parental consent forms are returned, whether or not the parents allow students to participate. Gift cards may be used to purchase classroom supplies or pay for a class party. Table 7 shows several studies where RTI has employed parent consent tracking incentives and the parent consent form return rates achieved.
Table 5. Study Participation Incentives
Respondent |
Type of Participation |
Amount |
Number of Respondents |
|
Year 1 |
Year 2 |
|||
Schools |
School participation in study |
$1,500 in Year 1 $1,000 in Year 2 |
54 |
54 |
Teachers |
Consent tracking (estimate 10 classrooms per school) |
$25 per classroom achieving at least a 90% signed parent consent form return rate |
540 |
— |
Students |
Parent consent incentive |
$0.50 token per student returning a signed parent consent form |
10,693 |
— |
Principal |
Baseline implementation questionnaire |
$15 gift card |
49 |
— |
Mid-implementation questionnaire |
$15 gift card |
32 |
— |
|
End-of-school-year implementation questionnaire |
$15 gift card |
49 |
— |
|
Prevention Coordinator |
Baseline implementation questionnaire |
$15 gift card |
49 |
— |
Mid-implementation questionnaire |
$15 gift card |
32 |
— |
|
End-of-school-year survey |
$15 gift card |
49 |
|
|
Follow-up questionnaire |
$15 gift card |
— |
49 |
|
Teachers |
Baseline implementation questionnaire |
$15 gift card |
98 |
— |
Cost questionnaires |
$50 gift card |
49 |
— |
|
First class’ fifth session mid-implementation questionnaire |
$15 gift card |
98 |
— |
|
Last class’ fifth session mid-implementation questionnaire |
$15 gift card |
98 |
|
|
First class’ ninth session mid-implementation questionnaire |
$15 gift card |
98 |
|
|
Last class’ ninth session mid-implementation questionnaire |
$15 gift card |
98 |
— |
|
Students |
Follow-up effectiveness questionnaire for school dropouts |
$25 gift card |
--- |
325 |
Follow-up effectiveness questionnaire for transfers |
$25 gift card |
--- |
406 |
|
Table 6. RTI Studies Involving School Incentives and Corresponding Response Rates
RTI Study |
School Incentive Provided |
Participation Rate Achieved |
Impact Evaluation of a School-based Violence Prevention Program (2004 – 2009) |
$1500 each year of participation (3 years) |
10% |
Middle School Coordinator Initiative (1999 – 2004) |
$1500 each year of participation for completion of student surveys (3 years) |
33% |
Program for International Student Assessment (PISA) (2004 – 2009) |
Conference for schools that enrolled in study (value of $1500) or check for $1500 (1 year) |
70% |
Table 7. RTI Studies Involving Consent Tracking Incentives and Corresponding Response Rates
RTI Study |
Incentive Provided |
Overall Rate of Consent Forms Signed and Returned |
Safe Schools/ Healthy Students Initiative (1999–2004) |
$25 per classroom achieving a minimum of 70% active parental consent forms signed and returned |
78% |
Middle School Coordinator Initiative (1999–2004) |
$25 per classroom achieving a minimum of 70% active parental consent forms signed and returned |
78% in Year 1 77% in Year 2 81% in Year 3 |
Impact Evaluation of a School-based Violence Prevention Program (2004–2009) |
$25 per classroom achieving a minimum of 90% active parental consent forms signed and returned |
89% in Year 1 |
Student incentives for completed parent consent forms. We will provide students with a $0.50 token the student can redeem in the school book store or cafeteria for returning their signed parental consent form, whether or not they obtain permission to participate in the study. Table 8 shows several studies that have employed these student consent return incentives and the response rates achieved.
Table 8. RTI Studies Involving Student Consent Return Incentives and Corresponding Response Rates
RTI Study |
Age of Respondents |
Incentive Provided |
Response Rate Achieved |
Middle School Coordinator Initiative (1999 – 2004) |
12 to 14 years |
$0.50 token (highlighter, pen, ice cream, school bookstore) |
78% in Year 1 77% in Year 2 81% in Year 3 |
Progress in International Reading Literacy Study (PIRLS) (2004 – 2009) |
Fourth-grade |
Book selection from Scholastic books, medal ($0.87 value) |
92% |
School staff incentives. We will give each principal, teacher, and prevention coordinator a gift card incentive in the amount of $15 each time that he or she completes an implementation evaluation questionnaire, for a total of $45 per principal, $75 per teacher, and $60 per prevention coordinator. Table 9 summarizes several studies that have used comparable incentives and the response rates achieved. Teachers completing the cost data collection survey will receive an additional incentive of a $50 gift card.
Table 9. RTI Studies Involving Adult Respondent Incentives and Corresponding Response Rates
RTI Study |
Respondents |
Incentive Provided |
Response Rate Achieved |
Parent Corps (2003 –2007) |
Parents |
$10 per questionnaire completed |
40% |
School Health Policies and Programs Study (1997 – 2001) |
State and District school staff |
$25 |
State staff: 100% completed at least one of seven surveys District staff: 75% completed at least one of seven surveys |
National Survey of Postsecondary Faculty (2002 – 2007) |
Faculty and instructional staff |
$30 check or gift certificate |
76% |
Student incentives for follow-up effectiveness survey. We will provide students who complete the baseline effectiveness survey but drop out of school before the follow-up effectiveness survey with a $25 gift card for completing the follow-up effectiveness survey by mail. We will provide students who complete the baseline effectiveness survey but transfer to a school not participating in the study before the follow-up effectiveness survey with a $25 gift card for completing the follow-up survey by mail.
10. Assurance of Confidentiality Provided to Respondents
All procedures have been developed, in accordance with federal, state, and local guidelines, to ensure that the rights, privacy, and confidentiality of school principals, teachers, prevention coordinators, and students are protected and that the relationships between students and school staff will be protected and maintained. The CDC National Center for Injury Control and Prevention’s human subjects coordinator has determined that CDC will not be engaged in human subjects research: CDC will not directly obtain data by intervening or interacting with participants and CDC will not have access to identifiable (including coded) private data. In addition, the RTI Institutional Review Board (IRB) reviewed all instruments, informed consent materials, and procedures to ensure that the rights of individuals participating in the study are safeguarded. A copy of the RTI IRB approval notice is included as Attachment F. A pilot test of these procedures was conducted, and no problems were identified (see Section B.4 and Attachment G for a summary of the pilot test).
All respondents will be assured that the information they provide will be treated in a confidential manner and will be used only for the purpose of this research, unless otherwise compelled by law. Copies of consent forms including this assurance of confidentiality provided in writing to parents and scripts to be read to students are provided in Attachment H. All teachers and school staff will be informed that what they disclose will not have any effect on their employment by or relationship with the school. All students will be informed that what students report in the surveys will not have any effect on their academic performance in the school, relationship with the school, or the services they receive at the school. Trained survey administrators will assure parents and students that student names will not be associated with responses provided. Respondents will be told that the information obtained from all of the surveys will be combined into a summary report so that details of individual questionnaires cannot be linked to a specific participant.
The effectiveness survey does not specifically ask about danger of immediate or potential physical harm. If, however, a student respondent verbally volunteers this information, RTI will report it to the appropriate authorities. Specifically, an incident report will be completed and distributed to the RTI project leader within 1 business day and to the RTI IRB within 2 business days. This exception to confidentiality will be fully described to students and parents through informed consent and assent forms and an assent script read to students. Data collectors will be asked to document all cases that would require reporting and to provide this information to the RTI project leader, the RTI IRB, and appropriate authorities. Because the survey contains sensitive questions about student physical and sexual dating violence perpetration and victimization, students will be reminded during the active parent consent and active adolescent assent process that their answers will not be shared with anyone outside the RTI project team, including their parent or guardian, unless required by law. Trained survey administrators will let students and parents know before the survey that any disclosure of potential harm to the student or others will result in a report to the proper authorities, as well as the parents of the student.
It is possible that another student could view survey responses while survey administration is in progress, so students will be spaced out around the room to avoid the possibility of another student being able to view survey responses. After completion of the survey, students will place questionnaires in an RTI-provided envelope. RTI field staff will seal the envelope, and it will not be unsealed until it arrives at RTI. School staff will not have access to any survey information provided by individual students.
RTI field staff will send the envelopes to RTI via pre-paid Federal Express, marked as confidential. No respondent names will be included in the Federal Express package of completed instruments to RTI. Assent/consent forms and completed surveys will be shipped to RTI separately and on different days. RTI must be notified and provided a tracking number for each shipment. If shipments do not arrive at RTI as scheduled, RTI will immediately initiate tracing through Federal Express. RTI will monitor this process throughout the data collection period.
To ensure data security, all RTI project staff (including data collection staff employed by RTI’s subcontractor, Headway Staffing Services) are required to adhere to strict standards and to sign confidentiality agreements as a condition of employment on this project. Effectiveness evaluation survey administrators will be thoroughly educated in methods of maximizing a parent and student understanding of the government’s commitment to confidentiality. Hard-copy data collection forms will be delivered to a locked area for receipt and processing. Individual identifying information will be kept separate from survey responses, and ID numbers will be assigned to participants for identification purposes. Mid-implementation surveys will be anonymous. RTI project staff will never leave completed consent/assent forms or questionnaires unattended. All completed consent/assents forms and the list of participant names and ID numbers will be stored in separate locked filing cabinets only accessible to authorized RTI personnel. Survey responses will be stored on a secure, password-protected computer shared drive. RTI maintains restricted access to all data preparation areas (i.e., receipt, coding, and data entry). All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only. Individual identifying information will be maintained separately from completed data collection forms and from computerized data files used for analysis. No respondent identifiers will be contained in public use data files made available from the study, and no data will be released in a form that identifies individual respondents.
11. Justification for Sensitive Questions
No sensitive questions will be asked as part of the cost evaluation surveys. Some questions included in student effectiveness evaluation instruments and in the teacher and prevention coordinator implementation instruments, however, might be considered sensitive by some respondents. Table 10 identifies the sensitive questions, explains the justification for their inclusion in the surveys, and describes how the data will be used. The informed consent protocol apprises parents and students that these topics will be covered during the surveys. These questions are included in the surveys because of their importance in understanding changes in dating violence among students, the potential mediating effect of dating violence norms on the main effect of the Safe Dates program on dating violence among students, and the potential moderating effect of alcohol or other drug use at the time of violence on the main effects of the Safe Dates program on dating violence among students. All sensitive questions are validated items previously used by Foshee and colleagues (1998). As with all information collected, these data will be presented with all identifiers removed, including school and school district identifiers.
12. Estimates of Annualized Burden Hours and Costs
The average annual response burden is estimated at 7,056 hours (9,368 hours in year 1 and 4,744 hours in year 2). Ninety-seven percent of the total burden is attributed to students and takes place over four classroom surveys spread out over two years when they fill out the implementation and effectiveness questionnaires. Tables 11 and 12 provide details about how this estimate was calculated. These estimates were generated based on our experience during the pilot phase of the time it takes to complete each survey and on the number of individuals expected to complete each one.
Student effectiveness surveys: Group-administered Teleform scannable questionnaires will be used. We anticipate 10,158 completed baseline questionnaires in Year 1 (5,925 hours) and 8,126 follow-up questionnaires in Year 2 (4,740 hours), giving an annual average burden of 5,333 hours across the two years. The figure of $6.00 per hour (an approximation of the hourly wage that students could earn) is used to value the time cost of survey participation. Based on this value, the estimated annual cost to students for collections of information is $35,553.00 in Year 1 and $28,441.00 in Year 2 (annual average: $31,997.00). However, because these surveys will be administered during the school day, the real cost to students is the opportunity cost of filling out the survey instruments in lieu of formal classroom instruction. This will occur during two classroom periods spread over two years.
Description of Questions |
Justification for Inclusion |
Use of Data |
Student dating violence victimization or perpetration, including psychological abuse, stalking, physical violence, or sexual violence |
Necessary to determine main effects of the Safe Dates program in preventing or reducing adolescent dating violence |
Used as dependent variable for multivariate analysis comparing students at treatment and control schools |
Attitudes related to dating violence, including psychological abuse, physical violence, or sexual violence |
Necessary to determine whether changes in attitudes explain effects of the Safe Dates program on violent behaviors among dating partners |
Used as mediating variable for multivariate analysis to assess attitude changes as the pathway of program effects on adolescent dating violence behaviors |
Alcohol or other drug use at the time that dating violence occurred |
Necessary to determine the context of dating violence incidents and whether students engaging in substance use at the time of violence are equally or less likely to benefit from the Safe Dates program than those not engaging in substance use at the time of violence |
Used as a moderating variable for multivariate analysis to assess interaction between exposure to the Safe Dates program and substance use at the time of violence as a significant predictor of adolescent dating violence |
Attitudes relating to school climate, support for teachers speaking openly about school initiatives, and administration support for Safe Dates program. |
Necessary to determine the extent to which teachers and prevention coordinators felt supported in implementing curriculum, and degree to which implementers supported the curriculum. |
Used as predictor variables for multivariate analysis to examine whether aspects of implementation climate predict fidelity of implementation of the Safe Dates program |
Table 10. Description of Sensitive Questions, Justification for Inclusion, and Use of Data
Principal baseline implementation survey: A Web survey will be completed by an anticipated 49 principals at treatment and control schools. The baseline implementation survey will be completed in Year 1, with a total hour burden of 12 hours. Using the U.S. Department of Labor’s Bureau of Labor Statistics’ (BLS, 2006) $38.45 estimate of principals’ average hourly wage, the total cost will be $471.01.
Prevention coordinator baseline implementation survey: A Web survey will be completed by an anticipated 49 prevention coordinators at treatment and control schools. The baseline survey will be completed in Year 1, with a total hour burden of just over 12 hours. Using the BLS estimate of $23.33 for prevention coordinators’ average hourly wage, total cost will be $285.79.
Teacher baseline implementation survey: A Web survey will be completed by an anticipated 98 teachers delivering the Safe Dates program. The baseline survey will be completed in Year 1, with a total hour burden of just over 24 hours. Using the BLS estimate of $24.70 for teachers’ average hourly wage, total cost will be $605.15.
Prevention coordinator mid-implementation survey: A Web survey will be completed by an anticipated 32 prevention coordinators at schools receiving the Safe Dates program. The mid-implementation survey will be completed in Year 1, with a total hour burden of 8 hours. Using the BLS estimate of $23.33 for prevention coordinators’ average hourly wage, total cost will be $186.64.
Teacher cost survey: A Web survey on the costs of implementing the program will be completed during the study period by an anticipated 49 teachers. The survey includes 11 questionnaires, with one to be completed after delivering each of the 11 Safe Dates components. Each questionnaire will take 20 minutes at a total Year 1 burden of almost 180 hours and cost of $4,437.77, using the BLS hourly wage estimate of $24.70.
Principal mid-implementation survey: A Web survey will be completed by an anticipated 32 principals at treatment and control schools. The mid-implementation survey will be completed in Year 1, with a total hour burden of 8 hours. Using the U.S. Department of Labor’s Bureau of Labor Statistics’ (BLS, 2006) $38.45 estimate of principals’ average hourly wage, the total cost will be $307.60.
Teacher mid-implementation surveys: Two Web surveys will be completed twice each by an anticipated 98 teachers delivering the Safe Dates program. These mid-implementation surveys will be completed in Year 1, with a total hour burden of almost 82 hours. Using the BLS estimate of $24.70 for teachers’ average hourly wage, total cost will be $2017.16. (It is important to note that for the 49 teachers completing cost survey instruments, some or all of the mid-implementation survey questions will be added the end of the cost instruments that correspond to Lessons 5 and 9 of the Safe Dates curriculum. For the remaining 49 teachers not completing cost survey instruments, the mid-implementation survey questions will be stand-alone Web surveys.)
First student mid-implementation survey: After lesson 5 of the Safe Dates curriculum, an anticipated 3,612 students will complete Teleform questionnaires on the program implementation. The survey will take 25 minutes to complete, with a total burden of 1505 hours (all in Year 1) and a total cost of $9,030.00 (also all in Year 1). A value of $6.00/hour is used in the cost calculation.
Second student mid-implementation survey: The second student mid-implementation survey will be administered using Teleform questionnaires after lesson 9 of the Safe Dates curriculum. A total of 3,612 students will be surveyed in Year 1, with a burden of 1505 hours at a cost of $9,030.00. A value of $6.00/hour is used in the cost calculation.
Principal end-of-school-year implementation survey: A Web survey will be completed by an anticipated 49 principals at treatment and control schools. The end-of-school-year survey will be completed in Year 1, with a total hour burden of just over 12 hours. Using the BLS estimate of $38.45 for principals’ average hourly wage, total cost will be $471.01.
Prevention coordinator end-of-school-year implementation survey: At the end of Year 1 an anticipated 49 prevention coordinators from the treatment and control schools will complete a 15-minute Web survey. Total burden will be just over 12 hours and total cost $285.79, using the BLS hourly wage estimate of $23.33.
Prevention coordinator follow-up implementation survey: In Year 2 an anticipated 49 prevention coordinators from the treatment and control schools will complete a 5-minute Web survey. Total burden will be just over 4 hours and total cost $95.26, using the BLS (2006) hourly wage estimate of $23.33
Table 11. Estimated Annualized Burden for Respondents
Type of Respondent |
Instrument Name |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per
(In Hours) |
Total Response Burden (Hours) |
Student
|
Effectiveness baseline survey |
10,158 |
1 |
35/60 |
5,925 |
First mid-implementation survey |
3,612 |
1 |
25/60 |
1505 |
|
Second mid-implementation survey |
3,612 |
1 |
25/60 |
1505 |
|
Effectiveness follow-up survey |
8,126 |
1 |
35/60 |
4,740 |
|
Principal
|
Baseline implementation survey |
49 |
1 |
15/60 |
12 |
Mid-implementation survey |
32 |
1 |
15/60 |
8 |
|
End-of-school-year implementation survey |
49 |
1 |
15/60 |
12 |
|
Prevention coordinator
|
Baseline implementation survey |
49 |
1 |
15/60 |
12 |
Mid-implementation survey |
32 |
1 |
15/60 |
8 |
|
End-of-school-year implementation survey |
49 |
1 |
15/60 |
12 |
|
Follow-up implementation survey |
49 |
1 |
5/60 |
4 |
|
Teacher
|
Baseline implementation survey |
98 |
1 |
15/60 |
24 |
Cost survey |
49 |
11 |
20/60 |
179 |
|
Fifth session mid-implementation survey |
98 |
2 |
25/60 |
81 |
|
Ninth session mid-implementation survey |
98 |
2 |
25/60 |
81 |
|
|
Total |
26,160 |
|
|
14,112 |
*Estimate of average hourly living allowance for participants
**Derived from Bureau of Labor Statistics (BLS) average salary estimates (U.S. Department of Labor, 2006)
Table 12. Estimated Annualized Cost for Respondents
Type of Respondent |
Instrument Name |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Respondent (In Hours) |
Cost/ |
Cost |
Student
|
Effectiveness baseline survey |
10,158 |
1 |
35/60 |
$6.00* |
$35,553.00 |
First mid-implementation survey |
3,612 |
1 |
25/60 |
$6.00* |
$9,030.00 |
|
Second mid-implementation survey |
3,612 |
1 |
25/60 |
$6.00* |
$9,030.00 |
|
Effectiveness follow-up survey |
8,126 |
1 |
35/60 |
$6.00* |
$28,441.00 |
|
Principal
|
Baseline implementation survey |
49 |
1 |
15/60 |
$38.45** |
$471.01 |
Mid-implementation survey |
32 |
1 |
15/60 |
$38.45** |
$307.60 |
|
End-of-school-year implementation survey |
49 |
1 |
15/60 |
$38.45** |
$471.01 |
|
Prevention coordinator
|
Baseline implementation survey |
49 |
1 |
15/60 |
$23.33** |
$285.79 |
Mid-implementation survey |
32 |
1 |
15/60 |
$23.33** |
$186.64 |
|
End-of-school-year implementation survey |
49 |
1 |
15/60 |
$23.33** |
$285.79 |
|
Follow-up implementation survey |
49 |
1 |
5/60 |
$23.33** |
$95.26 |
|
Teacher
|
Baseline implementation survey |
98 |
1 |
15/60 |
$24.70** |
$605.15 |
Cost survey |
49 |
11 |
20/60 |
$24.70** |
$4,437.77 |
|
Fifth session mid-implementation survey |
98 |
2 |
25/60 |
$24.70** |
$2,017.16 |
|
Ninth session mid-implementation survey |
98 |
2 |
25/60 |
$24.70** |
$2,017.16 |
|
|
Total |
26,160 |
|
|
|
$93,234.34 |
*Estimate of average hourly living allowance for participants
**Derived from Bureau of Labor Statistics (BLS) average salary estimates (U.S. Department of Labor, 2006)
13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers
Respondents will incur no capital or maintenance costs.
14. Annualized Cost to the Federal Government
The cost estimate for the completion of this project will be about $2,154,140 over three years and seven months. This is the cost estimated by the contractor, RTI International, and includes the estimated cost of work plan and schedule development ($43,654); Office of Management and Budget clearance (OMB) and CDC/RTI IRB clearance ($31,327); effectiveness, implementation, and cost instrument development and testing ($70,745); evaluation site selection and recruitment ($328,556); data collection (1,474,259); and teacher training and observation ($205,599). Annual cost to the federal government, calculated by dividing the total cost of the project by the time period (3 years, 7 months), is estimated to be $601,155.
15. Explanation for Program Changes or Adjustments
There is no change in burden requested, as this is a new information collection.
16. Plans for Tabulation and Publication and Project Time Schedule
Data analysis will focus primarily on describing program implementation and implementation drivers, assessment of overall program effectiveness, and cost-effectiveness analysis.
Implementation. We will examine the bivariate correlations among variables and conduct a series of regression analyses to test successive segments of the model that pertain to specific research questions (Klein, Conn, & Sorra, 2001). To test for mediation, we will follow guidelines set forth by Kenny, Kashy, & Bolger (1998) to determine whether full or partial mediation exists.
Effectiveness. We will use structural equation modeling to test for differences between treatment and control groups for baseline differences on demographic characteristics, mediators, and dating violence outcomes. Third, we will use structural equation modeling to evaluate the multivariate relationships of baseline demographics, mediators, and dating violence outcomes to attrition from baseline to follow-up. Variables found to differ between follow-up survey responders and nonresponders will be included as covariate in multivariate analyses of program effectiveness.
The primary focus of the effectiveness analyses will be to determine main program effects. Main program effect analyses will assess change attributable to the Safe Dates program with teacher training and observation and to the Safe Dates program without teacher training and observation among (a) student characteristics targeted by the program, and (b) dating violence outcomes. In general, separate analyses will be conducted on adolescents who report no baseline dating violence (to assess program effects on new initiation) and students who report dating violence at baseline (to assess secondary prevention effects). To determine the impact of the Safe Dates program, we will use an “intent-to-treat” model of analysis (Hollis & Campbell, 1999), which requires that all respondents initially enrolled in a given program condition be included in the first pass of an analysis strategy, regardless of whether respondents subsequently received program “treatment.” First, we will compare dating violence rates from students in the two treatment conditions and the control condition. Next, the hierarchical linear modeling (HLM 5.0) program (Bryk & Raudenbush, 1992) will be used to model program effects at the school and individual levels on student characteristics and dating violence outcomes. HLM will be used to adjust for potential violations of the assumption of independence of observations (i.e., individuals within a school are more similar than individuals across schools). This clustering of observations within schools can be modeled as separate variability at the school and individual levels. Analyzing only students actually in the program at treatment schools would not enable the use of a randomized experimental design because analysis of an appropriate control group of students would not be possible.
The procedures for estimating mediated effects in the context of HLM, outlined in Krull and MacKinnon (1999), will be used for parameter estimation. Mediation analyses at the school and individual levels will be conducted using the Asymmetric Confidence Interval method (MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002), a state-of-the-art single-sample method of estimating confidence intervals for mediated effects.
Moderation of program effects at the school and individual levels will be examined in HLM to assess whether program effectiveness depends on demographic characteristics and baseline levels of demographic characteristics and the targeted mediators (i.e., baseline by treatment interaction effects). Moderation is assessed by modeling interaction terms (e.g., treatment status by gender) to assess the differential impact of the program above and beyond main effects (e.g., treatment status, gender). These effects will be modeled at the individual and school levels.
Cost. For Safe Dates, costs may occur in two phases—start-up costs and implementation or ongoing operating costs. Start-up costs include the value of resources used in launching the program, and will include developing/purchasing program materials and initial training. An extensive Safe Dates curriculum has been developed, so the majority of the start-up costs will be the purchase of the product and training Safe Dates administrators (i.e., teachers). Implementation costs include the costs used in the ongoing delivery of the program. We will quantify these costs separately because start-up costs need not be considered if the evaluation question concerns whether or not to continue providing Safe Dates in schools that already offer the intervention.
For both start-up and implementation, costs will be partitioned into two components: variable and fixed costs. Variable costs are directly related to the level of services provided. For example, the amount of time required by teachers to deliver the Safe Dates program is a variable cost. This will likely vary based on the number of students who receive it and could vary depending on such things as whether the full Safe Dates program is administered, if only a portion of the curriculum is used, or teaching experience. Fixed costs, such as the costs to purchase the Safe Dates curriculum, do not vary with the number of individuals who receive the intervention and are generally one-time outlays.
For Safe Dates, the two main components of the variable costs include
Personnel. The largest component of the variable costs of Safe Dates, and of many similar programs, is the labor required to provide the program. Personnel costs include the dollar value associated with all time that educators, counselors, administrators and volunteers spend preparing for and performing Safe Dates activities. This includes training to learn more about Safe Dates, teaching the program to students or adolescents, community outreach, and administrative or staffing details. Regular staff time, overtime, and volunteer time should all be appropriately measured and valued.
Nonlabor. Nonlabor resources include materials used to prepare for and implement the Safe Dates activities. For example, some of the main nonlabor resources include photocopies, office supplies, materials used in the Safe Dates poster contest, and any incentives or rewards given to students for participation.
Using the human capital approach to quantify economic costs, we will value the time that personnel (excluding students) spend implementing Safe Dates at market wage rates for someone with that job classification (e.g., teacher’s salary). This approach recognizes that there is an opportunity cost, if not an accounting cost, of providing Safe Dates instead of other activities (e.g., classroom time for health or math, school enrichment activities). To derive these costs, we will need estimates of the amount of time that each individual (e.g., teacher or administrator) spends performing Safe Dates activities during the reporting period and estimates of the value of those individuals’ time (i.e., fully loaded salary, including wages plus benefits), which we will obtain from Bureau of Labor Statistics (2006) data.
For salaried staff, we will estimate hourly wage by dividing the estimated annual salary by 2,000 hours for administrators (the approximate number of work hours in a year) and 1,600 hours for educators (the approximate number of hours in a 180-day school year, plus 20 teacher workdays). If the individual receives health and retirement benefits, the hourly wage should be multiplied by 1.33 to estimate the value of both wages and benefits (Grosse, 2003).
The Safe Dates curriculum is distributed by the Hazelden Foundation (http://www.hazelden.org/OA_HTML/ibeCCtpItmDspRte.jsp?item=2770). The cost of the printed Safe Dates materials is $149. This expense represents a start-up cost of the program.
Our implementation cost estimates will combine estimates of the variable and fixed cost components to derive the total cost of implementing the Safe Dates program. Furthermore, we will estimate the cost per targeted participant by dividing the estimated total program cost by the number of targeted students. The number of targeted students will be obtained from teacher reports on cost data collection instruments. We will estimate the cost per class using similar methods.
Our cost-effectiveness analysis (CEA) will follow the methodology described in the health economics and policy analysis literature (e.g., Gold, Siegel, Russell, & Weinstein, 1996; Haddix, Teutsch, & Corso, 2003). The analysis will combine cost and outcome data to estimate the cost-effectiveness of the Safe Dates program with teacher training and observation relative to 1) the program without teacher training and observation and 2) a comparison (control) group of no Safe Dates program at the end of 13 months. We will first estimate the average cost and average effectiveness of Safe Dates on eight primary outcomes: psychological abuse perpetration, psychological abuse victimization, stalking perpetration, stalking victimization, physical violence perpetration, physical violence victimization, perpetration of sexual violence, sexual violence victimization, any perpetration and victimization of physical violence. We will express results using summary measures, such as the average cost per unit of average outcome change (e.g., cost per reduced incident of sexual violence victimization).
The final report will note that while there is a single cost of the program, focusing on only a single measure undervalues the full benefits of the program. As a result, we will quantify net benefits of the program via changes in quality of life as a result of participation in each of the three study conditions. We will draw on an existing body of literature on quality-of-life outcomes associated with intimate partner violence to calculate utility estimates for Safe Dates. For example, researchers have reported on SF-36 scores after intimate partner violence (Laffaye, Kennedy, & Stein 2003). We will replicate the methodology used by Wittenberg and associates (2006), using the shorter SF-12 format, an abbreviated version of the SF-36 that was originally designed for parents to complete about their children. We will revise the items to assure appropriate reading level and examples of activities and emotional states that are relevant to ninth graders for self-administration. Measures of central tendency (means and medians) and frequencies will be calculated to describe utilities. Although medians are preferred to describe utilities because of oftentimes skewed distributions, both means and medians will be calculated for these data to allow comparisons across transformation methods (Lawrence’s algorithm produces mean utility only). Linear regression models will be estimated with utility as the outcome variable and study conditions as the key independent variables. The explanatory power of the models will be assessed with the r-square statistic. These results will be used in the denominator of the cost-utility analysis. All cost utility analyses will be conducted using SAS.
The results of the analysis will be reported in a Data Summary. We will also publish an Evaluation Report, including a 1-page press release, a 2- to 3-page executive summary written in clear language and understandable by a wide range of audiences (parents, practitioners, policy makers, researchers), a 10-page executive summary, a report of less than 100 pages (including an overview of background literature to provide contextual information about the purpose of the Safe Dates program and evaluation approach, a detailed summary of evaluation methods and activities, the evaluation results, discussion of findings in comparison with those of other relevant program evaluations, strengths and limitations of the evaluation, and recommendations for future evaluations of this scope for practitioners, evaluators, and policy makers), and appendices. The report will also identify challenges encountered during program implementation and evaluation, as well as their solutions. The results of our study also will be used to develop peer-reviewed journal articles (e.g., American Journal of Public Health, Journal of Adolescent Health, and/or Prevention Science), conference presentations, research briefs, and Web-based papers for dissemination to researchers, schools, and the public.
The projected schedule for the entire project is presented in Table 13.
Table 13. Time Schedule for the Entire Project
Project Activity |
Time Schedule |
Start date |
August 10, 2006 |
Pilot test cost instruments |
January 2006 – May 2006 |
Recruit schools |
December 2006-August 31, 2008 |
Data collection preparation activities begin |
(1 week from OMB clearance) April 1, 2008 |
Deliver final effectiveness and implementation instruments |
(4 weeks from OMB clearance) April 30, 2008 |
Baseline effectiveness and implementation data collection |
(5 months from OMB clearance) September 15, 2008 - November 20, 2008 |
Cost and implementation data collection |
(7 months from OMB clearance) November 2008 – May 2009 |
Follow-up effectiveness and implementation data collection |
March 1, 2010 – April 30, 2010 |
Analyses and reporting |
After April 30, 2010 |
17. Reasons(s) Display of OMB Expiration Date is Inappropriate
No exemption is being sought.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exemption is being sought.
| File Type | application/msword |
| File Title | October 2007 |
| Author | Dvv1 |
| Last Modified By | Dvv1 |
| File Modified | 2007-12-05 |
| File Created | 2007-12-05 |