Summary of Pilot Test Results

Appendix G. Summary of Pilot Test Results

This appendix describes pilot test results of the effectiveness and cost instruments.

Effectiveness. A pilot test of the effectiveness survey was conducted by RTI from 2/10/07 to 2/20/07. RTI recruited students through communication with parents at Parent Teacher Student Association (PTSA)-sponsored events at Durham, North Carolina, area high schools. The inclusion criteria were (1) the child must be in the 9^th grade in a local school and (2) the child must speak English. Adolescents with family members who are RTI employees were excluded from participation. We screened adolescents to be certain that we had a mixture of males and females as well as daters and non-daters. Parents of adolescents were recruited at two PTSA-sponsored events. RTI sent a lead letter addressed to each of nine parents explaining the purpose of the study and inviting their adolescent to participate in the pilot test, along with a parental consent form, and advising that RTI staff would call to schedule a date for data collection.

One parent’s adolescent refused to participate in the study, and two parents did not respond to the lead letter or follow-up communication. After each refusal or failure to follow up, a new parent of an adolescent was contacted, resulting in a total of 12 parents contacted. Of these, 9 adolescent surveys were completed in person for a responses rate of 75%. For the pilot test, it was not feasible to conduct classroom-based data collection. Instead, surveys were administered individually or in small groups of two or three adolescents at RTI or at a local public library. We also did not use incentives for consent form completion. However, we offered a $25 gift card incentive for completion of the survey instrument because it was a substantial time burden outside of regular school hours for participants.

Findings. The average length of the survey was 33 minutes, with a range of 23 to 40 minutes. One adolescent reported never dating and skipped out of a significant portion of the questionnaire. Some adolescents left questions blank, but this occurred infrequently.

Adolescents were asked a series of questions at the end of the survey about their comfort level in answering questions, the seriousness of their answers, their honesty level, and length of the survey. Participants' self-reported comfort level was assessed using a 4-point Likert scale ranging from 1-very uncomfortable to 4-very comfortable. Participants were asked how seriously they answered questions during the survey. In addition, they were asked how honestly they answered the questions during the survey. Both questions had response options of 1-very, 2-somewhat, and 3-not at all. The length of the survey was assessed by asking participants whether they felt the survey was too short, about right, or too long.

Adolescents’ self-reported comfort scores ranged from 1 (very uncomfortable) to 4 (very comfortable) with a median score of 3. One participant wrote that he was uncomfortable answering questions about his race. The other participant who marked “very uncomfortable” responded that none of the questions made her feel uncomfortable, possibly indicating an error in her response. Participants answered most questions as directed, regardless of comfort level. Some participants skipped questions, but items skipped did not appear to be related to comfort level. For example, two participants left a question blank about exposure to dating violence prevention activities. Another participant left two items blank about frequency of dating violence experienced during the last 6 months, although she had already marked “never” to the same items in the “ever” column.

All adolescents reported that they had answered questions very seriously. Two adolescents reported that they answered questions somewhat honestly, and the remainder answered “very honestly.” Most adolescents said that the questionnaire length was about right, while two felt it was too long.

Specific Questions

1. One participant asked for clarification about the questions about frequency of dating violence. This participant wanted clarification about whether she had to mark “never” referring to the past 6 months if she marked “never” referring to “ever.”

2. Two participants left an item blank about exposure to dating violence prevention activities. We were unsure whether one of these participants did not follow a skip pattern preceding this question correctly or intentionally left the item blank because the instructions read: If you have not had any of them, don’t mark any of them.

Recommendations for Main Study

• Restate that all answers will be kept confidential prior to administering the survey. Restating confidentiality may reduce discomfort with questions about relationships and violence throughout the questionnaire.

• Add clarification instructions to questions about frequency of past dating abuse incidents. RTI added clarification instructions to questions about frequency of past dating abuse incidents to specify that even if the respondent marks “never” for the “ever” time period, he or she will need to mark an answer for the “during the last 6 months” time period as well.

• Reformat skip patterns so that when respondents are instructed to skip to another question, the subsequent questions appear at the beginning of a new page.

• Improve response options for the question about other dating violence prevention interventions adolescents received. Asking adolescents to mark “none of the above” will help accurately determine whether respondents skipped a question.

• Emphasize the importance of selecting a response for every question. Data collectors will remind respondents to fill in an answer for every question, which may result in less missing data.

Cost. We pilot tested the cost data collection forms with nine teachers at four high schools in two states. Eligible schools had (1) ninth graders, (2) at least one teacher in the school who agreed to deliver the Safe Dates program during the spring semester of 2006, and (3) at least one participating teacher who agreed to complete data collection forms about cost and time burden associated with program delivery. RTI recruited schools that had a prior history of cooperation with RTI and let schools determine which specific teachers would deliver the program. A large number of schools were contacted before the four participating schools were identified. Each school selected between one and four teachers to participate. RTI provided each school with a Memorandum of Agreement (MoA) explaining the purpose of the study and alerting them that the Safe Dates curriculum and cost questionnaires would be mailed by Federal Express. RTI followed up by contacting teachers directly by telephone or e-mail and resending forms by e-mail for teachers’ convenience. Each participating teacher was provided a copy of the Safe Dates curriculum and was asked to fax or e-mail completed forms within two school days of delivering each component. Most teachers did not complete forms within this time frame, so RTI began contacting teachers regularly by telephone and e-mail to remind them about the data collection schedule and to offer to answer questions about the study.

RTI received completed questionnaires from all teachers. Of the 99 cost forms (11 forms for each of the 9 teachers), RTI received 89 completed forms, for a response rate of 90%. We offered a $50 gift card incentive to each teacher for completing the series of forms and a $250 school incentive for study participation. Teachers delivered all 9 Safe Dates classroom sessions among 33 classes to a total of 935 students.

The cost survey forms included questions at the end of the survey about the availability and likely accuracy of the data requested, length of time to complete the forms, and level of difficulty.

Findings. Eight of nine teachers completed all nine classroom session cost questionnaires; one teacher completed eight of the nine classroom session cost questionnaires. Only six teachers completed the poster contest, and four teachers completed the play, so we only received completed questionnaires about these activities from these teachers. Most teachers skipped some parts of some questions on every questionnaire, and a few teachers provided partial answers to some of the questions. One teacher indicated that he was trying to work out with his administrator money to purchase binders for students, but we did not receive clarification about whether this was approved. In addition, one teacher reported that paper and ink were purchased by the school but did not estimate the cost of these materials, and another teacher reported that poster contest prizes were purchased but did not report how much they cost. Some teachers did not report the amount they spent on the poster contest prizes.

All teachers felt they had all the information they needed to complete the forms and that their responses were accurate. Most reported that the forms required 15 to 25 minutes to complete. All teachers reported that the cost forms were easy to understand and complete.

Forms that were completed more than two school days after a Safe Dates lesson was delivered contained data that was less detailed and complete compared with forms that were completed within two days of lesson delivery. For example, teachers who fell behind on the data collection schedule and completed several forms simultaneously provided estimates of time spent (and students present in class) that were equivalent across all class periods and lessons instead of reporting minor variations in attendance and time, as did teachers who did not fall behind on the data collection schedule. Two forms that were received from one teacher who fell behind on the data collection schedule appeared to be Xeroxed forms that another teacher at the school had handwritten and submitted previously, with only the name of the teacher changed.

All teachers volunteered that the students enjoyed the curriculum and that they participated more than was expected. Many of the teachers reported that they thought all nine sessions of Safe Dates were informative and helpful and that Safe Dates is a good curriculum. Overall, teachers reported they enjoyed teaching Safe Dates and that their school administrators agreed with their assessments. At one location, the school administrator sat in on several lessons because he found them so interesting and timely.

Several teachers indicated that it was difficult to fax the forms within the timeframe requested, asked if there might be another method for returning the forms, and asked whether the cost questionnaires could be shortened.

Recommendations for Main Study

• Convert the cost questionnaires to Web format. This format will provide prompts to respondents if they inadvertently skip questions, such as questions about cost and nature of materials purchased to deliver the Safe Dates program. It may also be possible to prompt respondents on subsequent questionnaires about approval for purchases or follow-up on items that they did not know the final answer about on a previous questionnaire. In addition, submitting responses via the Web will eliminate the time burden associated with faxing completed forms to RTI.

• Follow up with nonrespondents. RTI will ask teachers to provide a program delivery schedule in advance and will request that teachers complete the Web forms within one business day of delivering each program component. RTI staff will follow up with nonrespondents by telephone and e-mail remind teachers to complete the questionnaires and to offer help if needed.

• Replace a yes/no question and open-ended follow-up question about modifications to the Safe Dates lesson with Likert scale items. This will simplify and shorten the questionnaires.

4