Simulations Supporting Statement B_OMB Revisions_7.20.11vfinalv2

Supporting Statement B for

Simulations for Drug Related Science Education (NIDA)

Bethesda, MD 20892

Prepared by:

Robert Levine

Address: 3600 FAU blvd., Boca Raton, FL 33431

Telephone: 305-981-4830

Fax: 305-981-4831

Email: [email protected]

Submitted By:

Cathrine Sasek, Ph.D., Project Officer

National Institute on Drug Abuse

National Institutes of Health

6001 Executive Boulevard, Room 5230

Rockville, MD 20892

Phone: (301) 443-6071

Fax: (301) 443-6277

Email: [email protected]

Submission Date 07/20/11

Table of contents

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 3

B.1 Respondent Universe and Sampling Methods 3

B.2 Procedures for the Collection of Information 4

B.3 Methods to Maximize Response Rates and Deal with Non-response 7

B.4 Test of Procedures or Methods to be Undertaken 7

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting And/or Analyzing Data………………………………………………………….9

B.1 Respondent Universe and Sampling Methods

The respondent universe for this evaluation is high school students in the Broward County region. There are a total of 44 high schools and a total of 70,235 students enrolled in these schools. The county has approximately 50% males and 50% females enrolled in their high schools. Broward County is 30% White, 37% African American and 26% Hispanic (32 schools and 71,374 high school students enrolled).

Using data on knowledge of injury prevention behaviors gathered by past work in a school-based study of educational programs in injury prevention, the intra-class correlation due to students clustered within classes nested within teachers was calculated to be 0.021. The typical class size in the population to be studied is approximately 35. The design effect inflation factor (DE) of using the clustering randomization is then given by DE=1+(35-1) x 0.021=1.7. The funding for the study allows for a total of 360 students, or 180 students per group. The effective group size of the study is then n_adjusted = 180/1.7 = 106 students. In order to enroll approximately 180 students per group, or 360 students total, 12 classes will be randomized. At a class size of 35 students, this means that 420 students may be enrolled. A data-completion rate of 86% of initial enrollees would yield the desired total of 360 students.

Using a two-tailed alpha of 0.05, an effective sample of 106 students per group will have 90% power to detect an effect size of 0.45 in the knowledge change scores of the two groups. It is anticipated, based on the researchers’ past experience in educational intervention studies, that the standard deviation of the change scores will be approximately 15. An effect size of 0.45 then translates into a difference of approximately 7 points in change scores. This change represents approximately two questions more being answered correctly by the intervention group as compared to the control group.

The recruitment is focused on the teachers rather than the school due to monetary restriction for the project.  Enrolling all classes within only one school would be out of budget. A sampling frame will be developed from a list of all teachers who volunteer and randomly sample the required 12 teachers from this list. It is understood that this is a convenience sample and may result in limitations to the research findings due to selection bias. Because of the use of this sampling technique it is also understood that the findings could present limitations for the generalizability of these results and low external reliability. This will be noted in any reporting and peer review publication submissions. If the evaluation results show that the electronic module is highly effective however, commercialization may still be recommended since the funding purpose of this study is to develop research results for use in the public domain.

It is expected that 12 schools, with one teacher at each school, will enroll in the study.  The average class size is 35 students (12*35= 420 students).  It is expected that a 15% attrition rate due to absenteeism and other circumstances will occur, which would lead to 100*(357/420) = 85%  student completion rate. A sampling frame will be developed from a list of all teachers who volunteer. From all the teachers that volunteer, a random sample of the required 12 teachers from this list will occur. It will be ensured that each teacher has only one class enrolled.  That class will either receive the intervention or be a control group. Computer generated random numbers will assign the classrooms to either the intervention or the control group.

B.2 Procedures for the Collection of Information

The evaluation will use a hierarchical linear modeling approach, with students being level 1, classrooms being level 2 and teacher being level 3. A pre-test/post-test with entire classrooms of students to be assigned to either the intervention group (i.e. using the computed based module) or the control group (i.e. reading appropriate text material from a high school science text book) to evaluate the computer based learning module will be used. The protocol has been submitted to Copernicus Group IRB (One Triangle Drive, Suite 100, P.O. Box 110605, Research Triangle Park, NC   27709) for an expedited review (45 CFR 45.110).

Upon receiving approval from Copernicus Group IRB, a request to perform the evaluation to Broward County Board of Education will be submitted. Once approval from the school board has been granted, principals and science teachers for both public and private schools will be contacted for participation in the evaluation. Science teachers volunteering to participate will be able to contact the evaluation team to set up a meeting to discuss the educational module, assessments and time commitment. If the science teacher decides to have his/her class participate in the evaluation, letters will then be sent home with those students to inform their parents/guardians of their classroom participation in this evaluation. Parents/guardians will have the opportunity to contact the evaluation team regarding any questions they may have regarding the evaluation or educational module.

Once all recruitment has been completed, a schedule of dates and times to implement pre/post-tests and the intervention will be coordinated with the science teacher to ensure that the project is non-disruptive to the teachers planned educational activities and fits into their lesson plans. Once this has been determined, the evaluation team will go to the classroom to administer pre-tests to the classroom of students. It will be stated by the evaluation team member before passing out the pre-test, post-test or intervention that participation is voluntary and that the students grades, standing within the class or their standing within the school will be impacted on whether they choose to participate in the evaluation. All pre-tests will be administered and collected by the evaluation team. The pre-test will take approximately 25 minutes to complete by the students. Once collected they will be placed in a sealed manila envelope and be entered into the database by the research team.

One to two weeks later (exact time depending on the teachers schedule and lesson plans), the evaluation team will return to administer the intervention. At this time, preloaded laptops will be given to classrooms of students to participate in the learning module (intervention group) or standard reading materials will be handed out (control group). Both the intervention and control groups will utilize their given materials for the 50 minute classroom time period. At the end of the class, the research team members will collect the computers or reading materials. At this time, students who used the computer based learning module will be given a brief (3-5 minute) anonymous questionnaire regarding their opinion of the computer based learning module (i.e. did you like the graphics? Was the sound clear? Suggestions for improvement?, etc). This will occur within the 50 minute time block designated for the intervention. Evaluation team members will collect this feedback form from students before they leave the room for their next class.

Approximately 3-4 weeks later, the evaluation team will return to the classrooms to administer the post-test. This post-test will take approximately 25 minutes to complete and will be administered and collected by the team member. Once collected, they will be placed in a sealed manila envelope and be entered into the database by the research team. After all participating classrooms have completed post-tests, data analysis will begin to determine if there are any differences between the intervention and control groups in the areas of knowledge acquisition and attitudes towards science education and drug use.

A final post-test will be administered 6 months after the initial post-test to collect data. At this time, the evaluation team member will return to classrooms to administer and collect this attitudinal post-test. This post-test will take approximately 25 minutes to complete and will be administered and collected by the team member. Once collected they will be placed in a sealed manila envelope and be entered into the database by the evaluation research team. After all final post-tests have been administered, data analysis will begin, to analyze if there are any longer term differences between the intervention and control group. Additionally, final results will begin to be written up for the final report to the funding agency (National Institutes of Health’s National Institute on Drug Abuse).

B.3 Methods to Maximize Response Rates and Deal with Non-response

Evaluation team members will be present during the pre-test and post-test. The evaluation team members will be responsible for explaining to the students that their participation is voluntary and explaining the purpose of the evaluation. Evaluation team members will monitor the classroom while students complete the pre-tests and post-tests to ensure that students do not exchange pre/post-tests, and to monitor if pre-tests and post-tests are complete. While this evaluation is voluntary, we anticipate that outside individuals explaining the project will motivate students to complete the pre/post-tests and intervention. Since students will be completing the survey in class a high non-response is not expected. Non-response bias assessment will be conducted by examining the extent of the differences in responses among the class groups. Study resources may not permit the conduct of a sensitivity analysis, and it will not be possible to conduct interviews with a set of randomly selected non-responding students. However, results of non-response assessment will be noted in both reporting methods and any peer reviewed publication submission.

We expect an 85% response rate (and 15% attrition due to absenteeism). It is estimated that approximately 360 students total will complete with survey (12 classes will be recruited with a class size of 35 students, so 420 students could be available for participation, 12*35=420). A data-completion rate of 85% of initial enrollees would yield the desired total of 360 students to reach statistical power needed for the study.

B.4 Test of Procedures or Methods to be Undertaken

The pre-test and post-test have been developed by the evaluation team and consultants including standardized questions from validated questionnaires. The science attitude questions come from the Science Attitude Inventory (SAI II) and questions developed from the investigator’s group. The SAI is an established survey for measuring science attitudes. To date, it is the only science attitude survey.

The knowledge based and attitudinal questions were developed by the investigator. Pre-tests and post-tests have been developed in house and reviewed by an expert panel (expert panel: Dr. Leslie Miller – Expertise: Applying technology to education, Charlie O’Brien – Expertise: Science of Addiction, Michael Lang – Expertise: Science education, Charlie Parsons – Expertise: CEO of D.A.R.E. America, Dr. Eden Evins – Expertise: Clinical psychiatrist)

Knowledge based questions that will be used to measure knowledge acquisition were developed in line with information presented in the multimedia module and text based readings. Attitudinal questions for the section regarding drugs and the perception of harm from drugs have been adapted from the Florida Youth Risk Behavior Survey, and from prior evaluations measuring attitudes regarding alcohol and alcohol/substance abuse (which have been pre-tested with this population). Science attitudinal questions have been adapted from the Scientific Attitude Inventory and additional questions have been developed by the development team.

The pre-tests and post-tests were reviewed by the team, and changes have been made to make certain that knowledge questions use the same language used in the educational module, questions are ordered logically, and to check that the wording of the attitudinal questions is appropriate.

The qualitative feedback survey was developed in house, and is modeled after other user feedback surveys we have developed for multimedia educational evaluations in usability for projects that have been funded by the U.S. Department of Education and the Department of Defense. These questionnaires are to gather feedback regarding the user experience. Additionally, students/participants may not always verbalize any issues that they may be experiencing with the system, even if a user facilitator is in the room to field questions or concerns. Past experience in gathering user experience and usability data, written surveys where names are not associated with responses yield more information. The source of burden for this survey has been included in the sources of burden table for the follow up.

Methods used during data collection for this evaluation were adapted from past school evaluations conducted on alcohol education interventions and safe driving education interventions. The pre/post test method used has been successful in the high school population (for alcohol and safe driving educational interventions, unpublished data). The evaluation team will administer the intervention, pre-tests and post-tests. Additionally, the evaluation team present during this time will be able to monitor the completion of pre-tests and post-tests, and be able to answer any questions or to ensure that forms are not switched.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

All sample size and statistical aspects of the design were initiated by and will be analyzed by:

Robert Duncan, PhD

University of Miami Miller School of Medicine

Biostatistian

Professor of Epidemiology and Public Health

305-243-8208

Dr. Duncan has an extensive research and statistical analysis career spanning over five decades. Over the past several years, Dr. Duncan has been involved in the design, execution, and analysis of several school-based research projects, and has been the lead investigator in analyzing data on multiple evaluations and learning management systems. Dr. Duncan’s extensive experience in conducting, computerizing, and analyzing clinical and field studies is directly applicable to the activities described in this proposal.

Data collection oversight and analysis write up will be performed by:

Jill Graygo MA, MPH

Lead Evaluator

1900 SW 23 Terrace

Miami FL 33145

305-562-0278

Ms. Graygo is a Research Manager that currently oversees the development, study design, implementation, analysis and dissemination of multiple federally funded research studies. She has extensive knowledge in study design, ethical issues and Institutional Review Board approval process. Additionally, Ms. Graygo has also been involved in assessment tool development and evaluations for multiple projects/programs, and developing strategies for dissemination of results and /or interventions based upon findings.

10