NIDA’s Study of Substance Abuse doc.com Module Project
Supporting Statement A for
THE NATIONAL INSTITUTE ON DRUG ABUSE’S (NIDA)
STUDY OF SUBSTANCE ABUSE DOC.COM MODULE PROJECT
April 2011
Name: Elisabeth Davis, MPH
Address: Office of Science Policy and Communications
National Institute on Drug Abuse
National Institutes of Health
6001 Executive Boulevard
Bethesda, MD 20824-9591
Telephone: (301) 594-6317
Fax: (301) 443-7397
Email: [email protected]
Table of Contents
A. JUSTIFICATION A-5
A.1. Circumstances Making the Collection of Information Necessary A-5
A.2. Purpose and Use of the Information Collection A-6
A.5. Impact on Small Businesses or Other Small Entities A-11
A.12. Estimates of Hour Burden Including Annualized Hourly Costs A-17
A.13. Estimate of Other Total Annual Cost Burden to
Respondents or Record
Keepers A-18
A.14. Annualized Cost to the Federal Government A-18
A.15. Explanation for Program Changes or Adjustments A-19
A.16. Plans for Tabulation and Publication and Project Time Schedule A-19
References for Section A A-28
30-Day FRN
60-Day FRN
LIST OF TABLES
Table 1: Estimates of Hour Burden (p. A-17)
Table 2: Annualized Cost to Respondents (p. A-18)
Table 3: Timeline for Parts I and II of the Study (p. A-19)
Table 4: Levels 1 and 2 and Associated Co-Variables for HLM Analyses (p. A-25)
LIST OF ATTACHMENTS
Attachment A1: Email Invitation to Medical Students
Attachment A2: Email Invitation to Residents
Attachment A3: Penn Med Script for Survey Administration
Attachment A4: DUCOM Script for Survey Administration
Attachment A5: Email Notification Re Pre-Survey
Attachment A6: Email Notification Re Post-Survey
Attachment A7: Email Notification Re Module
Attachment A8: Penn Med and DUCOM IRB Letter
Attachment A9: IRB Approval (including all informed consent forms for medical students and residents)
Attachment A10: Penn Med Letter About Type of Consent
Attachment A11: DUCOM Letter About Type of Consent
Attachment A12: End of Study Email to Medical Students in Control Group
Attachment A13: End of Study Email to Residents in Control Group
A. JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
In compliance with Section 3506(c) (2) (A) of the Paperwork Reduction Act of 1995, this submission requests OMB approval for a 2 year clearance for the National Institute on Drug Abuse (NIDA) to conduct a research study on the efficacy of a novel medical educational intervention: The Substance Abuse doc.com Module. This project was developed to enhance the knowledge of healthcare professionals related to substance use disorders, their attitudes toward patients with substance use disorders, and their skills and their degree of confidence in communicating about substance use and abuse with their patients. This educational intervention was developed to influence, among provider populations, attitudes and positive patterns of behavior with this patient population. This study will be implemented by the University of Pennsylvania School of Medicine (Penn med) and Drexel University College of Medicine (DUCOM), as part of a contract with JBS International, Inc. that is funded by the National Institute on Drug Abuse (NIDA) (contract # NO1DA-9-1142).
The project is consistent with NIDA’s
mission to lead the Nation in bringing the power of science to bear
on drug abuse and addiction and is authorized under U.S.C. Title 42,
Chapter 6A, Subchapter III, Part C, Subpart 15 - 285o
(http://www.law.cornell.edu/uscode/html/uscode42/
usc_sec_42_00000285---o000-.html),
which outlines NIDA’s purpose to conduct and support research
on drug abuse and addiction. NIDA’s Office of Science Policy
and Communications (OSPC) is the sponsor of this project. OSPC has
responsibility for a wide variety of dissemination activities on
behalf of the institute. OSPC has funded four NIDA Centers of
Excellence (CoEs) for Physician Information to develop novel
educational work products to enhance education about substance abuse
and substance use disorders in medical schools and primary care
residency programs. These work products are accessible online under
NIDA Centers of Excellence on the NIDAMed Home Page
(http://www.drugabuse.gov/NIDAMed/).
This project, NIDA’s Substance Abuse doc.com Module Study, is an extension of NIDA’s and OSPC’s ongoing emphasis on the use of the Internet as the preferred means to communicate with busy primary care practitioners and other medical professionals about substance abuse.
This project will assess the efficacy of a specific interactive Web-based teaching module in the field of professional education of healthcare providers. This online module was developed by a team of investigators from DUCOM and Penn Med under a contract as part of NIDA’s CoEs. The term “doc.com” refers to a series of online interactive teaching modules that have been primarily produced by faculty and staff at DUCOM and that aim to improve communication between physicians and patients. The doc.com curriculum, which has more than 41 teaching modules, is sponsored by DUCOM and the American Academy on Communication in Healthcare (AACP) (http://webcampus.drexelmed.edu/doccom/user/individual_logon_1.asp).
If the NIDA substance abuse doc.com educational intervention is shown to have efficacy, it should result in an increase in the involvement of primary care physicians in the screening, diagnosing, managing and, when appropriate, referring patients with substance use disorders to treatment.
NIDA’s decision to emphasize providing more information and training to primary care providers as a critical link in the chain of how to appropriately care for patients with substance use disorders was recently supported by remarks by Dr. A. Thomas McClellan, then Deputy Director of The White House Office of National Drug Control Policy (ONDCP). His remarks at the American Academy of Addiction Psychiatry annual meeting in December 2009 were reported in the Psychiatric News (January 15, 2010, Volume 45, Number 2, Page 4 http://pn.psychiatryonline.org/content/45/2/4.2.full?sid=ecda1f71-6985-442f-8129-4bc9009dcc93). According to this article, the ONDCP will make increased involvement of primary care physicians in the screening and managing of patients with less severe forms of substance use disorders one of its main priorities.
Justification for the project is therefore based on two factors: (1) the need for NIDA to support and expand educational efforts related to substance use disorders that are directed at medical students and primary care physicians, and (2) the need for NIDA to obtain scientifically valid information about the efficacy of a novel educational intervention that is based on a work product of one of OPSC’s Centers of Excellence for Physician Information. Ultimately, NIDA expects to extend scientifically rigorous efficacy studies of this novel educational intervention to larger and more diverse populations of medical students and healthcare providers, including primary care physicians, to meet the educational needs of these groups.
A.2. Purpose and Use of the Information Collection
As stated in Section A.1, the goal of this project is to assess the efficacy of a novel educational intervention, the interactive Web-based NIDA substance-abuse doc.com module, in the field of professional education of healthcare providers.
This project will study the educational intervention in two groups of healthcare professionals: subjects in Group 1 are medical students and those in Group 2 are residents in primary care residency training programs (i.e., internal medicine or family medicine). The proposed research will formally and rigorously assess in both Groups 1 and 2 whether the intervention based on the NIDA substance abuse doc.com module can improve knowledge about substance use disorders, attitudes toward patients with these disorder, and measures of self-efficacy in communicating with such patients. These outcomes will be assessed by comparing the changes in results of surveys given to the students or residents before and after the educational intervention in the experimental group compared to changes in survey results by subjects in the control groups. The methods and sampling techniques for both Groups 1 and 2 are described in detail in Section A-16 and Section B.
Both subject groups of this project, medical students and residents, will complete surveys before and after exposure to the educational intervention (with the intervention being offered as part of standard educational training). Residents will also participate in a debriefing/faculty led seminar discussion. Using the information from these surveys, this project will assess whether exposure to a novel online teaching module (compared to the standard educational curriculum) leads to changes in knowledge, attitudes, and confidence in their communication skills. The questions related to perceptions of knowledge levels, attitudes, and communication skills are in the surveys that are described in Section B.2 of Supporting Statement B.
The hypothesis of this project is that exposure to a novel Web-based educational intervention will improve knowledge, attitudes, and confidence in communication skills in the two categories of respondents (Groups 1 and 2) compared to exposure to the standard educational curriculum. Specifically, medical students and residents who participate in the intervention arms of the project will report higher scores of knowledge about substance use disorders, more positive attitudes toward patients with substance use disorders, and enhanced confidence in their communication skills for providing screening, assessment, and treatment referrals than students and residents assigned to the control arms. Having these data will allow this project to test this hypothesis in a rigorous controlled research study.
These data are needed to assess the efficacy of the NIDA substance abuse doc.com module as the basis for a novel educational intervention for healthcare professionals (i.e., medical students and resident physicians). Not having these data would limit NIDA’s ability to confirm whether such a training module can effectively educate healthcare professionals on the screening, assessment, and treatment of substance use disorders. Further, if the hypothesis is supported, this would allow NIDA to publicize and distribute the educational intervention to healthcare professionals as an effective tool to improve the ability of healthcare professionals to care for patients with substance use disorders.
The data to be collected in this project will be used to:
Test the hypothesis described above
Provide the data in reports to NIDA OSPC.
Write manuscripts that will be submitted for publication in peer-reviewed journals.
Present abstracts or other types of presentations at relevant professional meetings, e.g., annual meeting of the American Association of Medical Colleges (AAMC).
The results from this project will be used by NIDA as a foundation for its efficacy research of medical education interventions related to substance use disorders. These results are expected to inform a larger, multi-site study of the educational intervention that this project will be testing. This anticipated study will provide more generalizable results because it will test the intervention in larger populations of medical students, residents in adult primary care residency programs, and practicing primary care physicians.
Specifically, we will use the results of this study as follows:
To get a more accurate estimate of the effect size of the intervention in the categories of participants and a priori subgroups. The latter include students or residents affiliated with Penn or Drexel at various practice sites.
To estimate durability of effect size over time.
To get a more accurate estimate of the variance of the effect size, as well as the variability of the effect size among the a priori categories of participants and subgroups.
To assess the patterns of distribution of the data outcome and other variables, such as their normality, etc, in the a priori categories of participants and subgroups.
To get a more accurate estimate of the intra-class correlation of response within clusters of participants that will be randomized.
To perform basic psychometric analyses on the results of the survey questions, including item analyses that may permit us to reduce the number of questions and make other refinements to the survey instrument.
To refine the module. For example, the module can provide basic data that indicate how the individual participants spent their time while accessing the module, e.g., which pages and or videos they opened, their sequence of pages and how much time they had each page open, and then test for associations in the analyses, e.g., students who spent time viewing the interviews with authentic patients in recovery may show evidence of more humane attitudes towards patients with substance abuse.
To assess the feasibility and generalisability of these methods in order to include medical students and residents in other US medical schools.
A.3. Use of Information Technology and Burden Reduction
Use of Information Technology by the NIDA Substance Abuse doc.com Module. Information technology in the form of the NIDA substance abuse doc.com module will be used in both Group 1 (involving medical students) and Group 2 (involving primary care residents). Burden will be minimized because the module is available online from any Internet-connected computer, enabling students and residents to participate in the study at a convenient time in their busy schedules. Participants in Groups 1 and 2 will be randomized to an intervention arm or a control arm. The intervention arm will be exposed to the new online educational intervention that is based on the NIDA substance abuse doc.com module.
The “doc.com” site is a collection of 41 modules on communication in healthcare, which is being developed jointly by DUCOM (delivery system) and the American Academy on Communication in Healthcare (content). Development began in 2004 and is ongoing. The 41 modules are delivered through a learning management system, which allows the building of on-line learning groups where a facilitator can give assignments, and monitor learners’ performance based on their assessments. However, the 41 doc.com modules are designed to also work outside of the learning management system––so they can be used locally and without Internet access––but then they lack the added functionality provided by the learning management system. Access to this series of 41 doc.com modules and their associated learning management system requires registration as described on the doc.com Web site, http://webcampus.drexelmed.edu/doccom/.
When DUCOM and Penn Med were chosen as a NIDA Center of Excellence for Physician Information to provide an online curriculum on substance use disorders, the modular design of all doc.com modules was used for this undertaking. This decision was based on experience in how to build it, and knowledge that the model is working well and successful, as is evidenced by the fact that the doc.com series is currently being used by 12,000 learners around the world.
The NIDA module, “The Clinical Substance Use Disorders,” was designed to be a standalone module outside of doc.com (i.e., a location that provided unrestricted direct access to the module) at the URL http://webcampus.drexelmed.edu/nida/. No registration is required to access the module at this URL. This module is also at the following sites:
NIDA’s NIDAMed home page (http://www.drugabuse.gov/NIDAMed/)
At NIDA’s CoE Web page (http://www.drugabuse.gov/CoE/)
By using the subheading for CoEs (http://www.drugabuse.gov/CoE/CoE.htm).
There is another URL (http://webcampus.drexelmed.edu/doccom/) where the NIDA module, “The Clinical Substance Use Disorders,” can be accessed freely and without registration. To find the NIDA module on this URL’s website, one scrolls down to the section that contains the four “demonstration” modules (where it is identified as “Demo Module 30: “The Clinical Substance Use Disorders”).
The module was located at the URL just outside of doc.com as a freely accessible demonstration module because it was produced using Federal Government funds and, as such, its content is in the public domain and can be reproduced or copied without permission.
Finally, the same NIDA module, “The Clinical Substance Use Disorders,” also resides inside of doc.com as doc.com Module 30. However, as a module residing inside of doc.com, like all of the other 40 modules that are inside of doc.com, it can only be accessed by registering as instructed at the URL http://webcampus.drexelmed.edu/doccom/?m=30.
The NIDA substance-abuse Web module that is being evaluated in the current NIDA evaluation study will be located on a new URL (i.e., a “research-use only” URL) that the DUCOM research team will create using existing DUCOM servers. Access to this new “research-use only” URL will be limited to the study participants in the intervention group by the method described below. Study participants will not be asked to provide any personally identifiable information available on the “research-use only” NIDA module.
The NIDA substance abuse doc.com module was based on doc.com modules because of the highly technical and innovative manner in which these modules are teaching basic and advanced communication skills to medical students and residents. In addition, the doc.com modules, including the NIDA substance abuse doc.com module, have the capacity to teach relevant content (knowledge) and to influence attitudes of the learners. Thus, it is a unique and technologically advanced teaching tool that can comprehensively address the learner’s needs by teaching knowledge, attitudes, and communication skills.
The doc.com modules are conceptually grounded in behavioral and communication science. The three domains of knowledge, attitudes, and communication skills correspond to the three main stages of change that have been identified by Everett M. Rogers in his theory of diffusion of innovation (Rogers, 2003). These three main stages are symbolized by the acronym, “KAP” that stands for Knowledge, Attitudes (persuasion), and Practice (skills).
A novel technological aspect of the NIDA substance abuse doc.com module is the “Patient Interviews” section. This section contains digital videos of structured interviews with five patients who are in recovery from drug addiction. All five of the patient interviewees were asked the same set of 22 questions about the history of their former drug use, what helped them overcome the addiction, and their lives, dreams, and fears. In addition to hearing directly from patients in this diverse group of individuals who previously abused drugs, learners are instructed how to use the "Patient Interview Matrix" feature of this section. Learners can start by choosing one of the patients and then seeing how he or she responded to one or more of the 22 questions that are displayed on the right side of the screen by clicking the question. Alternatively, learners can focus on a single question and hear how all five patients answer it by clicking on the patients on the left side of the screen.
Use of Information Technology by the Data Coordinating Center. The Data Coordinating Center (DCC) of this project will enable the surveys to be posted online on a password protected Web site. The DCC will use software (i.e., Oracle Clinical) currently utilized for clinical trials at multiple sites. Oracle Clinical provides an integrated data management system and remote data entry capabilities (http://www.oracle.com/us/industries/life-sciences/046720.html). Because Oracle Clinical does not require pre-loading of any software into the subject’s computer, subjects can access a DCC controlled Web site by using any Internet capable computer. Thus, this system should reduce the burden on respondents by being highly convenient for respondents to access.
Emails will be sent to all potential participants in Groups 1 and 2 as an introduction to the study and an invitation to participate (see Attachments A1 and A2).
Research team members (investigators) will also introduce the study by meeting with groups of potential participants (i.e., Penn Med or DUCOM residents or medical students) to discuss its details and be available to answer questions (see Attachments A3 and A4).
All participants in the study will be required to provide their names and email addresses to the Project Managers at Penn Med or at DUCOM depending on their medical school affiliation. The justification for requiring these data is that they are needed so that the Project Manager affiliated with the participant’s institution can email each participant the appropriate link, log-in, and password to allow that individual participant to access the appropriate online site at the correct time and sequence as determined by the study protocol.
For example, the Project Managers will email the following only to participants who are in the intervention group (i.e., those assigned to view the Web module): 1) the link to the new “research-use only” URL of the NIDA substance-abuse web-module; 2) a 7 digit study ID number unique for that participant (see next paragraph below for details); and 3) a random 6 letter password (that the Data Coordinating Center previously assigned to be associated with that participant’s unique 7 digit study ID number).
Group 2 (residents) will complete the informed consent form using paper and pencil during the educational sessions when the study is introduced. Those who complete the informed consent form, thus agreeing to participate, will receive an email providing them with a user name (a unique study identifier [SID]) and password that will direct them to click on a link that will take them directly to the DCC’s Web site and allow them to access the Pre-Survey. For the Post-Survey, respondents will be sent a similar email with their SID that directs them to the DCC Web site and instructs them to click on a link to access the Post-Survey. In addition, an email will be sent with instructions that direct participants in the intervention arm to the module when the appropriate time in the study arises (see Attachments A5, A6, and A7). Taking surveys online in this convenient manner should reduce the burden on these respondents.
Medical students who participate in Group 1 will complete surveys using either electronic or paper and pencil methods. Medical students at Penn Med will complete all four surveys on paper.
DUCOM medical students will complete the surveys in the following manner:
Pre-Survey 1: electronic administration of survey.
Pre-Survey 2: paper and pencil administration of survey.
Post-Survey 1: electronic administration of survey.
Post-Survey 2: electronic administration of survey.
See Section A.6 for details of survey administration (i.e., when in the academic program the surveys will be completed).
NIDA will not have access to the data and information collected from this study.
Use of Email as Main Method of Communication. For both Groups 1 and 2, when appropriate, electronic communications (i.e., email) will be utilized with the subjects as the primary means of communication. Each subject will be provided with a confidential and unique study identification number (SID) and a password for each stage of the study. Study participants will not be asked or required to register their email addresses as the log-on ID.
For example, subjects will be given a password and URL link to be able to access the Pre-Survey at the appropriate time that is specified by the protocol (and not be able to access the Post-Survey at that time). Then, after completing the Pre-Survey, subjects in the intervention arm will be automatically emailed another password and URL link (to decrease burden) so that they can access the NIDA substance abuse doc.com module via a special URL link for use only for study participants. This is to confirm that the subject has visited the site and to measure how much time the subject has spent on the site.
Finally, subjects will be automatically emailed another password and the URL link at the proper follow-up time as specified by the protocol to access and complete the Post-Survey.
There have been no prior studies of the NIDA substance abuse doc.com module in any study population. One reason for lack of evaluative studies is that the module has only recently been completed and released to the public domain. For example, the Patient Interview section was completed in the summer of 2009.
There has been one prior study of the efficacy of four doc.com modules (not including the NIDA substance abuse doc.com module) as tools for teaching basic doctor-patient communication skills to first-year residents in internal medicine, but not medical students or more advanced residents (Spagnoletti et al., 2009). The results of the study supported the efficacy of the four modules that were tested.
Because the NIDA substance abuse doc.com module has not yet been studied, there is no similar information available for this project and hence there is no duplication of effort. No other institutes at the National Institutes of Health or the U.S. Public Health Services are currently evaluating doc.com modules.
A.5. Involvement of Small Businesses or Other Small Entities
No small businesses will be involved in this study.
Resident physicians are included in this study. Although physicians are considered to be small businesses, all of the population of physicians under study will be physicians still in their post-graduate training years (i.e., residents). All of these residents are employed by large hospitals or health systems, none of which are considered small businesses. None of these residents are in private practice or practice in small groups of physicians.
As noted in Section A.2., the collection of these data is critical to assessing the NIDA substance abuse doc.com module’s educational efficacy on enhancing the learner’s knowledge, attitudes, and confidence in their communication skills related to substance use disorders.
Although this is a single-time study, periodicity will occur in order to obtain accurate results over a period of time (e.g., before and after exposure to the educational intervention). According to the protocol for Group 1 of the study, medical students will be given the survey twice before the intervention (“Student Pre-Survey 1” and “Student Pre-Survey 2”). The first survey (i.e., “Student Pre-Survey 1”) is needed so that the study can establish a baseline of knowledge, attitudes, and degree of confidence in communication skills at an important time point in the medical school curriculum (i.e., at the end of their pre-clinical years) and just before the start of the first clinical year. Having results from the Student Pre-Survey 1 will allow researchers to use baseline data to study the change in results from Student Pre-Survey 1 to Student Pre-Survey 2 as co-variables in the multi-variable analyses (see Table 4). The pre-surveys will be administered as follows:
Medical Student Pre-Survey 1:
DUCOM: Prior to the first clinical year.
Penn Med: Prior to the first of clinical year.
Medical Student Pre-Survey 2:
DUCOM: On the first day of ambulatory medicine rotation.
Penn Med: On the first day of family medicine rotation.
The medical students will also be given the survey twice after the intervention (“Student Post-Survey 1” and “Student Post-Survey 2”). The Student Post-Survey 2 is needed so that the study can evaluate how long the improvements in knowledge, attitudes, and confidence in communication skills persist after the intervention. This is a critical point since it may indicate that the NIDA substance abuse doc.com module is efficacious in the short term but not in the long term. The medical student post-surveys will be administered as follows:
Medical Student Post-Survey 1:
DUCOM: On the last day of ambulatory medicine rotation.
Penn Med: On the last day of family medicine rotation.
Medical Student Post-Survey 2:
DUCOM: At or near the end of the year of clinical rotations.
Penn Med: At the end of the year of clinical rotations.
The protocol for Group 2 of the study stipulates that the residents be given the survey twice. The resident surveys will be administered as follows:
Resident Pre-Survey:
DUCOM: At or near the start of outpatient primary care rotation.
Penn Med: On the first day of primary care medicine rotation during orientation.
Resident Post-Survey:
DUCOM: At or near the end of outpatient primary care rotation.
Penn Med: On the last day of primary care medicine rotation or within one week of completing the rotation.
Both of these surveys are necessary, as the research findings depend on the changes in results of Resident Pre-Survey to Resident Post-Survey in the intervention arm compared to the changes in results between the same two surveys in the control arm.
For both Groups 1 and 2 of the study, burden will be minimized by shortening the Post-Survey(s), (e.g., eliminating the demographic questions) and allowing the residents to conveniently complete both surveys online from any Internet connected computer. This is feasible because the respondents will be tracked individually and the information for all subjects will be present in the database as a consequence of their completing a Pre-Survey at an earlier date.
A.7. Special Circumstances Relating to the Guidelines in 5 CFR 1320.5
This information collection fully complies with 5 CFR 1320.5(d)(2) guidelines except that there may be limited generalizability of this project’s results to the general universe of medical students and residents as was described in more detail in Section A.2.
The 60 Day Federal Register Notice of the proposed research was published in Vol 75 No. 242, p79008 on December 17, 2010. No public comments were received.
Consultations on the design, survey questions, data management, and analysis of this study have occurred throughout the planning phase of this project. These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall project design, use of cluster randomization to decrease “contamination” between the intervention and control arms in the study, and data analysis plans; to obtain advice and recommendations concerning study-related informational technological applications; and to structure the study and instruments so as to minimize overall and individual response burden. Consultations have occurred with the following individuals in connection with this study (listed in alphabetical order):
Charles A. Dackis, M.D., Associate Professor of Psychiatry and Director, The Charles O’Brien Center for Addiction Treatment, Department of Psychiatry, University of Pennsylvania School of Medicine (215-662-8752). Years of consultation: 2009–2010.
Christof Daetwyler, M.D., Associate Professor of Family Medicine, Technology in Medical Education (TIME) and Clinical Skills Teaching and Assessment, Drexel University College of Medicine (215-991-8565). Years of consultation: 2009–2010.
Robert Gallop, Ph.D., Associate Professor of Statistics, Department of Mathematics and Applied Statistics, West Chester University of Pennsylvania, West Chester, PA 19383 and Senior Biostatistician, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine (610-436-2419). Years of consultation: 2009–2010.
John H. Holmes, Ph.D., Associate Professor of Medical Informatics in Epidemiology at the Hospital of the University of Pennsylvania (HUP), and Senior Scholar, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine (215-898-4833). Years of consultation: 2009–2010.
Ann King, M.A., Assessment Scientist in the Measurement Consulting Services Unit at the National Board of Medical Examiners (NBME) (215-439-8604). Years of consultation: 2009–2010.
Richard Landis, Ph.D., Professor and Director, Division of Biostatistics, Department of Biostatistics and Epidemiology, University of Pennsylvania School of Medicine (215-573-4922). Years of consultation: 2009–2010.
Jennifer Lapin, Ph.D., Director, Graduate Medical Education (GME) Research and Evaluation, University of Pennsylvania School of Medicine (215-746-3568). Years of consultation: 2009–2010.
Gail Morrison, M.D., Vice Dean for Education, University of Pennsylvania School of Medicine (215-898-8034). Years of consultation: 2009–2010.
Dennis H. Novack, M.D., Professor of Medicine, Associate Dean of Medical Education, Drexel University College of Medicine (215-991-8537). Years of consultation: 2009–2010.
Charles O’Brien, M.D., Ph.D., Professor of Psychiatry and Director of the Treatment Research Center (TRC), Department of Psychiatry, University of Pennsylvania School of Medicine (215-222-3200, ext 4). Years of consultation: 2009–2010.
Steven J. Peitzman, M.D., FACP, Professor of Medicine, Drexel University College of Medicine (215-991-8537). Years of consultation: 2009–2010.
Judith A. Shea, Ph.D., Professor of Medicine in the Division of General Internal Medicine, Department of Medicine, Associate Dean of Medical Education Research and Director of the Office of Evaluation and Assessment in the Academic Programs Office, University of Pennsylvania School of Medicine (215-573-5111). Years of consultation: 2009–2010.
Arnold J. Smolen, Ph.D., Associate Dean for Information Technology, Drexel University College of Medicine (215-991-8564).Years of consultation: 2009–2010.
Barbara J. Turner, M.D., M.S.Ed., Professor of Medicine, Division of General Internal Medicine, and former Director, Primary Care Physician Scientist Fellowship, Department of Medicine, University of Pennsylvania School of Medicine (215-898-2022). Years of consultation: 2009–2010.
Although participation as a research subject in this project is voluntary, respondents are likely to perceive a time cost and burden associated with their participation. The use of incentives to increase response has been well documented by Dillman (1978; 2000). Therefore, several non-monetary incentives will be offered to encourage participation in the study and to increase response rates (i.e., completing the surveys and other outcome measurements as applicable, as well as completing the educational interventions [if randomized to the intervention arm]).
Incentives will differ between medical students and residents. However, all incentives will be proportionate and reasonable to the time demands of the various activities in which subjects may participate. All incentives have been reviewed by Penn Med’s Institutional Review Board (IRB), which determined that the incentives are not unduly influential or coercive. Note: DUCOM and Penn Med have arranged that DUCOM will rely on Penn Med’s IRB for review and continuing oversight of its human subjects research for this study (see Attachment 8 for the agreement between Penn Med and DUCOM). Penn Med’s IRB and DUCOM’s IRB will be jointly responsible for protecting the rights of study participants. For the purposes of this document, only Penn Med’s IRB will be referred to in the rest of the text. In addition, the following attachments are provided that relate to the IRB approval:
Attachment A9: IRB Approval (including all informed consent forms for medical students and residents)
Attachment A10: Penn Med Letter About Type of Consent
Attachment A11: DUCOM Letter About Type of Consent
Incentives for both groups will be gift cards for Starbucks or amazon.com. The incentive amounts described below are consistent with incentives used previously by Penn Med and DUCOM when they participated in an American Medical Association funded study called Sound Prescribing: A Lifelong Curriculum For Physicians. The Penn Med IRB approved that study (#808019) and the DUCOM IRB accepted the Penn Med IRB approval. An 83% response rate was achieved using those incentives. In addition, these amounts are consistent with amounts used for the NIDA Physicians Outreach project (clearance was obtained under OMB # 0925-0574 NOA). Although the participants in the NIDA study were exclusively practicing physicians, some of them were early in their careers (i.e., more similar to residents). Experience at the two participating medical schools has shown that the monetary value for these incentives is consistent with what is required for maximizing participation.
The incentives that will be used in this study are :
Group 1 (Medical Students):
No incentives for medical students: Incentives to complete these surveys were strongly discouraged by the Deans’ offices which want to avoid setting unwanted precedents.
Group 2 (Residents):
Pre-Survey (intervention and control): $10 gift card
doc.com module (intervention only): up to three $25 gift cards for a maximum of $75 in gift cards*
Post-survey (intervention and control): $20 gift card
* Residents will be given a $25 incentive for each 20 minute section of the doc.com module they complete to encourage them to return and complete the module if they are called away.
A.10. Assurance of Confidentiality Provided to Respondents
The current project will fully comply with the Privacy Act of 1974 (5 U.S.C. Section 552a, 1998) (http://oma.od.nih.gov/ms/privacy/pa-files/0156.htm). The Privacy Act may apply to some data collection activities (e.g., the study will require identifiable responses by subjects due to the planned follow-up evaluation).
All respondents will be assured that their participation is voluntary, that no adverse consequences will accrue to students or residents who don’t participate in the research or withdraw from the research, and that their comments and opinions will be kept private (subject to exceptions from IRBs and the sponsor or if required by law to provide access). In addition, emails to inform medical students and residents about the study and any other introductory materials about the study, will indicate NIDA’s Federal status and the purpose of the study (see Attachments A1, A2, A5, A6, and A7). Also, Section B contains additional information on study procedures related to email communication.
Because the study meets the Common Rule definitions for human subjects research, 45 CFR 46 (Regulations for Protection of Human Subjects) will apply. The Penn Med IRB requires that the data in the study be kept secure. Participants will be informed that the information they provide will be kept confidential, and will not be disclosed to anyone but the researchers conducting this study, except as required by law (see Attachment A9). NIDA will not at any time be in possession of the data. In order to protect privacy of the subjects, each subject will be assigned a unique study ID number. All documents that identify participants by name will be kept in a locked file cabinet in the office of the Research Project Manager(s). The documents will be stored for 3 years (along with all data) and then destroyed. All of the databases related to the study will therefore not contain the subject’s name or other personal identification (e.g., Social Security number).
The data for Groups 1 and 2 of this study will be kept in a secure relational database (Oracle Clinical) in the DCC at Penn Med during the study and for 3 years after the study ends. Some of the data from medical students may temporarily be located in another secure relational database in the School of Medicine Information Technology servers at Penn Med and similar servers at DUCOM prior to being transferred to the DCC. These will be password protected databases with well established security systems to prevent unauthorized access.
All of the provisions of the Privacy Act listed above will be included in the Informed Consent Form that the Penn Med IRB has approved. At this time there is an information technology (IT) system in place for maintaining data at the host institution of the study, but it cannot be assigned to the study until it is clear that the study will be conducted. Therefore, a Privacy Impact Assessment (PIA) cannot be completed at this time; however, if the study receives clearance and is conducted there are plans to develop a new system or use the institution’s existing system for data collected. NIDA will conduct a PIA as required when the study is under way and will report annually on the security in conformance with the e-Government Act. This should suffice for the Privacy Act mandates listed above.
A.11. Justification for Sensitive Questions
The study will collect personally identifiable information and will comply with all requirements of the Privacy Act. Personally identifiable information such as names and email addresses of the participants will be collected by the Project Managers (research coordinators) for the purpose of assigning unique study ID numbers to participants and randomly assigning them to an intervention arm or control arm, and to conduct necessary follow-up for completion of the study tasks. As noted in Section A-10, this information (e.g., the link between the subject’s name and study ID number) will be kept in a locked in a file cabinet in the Research Project Manager(s) office. Only the Research Project Manager(s) will have access to these documents. Demographic information in the survey instruments (e.g., age, race/ethnicity, gender) complies with the Privacy Act. The surveys do not ask for a name, Social Security number, date and place of birth, address, email address, and other similar sensitive information that can be linked or is linkable to an individual. None of the questions in the surveys will request any personally invasive or sensitive information regarding alcohol or drug use.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
Table 1 contains estimated response burdens for each of the two subject populations participating in the project’s research study.
Estimates for response burden were calculated based on the methodology being used with each target population and are based on previous pilot experience of nine representatives of the two populations taking the surveys. For example, for the Pre-Surveys and Post-Surveys for both students and residents, the average time of completion was 10 minutes; therefore, burden estimates of 0.17 hour (10 minutes/60 minutes) were used for each survey taken for the estimated 708 individuals expected to participate in the project.
Study |
Type of Respondents |
Estimated Number of Respondents |
Frequency of Response |
Average Time per Respondents |
Estimated Total Burden Hours |
Group 1
|
Medical Students (Pre-Survey) – |
400 |
2 |
0.17 |
136 |
Medical Students (Post Survey) |
2 |
0.17 |
136 |
||
Group 2 |
Primary Care Resident Physicians (Pre-Survey completed once; Post-Survey completed once) |
308 |
1 |
0.17 |
52.36 |
Primary Care Resident Physicians (Post Survey |
1 |
0.17 |
52.36 |
||
|
Total |
708 |
....................... |
................. |
377 |
Table 2 summarizes the estimated annualized cost burden to the subjects. Wage estimates were obtained from the U.S. Department of Labor, Bureau of Labor Statistics using wage estimates from 2008. A 4% annual cost of living expense was added to the 2008 wage estimates to obtain the 2011 wage estimates. The mean hourly wage estimates were calculated for the selected occupational category because neither medical students nor residents were listed as a distinct category by the Bureau of Labor Statistics.
Table 2: Annualized Cost to Respondents
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Respondent |
Mean Hourly Wage Rate (adding 4% annual cost of living expenses) ‡ |
Respondent Cost |
|
Medical Students |
Healthcare Practitioner and Technical Workers, All Other (29-9099) |
400 |
4 |
0.17 |
$27.31* |
$7428 |
Primary Care Resident Physicians |
Healthcare Practitioner and Technical Workers, All Other (29-9099) |
308 |
2 |
0.17 |
$27.31 |
$2668 |
Total |
|
708 |
................. |
................. |
................. |
$10,096 |
‡Wage Estimates from 2008: A 4% yearly cost of living expenses was added to derive 2010 wage estimates from 2008 wage estimates, i.e., 2009 hourly wage is $25.25 ($24.28 x 1.04); 2010 hourly wage is $26.26 ($25.25 x 1.04), and 2011 hourly wages will be $27.31 ($26.26 x1.04). Most if not all of the study will be done in 2011.
*The mean hourly wage estimates for this category were chosen because neither medical student or resident physician categories were listed in the 2008 National Occupational Employment and Wage Estimates.
A.13. Estimate of Other Total
Annual Cost Burden to Respondents or
Record Keepers
There are no capital, start-up, operational, purchase of services, or maintenance costs to the respondents in providing the information required by this research. They will use technology already available on their institutions’ computers.
A.14. Annualized Cost to the Federal Government
This project has been funded with ARRA money. The cost to the Federal Government of this 2-year project is $1,006,750, or $503,375 per year. These costs cover all aspects of survey design, testing, data collection, and analysis, and the time needed from one NIDA staff member. It is estimated that one full-time equivalent NIDA staff member will spend 208 hours over the 2-year project period to manage and administer the project. Assuming an annual salary of $90,000, government personnel costs will be $6,750 over a 2-year period, or $3,375 per year.
A.15. Explanation for Program Changes or Adjustments.
This is a new project and is a new collection of information.
A.16. Plans for Tabulation and Publication and Project Time Schedule
The project will be conducted over a 2-year period, and is currently funded from August 2010 to September 2012. This 25 month period that the study is currently funded for includes study preparation, data collection, analysis and reporting. Anticipating OMB approval July 2011, data collection from all study groups will be conducted July 2011 – June 2012, and is anticipated to be closed at the end of June 2012. If the academic cycle (medical school clinical rotation) in either school is missed, data collection will be continued for another semester or clinical rotation to obtain the required sample size. This could require data collection for an additional 6-9 months. A no-cost extension may be needed to complete the study.
Table 3: Project Time Schedule for Groups 1 and 2 of the Proposed Study
Activity
|
Time Schedule |
Cluster randomization of groups of subjects during specified time in academic program (e.g., DUCOM Group 1 ambulatory medicine rotations) to intervention or control arms; data collection |
Within 12 months after OMB approval (An additional 6-9 months may be needed depending on OMB approval and timing of clinical rotations) |
Data clean up and query resolution |
On-going, during the first 12 months after OMB approval
|
Closing the database |
13th month after OMB approval |
Analyses, report and executive summary writing, presentation to NIDA, other presentations, manuscript writing and submission |
7–15 months after OMB approval
|
Publication Plans. Results of the study completed under the project will be presented to NIDA in a briefing to the agency, accompanied by a final written report and executive summary. Results will also be written as manuscripts to be submitted to peer-reviewed publications after review by NIDA. Peer-reviewed publications that accept the manuscript will make the manuscript and results available for dissemination to the public online (if that is the journal’s policy) and in hard copy format. Examples of journals that may be targeted for publication include Academic Medicine and Journal of General Internal Medicine. Other journals that have high impact and wide readership include JAMA (perhaps submitting a manuscript for inclusion in JAMA’s annual medical education issue in September), the Archives of Internal Medicine, the Annals of Internal Medicine, and British Medical Journal.
Consideration will also be given to submitting to journals that focus on medical education research, especially as it relates to communication skills. Examples include Journal of Communication in Healthcare, Medical Education, Health Communication, Journal of Family Practice, Family Medicine, and Patient Education and Counseling.
Publications will also be submitted to PubMed as required per current NIH policies and made available to the public as well as medical educators through that mechanism. Links to these publications and their citations will be located on the NIDAMed Web site from which they can be viewed directly or downloaded. A copy of the executive summary will be emailed to individual participants who express an interest in receiving it.
Presentations will be made at national and regional conferences of organizations interested in the use of new medical education interventions to teach about science-based drug abuse treatment and prevention and about communication skills of physicians related to substance abuse. Likely conferences include those sponsored by the American Association of Medical Colleges (AAMC), the Society for General Internal Medicine (SGIM), and the American College of Physicians (ACP).
Exact timing of presentations depends on the submission and approval cycles for the various sponsoring organizations. Likewise, publication of articles in refereed journals depends on the results of evaluations by the reviewers (e.g., how many revisions are required).
Analysis Plan. As described in Section A.1, the overall goal of this project is to assess the efficacy of a specific interactive web-based teaching module, the NIDA substance abuse doc.com module, in the field of professional education of healthcare providers.
Research Design. The study utilizes a prospective randomized controlled cluster design.
Medical Students. In the design for Group 1, the “cluster” refers to a group of 10–15 medical students who have previously been assigned to the same outpatient clinic for the same 4-week period during their first year of clinical training. This first year occurs from January of the second year of medical school to December of the third year of medical school at Penn Med and from July at the start of the third year of medical school to the beginning of June at the end of the third year of medical school at DUCOM. This 4-week period of clinical training is referred to as a “clerkship.”
The study will randomize all of the students on the same clerkship at the same medical school during a specific 4-week period to either the intervention arm or the control arm. At Penn Med the study will use students on their family medicine clerkship. At DUCOM the study will use students on their ambulatory medicine clerkship. The study will randomize during the first full 4-week period that occurs after OMB approval. The study will continue to study students in 4-week clerkships until the end of the data collection period (see Table 3). The study will use a permuted block design for this cluster randomization process with a 1:1 randomization of clusters between control and intervention arms.
In Group 1, the study will have certain safeguards in place to address concerns about possible contamination between subjects in the control arm and in the intervention arm. One example of such contamination includes subjects in the control arm visiting the public-accessible NIDA substance abuse doc.com module on their own. It is not feasible to remove the substance abuse doc.com module from the public domain during the period when this study is active. NIDA has decided that the site needs to remain accessible to all because it serves a valuable public health function. It is also important to note that medical students in their first year of clinical training are not likely to visit the site outside of the research protocol if they have agreed not to, and they are generally fully occupied with other tasks related to their clinical training as well as being under pressure to perform well in their clerkships to increase their chances of matching with a good residency training program. Medical students are under severe time constraints. Even so, the investigators recognize this possible threat to validity and will follow Steps 1–5 listed below in an attempt to minimize the risk of contamination to the study.
The study’s informed consent form will include an explicit statement that subjects in the control arm agree to refrain from asking subjects in the intervention arm about the intervention or what those subjects learned from the intervention as well as to refrain from visiting the NIDA substance abuse doc.com module on their own while the study is ongoing (i.e., during the month of their clerkship and up to the end of their primary clinical year when they take Student Post-Survey 2). Study personnel who will obtain informed consent will be trained (as per written instructions in the study’s Manual of Operations [MOP]) to emphasize this point verbally when explaining the consent form to subjects.
The consent form will also include an explicit statement that the subjects in the intervention arm agree to refrain from discussing the intervention or what they learned from the intervention with subjects in the control arm as well as to refrain from visiting the NIDA substance abuse doc.com module on their own outside the experimental protocol (i.e., accessing it without the study-designated password). This statement aims to prevent subjects in this arm from visiting the site off-protocol (i.e., getting more exposure––a “higher dose”).
The MOP will include an orientation to the study for the directors of the clerkships who will be involved in the study. This orientation will explain the research protocol to the clerkship directors and the need to prevent contamination of the students in the two research arms. The clerkship directors will be asked to reiterate the precautions noted in 1 and 2, above, when they discuss the study and consent forms with the students. Clerkship directors will also reemphasize the importance of these prohibitions at the time that the students in the control arm receive the control educational addition to the standard clerkship curriculum.
The consent form will ask subjects not to access the NIDA substance abuse doc.com module outside of the study’s protocol or otherwise receive information about the intervention outside of the protocol.
If any subjects indicate on their surveys that they have accessed the NIDA substance abuse doc.com module outside of the study protocol or have otherwise indicated that they have been “contaminated,” the analysis will stratify the data and consider them to be an a priori subgroup in order to take this variable into account.
In the opinion of the investigators, it is highly unlikely that medical students during their first clinical year of clinical training will be motivated to visit the NIDA substance abuse module on their own (i.e., outside of the research protocol) especially after they’ve agreed to not do so. The investigators feel that these circumstances, plus the precautions noted above in Steps 1–5, should minimize the risk of contamination and adjust for it if it does occur.
In order to provide the students who are randomized to be in the control arm of the study with an educational incentive to participate and maintain equity between the intervention and control arms, the study will provide a substitute educational addition to the standard clerkship curricula for the students in the control arm. These substitute additions will not include a discussion of care of patients with substance use disorders. Students will be informed that the proposed modules and substitute educational additions may or may not benefit their medical school/resident education and treating patients with substance use issues. In order to avoid any feelings of coercion to medical students or residents, participants are being offered an incentive to participate in this study (see Section A-9). In addition, a statement is included in the email invitations that specifically states that no penalty shall be incurred for those who choose not to participate.
Medical students in the control arm at Penn Med will be given a case study that the clerkship director will choose from the online repository of teaching cases (http://www.med-u.org/virtual_patient_cases/fmcases) created by The Society of Teachers of Family Medicine.. The case that will be used is titled Family Medicine Computer-Assisted Simulations for Educating Students (fmCASES). According to the Web site, this is “a comprehensive set of peer-reviewed online cases to teach the core curriculum of the family medicine clerkship. In partnership with iInTIME, a team of STFM educators has developed a set of 29 virtual patient cases to build clinical competency, fill educational gaps, and model the core values of family medicine” (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2713170/).
At DUCOM, the students in the control arm will be instructed to read the online doc.com module on somatization (http://webcampus.drexelmed.edu/doccom/user/). In addition, Dr. Novack, a DUCOM faculty member, will give them a lecture on that topic. All DUCOM students have been registered to have full access to the restricted (i.e., non-public accessible) doc.com modules, including the module on somatization.
Given that the intervention occurs as part of the medical students’ educational training, providing all participants with exposure to the same information is important. For medical students in the control arm and intervention, they will receive an email at the completion of the study directing them to the module (e.g., link for NIDA site to be provided or control site) so they have exposure to the content offered in the education session and may contact the PI at their institution if they have questions (see Attachment A12) .
Residents. As noted above, the study utilizes a prospective randomized controlled cluster design. For Group 2, the “cluster” refers to a group of 4–10 resident physicians who have previously been assigned to the same outpatient clinic for the same time period. All residents will be in residency training programs that are affiliated with Penn Med (Family Medicine and Internal Medicine residents) or with DUCOM (Internal Medicine residents). These residents will be in their first, second, or third year of clinical training. These years are designated as Post-Graduate Years 1, 2, and 3 (PGY-1, PGY-2, and PGY-3).
At Penn Med, residents are assigned to 2-week blocks of time that are called “firms.” These residents are assigned to one of three clinics in which they only have responsibilities for seeing outpatients (i.e., these are dedicated ambulatory care rotations). It is during these 2-week blocks of time that the study will randomize as “clusters” in the same manner as described above for Group 1 (i.e., permuted block randomization with a ratio of 1:1 of clusters to intervention and control arms).
The number of residents that are assigned to these 2-week blocks of time at any one time varies among clinics from 4 to 10. The study will start to randomize these clusters after receiving OMB approval for the study. The study will finish randomization so that the residents in the last cluster to be randomized can complete the study protocol before May 31, 2011 (Note: Date will be adjusted accordingly based on OMB approval).
The study of residents (i.e., Group 2 subjects) will use Steps 1–5, as described in more detail above for Group 1 subjects, to minimize the risk of contamination in the study and its effects on the analysis. Briefly, these include:
Explicit statements in the consent form for subjects in the control arm to avoid communication about the intervention or Web site and to refrain from non-protocol visits to the Web site. This will be emphasized by study personnel who are obtaining consent and will be included in training of study personnel as per the study’s MOP.
Explicit statements in the consent form for subjects in the intervention arm to avoid communication about the intervention or Web site and to refrain from non-protocol visits to the Web site. This will be emphasized by study personnel obtaining consent.
Written training instructions in the MOP to orient the attending physicians in the residents’ clinics who will be overseeing the residents in the study to re-emphasize the precautions mentioned in the consent forms.
Questions on the residents’ Pre- and Post-Surveys that ask about visits to the Web site off protocol and other exposures that reflect contamination between arms.
The analysis phase will consider subjects who indicate that they have been “contaminated” as an a priori subgroup so the analysis can adjust for these effects if they occur.
As with the students in Group 1, the investigators hold the opinion that the risk of generally over-worked and sleep-deprived residents visiting the Web site on their own during their free time is minimal, especially after they have agreed to not do so.
During the study, residents in the control arm will not add a new non-substance abuse related educational exposure (e.g., module, debrief/faculty led seminar) but will rely on the standard curriculum for the control arm. The belief is that such an addition is not necessary and may be detrimental to successful completion of this study. The study design without an addition will be simpler to implement and be less burdensome to participants in the control arm. The investigators do not believe that lack of an additional education exposure will compromise the research design for the residents. The anticipation is that participation will be improved and encouraged because prior medical education research involving residents has indicated that minimizing time expended on the research study should be a priority (unpublished data, Paul N. Lanken, MD, Principal Investigator at Penn Med).
However, given that the intervention occurs as part of educational sessions with a debrief/faculty led seminar, providing all participants with exposure to the same information as part of their training is important. Therefore, residents in the control arm will receive an email at the completion of the study directing them to the doc.com module (link for NIDA site to be provided) so they have exposure to the content offered in the education session, told that facilitator’s guide provides consistent information covered in the debriefing/faculty led seminar and to contact the PI at their institution if they have questions (see Attachment 13).
The standard curriculum that the residents in both control and intervention arms will receive varies from residency program to residency program. For example, in the standard (“core”) curriculum for the Penn Med-affiliated Internal Medicine residency, there are currently no formal didactic teaching sessions related to substance use disorders. The residents learn about care of patients with substance use disorders by taking care of such patients in the hospital, emergency room, and outpatient clinics. They are given instruction by their clinical instructors in these clinical sites as applied to specific patients.
In contrast, the standard curriculum for the Penn Med-affiliated Family Medicine residency includes at least one yearly lecture about care of patients with substance use disorders by a psychiatrist who is a Penn Med professor. Each week, two Penn Med psychiatrists spend three half days (8–10 hours) per week to serve as resources for the residents in their outpatient clinics about the care of patients with any type of behavioral health disorder, including substance use disorders. Finally, all PGY-2 residents (second year residents) are required to spend approximately 8 hours over a 2-week period as observers in a substance abuse clinic with a full-time Penn Med faculty psychiatrist who specializes in treatment of substance use disorders. In addition, there are case-based discussions relating to care of patients with substance use disorders with other faculty preceptors in the Family Medicine outpatient and inpatient practices.
The study will use the randomized permuted block method to assign groups of medical students or residents to intervention or control arms to take into account the changes over time in the medical students or residents. The subjects will not be blinded to the randomization assignment.
Data Collection Plan. The project will use a repeated measures approach to assess the educational intervention’s efficacy.
The study will achieve its specific aims, as described above, by collecting information from individual subjects (either medical students in Group 1 or primary care residents in Group 2) by use of surveys that are given before (“Pre-Survey”) and after (“Post-Survey”) the intervention in the intervention arm and at the equivalent time points in the control arm. The study will compare the changes in knowledge, attitudes, and skills, as measured by differences between the Pre-Survey and Post-Survey in the intervention arm with the changes that occur in the control arm.
Data Analysis Plan and Plans for Use of Quantitative Data. Data analysis will seek to answer the overall hypothesis of the study and achieve the specific aims of the study as described earlier.
All survey items have been designed to determine the subjects’ knowledge, attitudes, and degree of confidence related to communication skills regarding patients with substance use disorders. Therefore, the data analysis will consist of assessing medical students’ or residents’ knowledge, attitudes, and skills in these three domains. For each item on the surveys and for each difference between Post-Surveys and Pre-Surveys, data distributions will be reviewed to determine if the distribution of data meet the assumptions of any statistical tests being used (e.g., if the data are normally distributed). If violations to statistical assumptions are discovered, data may need to be transformed (e.g., log transformation) before additional analyses are conducted or, alternatively, non-parametric methods will be used. After data distributions for each variable are assessed, basic descriptive statistics will be calculated. Frequencies (e.g., raw numbers, percentages) will be presented for nominal and ordinal level measures (e.g., demographic data). Measures of central tendency (e.g., mean, median, mode, standard deviation, and range) will be presented for interval and ratio level variables. As a general practice, means and standard deviations will be presented for normally distributed variables; medians will be presented to show central tendency for skewed distributions.
Descriptive statistics of the scores from the questions from the three domains of the surveys and exploratory graphing such as frequencies, means, standard deviations, box and whisker plots, stem and leaf diagrams, and scatter plots will be used to assess the normality of the data in terms of the presence of skew and/or outliers. The continuous outcome data will be transformed if necessary by using an appropriate transformation such as the log transform for skewed long tailed data.
The dependent variables in both Groups 1 and 2 of the study are changes in scores for these three domains: knowledge, attitudes, and confidence in communication skills. For Group 1, the changes will be calculated by comparing the Student Post-Survey 1 results to the Student Pre-Survey 2 results for each medical student. For residents in Group 2, the changes will compare results from the Resident Post-Survey to the Resident Pre-Survey. The surveys are referenced in Section B. The specific scores for each dependent variable will be derived during the preparation phase based on a modified group process that involves the project’s investigators and consultants as well as preliminary data from similar surveys that were completed by medical students from the four NIDA CoEs for Physician Information. Dr. Judy Shea, part of the research team, has unpublished preliminary results of a factor analysis that involves the attitudinal questions (Part 3) of these surveys.
The independent variables include variables related to the individual research subjects (Level 1) and variables related to the “clusters” (Level 2) as described above. A preliminary list of independent variables for both individual subjects and their clusters is shown in Table 4.
Table 4: Levels 1 and 2 and Associated Co-Variables for Hierarchical Linear Modeling (HLM) Analyses
|
Category of Research Subjects |
|
|
Medical Students |
Residents |
Level 1
|
Individual Research Subject |
Individual Research Subject |
Level 1 Associated Co-Variables (Independent Variables) |
|
|
Level 2 |
Cluster of Research Subjects (medical students on the same clerkship during the same month) |
Cluster of Research Subjects (residents in the same outpatient clinic during the same time period) |
Level 2 Associated Co-Variables (Independent Variables) |
|
|
The data analysis plan for both Group 1 and Group 2 of the study will use HLM because the data are hierarchically structured (Goldstein, 1995; Raudenbush & Bryk, 2002). The two levels of hierarchy in the research design for both the students and residents, as described above, are the subjects themselves in Level 1 and their respective clusters as defined above for medical students and residents in Level 2 (see Table 4). Traditional statistical methods, such as multiple linear regression and logistic regression and students t test, have an underlying assumption that the observations for each subject are independent (i.e., that the observations in any one subject are not systematically related to observations in any other subjects). This assumption is likely violated when the subjects are in these hierarchical clusters (e.g., members of the same clusters may have similar responses, as is the case for this study’s research design). HLM is used to take this clustering into account in the statistical modeling framework. Ignoring the nesting may result in inaccurate levels of significance if there is a positive correlation among cluster members. Therefore, any inference from models which assume independence of the data will be susceptible to error.
The benefits of using HLM in this analysis include:
The ability to inspect if the results are biased due to drop out or missing data.
The use of all available data.
The ability to accommodate the various levels of nesting within the data structure and model the potential correlation between members of the various levels.
The ability to accommodate various data such as binary, ordinal, or count data. HLM applied to this type of data is usually referred to as Hierarchical Generalized Linear Models (HGLM).
Allows for the inclusion of covariates in the model such as demographics. The site can be thought of as possibly a third level of the data (members within cluster, within site) but being limited with respect to the number of sites (Penn Med versus DUCOM) the site will be treated as a potential covariate and all models will be adjusted for “site.”
The advantage of HLM is the ability to account for the clustering or correlation attributable to members within a section. If there is no correlation, then there is no benefit to the HLM approach. The first step will be to examine if there is a sufficient source of variance attributable to variability between sections to warrant section being treated as a random effect, hence accounting for section-to-section variance and correlation within section through the Hausman test (Hausman, 1978). If sufficient, then the HLM approach will be implemented. If there is not sufficient variability between sections, sections will be treated as a fixed effect, which will account for the on-average values of the outcome measures to vary between sections and perform an analysis of covariance (ANCOVA) model to contrast the two arms.
As with any statistical model, there are potential limitations with this approach; therefore, a goodness of fit diagnostics on the model will be performed. If there is no evidence of correlation of members within a cluster, an independent sample t-test will be implemented. Although highly unlikely, if the correlation of members within a cluster is negatively correlated, an independent sample t-test will again be implemented, which will result in more conservative inferences as compared to an HLM.
The research team has a high level of experience with these models, including two recent papers published on this topic by Dr. Gallop (who is a biostatistician for this study) and Dr. Tasca (Tasca & Gallop, 2009; Gallop & Tasca, 2009).
The expiration date for the OMB clearance will be displayed on all instruments approved for this study.
There are no exceptions to the certification statement identified in OMB Form 83-I, item 19, “Certification for Paperwork Reduction Act Submissions.”
Dillman DA. Mail and Telephone Surveys: The Total Design Method. New York: Wiley-Interscience, 1978.
Dillman DA. Mail and Internet Surveys: The Tailored Design Method (2nd edition), New York: Wiley-Interscience, 2000.
Gallop R and Tasca GA. Multilevel modeling of longitudinal data for psychotherapy researchers: 2. The complexities. Psychotherapy Research 2009; 19, 438-452.
Goldstein H. Multilevel Statistical Models. New York, Halstead Press, 1995.
Hausman JA. Specification Tests in Econometrics, Econometrica 1978; 46:1251-1271.
Raudenbush SW and Bryk AS. Hierarchical linear models (2nd Ed.).London: Sage, 2002.
Rogers EM. Diffusion of Innovations, (5th ed.), New York: Free Press, 2003.
Spagnoletti CL, Bui T, Fischer GS, Gonzaga AMR, Rubio DM, and Arnold RM. Implementation and evaluation of a web-based communications skills learning tool for training internal medicine interns in patient-doctor communication. Journal of Communication in Healthcare 2009; 2(2):159-172.
Tasca GA and Gallop R. Multilevel modeling of longitudinal data for psychotherapy researchers: 1. The basics. Psychotherapy Research 2009; 19, 429-437.
A -
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | JUSTIFICATION |
| Author | lankenp |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |