0990- EIVRI studyssPartA

Supporting Statement

For the

Evaluation of the Effectiveness of an Educational Interactive Video on Research Integrity

Office of Research Integrity (ORI)

Office of Assistant Secretary of Health (ASH)

Office of the Secretary of Health and Human Services (OS)

U. S. Department of Health and Human Services (HHS)

Contact Person:

Rhonda J. Moore, Ph.D.

Health Science Administrator

Office of Research Integrity

Office of Public Health and Science

U. S. Department of Health and Human Services

1101 Wootton Parkway, Suite 750

Rockville, Maryland 20852

Phone 240-453-8435

Fax 240-276-9574

Email [email protected]

Contractor:

DSFederal, Inc.

6210 Wedgewood Road

Bethesda, MD 20817

Phone 240-813-5799

Fax 301-916-9350

Project Director: Sophia J. Parker

Supporting Statement for an Evaluative study of the Effectiveness of an Educational Interactive Video on Research Integrity

A. Justification

1. Circumstances Making the Collection of Information Necessary

• The Office of the Assistant Secretary for Health (OASH) and specifically the Office of Research Integrity (ORI), requests approval for a new data collection to examine the effectiveness of an educational interactive video on research integrity.

• The Department of Health and Human Services (DHHS) through the Public Health Service Act section 493 directed the Secretary to create a regulation to protect against biomedical and behavior research fraud. In response, the Office of Research Integrity (ORI) was created (42 USC 289 - Sec. 289b. Office of Research Integrity) and the Secretary issued 42 CFR part 50 and 93 which created regulations requiring institutions to report their investigations of research misconduct.

• The Office of Research Integrity (ORI) promotes integrity in biomedical and behavioral research supported by the U.S. Public Health Service (PHS) at about 4,000 institutions worldwide. ORI monitors institutional investigations of research misconduct and facilitates the responsible conduct of research (RCR) through educational, preventive, and regulatory activities.

• A primary mission of the ORI Division of Education and Integrity (DEI) is to educate institutions about regulations, but perhaps more importantly to provide institutions with educational resources that will help each institution to promote an environment that fosters research integrity. Educational resources previously provided by DEI include textbooks, pamphlets, interactive videos, newsletters, reports, a website and blog, conferences and workshops, and journal articles.

• In 2000, the Division of Education and Integrity (DEI) at ORI was directed to “focus more on preventing misconduct and promoting research integrity through expanded education programs.” Specifically, DEI was directed to “conduct policy analyses, evaluations, and research to improve DHHS research integrity and build the knowledge base in research misconduct, research integrity and prevention” (Federal Register: May 12, 2000, Volume 65, Number 93, pages 30600-30601)

• In Fiscal Year 2009, ORI contracted with a company to create an educational interactive video for educating viewers about responsible conduct of research (RCR) and research integrity issues. The intended audience for this study includes graduate students, post-doctoral fellows, faculty, research staff, administrators and research integrity officers (RIOs). This research effort is for an independent evaluation of the effectiveness of the interactive video and meeting ORI goals. We believe this evaluation study will provide insight into this regard.

2. Purpose and Use of Information Collection

1. One of ORI’s goals is to educate the public about RCR, research integrity and the prevention of research misconduct. The development of an interactive video can aid in the effective dissemination of information. Therefore, it is important to know whether this interactive video is effective and useful in actually meeting ORI goals.

• This web-based survey evaluation is targeted to the solicitation of opinions from respondents (i.e., research instructors/faculty, Research Integrity Officers (RIOs) and Research Administrators) who have experience with the ORI educational programs or who may have experience with RCR programs in the near future;

• The information to be collected will be used by the ORI to help gain additional insight regarding the effectiveness of this educational interactive video, and determining whether it meets our customers’ ( i.e., Research instructors, Research Administrators and Research Integrity Officers) and ORI’s mission and goals.

• The information gained from this evaluation will also help ORI identify strengths and weaknesses in existing RCR educational resources and will assist our office in making improvements in our RCR educational programs based on the feedback.

• The information gained from this evaluation will also help ORI to better refine methods for offering, presenting and delivering information most effectively, and better understand the type of and quality of services our target audiences need.

• This evaluation can also aid in the development of future educational materials and training opportunities for RIOs, faculty, research administrators and students for promoting RCR and research integrity.

3. Use of Improved Information Technology and Burden Reduction

• We have done several things to reduce the burden of the study on respondents. Only individuals who have already reviewed the DVD will be requested to take the web based survey. To further investigate the effectiveness of the educational interactive video, we will also use a demographics form and a questionnaire accessible via a web-based survey evaluation tool.

• Further, w have considered a variety of modes of data collection for this study and have concluded that the proposed web-based self-administered questionnaire or customer satisfaction survey will be the most time and cost-effective plan for data collection from this population. This web-based survey can also facilitate gathering of information, and given the short time frame of the survey (less than 25 minutes) significantly lessen the burden on respondents.

4. Efforts to Identify Duplication and Use of Similar Information

• ORI is a primary source of funding for research on RCR, research integrity and research misconduct. For that reason, ORI staff members are the most knowledgeable about the availability of other existing interactive DVDs that would satisfy the needs of this study. ORI staff members have no knowledge of any other existing interactive DVDs addressing the problems of RCR and research integrity as a training tool.

5. Impact on Small Businesses or Other Small Entities

• No small businesses will be involved in this study.

6. Consequences of Less Frequent Collection of the Information

• This study is a web-base survey data collection activity taken at one point in time. If this survey is not conducted, it will result in the failure to determine a meaningful statistical evaluation of the educational interactive simulation.

• This study is web-based survey of research faculty/instructors, Research Integrity Officers (RIOs) and Research Administrators’ perceptions of the effectiveness of this educational interactive video. The data collection is voluntary and the burden on participants is not high..

• There are no legal obstacles to reduce the burden.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

• This request fully complies with the Guidelines of 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice/Outside Consultation

• A 60-day Federal Register Notice was published on Thursday, February 24, 2011 /(Federal Register /Vol. 76, No. 37 /, page 10365). No public comments were received.

9. Explanation of any Payment/Gift to Respondents

• No payments or gifts will be provided to the respondents.

10. Assurance of Confidentiality Provided to Respondents

• Data will be kept private to the extent allowed by law. DSFederal and CSR Inc., (the contractors) will give respondents its assurance that the interview information will be reported in such a way that it will not be possible to identify the respondents or their institutions. In addition, interviewers and other project staff with access to identifying information will be required to sign DSFederal data privacy pledges. DSFederal will maintain identifying and interview information in separate files on a shared drive that will limit access only to persons working on the project. During data collection and cleaning, the identification and interview files will have a numeric link between them. That link will be removed when the data have been edited and a final data set is produced. No information that will allow identification of respondents/non-respondents will be provided to ORI.

• DSFederal will submit this study plan to the CSR-centralized IRB for their approval before conducting our data collection (see attached documents including approval).

11. Justification for Sensitive Questions

• No sensitive questions are included in this study.

12. Estimates of Annualized Hour and Cost Burden

12A.Estimated Annualized Burden Hours

This web based survey evaluation will be implemented using an existing ORI database made up of over 40,000 NIH funded principal investigators (research faculty/instructors). Over 2000 of the individuals on this same list are also instructors (non-NIH funded), research administrators, and Research Integrity Officers (RIOs)(from educational institutions). For the purposed of this study, this group is the respondent universe. Based on prior research in this field, we do not anticipate the response rate to be particularly high. However, we estimate that only about 15% of this sample (lowest range, approximately 6000 individuals) will potentially complete this survey. This survey will take approximately 21 minutes to complete. The response rate is estimated at 15% based on web based survey research that indicates that a response rate for web based surveys is between 15% and 22%. These numbers are also considered to be high for studies with optional responses and no incentive^.1-2 We are using the 15% to calculate the estimated response rate for the sample. Here aResponse Rate of 15%  (40,000 X .15) equals 6,000 participants.. There are 32 questions, and 26 are multiple choice questions that may take approximately 20 seconds (or about 9 minutes total). There are also six short answer questions which could take approximately two minutes each to answer. Therefore, the estimated time to complete the survey is approximately 21 minutes.

Type of Respondents

No. of Respondents

No.

Responses

per

Respondent

Average

Burden per

Response

(in hours)

Bur Total Burden Hours den

Research Faculty/Instructors,

Research Integrity Officers (RIOs) Research Administrators

6000

1

21 /60

2100

 

 

 

 12B. Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost to the federal government for contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 14. The Department of Labor website can be used to determine appropriate wage rates for respondents.

Estimated Annualized Burden Costs

Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Faculty/ Principal Investigators	2100	141.00*	296,100.00
Research Integrity Officers (RIOs)
Research Administrators
Total			296,100.00

*Please note that this estimate of wages for study participants was derived from the Healthcare practitioner and technical occupational category. Individuals who teach RCR and Research integrity may be faculty/principal investigators and researchers, Research integrity officers or research administrators. In many cases, the work may overlap as someone can be a faculty member and a RIO at the same time. As the majority of participants may be from the biomedical and health care field (given US PHS funding) and given the overlap, this was one of the rationales for choosing this occupational category as the closest estimate for wages. See also, US Department of Labor Force Statistics from the Current Population Survey. Labor rates by Education.(2010), pp.16. Available at: http://www.bls.gov/cps/cpswom2010.pdf [Date accessed: 04/2/2012].

13. Estimates of other Total Annual Cost Burden to Respondents or Record keepers/Capital Costs

• Respondents will not incur capital costs or capital maintenance costs as a result of participating in this information collection effort.

14. Annualized Cost to Federal Government

• ORI has contracted with DSFederal to conduct the data collection and analysis, and to prepare a report. The contract will cover a one year period at a total cost of $199,050.94 to the government. Annualized contractor costs are estimated to be $199,050.94 per year.

• It is expected that the ORI project officer will spend a total of 10 percent time on the project (208 hours [10%*40hrs/week*52weeks/year=208hrs] time). At the average rate of $41.92 per hour, we estimate that it will cost the government $8719.20. Annualized the cost to the government for the ORI staff time is estimated to be $10,000.

• It is anticipated that the ORI consultant, DSFederal and CSR, Inc (subcontractor to DSFederal) assisting on this project will spend a total of $199,050.94 participating over the one year. The total cost to the government is expected to be $207,770. Annualized the cost to the government for the ORI consultant time is estimated to be $199,050.94.

• The total cost to the government would be $207,770.00; annualized it would cost the government $207,770.00.

15. Explanation for Program Changes or Adjustments

• This is a new data collection effort.

16. Plans for Tabulation and Publication and Project Time Schedule

• ORI may decide disseminate the results of this study. Please see the OMB Sec B Statistical methodology section of this OMB package.

• It is important to note that it is critical to obtain OMB clearance to proceed with the study by the start of May 2012 so that the study can be completed before the expiration of the contract (September 29, 2012). ORI requested and received a no cost extension (to obtain OMB clearance) and this is the final end date for this project. The schedule for preparation of the manuscript follows:

• Project Timeline for the Proposed Evaluation Study

Task Name	Duration	Start	Finish
Meet with ORI to review new SOW timeline	1 day	Thu 7/28/11	Thu 7/28/11
Review “The Lab” and Facilitator Guide to “The Lab”	1 day	Thu 9/15/11	Thu 9/15/11
Develop a Proposed Set of Study Questions for Evaluation	12 days	Thu 9/15/11	Fri 9/30/11
Determine the Hypotheses to Be Tested	12 days	Thu 9/15/11	Fri 9/30/11
Identify and Review Relevant Literature	7 days	Thu 10/6/11	Fri 10/14/11
Conduct a Comprehensive Review of Existing Data Sources	15 days	Thu 10/20/11	Wed 11/9/11
Identify the Key Variables	12 days	Thu 11/10/11	Fri 11/25/11
Determine the Exact Experimental Design of This Evaluation	22 days	Thu 9/15/11	Fri 10/14/11
To work with those research integrity offices to place either the questionnaire or a link to it on their web site	22 days	Thu 9/15/11	Fri 10/14/11
To present the proposed design along with potential pros and cons to the COTR at the September kick-off meeting.	1 day	Thu 9/15/11	Thu 9/15/11
Prepare Drafts of All Data Collection Instruments for Approval	30 days	Thu 9/15/11	Wed 10/26/11
Create the Data Collection Forms	5 days	Mon 10/17/11	Fri 10/21/11
Assist ORI in Acquiring Necessary IRB and OMB Approvals	38 days	Thu 10/27/11	Mon 12/19/11
OMB submission	5 days	Thu 10/27/11	Wed 11/2/11
IRB application	4 days	Tues 04/10/2012	Fri 04/13/2012
Develop a Web-based Interactive Survey	5 days	Thu 10/27/11	Wed 11/2/11
To decide to which survey program to use and subscribe in order to begin the preparation of the on-line survey	5 days	Thu 10/27/11	Wed 11/2/11
Conduct Pilot Test(s)	10 days	Fri 04/13/2012	Fri 4/20/2012
Conduct Web-based Interactive Survey	60 days	Mon 4/30/2012	Fri 6/22/12
Via electronic invitation and using an existing ORI database of over 40,000, invite research integrity officers (RIOs), faculty and instructors and research administrators to Participate in the Evaluation	1 day	Mon 04/30/12	Mon 04/30/12
Analyze Data Provided by ORI on the Use of the Interactive Video on the ORI Website	35 days	Mon 6/25/12	Fri 7/30/12
Transmit All Raw Data and Analyses to ORI	1 day	Fri 7/30/12	Fri 7/30/12	July
Provide a Final Written Report	1 day	Fri 9/28/12	Fri 9/28/12
Prepare a Final Power Point Presentation of the ORI Evaluation and Submit to the COTR	14 days	Fri 9/14/12	Fri 9/14/12	September
Present the Results of These Findings via Written Report to the COTR	1 day	Fri 9/28/12	Fri 9/28/12
Monthly Progress Report	142 days	Thu 9/15/11	Fri 3/30/12
Transmit the Progress Report or Other Written Documentation of Work	142 days	Thu 9/15/11	Fri 3/30/12
Meet with the ORI in Person or by Telephone Every 2 Weeks during the Contract or As Otherwise Agreed	142 days	Thu 9/15/11	Fri 3/30/12
Provide Project Management over All Tasks	142 days	Thu 9/15/11	Fri 3/30/12
Per the Request of the COTR, Complete a Final Face-to-Face to face Presentation of the Study Findings	1 day	Fri 9/28/12	Fri 9/28/12	September

17. Reason(s) Display of OMB Expiration Date is Inappropriate

• DSFederal is not requesting to suppress the OMB expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

• There are no exceptions to the certification.

1 Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis Field Methods August 2008 20: 249-271

2 Sills, S. and Song, C. 2002. “Innovations in Survey Research: An Application of Web‐Based Surveys.” Social Science Computer Review. 20 (1):22‐30.

17