Supporting Statement B - OMB - IDRWG data collection 3.23.10

Supporting Statement B for

Process Evaluation of the NIH Roadmap Interdisciplinary Research Work Group Initiatives, NIDCR

Sue Hamann, Ph.D.

Science Evaluation Officer

Office of Science Policy and Analysis

National Institute of Dental and Craniofacial Research (NIDCR)

National Institutes of Health

31 Center Drive, MSC 2190

Building 31, Room 5B55

Bethesda, MD 20892-2190

Phone: 301-594-4849

Fax: 301-496-9988

[email protected]

Table of contents

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for the Collection of Information 1

B.3 Methods to Maximize Response Rates and Deal with Non-response 2

B.4 Test of Procedures or Methods to be Undertaken 3

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 3

List of Attachments:

• Attachment 1. Investigator Survey

• Attachment 2. Trainee Survey

• Attachment 3. Introductory email

B.1 Respondent Universe and Sampling Methods

The universe of respondents for which the clearance is sought includes three groups: (1) IDRWG Principal Investigators (2) Other Investigators on the IDRWG Grants; and (3) Trainees.¹

IDRWG Grantees and other investigators The estimated number of principal investigators, 50, was calculated based on the data available on the NIH website.² Two additional investigators (100 total) would be selected from each project.

Trainees. The total number of trainees who have participated in all IDRWG programs is estimated at 300 based on information contained in NIH records. This population undergraduates, graduate students, postdocs, and mid-career professionals. We propose to survey the entire population of 300 students. We want to estimate population proportions of characteristics of interest with a margin of error of plus or minus 3.5 percentage at 95% confidence level. For a large population, we would require a sample of 784. However, with a finite population size of 300, the required number in the sample would be 217 students. If we assume an 80% response rate we would need to select 272 students to reach this target number. Since the sample is 91% of the population we instead propose to select all the students in the population. In case the response is rate is slightly lower than expected, this sample still gives us the required sample for estimating population percentages with the desired margin of error assuming that nonresponse is random.

B.2 Procedures for the Collection of Information

This request for clearance includes data collection from investigators and trainees via on-line surveys. During the design and feasibility phase of the project, the protocols were developed and pre-tested on representative respondents. Although surveys were not pre-tested in this current format, many of the questions were crafted in from similar surveys conducted on the National Science Foundations evaluation’s of the Interdisciplinary Graduate Research Training and the Science and Technology Centers programs.

Survey of Investigators. The survey (see Attachment 1) will be administered on-line. Individuals who received an IRWG grant will receive an email invitation describing the goals of the study and the procedures being used. The invitation email will also contain a link to the survey. Two reminder emails will be sent only to non-respondents two and four weeks following the initial invitation. All individuals who complete the survey will receive a “thank you” note, also via email. The survey will be closed at the end of eight weeks. Individuals who did not respond after two reminder emails will be contacted by telephone and given an option to respond to the survey during the call or to schedule a follow-up call.

Given the relatively small sizes of the study populations and the prevalence of qualitative information sought, the use of simple descriptive statistics—such as counts, ranges, and frequency—is most appropriate for the analyses of the data. For example, a relative frequency of yes/no responses will be calculated for the question of whether the grantees applicants would have been able to fund their projects through other sources.

B.3 Methods to Maximize Response Rates and Deal with Non-response

Several methods will be used to maximize response rates and to deal with non-response, such as:

• Providing a sufficient timeframe for data collection;

• Using a survey package that allows monitoring of responses and identification of non-respondents;

• Designing the survey instrument in such a way that each respondent is only presented with questions that are relevant to his or her specific situation (by using skip patterns); and

• Having a reminder email to non-respondents sent from an NIH email address.

With these methods in place we expect to achieve a response rate of 80 percent. We base this estimate on our experience in the pilot phase, and on similar surveys conducted with samples of graduate students and career researchers who participated in federally funded science programs. The table below presents the response rates for various evaluation studies of programs that surveyed graduate students and early career individuals, which were used to estimate the expected response rates for this project.

B.3-1. Response Rates in Previous Studies Used to Predict Response Rate for Current Effort

Program	Response Rate	Length of Time Between Participation and Data Collection
CAREER Fellows	84%	0-10 years
IGERT Former Students	74%	0-10 years
GK-12 Fellows	MS 83% PhD 92%	0-5 years

The table below presents the projected number of completed surveys if we obtain an 80 percent response rate. Section B.1 provides our rationale for surveying the full population of trainees.

B.3-2. Projected completed surveys

Respondent group	Population size	Projected responses with 80% response rate
IDRWG grantee (PI)	50	40
Grant investigators	100	80
Trainee	300	240

B.4 Test of Procedures or Methods to be Undertaken

Because of the importance of the initiatives and this evaluation to the NIH, a feasibility study was conducted that resulted in the design for the current Process Evaluation, which is the subject of this application. The proposed survey instruments were pilot tested with the appropriate respondent groups including investigators and trainees. Respondents were asked to comment on the clarity and content on the questions.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor for collection and analysis of data in this study is Abt Associates Inc., Cambridge, MA. Two individuals from Abt Associates participated in the development of the study plan. These individuals have knowledge of statistical methods, experience in evaluation of research programs, and expertise in scientific research. Contact information for these individuals is as follows: Luba Katz, PhD ([email protected]; 617-349-2313) and Alina Martinez, EdD ([email protected]; 617-349-2312).

1 The proposed study design also includes interviews with the NIH staff, for which clearance is not sought.

2 http://nihroadmap.nih.gov/interdisciplinary/fundedresearch.asp