Mini SSB NIH External Constituency Surveys

Mini Supporting Statement B

NIH External Constituency Surveys

Section B

B.1. Respondent Universe and Sampling Methods

There are two populations of interest under the Peer Review Capacity Surveys: an applicant population, and a reviewer population. These populations are defined as follows:

Applicant Population

The applicant population comprises those individuals who have submitted a grant application and/or had active grant support from NIH or the National Science Foundation in the past five years. All members of the population meet these criteria.

To ensure a sufficient number of respondents who have had NSF funding and a sufficient number of investigators based in Canadian Institutions to permit valid comparisons, we will sample 300 individuals who have had research support from NSF in scientific areas relevant to the NIH research portfolio in the past five years, using the NSF public awards database. To obtain a sample of respondents with a history of research support from CIHR, we will sample 300 individuals from the NIH applicant sampling frame whose applications were sponsored by institutions in Canada. The survey includes questions about prior research support from and reviewer service to the three agencies that will be used to sort respondents into a primary focal group of respondents who identify as NIH investigators/reviewers, and two out-groups of respondents who identify as primarily as NSF and CIHR-supported scientists.

Reviewer Population

The reviewer population comprises those individuals who meet the criteria shown above for applicants and who have served in NIH study sections in the past five years. The target population of reviewers includes regular (appointed/permanent) and ad hoc (temporary) reviewers.

The total number of applicants and applicant/reviewers in the NIH sampling frame from non-Canadian institutions (62,433) is equal to the sum of the number of individuals who are applicants only (29,251), and the number of individuals who are both applicant and reviewer (33,182).

Table B.1-1. Applicant and Applicant/Reviewer Population Counts

Stratum
	Applicants Only	Applicant/ Reviewers	Total Applicant Population
American Indian/ Alaska Native, Hispanic	21	14	35
Asian, Hispanic	26	25	51
Black, Hispanic	39	16	55
Multiracial, Hispanic	65	54	119
Other, Hispanic	1,293	1,183	2,476
American Indian/ Alaska Native, non-Hispanic	60	54	114
Asian, non-Hispanic	4,528	5,324	9,852
Black, non-Hispanic	769	584	1,353
Multiracial, non-Hispanic	344	272	616
Other, non-Hispanic	22,073	25,638	47,711
Pacific Islander, non-Hispanic	33	18	51
Total	29,251	33,182	62,433

Note: A total of 16,154 persons with unknown ethnicity were assumed to be non-Hispanic for purposes of sample selection. 8,921 persons with unknown race were included in the “other” race category, together with 41,266 Whites.

Sample Selection

Determining Overall Sample Sizes

The total number of individuals (4,779) who will be sampled and subsequently surveyed is defined by the power estimate described below. This includes 4,179 applicant/reviewers in the NIH pool, 300 in the NSF pool, and 300 in the Canadian-based investigator pool. The number of individuals is within the burden limits under NIH’s Office of Management and Budget (OMB) Generic Clearance No. 0925-0627. Since demographic factors will not be considered in making their selection, the 300 individuals sampled from the NSF awards data base and 300 individuals from the NIH applicant sampling frame whose applications were sponsored by institutions in Canada are deducted from the allocation scheme described below.

Broad Allocation Scheme

The total number of individuals to be sampled will be allocated to the following sets of individuals:

1. Applicants only (29,251)

2. Applicant/Reviewers (33,182)

Within each set, sample sizes must be sufficient in order to allow for estimates within race and ethnicity groups to meet precision requirements (discussed below).

Initial Sample Sizes Based on Precision Requirements

The following four steps were taken for the two groups of individuals: Applicants only, Applicant/Reviewers.

The following four steps will be taken for the two groups of sample members—applicants only and those who are both reviewers and applicants:

1. A cross-tabulation will be created of the number of individuals by race (Asian, Black, Native American, Pacific Islander, “other,” and multiracial) and ethnicity (Hispanic or non-Hispanic).

2. Using the nQuery Advisor software (Elashoff, 2005), the number of individuals required to be sampled in each race-by-ethnicity group will be estimated such that, within each group, a two-sided 95% confidence interval for a population proportion of 50% will have a half-width of 5%.

3. For those groups with population counts of less than 30 or for which nQuery reports that the sample size is not large enough for the sample calculation to be approximately normally distributed, all group members in the relevant sample will be included. Such groups are said to be selected with certainty.

4. For those groups not selected with certainty, nQuery will report the required sample size.

The following table (Table B.1-2) shows the number of individuals selected with certainty or estimated by nQuery as being required to meet the precision requirement outlined in Step 2 above.

Table B.1-2. Initial Sample Sizes Based on Precision Requirements

Stratum	Applicants Only		Applicant/Reviewers
	(1) Population Count	(2) Sample Size	(3) Population Count	(4) Sample Size
American Indian/ Alaska Native, Hispanic	21	21	14	14
Asian, Hispanic	26	26	25	25
Black, Hispanic	39	39	16	16
Multiracial, Hispanic	65	65	54	54
Other, Hispanic	1,293	649	1,183	691
American Indian/ Alaska Native, non-Hispanic	60	37	54	46
Asian, non-Hispanic	4,528	334	5,324	335
Black, non-Hispanic	769	321	584	344
Multiracial, non-Hispanic	344	214	272	227
Other, non-Hispanic	22,073	338	25,638	345
Pacific Islander, non-Hispanic	33	20	18	18
Total	29,251	2,064	33,182	2,115

Columns 1 and 3 shows the number of individuals who are applicants and the number who and applicant/reviewers, respectively, and Columns 2 and 4 list the numbers of individuals who will be sampled to achieve the stated precision requirement.

Sample Power Analysis

Power estimates were calculated for comparing various race and ethnicity groups for each of the following samples: 1) Applicant only; 2) Applicant/Reviewer.  The power estimates ranged from a minimum of 10% to a maximum of 48% for detecting a difference of 5%.  The power estimates ranged from a minimum of 533% to a maximum of 97% for detecting a difference of 10%.

Response Rates

The response rates for the survey will be calculated in accordance with the recommendations that the American Association for Public Opinion Research (AAPOR) has published in its Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (2008). The formula for the response rate is as follows:

_,

where I = complete interview, P = partial interview, R = refusal, NC = noncontact, and O = other nonresponse. Notably, this formula differs from the AAPOR formula RR4 in that, because all individuals in the NIH-provided sampling frame are assumed to be eligible for the study, no estimate of the number of eligible individuals among those with unknown eligibility is included in the denominator. Adjustments to the response rate formula can be made if ineligibility of some individuals is later determined.

Sample Weights

Discussed here is the method to be followed to create the final sample weights and final estimates for the peer review surveys. One nonresponse-adjusted sample weight will be created for the applicant sample; another weight, for the reviewer sample. These weights will consist of a product of two factors: the base weight and the nonresponse adjustment, defined as follows:

• The base weight (for a given sample) is the inverse of the unconditional probability of selecting a sample member into the sample. This weight accounts for the stratification used in the sample design. Notably, if all sampled individuals respond, then no nonresponse adjustment is necessary.

• The nonresponse adjustment (for a given sample) is an adjustment imposed on the sampling weight of the respondents to account for those applicants who do not respond to the survey. In general, this adjustment will be greater than 1 so that each respondent will represent himself or herself, as well as some portion of the nonrespondents.

There are numerous ways of constructing a nonresponse adjustment. For each of the applicant and reviewer samples, the plan is to adjust the base weights within strata and to use a simple ratio adjustment. In order to perform this adjustment, we will need to know which stratum each respondent belongs to.

Estimation Procedure

After the data are collected, analysis of the data must rely on software that can account for the sample design. Data analysis will be performed with SUDAAN software (2008). SUDAAN can manage correlated observations in a general sense, with nonparametric and parametric approaches being available. Base SAS software will be used for data manipulation and tabulation of results (2008).

B.2. Information Collection Procedures/Limitations of the Study

Data Collection Procedures

Sample members will be asked to complete the surveys online. The basic steps involved in the data collection process for all three surveys include:

• A lead letter will be sent to each sample member via email during the week prior to the day the survey opens (Attachment 3).  The lead letter will be signed by a senior NIH official. It will explain the purpose of the survey and why they were selected to participate. 

• Three to five days after the lead letter is sent, an e-mail invitation will be sent to all sample members (Attachment 4).  It will again invite the sample member to participate in the survey and will provide a hyperlink to the survey Website.

• One week after the e-mail invitation, a reminder e-mail will be sent to all sample members (Attachment 5). The e-mail will encourage those who have not yet logged in to the Website to participate in the survey.

• One week after the first e-mail reminder, a second e-mail reminder will be sent to all non-respondents (Attachment 5).   The e-mail will reinforce the purpose and relevance of the survey. 

• One week after the second e-mail reminder, a third e-mail reminder will be sent to all remaining non-respondents (Attachment 5).   

• In addition, a final reminder letter (Attachment 6) will be mailed by express mail (FedEx) along with a hardcopy version of the survey.   Enclosed with the letter will be a postage paid, business reply envelope for returning the completed questionnaire.

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

The ability to gain the cooperation of potential respondents is key to the success of these two surveys. Consistent with sound survey methodology, the design of the survey will include approaches to maximize response rates, while retaining the voluntary nature of the effort. We will use the following approaches to maximize response rates for the surveys:

• Participation will be made as easy and non-burdensome as possible by designing each questionnaire to take no more than an average of 15 minutes to complete.

• The online instruments will be designed to be clear and easy to understand. Thorough usability testing of the survey instruments will be conducted to eliminate technical errors and to ensure ease of navigation and use.

• Advanced outreach will raise awareness about the surveys and to encourage participation (e.g., announcements on NIH Websites and newsletters).

• The lead letter and introductory e-mail invitations will inform sample members of the study. They will contain enough information to generate interest in the surveys. The letter and email will provide a point of contact at RTI for additional information.

• Follow-up e-mails will remind sample members about the survey, and encourage participation. These reminders will always include a link to the survey.

• A final reminder letter will include a hardcopy version of the survey to provide an alternative mode for answering the questions.

B.4. Tests of Procedures of Methods

The survey instruments have also been tested through a modified Question Appraisal System (QAS). With the QAS, the questions in the instrument were analyzed in relation to the tasks required of the respondents (to understand and respond to the questions) and evaluate the structure and effectiveness of the questionnaire form itself. RTI International’s Question Appraisal System (QAS-04) was used to guide this instrument review. This coding system constitutes an item taxonomy that describes the cognitive demands of the questionnaire and documents the question features that are likely to lead to response error. These potential errors include comprehension, task definition, information retrieval, judgment, and response generation. This appraisal analysis was used to identify possible revisions in item wording, response wording, questionnaire formats, and question ordering/instrument flow.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Dr. Jeff Laufenberg

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-316-3107

E-mail: [email protected]

LIST OF ATTACHMENTS

Attachment 1 - Survey Instrument

(Attach 1_NIH Enhancing Peer Review _Capacity Survey_Reviewer Screenshots_8-6-2014)

Attachment 2 - Privacy Act Determination Letter

(Attach 2_NIH Enhancing Peer Review _Capacity Survey_Reviewer Screenshots_8-6-2014)

Attachment 3 – IRB Exemption Letter

(Attach 3_NIH Enhancing Peer Review _Capacity Survey_IRB Exemption_8-6-2014)

Attachment 4 – Inivitation and Reminder Letters

(Attach 4_NIH Enhancing Peer Review _Capacity Survey_Invitation _ Reminder Letters _8-6-2014)