OMB Mini Supporting Statements A for Phase 2

Mini Supporting Statement

NIH External Constituency Surveys

Section A

A.1 Circumstances Requiring the Collection of Data

This is a request to conduct voluntary customer satisfaction surveys of the National Institutes of Health’s (NIH’s) Enhancing Peer Review Initiative. These surveys will help fulfill the requirements of:

• Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and

• The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service their customers want as well as their level of satisfaction with existing services.

A.2 Purposes and Uses of the Data

Three surveys are planned -- a Reviewer Survey, an Applicant Survey and an Advisory Council Survey. The primary objective of these surveys is to assess these stakeholders’ experience with the peer review enhancements. The findings from the surveys will provide an important source of information for developing recommendations to further refine the enhanced peer review process. The information collected in these surveys is needed by NIH to obtain customer feedback about their satisfaction with the changes being implemented.

The Reviewer Survey will focus on respondents’ experience with the shortened grant application format that was introduced in January 2010. Other questions will assess the ongoing experience of the community of NIH peer reviewers to changes to the peer review system that were made in January 2009 (e.g., enhanced review criteria, templates for structured critiques, scoring of individual review criteria, use of a 9-point scoring scale, and clustering of New Investigator/ Early Stage Investigator applications for review).

The Applicant Survey will ask respondents to report on their most recent application experience. Questions will also focus on the shortened applications and the usefulness of the 9-point rating scale, scoring of individual review criteria and overall impact/priority, and other key elements.

Advisory Council members will be questioned about whether the information contained in the artifacts of the review process, overall impact scores and summary statements are sufficiently detailed to permit them to execute their roles as Advisory Council members.

A.3 Use of Information Technology to Reduce Burden

The mode of data collection for these surveys was carefully considered with respondent burden in mind. It was determined that automated information technology will be used to collect and process the information. The surveys will be conducted online. Invitations to participate will be sent to the selected sample members via email and later by mail, if needed.

A.4 Efforts to Identify Duplication

Collected information will be limited to that which is needed to assess customer satisfaction. Some of the data we are seeking is available through NIH data systems where administrative information relating to research grants and contracts is stored. For applicants, for example, this includes administrative data on individual grant applications (e.g., date of submission, type of application, and application status). For reviewers and Advisory Council members, NIH maintains data on the number, dates, type of review activities, and other topics. However, these data may not be linked to the customer satisfaction survey responses to achieve the goals of this effort. The proposed survey instruments minimize the duplication to the maximum extent possible. Only essential demographic data are requested.

A.5. Small Business

Not Applicable

A.6 Consequences of Not Collecting the Information

Individual applicants and reviewers will be asked to complete the survey only once in FY2012. For prior and subsequent years, unique samples of applicants and reviewers are involved. Absent this survey frequency, changes to the peer review system might not be adapted to meet customer needs based on customer satisfaction, because formal data on satisfaction with the system would not be ascertained.

A7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 C.F.R. 1320.5

This data collection fully complies with 5 C.F.R. 1320.5.

A.8. Consultation Outside the Agency

Not Applicable

A.9. Payments or Gifts to Respondents

No payment or gift will be offered to survey participants.

A.10. Assurance of Confidentiality

The NIH Privacy Act Officer has reviewed this OMB request and determined that the Privacy Act is applicable (Attachment 4).

Concern for privacy and protection of respondents’ rights will play a central part in the implementation of the surveys. Strict procedures will be followed for protecting the anonymity of information gathered from the participants. Participation will be fully voluntary, and the choice not to participate will have no impact on eligibility for or receipt of future funding.

Safeguarding procedures that we will implement include:

• The safeguarding protections offered to survey participants are described in the informed consent language in the introduction to the survey instruments. Respondents will be informed their participation is voluntary and that no consequences will be associated with not responding or with responding. Individuals contacted in the course of these surveys will be assured of their confidentiality under 42 USC 1306, 20 CFR 401 and 422, 5 USC 552 (Freedom of Information Act), 5 USC 552a (Privacy Act of 1974), Privacy Act System of Records Notice: 09-25-036, and OMB Circular No.A-130.

• All data will be analyzed and reported in an aggregate form that does not personally identify any applicants or reviewers.

• An independent contractor, RTI International (RTI), will collect and collate the surveys electronically. RTI will also be responsible for initial analysis and reporting of the data. The data sets that will be transferred back to NIH staff will be fully de-identified. RTI has the required security clearances in order to assure confidentiality and protection of the data.

• RTI’s Institutional Review Board (IRB) has determined that these surveys are exempt from IRB review (IRB ID Number 12444) based upon information provided by the RTI project manager (Attachment 5). In addition, all study staff members will receive Human Subjects Protection Awareness training. This training will promote awareness of the human subjects’ protection offered by the survey design, ethical issues and concerns, and regulations and assurances by which the survey is governed.

• Access to data will be restricted to project staff members on an as-needed basis.

RTI will observe high standards of information technology (IT) security to protect the confidentiality, integrity and availability of all computer-based systems and the data they contain. RTI IT security policies and procedures are designed to protect information systems and data from a wide range of risks and will educate their staff to be aware of their responsibilities for ensuring information security and to comply with these policies. RTI also participates with agencies to ensure that their policies conform to agency information security requirements and applicable laws and regulations as required by contract. RTI has System Security Plans for its infrastructures in which it documents how they secure their systems using administrative, technical, and physical controls.

All computer-based systems employed by RTI will comply with the Privacy Act of 1974. The system security features will include:

• User ID and Password authentication required to access all computer systems

• The Website will operate on a certified and accredited Internet-accessible Standard Security Infrastructure which has received an Authority to Operate in accordance with NIST special publication 800-37 (Guide for the Security Certification and Accreditation of Federal Information Systems).

• Web content delivery will be on FIPS 140-2 compliant hardware.

• Access from the Internet is available to authorized staff only and is controlled by RTI’s Internet firewalls. Remote access to RTI’s data networks is provided through the use of client-computer-installed VPN software, a clientless SSL/VPN portal, and direct dial-in connections. The use of RSA SecurID two-factor authentication for remote access is supported.

A.11 Questions of a Sensitive Nature

The NIH is committed to providing high-quality service to its customers.  Given the diversity of its constituents, it is important for NIH to collect survey data from a wide range of customers. The Applicant Survey and Reviewer Survey contain questions regarding respondents’ race, ethnicity, gender, and age. All three surveys contain questions about work-related information (type of employer organization, job title, education).  This information will allow NIH to analyze the survey data by key analysis subgroups and support NIH’s long-standing efforts to strengthen the diversity of the membership of its applicants and reviewers.  

Respondents may skip any or all of the questions concerning race, ethnicity, gender, age and work-related information in the surveys.  Those who choose to provide these demographic data will do so voluntarily.  The surveys will not collect any personally identifiable information.  Thus, any demographic information gathered by the surveys will not be linked to individual respondents.     

A.12 Estimates of Response of Burden

The total number of participants who will be sampled is 4,710. These participants are university faculty and other members of the NIH research community. The total sample size is expected to be 250 for the Advisory Council survey, 2,518 for the applicant survey, and 1,942 for the reviewer survey. It is estimated that the Advisory Council Survey will each take an average of 15 minutes to complete. The Applicant Survey and Reviewer Survey are estimated to take an average of 30 minutes to complete. The annual hour burden is, therefore, estimated to be 2292.5 hours for approximately 4,710 respondents (Table A.12-1).

Table A.12 – 1 Estimates of Annual Hours Burden

Types of Respondents	Number of Respondents	Frequency of Response	Average Response Time	Annual Hour Burden
Adult Science Professionals – Applicant Survey	2,518	1	0.50	1259
Adult Science Professionals – Reviewer Only	1,942	1	0.50	971
Advisory Council Survey	250	1	0.25	62.5
Total	4,710			2292.5

Estimated costs to the respondents consist entirely of their time. Costs for time were estimated using a rate of $41.00 per hour for adult science professionals. A rate of $95.00 per hour for Advisory Council members is based upon the 2011 NIH salary cap for senior investigators, $199,700 per year. The estimated annual cost burden for respondents for the first year for which the generic clearance is requested is $97,430 (Table A.12-2).

Table A.12 – 2 Annualized Cost to Respondents (Based on Expected 80% Response)

Types of Respondents	Number of Respondents	Frequency of Response	Average Time per Respondent	Hourly Wage Rate	Respondent Cost
Adult Science Professionals – Applicant Survey	2,518	1	0.50	$41	$51,619
Adult Science Professionals – Reviewer Survey	1,942	1	0.50	$41	39,811
Advisory Council Survey	250	1	0.25	$96	6,000
Total	4,710				$97,430

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

We do not require any additional record keeping.

A.14. Estimates of Costs to the Federal Government

For the first year, the approximate annualized cost to the government for this data collection effort is approximately $581,801 (Table A.14-1). Total government personnel costs will be $137,220, taking into account benefits. This figure assumes a median GS-15 annual salary of $140,259 - $144,385 for an NIH professional to manage the projects, an upper level GS-14 salary of $136,771 for additional NIH staff members to provide expert reviews and analysis. Salaries are based on the January 2011 General Schedule for the Washington, DC metropolitan area (http://www.opm.gov/oca/11tables/html/dcb.asp). Details are provided in the table below. Estimated annual federal personnel costs for the second year are $54,331, which adjusts for a reduced level of effort for federal staff and a shorter project duration in the second year.

Contractor support will be required to carry out the data collection efforts. It is estimated that the first year of this effort will cost approximately $430,451. Survey efforts in subsequent years will cost an estimated $215,225 for the second year. The cost reduction in the second years is because IT systems and surveys will have mostly been finalized in the first year and the project duration extends over only 6 months in the second year. The NIH anticipates undertaking no more than one project in 18 months.

Courier costs for paper surveys total $14,130 assuming a Courier shipping rate of $15.00 per letter and a 20% rate of follow-up requests.

Table A.14-1. Annualized Costs

Activity	Cost Year 1	Cost Year 2
Administration of the Clearance
NIH staff (1 GS-15) – 20% FTE @ $140,259/yr in May 2011 – December 2012	$36,466	$18,771
NIH staff (16 GS-14)– 5% FTE beginning @ $136,771/yr in May 2011 – November 2011; 2.5% thereafter	100,754	35,560
Contract Support for Data Collection
3 surveys, 18-month project period	430,451	215,225
Courier Cost for Paper Surveys
942 surveys (4,710*20%) x $15.00	14,130	0
Total	$581,801	$269,556

A.15. Changes in Burden

Not Applicable

A.16. Plans for Publication, Analysis and Schedule

The analysis plan is designed to examine the degree to which survey responses differ across key analysis subgroups or combinations of those groups. Key analysis groups are defined by combining the following information to form groups of interest, such as review outcome, employment information, race and ethnicity.

Comparisons across key subgroups will focus on topics such as experience with the peer review process, satisfaction ratings about the peer review process, as well as the format of grant applications. Analyses will focus mainly on descriptive information including two-way tables to compare groups of interest.

Data collected for this study will be aggregated. No results will be reported that identify respondents by name or another identifier that allows respondent’s identity to be disclosed. Specific procedures for analyzing the data are described in the following paragraphs.

Descriptive Information

Analysis will begin with a description of the applicants, peer reviewers and Advisory Council members who responded to the peer review surveys. The three surveys are provided in Attachments 1, 2 and 3, respectively. One analysis table will be created with the demographic variables collected in Section C on the applicant questionnaire, and Section E on the peer reviewer questionnaire. No gender, race, or ethnicity information will be collected from Advisory Council Members. The data will be presented in two tables. One table will contain two columns, one column for applicant questionnaire data and one column for peer reviewer questionnaire data. The second table will contain the Advisory Council questionnaire data.

Data will be presented in tabular format with frequencies and percents for categorical variables; means, minimum and maximum values will be displayed for continuous variables. Table A.16-1 is an illustration of the table that will be compiled during analysis for the descriptive and demographic related questions shown above. The overall numbers of respondents reported in each column will be given in the column headers.

Table A.16-1: Demographic Information – Sample Table Shell

Demographic Question	Applicant Questionnaire	Reviewer Questionnaire
	N =	N =
Ethnicity
Hispanic	n (%)	n (%)
Non-Hispanic	n (%)	n (%)
Type of Employer Organization
Institution of Higher Education	n (%)	n (%)
Hospital/Medical Center	n (%)	n (%)

Assessing Unit and Item Non-response

After an overall descriptive summary of the sample respondents, a Unit and Item non-response analysis will be carried out. While sampling weights will be adjusted for unit non-response within sampling strata, if the response rate within sampling strata is low (less than 75%), then the sample respondents may not be representative of the relevant target population. In order to assess whether or not unit response rates are low, response rates will be tabulated for each race and ethnicity group within the three selected samples (Applicant only, Reviewer only, and individuals who are both Applicant and Reviewer).

Even when unit response rates are high, item nonresponse amongst respondents may reduce the degree to which inferences about such an item is trusted. Since there are a variety of analyses that may be carried out using the peer review surveys’ responses, one could calculate item nonresponse for a variety of analytical subgroups. We will tabulate item response rates, separately for the Applicant, Reviewer and Advisory Council questionnaires, overall and within some key analytical subgroups, where applicable (e.g., race and ethnicity).

Analysis of Survey Responses

Survey responses to various questions will be analyzed by comparing survey responses between the key groups described in the first section. Categorical responses will be analyzed by cross-tabulating weighted responses across given groups (such as race or ethnicity). Statistical differences will be assessed by performing sample survey appropriate Chi-square tests of proportions to test for independence of survey responses across the groups. Continuous responses will be analyzed by reporting weighted means across given domains. Statistical differences will be assessed by performing sample survey appropriate t-tests to test for differences in mean response across the domains. Two-way tables will be created for all satisfaction/opinion questions in order to compare the groups of interest. All categorical variables will contain the frequency counts of the responses as well as their respective percentage of non-missing data. All continuous variables will be displayed with means along with the number of non-missing responses, minimum and maximum values.

Tables A.16-2 and A.16-3 are examples of tables to display the results of the analysis.

Table A.16-2. Experience of Applicants – Sample Table

Question	Applicant Questionnaire
	N =
Application assigned numerical impact/priority score	n (%)
Application received NOA -- funded	n (%)

Table A.16-3. Experience of Peer Reviewer – Sample Table

Question	Reviewer Questionnaire
	N =
Capacity as a NIH reviewer
Regular (appointed)	n (%)
Ad hoc (temporary)	n (%)
Both regular and ad hoc	n (%)
Reviewer for Components of NIH
Center for Scientific Review	n (%)
One or more NIH Institutes/Centers (ICs)	n (%)
Both CSR and ICs	n (%)

Plans for Publication

A written report with accompanying charts will be provided to NIH management for internal use. There are no plans to publish the results of these surveys.

Project Time Schedule

The project time schedule is provided in Table A.16-4. OMB clearance is being requested for one year.

Table A.16-4. Project Time Schedule

Activity	Time Schedule
Launch survey website and email invitations	February 14, 2012
Conduct data collection	February 14 to March 31, 2012
Create analysis file and analyze data	April 1 to June 11, 2012
Document findings	June 12 to September 15, 2011

A.17. Approval to Not Display Expiration Date

We are not requesting an exemption to the display of the OMB Expiration date.

A.18 Exceptions to Item 19 of OMB form 83-I

These surveys will comply with the requirements in 5 CFR 1320.9.

8