Mini Supporting Statement B

Mini Supporting Statement B

B.1 Respondent Universe and Sampling Methods

The proposed survey will be administered by the NIH Center for Scientific Review as part of a program evaluation effort. The survey sample will include approximately 1,200 individuals who participated in the Early Career Reviewer (ECR) Program in 2014, 2015 and 2016 by serving as a grant application reviewer. In 2014-2015, 820 individuals participated on review panels as ECRs. We anticipate an additional 380 ECRs will serve as reviewers in 2016. These ECRs will be eligible to complete the survey. We anticipate a completion rate of 60% for an estimated final sample of 720.

A previous version of this survey was administered in 2012 to 236 ECRs. Responses were received from 142 (60%) individuals. It is based on that experience that we estimate a 60% response rate for this proposed survey.

B.2 Procedures for the Collection of Information

All eligible ECRs who served on study sections from 2014 through 2016 will be invited to participate in the survey. ECRs who served on study sections in 2011 through 2013 will not be invited to participate in the survey for the following reasons: (1) those serving in 2011 and early 2012 were previously surveyed, (2) the ECR program procedures changed over the first few years of the program and stabilized in 2013, and (3) recall of specific experiences with the program may be more difficult for those serving prior to 2014. Surveys will be tagged for the year of ECR participation as a reviewer to evaluate differences in responses due to time lag between ECR service and survey completion.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

• The Director of the ECR Program will send a personal note via email thanking ECRs for their service as a reviewer and inviting them to participate in the survey.

• The survey will be sent to the most recent email address on file for each ECR. For any returned emails due to inaccurate address, an effort will be made to locate the most recent email address and resend the invitation.

• A deadline for completion of the survey will be set for two weeks after the sent date of the email. The deadline will be adjusted to allow two weeks for completion for any cases in which a correction in email address is needed.

• A follow-up reminder email will be sent one week after the sent date of the original email.

• A second reminder email may be sent after the two-week deadline if the response rate falls below 50%. For the purpose of this program evaluation, a 50% response rate is deemed adequate.

B.4 Test of Procedures or Methods to be Undertaken

The survey of ECRS that was administered in 2012 provided a sufficient test of the survey procedures. Information derived from that survey informed the deletion or modification of questions from the original survey. The original survey included 10 forced-choice questions and 6 open-ended questions. The free responses to the open-ended questions informed additional forced-choice questions in the updated survey. Given that not all respondent completed the open-ended questions, the number was reduced from 6 open-ended questions to only 1 open-ended question that provides the respondent to provide additional comments.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Monica Basco, Ph.D., Coordinator of the ECR program, developed the revised survey based on review of the results of the original survey. Dr. Basco has had extensive training in measure development and statistical analysis of similar data. Mary Ann Guadagno, Ph.D. leads CSR’s planning, evaluation, analysis, and survey efforts and Charles Dumais provided analysis and reporting of the original survey. Both have reviewed the survey and provided input.