2 Reviewers Focus Group Guide

OMB Control Number: 0925-0474 Expiration Date 2/28/2018

Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0474). Do not return the completed form to this address.

Reviewers' Experiences with the NIH Peer Review Process

Focus Group Opening Script

Hello and welcome. Thank you for meeting with us today. My name is Dr. Ami Lynch and I am from Social Solutions International, Inc., a research firm that works on a wide range of issues to improve the health and well-being of people and communities. Thank you for being here to share your thoughts and ideas. My role today is simply to guide our discussion and to make sure that everyone has an opportunity to share their experiences and opinions. 

Opening Comments: 

Social Solutions has been contracted by the NIH Center for Scientific Review to help gain a better understanding of the experiences of Reviewers with the NIH Peer Review Process. The purpose of this focus group is to better understand Reviewer’s experience and satisfaction with the peer review process. CSR’s mission is to ensure that every applicant receives a fair, thorough, competent, and timely review. They want to know your experience and honest feedback about the NIH peer review process. Ultimately, results of this focus group will be combined with the views of other key stakeholders to enable NIH decision-makers to identify what all stakeholders like best about the process and what could be done better. Emerging critical issues and sentiments will be further explored via future quantitative research. CSR hopes this research may lead to the identification of possible new directions for the peer review process. Your input today will be critical to this process.

Focus Group Overview:

This focus group discussion should last approximately 90 minutes. We would like to tape record the discussion so we can keep track of what you tell us. We will not refer to you by name during the audio recording, and ask that other participants refrain from doing so as well. This will ensure that no comments you make can be connected to your name to protect your privacy to the extent permitted by law. When you wish to join in on the discussion and respond to a question, please raise your hand. What you say is very important and we do not want to miss anything. We are only recording the discussion to make sure we don’t miss any of the information you share.

Your names will be tracked to account for the distribution of incentives, but this information will not be shared with anyone at NIH or CSR. The information you share with us today will not be linked to your name in anyway, will not affect your ability to apply for or secure funding from NIH in the future, and will never be associated with your NIH applications, should you apply.

We would like to ask you not to discuss any of the information you hear from this discussion group with anyone outside of this room. We want you to feel comfortable sharing your personal opinions and experiences with the group so that we can learn from you.

Your participation in this group is entirely voluntary, and you are free to leave at any point if you find it necessary. I will be available after the session this evening to answer any questions you may have about the discussion or the project in general. If you need to get up and stretch, get food or drink, and/or use the rest room during the focus group, please feel free to do so. We just ask that you do so quietly because of the recording.

Following the Focus Group, Social Solutions may invite select participants to appear in a video clip to be presented to CSR with the study findings. Participants will be asked to repeat their particularly insightful comments they made during the focus group. If approached, you are under no obligation to appear in a video clip. If you do chose to do so, Social Solutions cannot be responsible for any of the statements you choose to make on video.

If you DO NOT wish to be contacted with this further opportunity, please let us know at the conclusion of the discussion.

Discussion Group Rules:

First, since each and every person's comments are important, we ask that only one person speak at a time. This will help us to hear each other. There is no such thing as a right or wrong answer in this group. We have no expectations about what is going to be said. Tell us what you honestly think, and feel free to share whatever is on your mind. If you disagree with something that you hear, let us know. If you agree, don't just say, "I agree", but try to add your own perspective or opinions. We want to hear as many different opinions, ideas, and stories as possible.   

Consent

As you came in, you were asked to sign an informed consent statement. In order to participate today, we must have your written consent. Is there anyone who has not signed the consent statement?

[Social Solutions researchers will ensure that the number of participants matches the number of signed consent forms prior to proceeding. If the numbers do not match, Social Solutions will confirm whether they have completed theirs privately with each participant, to avoid identifying participant to each other.]

Does anyone have any questions at this time?

I am going to turn on the tape recorder now so we can get started…

Reviewers' Experiences with the NIH Peer Review Process

Focus Group Probes/Questions

1. Let’s first think about the success of the NIH/CSR review process, in terms of the overall outcomes and what is or is not being achieved. What are the markers or end results that would indicate that the NIH/CSR peer review process is successful? Would you characterize the current review process as successful? Why or why not?

2. If we think now about quality in terms of the effectiveness of each step of the peer review process, what key factors are in place that help ensure success in the end? What key factors are missing or need to be strengthened to contribute to a more successful review process?

• Probes: Efficient use of time in the review? Constructive and effective discussions? Personal dynamics within the Study Section? SRO role/assistance? Appropriate Study Section assignment? Appropriate Study Section roster? What are some of the typical challenges faced that affect the quality of a reviewer’s performance?

3. CSR’s mission is to ensure “NIH grant applications receive fair, independent, expert, and timely reviews—free from inappropriate influences—so NIH can fund the most promising research.” Is this mission reflected in the day-to-day reality of NIH peer review? How can NIH strengthen the quality of the peer review process to align more with CSR’s mission?

• Probes: How well does the peer review process embody NIH’s mission regarding fairness? Independence? Expert? Timeliness?

4. How many of you have served as a reviewer on a Study Section using an alternative review format? If so, which ones and how many times have you used them? What are your impressions of different review formats (in person, virtual/remote)? Are there emerging technologies that could be useful to NIH peer review? Is there a direction in which you would prefer to see the peer review process go in terms of review format?

• Probes: What are the advantages/strengths? What are the disadvantages/challenges? CSR Internet Assisted Review (IAM)? Video Assisted Meetings (VAM)? Teleconference reviews? Hybrid meetings? Editorial Board Review? Which is most challenging? Which is most effective? Do alternate review formats affect quality? If so, how? How would CSR/NIH benefit from using emerging technologies?

5. Do you feel you are able to identify the best applications? Why or why not?

• Probes: Does the NIH peer review process enable you to identify applications with greatest scientific merit/impact? Why or why not?

6. Are there barriers to providing feedback to applicants? If so, what are the barriers?

• How do you approach providing feedback on applications? How much time are you able to spend on each one? What is most important to convey to applicants?

7. When three individuals review an application, their opinions of the application sometimes vary. What do you think/do when there is a divergence of opinion regarding the assigned reviewers’ scores? Have you ever been the outlying reviewer? If so, what do you think/do?

• How does voting out of range affect scoring? How do reviewers calibrate their scores before, during, and after a study section meeting? How effective is this process? What, if anything, can NIH do to help reviewers calibrate their scores to one another within a study section?

8. Why do you choose to serve as an NIH reviewer? What could NIH do to make investigators more willing or likely to serve on Study Sections?

• Probes: What role does your sense of service as a member of a community play? What do think about holding review meetings in other parts of the country? The current stipend? Use of alternate review formats?

9. What do you think of the 9-point scale for evaluating applications?

• Probes: How do you use the scale? How can reviewers better utilize the entire range of scores to mitigate score compression? Is the final score meaningful to you? Do the criterion scores provide helpful feedback? Could there be a better way to evaluate applications than via the 9-point scale? If so, what would you suggest? Why?

10. Is there anything else that would be helpful for NIH to know about (your experiences with/impressions of) the current peer review process? Anything about the topic that we didn’t ask? Is there anything else you want to share?

Thank You.