Form 7 Attachment 4 PI Site Visit Interview Protocol

Attachment 4

PI Site Visit Interview Protocol

OMB No.: 0925-XXXX

Expiration Date: XX/XX/20XX

Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-XXXX). Do not return the completed form to this address.

Rights of study participants are protected by The Privacy Act of 1974. Participation is voluntary, and there are no penalties for not participating or withdrawing from the study at any time. Refusal to participate will not affect your benefits in any way. The information collected in this study will be kept private to the extent provided by law. Names and other identifiers will not appear in any report of the study. Information provided will be combined for all study participants and reported as summaries. You are being contacted to participate in this data collection so that we can evaluate the Science Education Partnership Award Program.

SEPA Principal Investigator Interview

Date and time of interview: INSERT HERE

Principal investigator: PREPOPULATE

Institution: PREPOPULATE

Project title: PREPOPULATE

Funded year: PREPOPULATE

End date: PREPOPULATE

Project abstract: PREPOPULATE

Introduction

• Thank you for taking the time to talk with me today about your SEPA project.

• The purpose of this interview is to get your views on various aspects of your program’s partnerships (selecting partners, the role of partners in developing the project approach, how implementation went, support provided by NIH) and your views about the evaluation (NIH guidance and/or evaluation TA received, factors that influenced the evaluation design, and the usefulness of the evaluation).

• The interview will take about an hour.

• Your participation is voluntary.

• We would like to audiotape this interview. It helps us take more accurate notes. Is that okay? At any point during the interview, we can stop or you can ask us to turn off the audio recorder. [If permission to tape is granted.] Recorded interviews will be stored in a secure location and will be destroyed after the end of the study.

• The data will be reported in aggregate and individuals associated with comments will not be disclosed.

• We are also interested in documents or other sources of information regarding any of the topics we discuss.

A. Project Background

I’m interested in knowing more about the background of your project. Is this the title of your project? [TITLE OF SEPA PROJECT].

1. What problem or need was the project designed to address?

2. What approach(es) did you use to address this problem/need?

3. How was the approach developed?

[Probe: Did you conduct a needs assessment or feasibility study to assess the need for/practicality of your approach?]

4. Was the project approach influenced by other SEPA projects?

[Probe: If yes, which projects? How did you find out about these projects? In what ways did these influence the approach developed for this project?]

5. Was the project approach influenced by other NIH projects?

[Probe: If yes, were any of these CTSA, RCMI or IDeA (INBRE or COBRE) projects? If

yes, which ones? How did you find out about these projects? In what ways did these influence the approach developed for this project?]

6. Was the project approach influenced by other federally funded STEM Ed projects?

[Probe: If yes, which projects? How did you find out about these projects? In what ways did these influence the approach developed for this project?]

7. Was the project approach influenced by other (non-federally funded) projects?

[Probe: If yes, which projects? How did you find out about these projects? In what ways did these influence the approach developed for this project?

8. To what extent was the project successful addressing the problem or need?

[Probe: What makes you think so? Please provide specific examples/information.]

B. Project Partnerships

Next, I want to ask about your experience working with project partners.

For purposes of the evaluation, a partner organization is an entity, other than the lead organization, which has devoted time and made meaningful contributions to the SEPA project.  For example, partner organizations may contribute in the areas of developing curricula, providing supplemental education, developing professional development opportunities and workshops, developing exhibits and films, developing outreach, or other areas of the project.  These contributions may be in one or several phases of the project, such as design and development; implementation; analysis; reporting; and/or dissemination of findings.  However, Evaluators and Advisors to the project are not considered organizational partners.

Selecting partners

9. How did you select the project partners?

[Probe: Had you previously worked with any of these partners? If yes, please briefly describe your previous work each one.] If no, how did you learn about each of these partners?

9. Do you know whether these organizations previously worked together on other projects?

[Probe: Please describe partners’ previous work.]

9. Did you add partners after your grant was awarded?

[Probe: If yes, Why? At what point in the project did you bring these partners on board? How did you select them? Had you previously worked together? If yes, please briefly describe your previous work each new partner.]

Role of partners

9. What role, if any, did partners have in the project?

1. Conceptualizing and developing the approach

[Probe: What specifically did each partner contribute to development of the approach? Did they lead this effort? If no, ask who did]

2. Implementing the approach

[Probe: What specifically did each partners contribute to development of the approach? Did they lead this effort? If no, ask who did]

3. Analysis and/or reporting

[Probe: What specifically did each partners contribute to development of the approach? Did they lead this effort? If no, ask who did]

4. Planning for the future

[Probe: What specifically did each partners contribute to development of the approach? Did they lead this effort? If no, ask who did]

5. Organizing input from the partner organizations

[Probe: How was this handled? Did a partner lead this effort? If no, ask who did. ]

6. Role(s) played in other areas of the project (please describe)

[Probe: What specifically did partners do related to these areas? Did they lead this effort? If no, ask who did.]

9. To what extent were partners’ ideas and/or suggestions taken into account when developing the

project approach?

[Probe: Please provide a brief example to illustrate how partner input influenced development

of the approach.]

9. What were you able to accomplish because of your project partners’ expertise and experience that you could not have accomplished without them?

9. What did you learn from working with each project partner?

9. For grants that have ended, ask: Have you or would you consider continued work with this project partner?

For grants still in progress, ask: Would you consider continued work with this partner?

[Probe: If yes, on what types of activities? If no, why not?]

Project structure and communication

9. To what extent were steps taken to provide a shared vision for/commitment to the project?

[Probe: Please briefly describe the steps that were taken and when.]

9. To what extent were there established lines of communication among organizations?

[Probe: Please describe the communication process.]

9. How often did you meet/communicate with each partner?

[Probe: If more than one partner ask, with all partners combined? How were the group’s communications facilitated?

• Synchronous: conference calls, on-line, face-to-face

• Asynchronous: email

9. Did any of partners attend the annual SEPA conferences?

[Probe: If yes, did they make presentations about or on behalf of the project (e.g., for project outreach or dissemination)? Please describe presentations made by partners, include title, venue and year.]

Implementation experiences

9. What factors helped and/or hindered your efforts to work with project partners?

9. What challenges, if any, did you encounter coordinating the team’s work?

[Probe: Why did these challenges occur and how did you deal with them?]

9. Did any partners leave before the project before it ended?

[Probe: If yes, ask why? Were partners replaced or was their work completed?]

Guidance provided by SEPA

9. To what extent did the SEPA program support or promote your project?

[Probe: How?]

9. To what extend did the SEPA program support or promote collaborations among SEPA projects?

[Probe: What types of supports were provided?]

9. To what extent did other NIH programs support or promote your project?

[Probe: Did you have interactions with CTSA, RCMI or IDeA (INBRE or COBRE) programs? If yes, please describe types of supports provided.]

9. To what extend did other (non-NIH funded) federal programs support or promote collaborations with other SEPA projects?

[Probe: What types of supports were provided?]

9. Would you have still worked with partners if it wasn’t a SEPA requirement?

[Probe: Why?]

Lessons learned

9. Were there lessons you learned about working with project partners that would be of use to other SEPA projects?

[Probe: Please briefly describe what you learned.]

9. What advice would you provide to other PIs about selecting partners?

C. Evaluation

The next set of questions relates to your project’s evaluation.

PI’s involvement in evaluation

9. To what extent were you involved in the evaluation of your SEPA project?

[Probe: Please briefly describe your role.]

9. Why did you choose this evaluator for your SEPA project?

[Probe: Did you have an open solicitation, rely on previous connections, referrals, etc.?]

9. At what point in the project did you involve the evaluator?

[Probe: at the beginning, mid-way through, at the end, throughout the project]

9. To what extent did the evaluator lead development of the evaluation design?

[Probe: Did they contribute to the proposal? If no, did they contribute in other ways? Please describe.]

9. How would you describe your working relationship with the evaluator?

SEPA program guidance and TA to evaluation

9. Did the program solicitation provide clear and adequate guidance as to what was required in the evaluation design?

[Probe: If yes, ask if they remember specific requirements.

9. Did the program provide evaluation TA?

[Probe: If so, did it meet your project’s needs? If not, what additional support would have been helpful.]

9. At any point, did your project need evaluation technical assistance?

[Probe: If yes, what TA would have been helpful?]

9. Did your project receive evaluation technical assistance?

[Probe: If yes, where/from whom did you get TA from? Was it sufficient? Why (or why not)? What else would have been helpful?]

Evaluation Design and Implementation

9. To what extent did the evaluation adequately address/measure the project’s goals?

[Probe: Please briefly describe what makes you think so.]

9. If the evaluation design had shortcomings, what were they and why did they arise?

9. Was the evaluation implemented as intended?

[Probe: If not, what were the major changes? Why did they occur?]

9. Were the evaluation findings useful to the project?

[Probe: If yes, in what ways? If no, why not? What would you have wanted to learn from the evaluation? How would you have used this information?]

9. What aspects of the evaluation were most useful or least useful?

[Probe: Please describe why you think so. What would you want future evaluations to do more of? Less of?]

Lessons learned

9. Was having an evaluator/evaluation beneficial for the project?

[Probe: In what ways?]

9. What lessons did you learn from conducting the evaluation?

9. What advice would you provide to other PIs about selecting and working with evaluators?

9. What challenges were involved in conducting the evaluation?

9. If you have a chance to redo this evaluation, what would you do differently?

[Probe: Why?]

THANK YOU FOR TAKING TIME TO TALK WITH ME TODAY. OUR CONVERSATION WAS VERY HELPFUL FOR THE EVALUATION. I ENJOYED TALKING WITH YOU.