REU Site Visits - Student focus group

REU PRINCIPAL INVESTIGATOR PROTOCOL DRAFT

APPENDIX C

REU Site Visit Protocol

REU Principal Investigator (PI)/Coordinator Interview

60 minutes

Interview Introduction

Hello, my name is [name] and I am a [position] at Mathematica Policy Research.

Thank you again for taking the time to speak with me today. As you know, Mathematica developed and tested an electronic data system for the REU program. Thanks again for participating in the pilot!

We would like to speak with you to gain a deeper understanding of the REU program. We are particularly interested in understanding how REU Sites are organized and select and work with students. We would also like to get your feedback on the data system.

I’d like to emphasize that everything you say will remain confidential. We will not attribute any information you provided to you or your discipline in our work, such as our report to NSF or presentations.

Do you mind if I tape this conversation? It will help me transcribe it. Rest assured that no person outside Mathematica’s research team will have access to interview recordings, notes, or transcripts.

[If they agree, turn on the recording and, once on, thank them for letting you record.]

Do you have any questions before we begin?

BACKGROUND

1. Please tell me about yourself. How long have you been [position] at [university]?

Probe: What do you typically teach? How long have you been a professor? Site PI?

2. You are the PI of this Site. I am wondering, what motivated you to apply to lead an REU Site?

Probe: What is the goal of your REU Site?

3. What is your role at the Site?

Probe: Do you work with students directly?

4. What does it take to run an REU Site? (Thinking in terms of infrastructure, staffing, collaborations, and funding.)

Interviewer: We also want to understand if there is such a thing as a “Site” created to participate in the REU program, or if the funds contribute to an existing operation. If so, we want to understand what this means in terms of what needs to change, if anything, to participate in the program.

5. Do you collaborate or partner with other REU Sites? If so, with whom? In what ways?

Probe: Ask about why it was important to collaborate with other Sites; some Sites may have interdisciplinary collaborators, find out why?

Find out if they have other Site collaborators outside of the REU program, such as faculty at {university name}? What do they bring to your Site?

RECRUITMENT AND SELECTION

Let’s talk for a moment about how you recruit and then select students to participate in a research experience at your site.

6. How do you recruit students?

Probe: Do you advertise the opportunity to join your Site, perhaps at conferences? Do you ask alumni to spread the word? Use social media? Connect with fellow faculty?

7. Do you look for a particular type of student—say, students with specific past experiences, academic background, or technical skills? If so, how does that affect your recruitment efforts?

Probe: For example, Sites seeking to recruit African American students may advertise at HBCUs.

8. [For registration Sites:] How would a student apply to your Site? Can you walk me through the steps and requirements?

[For common app Sites:] I realize that this year you participated in the pilot, so you used the common app created for the REU program, but I am curious to hear about how students applied to your Site before, in prior years. Can you walk me through the steps and requirements?

9. How do you decide which students to admit? Can you walk me through the process and who is involved?

Probe: What role do collaborators play? (Probe: faculty, visiting scholars, etc.)

OFFERINGS FOR STUDENTS

Let’s talk about the students selected to come to your Site.

10. For these students, what does it mean to join your REU Site for a summer research experience? What would they do or experience?

11. What supports would students receive while doing research at your Site?

Probe: For example, do students receive opportunities for professional development, such as attending invited lectures/seminars/conferences or participating in workshops? Let’s consider adding a list here for the interviewer to checkmark and get examples as appropriate.

12. Would you or someone else mentor students while doing research at your Site? If so, tell me about it.

Interviewer: Site use different approaches to mentor students, which may be provided one-on-one or as part of a group by PIs, Co-PIs, Site faculty, Site PhD students, Site post-docs, or peers (learn about details: who, what mentoring looks like, where it happens, etc.) If mentoring is mentioned under Q4, then administer the probes.

13. What happens once the summer ends? Do students remain engaged with your Site or research in any way during the academic year? If so, how?

TRACKING OUTCOMES

Earlier you mentioned that you hope to (reference here a specific goal stated earlier, such as to increase the number of minority scientists or to help students figure out if this is the right career for them). With that in mind, I wonder:

14. Have you succeeded?

If not explained as part of the answer: How did you gather this information? [Adapt depending on answer to Q13:] Do you track or stay in contact with students after they complete their research experience at your site? How?

If Site is relatively new, revise lead question to read: How will you know if you’ve achieved your goal?

15. As you know, NSF is seeking to increase its ability to collect and analyze data about the REU program by creating a data system. If you could access that data system, what information would you like to obtain?

Probe: For example, what type of data would be helpful for your renewal application? What types of data would you like to have and currently do not?

REU WEBSITE PILOT

I’d like to get your feedback on the REU website you helped us pilot test.

16. What did you think of the website?

Probe: What did you like? What didn’t you like?

17. [For common application users:] Compared to the way in which you managed applications before, was the process easier or more difficult through the pilot? In what ways?

[For registration users:] Compared to the way in which you gathered application information before, was the process easier or more difficult through the pilot? In what ways?

[Probe for both groups]: Compared to student information you submitted to NSF in the past, was it easier or more difficult to provide information through the website instead?

18. [For common application users:] NSF is considering two alternatives: to adopt the common application or to adopt only the registration (which is the first part of the common app that collects basic background information from students). Based on your experience, what would be your recommendation to NSF? Why?

[For registration users:] NSF is considering two alternatives: to adopt the registration or the common application (which includes the registration as well as other information that is typically part of an application, such as student transcripts). Based on your experience, what would be your recommendation to NSF? Why?

Interviewer: Respondents will be familiar with the registration and the common application and should have received background information ahead of the site visit.

END

19. Is there a question I should have asked you and did not?

20. Do you have any questions for me?

REU Site Visit Protocol

REU Mentor Individual or Small Group Interview

30 (individual) - 60 minutes (small group)

Interview Introduction

Hello, my name is [name] and I am a [position] at Mathematica Policy Research.

Thank you again for taking the time to speak with me today. As you know, Mathematica developed and tested an electronic data system for the NSF Research Experiences for Undergraduates or REU program.

We would like to speak with you about your work at [name of REU Site]. We are particularly interested in understanding your role as mentor(s) to the students who are conducting research here this summer as part of the NSF REU program.

I’d like to emphasize that everything you say will remain confidential. We will not attribute any information you provided to you or your discipline in our work, such as our report to NSF or presentations.

Do you mind if I tape this conversation? It will help me transcribe it. Rest assured that no person outside Mathematica’s research team will have access to interview recordings, notes, or transcripts.

[If they agree, turn on the recording and, once on, thank them for letting you record.]

Do you have any questions before we begin?

[If this is a small group interview, then treat it like a focus group and state:]

Before we begin, I’d like to ask that you feel free to respond or react to one another’s comments – regardless of whether you agree or disagree—and that only one person talk at a time.

1. Please tell me about yourself. What is your position at [university name]? In what ways are you involved with [REU Site name]?

Probe: How long have you been [position] at [REU Site name]? How long have you been a mentor to REU students?

Interviewer: Sites use different approaches to mentor students, which may be provided one-on-one or as part of a group by PIs, Co-PIs, Site faculty, Site PhD students, Site post-docs, etc. (learn about details: who, what mentoring looks like, where it happens, etc.)

2. What is your role as student mentor?

Probe: How many students do you mentor? How are students matched with mentors? Do you collaborate with others in supporting students?

3. [If applicable] How did you prepare to become a mentor? Did someone explain to you your role and responsibilities?

4. Can you describe how you work with students in a typical week?

5. What other supports do students receive while doing research at [Site name]?

Probe: For example, do students receive opportunities for professional development, such as attending invited lectures/seminars/conferences or participating in workshops? Let’s consider adding a list here for the interviewer to checkmark and get examples as appropriate. Who offers these supports?

6. What happens once the summer ends? Do students remain engaged with you, others at [Site name] or research in any way during the academic year? If so, how?

7. In your view, what needs do students have when joining [Site name] to participate in research?

Probe: Are these needs being met? If yes, how. If not, why not?

Do you think that students need additional support that they are not current receiving? If so, please explain.

8. In what ways do you feel students contribute to the research being conducted at this [Site name]?

9. Is there a question I should have asked you and did not?

10. Do you have any questions for me?

REU Site Visit Protocol

Student Focus Group

60 minutes

Introduction

Welcome and thank you for taking time to join us today.

My name is [name] and I work at Mathematica Policy Research. The National Science Foundation (NSF) asked us to develop and test an online [common application / registration] for students interested in participating in the REU program. You were invited to participate in this focus group because you are participating in a research experience at an REU Site, this Site, and used that online system [to apply / to complete part of your application].

I would like to ask you a few questions about how you learned about the REU program, what you think of the online [common application / registration], and your experiences at this Site so far. Keep in mind that there are no right or wrong answers. I just want to hear about your experiences and opinions.

Also, please rest assured that we, the study team, will keep your answers confidential—that is, we will not identify you to any staff at the REU Site or in any papers, reports, or presentations. And nothing you say will impact your participation at this Site, the REU program, or other NSF programs in the future.

Do you have any questions before we begin?

I would like to ask your permission to record this meeting. This will help me transcribe your responses accurately later, and concentrate on the conversation now. The recording will also be confidential and only used to transcribe this meeting.

Do I have your permission to record? [Turn on the recording and, once on, thank them for letting you record.]

Before we begin, I’d like to ask that:

• you please speak up so everyone can hear your comments,

• feel free to respond or react to one another’s comments – regardless of whether you agree or disagree

• only one person talk at a time

Now let’s go around the room and introduce yourselves. Please state your first name, major, year in college you just completed (say, freshman or sophomore), and college or university in which you are getting your bachelor’s degree. [Draw diagram with student names]

Let’s begin.

1. HOW DID YOU FIND OUT ABOUT THE REU PROGRAM?

Probe: Revise question to read “about this REU Site” if draw confused looks

2. WHAT DID YOU THINK ABOUT THE APPLICATION PROCESS?

Probes: (For registration pilot students) Did you use the online registration? What did you think of it? Did you provide your ID with your registration? Were you confused by having to do this, that is, but having to complete the online registration and provide your ID number in your application to the Sites? Do you have any suggestions for improving the registration?

(For common application pilot students) Did you apply through the common application? What did you think of it? Was it difficult to use in completing your application? Did it provide all of the information you needed to select Sites to which you would submit applications? Do you have suggestions for improving it?

3. IN WHAT WAYS ARE YOU INVOLVED IN RESEARCH HERE AT [Site name]?

Probe: Do you have a mentor to guide you? Who? How often do you meet? Are these meetings / is your mentor helpful? How?

4. WHAT DO YOU DO IN A TYPICAL WEEK?

Probes: In addition to working at the lab or doing research with your professor, what else do you do? Do you attend seminars, get career advice, participate in workshops on how to apply to grad schools?

Which of these activities would you say have been the most useful? Why? Which would you say are less helpful? Why?

5. IN WHAT WAYS IS THIS EXPERIENCE HELPING YOU?

Probes: Is this experience helping you decide whether or not to apply for graduate school? Or even pursue a career in [science/engineering/etc.]?

Have your plans changed since joining this Site? How? Why?

6. WHAT ARE YOUR PLANS FOR THE FUTURE? (if time permits)

Probes: What do you plan to do after you graduate from college?

Last updated: 5/25/2018 Page 7