WWC Interviews with K-12 Education Leaders

WWC Decisionmaking Study

User Experience Interview Protocol

Date:
Interviewer Name:
Participant ID:

Introduction (5 minutes)

Hello. Thank you for agreeing to talk with me today. This is [your name] from the American Institutes for Research or AIR. AIR is collecting data for the Institute of Education Science (IES) within the U.S. Department of Education about how K-12 education leaders use research evidence in their decision-making.

Before we start, let’s make sure that we are all set with technology.

Technology check: Participant logged into GoToMeeting, camera on.

I see that you are logged into GoToMeeting and that you have your camera turned on. Thank you for doing that. Later in our conversation I will make you the presenter so that you can share your computer screen with me. I will walk you through how to do that if you are not familiar with it. To confirm, do you have a browser and internet access on your computer?

Technology check: Participant logged into GoToMeeting, camera off.

I see that you are logged into GoToMeeting, but your camera is turned off. Do you mind turning on your camera? Being able to see you will help me guide you through the questions and tasks that we need to cover. [If participant agrees to turn the camera on.] To turn on the camera, click on the camera icon in the corner of your screen (since the participant is logged in, if needed you can share your screen to show them how to click on the camera icon). Thank you for doing that. Later in our meeting I will make you the presenter so that you can share your computer screen with me. I will walk you through how to do that if you are not familiar with it. To confirm, do you have a browser and internet access on your computer?

Technology check: Participant not logged into GoToMeeting.

I see that you are not logged into GoToMeeting. We will use GoToMeeting later in our conversation. Please click on the link in the invitation for this interview to launch GoToMeeting (direct the participant to click on download if it comes up). Do you mind also turning on your camera?

Being able to see you will help me guide you through the questions and tasks that we need to cover. [If participant agrees to turn the camera on.] To turn on the camera, click on the camera icon in the corner of your screen (since the participant is logged in, if needed you can share your screen to show them how to click on the camera icon). Thank you for doing that. Later in our meeting I will make you the presenter so that you can share your computer screen with me. I will walk you through how to do that if you are not familiar with it. To confirm, do you have a browser and internet access on your computer?

Background (2 minutes)

Before we begin the interview, I would like to share some background about the project and provide an overview of what will happen during our conversation today. As you know from our emails, I will ask you questions about how you find solutions to problems about teaching and learning you encounter at school. I will also ask you to browse the WWC website and complete several tasks.

Today’s conversation will take approximately one hour. We would like to record both the audio and video of your computer screen, but not your camera image. Only AIR staff will have access to the recording. The recordings will be transcribed and will be deidentified so that everything you say and do will be completely confidential. When we report the results of the interviews, we will combine findings across many participants to prevent individuals from being identified through their responses. We may include quotes, but they will not be attributed to individuals.

Do you have any questions? (Answer questions if any.)

Think Aloud Demonstration (3 minutes)

During our conversation I will ask you to think aloud as you navigate the WWC website. I want to understand your thought process, so please say out loud whatever you are looking at, thinking, doing, and feeling as you explore the website. What I am looking for is a running dialogue of what’s happening inside your head.

Here is an example of what I mean. Suppose someone asked me: Can you show me how to find information about the Every Student Succeeds Act (ESSA) on the U.S. Department of Education website? Here is how I would think aloud.

First I’m going to Google the U.S. Department of Education because the question asked about their website. The first option looks like it’s the Department’s website. I am going to click on it because the domain name is ed.gov which I know is an official government website. I see information about student loans, grants, the picture of Secretary of Education… I also see “Laws” on top of the page, and I’m going to click on it. The information that I see is all about ESSA: the legislation, regulations, guidance… Although I am not sure where I would go from here – maybe I would click on the ESSA hyperlink in the body of the paragraph.

Does this make sense? Do you have questions? [Answer questions if any.]

Let’s get started with the interview. I am going to begin the recording now.

** BEGIN RECORDING NOW **

Part I: Background (1 minute)

First, I am going to ask you a few questions about your background.

1. Can you please confirm your title and role at [organization]?

2. How long have you been in this role? ­­­­­­­­­­­­­­­­­­­­­­­­

Part II: Use of Research Evidence (6 minutes)

Now I am going to ask how you solve questions related to improving teaching and learning. I would like to focus on how you use research evidence to solve these questions. I want to ensure that we have a shared definition of what research evidence means. Research evidence is evidence produced through a study or an evaluation of a program, or practice, or policy. You typically find research evidence in journal articles or reports, although findings do not have to be publicly available to be research evidence. Things like student test scores, grades, teacher evaluations, and data dashboards are not research evidence, although they may be used to inform decision making. Does this distinction make sense? [Answer questions if any.]

3. How often do you use research evidence when making decisions about teaching and learning? For example, decisions about improving student learning.

• Frequency of use indicated  Go to Q4

• Never  Why not? [Probe for barriers to using research evidence in role/organization]  Go to Q7

4. What types of research evidence do you use?

5. Do you look for research evidence yourself or does your staff look for it?

6. Where do you [or staff] look for research evidence?

7. Have you ever accessed the WWC website?

8. Have you used the WWC resources (even if you haven’t accessed the website)?

• YES Can you give me an example?

• NO  Why not? [Probe for reasons]

Part III: Structured Scenarios (35 minutes)

Now I am going to ask you to access the WWC website and think aloud. I am going to make you the presenter so that you can share your screen with me. [Right click the participant’s name to make them the presenter.] You should be able to share your screen now. [Skip to the next paragraph if the participant knows how to share screen.] You need to click on the “share” icon and then select the screen on which you can see your browser window. That will allow me to see your screen [have a screenshot of the “share” icon that you could show on your screen if needed]. Great, I can see your screen. [May need to repeat if can’t see the participant’s screen.]

[If the webcam is on, direct the participant to minimize the camera or move it to the screen that is not being shared to make sure that it’s not recorded.] Please go to the WWC website. I posted the link in the chat box [post https://ies.ed.gov/ncee/wwc/].

Now we are going to do a few specific tasks.

Task 1: Search for Evidence Using Find What Works. For the first task, think about a typical scenario in your professional role in which you may need research evidence. For example, maybe you are seeking a program to help students or teachers, maybe it’s in math, maybe it’s literacy, maybe it’s related to professional development, things like that… Does anything come to mind? [If yes] What is your scenario? [If not] How about you search for a math or literacy tutoring program for middle school?

I would like you to use the Find What Works tool on the WWC landing page to find research evidence for this specific scenario. The Find What Works tool is the blackboard looking rectangle that you see at the top of the page. I’ll let you know when the time is up by raising my hand. You may notice embedded pdfs and/or videos; because we have limited time for the interview, please spend no more than one minute looking at them. I’ll ask some follow up questions after we are done with this task.

If participant goes to the search bar, do not redirect them at first, but once they have reached the next webpage, redirect them back to the WWC site. You may explain that the search bar navigates away from the WWC website, which is the focus of interview

If participant does not explore filters, direct them to click/explore the filters.

Allow up to 7 minutes of searching and looking at evidence and 3 minutes for follow up questions. Remember to remind the participant to think aloud. If the participant clicks on PDFs or videos, allow viewing 1-2 pages or the start of video and then ask them to navigate away.

9. Have you used Find What Works prior to today?

10. Why did you choose the resource or resources to find what you were looking for?

11. Can you summarize what you think worked well (e.g. navigation, search process, content, layout, flow)?

12. Was there anything you couldn’t find?

13. If you were using Find What Works on your own, for instance at work, would you invest additional time to look for what you need?

14. What would you change to make Find What Works more helpful (e.g. navigation, search process, content, layout, flow)?

1. Is there anything specific about the filters you would change?

Task 2: Quick Link – Practice Guides. Please go to the main page and scroll down to the Quick Links box. Click on the Practice Guides link and spend a few minutes browsing the information there. As before, I’ll let you know when the time is up by raising my hand. After you’re done, I’ll ask some follow up questions. Allow 5 minutes for browsing and 3 minutes for follow up questions. Remember to remind the participant to think aloud. If the participant clicks on PDFs or videos, allow viewing 1-2 pages the start of video and then ask them to navigate away.

15. Did you know about the WWC Practice Guides prior to today?

16. How would you describe the information available through the Practice Guide/s?

17. Is there a specific reason why you selected the Practice Guide that you did? If yes, please explain.

18. How likely are you to use the information available at the Practice Guides link in your professional role? Can you tell me why?

19. Would you change anything about the Practice Guides link (e.g. navigation, search process, content, layout, flow)? Please explain.

Task 3: Quick Link – Resources. Now go back to the main page and click on the Resources link in the Quick Links box and spend a few minutes browsing the information there. I’ll let you know when the time is up by raising my hand.

Allow 5 minutes for browsing and 3 minutes for follow up questions. Remember to remind the participant to think aloud. If the participant clicks on PDFs or videos, allow viewing 1-2 pages or the start of video and then ask them to navigate away.

20. How would you describe the information available through the Resources link?

21. Is there a specific reason why you selected the Resources that you did? If yes, please explain.

22. How likely are you to use the information available at the Resources link in your professional role? Can you tell me why?

23. Would you change anything about the Resources link (e.g. navigation, search process, content, layout, flow)? Please explain.

Part IV: Free Navigation OPTIONAL

We have some time left before my final questions. Please browse the WWC website for XX minutes and explore any links or topics that may be of interest to you. As you navigate, please think aloud and tell me your thought process. I will keep track of the time and will raise my hand when the time is up. [Throughout, remember to remind the participant to think aloud. Allow the number of minutes for free navigation that would still leave 5 minutes for Closing.]

Part V: Closing (5 minutes)

We are finished browsing. You are welcome to stop sharing your screen. I have a few concluding questions.

24. Given what you have seen on the website today, what is the most relevant for what you or your staff need in your professional roles? [Prompt for either content and/or navigation features if not already discussed]

25. How likely are you or your staff to use this website on your own? Why? [If the participant provides reasons unrelated to the website, for example lack of time, probe if there are reasons related to the website.]

26. If you were suggesting this website to a colleague, is there anything specific that you would highlight? [Prompt for content, tools, resources]

1. Is there already a similar website that you suggest to colleagues? If yes, which one?

27. Is there anything we did not discuss today you would like to share?

Thank you again for your time.

1000 Thomas Jefferson Street NW, Washington, DC 20007-3835 | 202.403.5000 | www.air.org