2017-2018 National Teacher and Principal Survey (NTPS) Portal Usability Testing

National Center for Education Statistics

Attachments 2-4

Attachment 2 – Recruitment Screener

Attachment 3 – Protocol Materials

Attachment 4 – Portal Instrument Screen Shots

2017-2018 National Teacher and Principal Survey (NTPS)

Portal Usability Testing

OMB# 1850-0803 v.189

February 2017

2017-18 NTPS Portal Usability Testing

Attachment 2

Recruitment Screener

Hello. My name is [XXXXXX] and I work for the U.S. Census Bureau. The U.S. Department of Education is conducting a study of schools to evaluate the effects of school workplace conditions, salaries, and training opportunities on the educational workforce, among other things.

We are looking for volunteers to help us improve the web survey used in this study. We want to know how easy or hard it is to complete the web survey, and we need help from people like you to do that. The information gained from the study will be used for potential revisions to the web survey. Your participation is voluntary, and you will receive no sales pitches or follow-up calls based on your participation. I’d like to ask you a few questions to see if you qualify for the study.

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

4. Do you currently work at a school?

___ Yes

___ No – Thank you for your interest, but we need participants who work in a school. END CALL

5. What is your current job description?

_________________________________________________

6. Are you a school administrator or staff person who can provide information about the teachers at your school, including teacher’s names, email addresses, subject matter taught, and full- or part-time status?

____ Yes

____ No – Thank you for your interest, but we need participants who have access to that data. END CALL

7. Do you have access to the files containing that information at the school?

____ Yes

____ No or Not sure - Thank you for your interest, but we need participants who have access to that data. END CALL

8. What is the name of the school where you work?

________________________________________________

9. What grades does your school serve?

________________________________________________

10. How many students are enrolled in your school?

________________________________________________

11. Is your school public or private or something else?

________________________________________________

12. Have you participated in a research study or focus group in the past 6 months? (RECRUIT NONE.)

___ Yes– Thank you for your interest, but we need participants who have not participated in a study recently. END CALL

___ No

13. Based on your answers, we would like to invite you to participate in our study. You will receive $75. The interview is approximately 90 minutes in length. We would like to conduct the session at your school so you have access to your records and can use your computer. Is that okay?

__Yes, it is okay to come to the school and I will have access to my records -> Go to 14

__No, I cannot conduct this session at my school

We can conduct the session at the U.S. Census Bureau Headquarters or at a local library, but I first need to contact the researcher who will work with you. What is your telephone number and a good time to call?

Telephone # _______________________________

Good time to call ___________________________

GET BACK TO THEM AFTER TALKING WITH RESEARCHER

14. We are conducting interviews between __________ and ___________. When are you available to participate in this research study during that time period?

	CIRCLE
Yes	1	 (CONTINUE, and collect date, time and location)
No	2	 (THANK AND TERMINATE)

Date:

Time:

Location:

2017-18 NTPS Portal Usability Testing

Attachment 3

Protocol Materials

Table of Contents

Usability Testing Protocol 5

Practice Think Aloud 6

Interview – Starting with Invitation letter for the teacher listing 7

Satisfaction Questions 7

Debriefing Questions 7

Vignette 1 8

Vignette 2 8

Vignette 3 8

Additional Debriefing Questions 8

Finishing up session 8

Interviewer Log Sheet 9

Questionnaire for User Interaction Satisfaction (QUIS) 10

Usability Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Invitation Letter with Username

• Satisfaction Survey

• Vignettes

• Incentive

• Payment Receipt

• Pens and Paper

• Webcam with gooseneck clamp

• Laptop

• Guest Wireless Username and Password

• Extra Equipment and Charging Cords

INTERVIEWER USABILITY TESTING SCRIPT

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult our various web survey products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand NCES consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

OK great.

Thank you.

There are several parts to today’s session.

As you know when we talked to you to see if you were eligible to participate, we had to make sure you had access to the list of teachers at your school. As the first part of today’s session we’d like you to describe that list in your own words.

Probe if they are not forthcoming:

Do you have a list?

(If yes, then ask remaining questions; If No, then ask what type of information they have about their teachers)

What do you call it?

What type of information is on it?

Where did that information come from?

Who created it?

Who uses it?

Do you have to report the information from that list to anyone? Who?

Do other schools use a similar format?

Thank you for that information.

The second part of today’s session is for you to help us evaluate the design of a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Teacher Principal Survey. You will do this by completing it using your computer/this laptop.

If you run into any difficulties as you work in the web instrument, please don’t blame yourself. Any difficulties are the result of the design of the web instrument, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants, to help improve the web instrument. I did not create it, so don’t feel like you have to hold back your thoughts to be polite to me. We appreciate your help so we can make the web instrument work well for everyone.

Think Aloud: I want you to respond to the web instrument exactly the way you would if you were at school, but with one major difference. I would like you to think aloud as you work within the instrument. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?

• What are you including?]

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session, I will have some additional questions about how satisfied you were with the survey. Now, we may open up some of the screens and talk about them.

Interview – Starting with Invitation letter for the teacher listing

[Hand participant the mailing materials (invitation letter with their user name/ID info. Make certain to write down the PIN or any information that the user may need later in survey]

This is an example of mailing materials you would receive at school if your school was selected for the NTPS survey. Imagine that you received this at your school address. Please take a moment to read it. Notice that the address is not your real school address. If you were to receive the survey for real, the mailing materials would have your real school address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your school’s address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you and your school. Please go ahead and do what the letter says to get started. Remember that I am interested in what you have to say as you go about interacting with the web instrument. We especially want to see how you enter the teacher information into the form, so please start with that piece.

[Interviewer instruction: Stop them about 40 minutes into the instrument. If they pause, tell them that we’d like them to work within the instrument for 30-40 minutes. If we have to use mocked up data, provide that to them with fictitious information.]

Okay, we are ready to move onto the next task. We’d like you to logout of the instrument.

Satisfaction Questions

Now I would like you to answer these questions about the web instrument.

[Open the satisfaction survey on the laptop and have participant answer the questions.]

Debriefing Questions

So, now that you have worked with the web instrument for 40 minutes, in your own words, tell me the purpose of the instrument and the important features of the instrument.

If manual entry:

I noticed that you manually entered your teacher’s names. Why did you choose that way of entering teacher names? Do you have any comments on the data requested?

If file upload:

I noticed you used the file upload feature to enter your teacher list. Why did you choose that way of entering teacher names? Do you have any comments on the file structure or template?

Thank you.

Now we have three tasks we would like you to do.

[Interviewer instruction: If they covered the task during the main session, then you can skip the vignette.]

Vignette 1

Now I’d like you to go back into the instrument and enter the information about you and your school.

[Interviewer instruction: You may either text them or type the correct URL for them if they had difficulty in the start of the session. See if they can re-enter with their ID and PIN or security questions. Note what participant does if they don’t have their PIN. Give the respondent their PIN or security questions if they didn’t have it.]

AT THE END: We noticed that you did/did not write down your PIN. Would you have done that if you had received this survey for real?

Vignette 2

Now I’d like you to explore the other information in the website and provide feedback on that information to me. I’m especially interested in whether you would use the data provided within the instrument, and how you would use it.

[Interviewer instruction: See where they go naturally, but direct them to the resource center if they do not seem interested in going there. Use general probes to find out about their thoughts on the usefulness of the feature.]

AT THE END: Do you find these data useful to you in your job?

Vignette 3

Now using this new ID and PIN, (hand ID and PIN to participant), I’d like you to first enter these data for three teachers. Please think aloud while you do this task.

(Once participant has finished that): Now I would like you to upload a file of teacher data which is <here>. Please think aloud while you do this task.

AT THE END: What you expect to happen to the teacher data previously entered?

Additional Debriefing Questions

Do you have any questions about the Web instrument or how you would use it?

I have slides of the web instrument available. We can go over any of these screens you would like.

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session

[Give the respondent incentive envelope [$75], and thank them for their time and effort. Collect materials and leave or escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $75.

Interviewer Log Sheet

P#___

TA:______________________

Date: _____________________

Location: ________________________

PIN:

Main data entry:

Manual OR File Upload (circle one)

Wrote PIN down/Answered security questions

Notes:

Vignette 1:

Re-enter survey: Success or Fail

Notes:

Vignette 2:

Notes:

Vignette 3:

Notes

Questionnaire for User Interaction Satisfaction (QUIS)

Please Note: The questionnaire will be electronic with word labels on either end of 7 radio buttons. See image below for an idea of what the final format will look like.

1. Overall experience with the survey

Unpleasant to Pleasant

2. The survey screens are visually pleasing

Never to Always

3. The survey screens look professional

Never to Always

4. Characters (words/text) on the screen

Barely legible to Very legible

5. Instructions for correcting errors

Confusing to Clear

6. Entering your answers into the survey

Difficult to Easy

7. You knew what to do with this survey

Never to Always

8. Additional Comments__________________________________________________________________

2017-18 NTPS Portal Usability Testing

Attachment 4

Portal Screen Shots to Be Used during Portal Usability Testing

21