2017 National Household Education Surveys Program (NHES) Usability Testing

Attachment 2 Recruitment Screener

NHES:2017 Usability Testing

Hello. My name is [XXXXXX] and I work for the U.S. Census Bureau. The U.S. Department of Education is conducting a study about children’s care and education, and adults’ education and work credentials. The Census Bureau is assisting with the research.

We are looking for adult volunteers to help us improve the web survey used in this study. We want to know how easy or hard it is to complete the web survey, and we need help from people like you to do that. The information gained from the study will be used for potential revisions to the web survey. Your participation is voluntary, and you will receive no sales pitches or follow-up calls based on your participation. I’d like to ask you a few questions to see if you qualify for the study.

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

4. How old are you?

___ years old

___ Prefer not to answer

5. [If Q4 is 65 or older then ask] Is everyone living in your household 65 or older?

1. Yes

2. No

3. Prefer not to answer

6. Are you Hispanic or Latino?

1. Yes

2. No

3. Prefer not to answer

7. Which of the following categories best describes your racial background? (Circle all that apply)

1. American Indian or Alaska Native

2. Asian

3. Black or African American

4. Native Hawaiian or other Pacific Islander

5. White

6. Prefer not to answer

8. What is YOUR EDUCATION level?

1. Less than High School

2. High School Graduate

3. Some College, trade, technical or vocational school

4. College degree

5. Graduate education

[In order to determine which survey will be the best fit for you, we’d like to ask you a few questions about your household]

9. How many people live in your household?
   
   

__ __

10. Are you a parent or guardian of any children who are in Kindergarten to 12^th grade, or who are ages 5 to 17?

1. Yes (continue)

2. No (go to Question 11)

Name or Initials	Grade (K-12 or no grade)	How old is this child?	Is this child male or female?	Is this child attending a public school? (Y or N)	Is this child attending a private school? (Y or N)	Is this child homeschooled for grades equivalent to K to 12th. (Y or N)

11. Are you a parent or guardian of any children ages 5 or younger?

1. Yes (continue)

2. No (go to Question 12)

Name or Initials	Is this child NOT YET in kindergarten? (Y or N)	How old is this child?	Is this child male or female?	Is this child in a care arrangement, e.g., family day care, day-care center, preschool, prekindergarten, Head Start or other early childhood program? (Y or N)

12. Have you ever completed one of these types of work experience programs to help prepare you for future employments - an internship, student teaching, co-op, practicum, clerkship, externship, residency, clinical experience, or apprenticeship?

1. yes

2. no

13. Have you ever earned any of the following types of certificates? (Check all that apply.)

1. A certificate for completing a training program from an employer, employment agency, union, software or equipment manufacturer, or other training provider, but NOT from a school.

2. A certificate for completing a vocational program at a high school.

3. A high school equivalency certificate, such as a GED®.

4. A certificate (sometimes called a diploma) for completing a program of study at a community college, technical college, or other school after high school. Do NOT include college degrees

14. Do you have a currently active professional certification or a state or industry license? A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teachers, or an IT certification. Do not include business licenses, such as a liquor license or vending machine license.

1. Yes

2. No

15. Do you have access to the internet on a computer, smart phone or tablet?

    1. Yes [go to 17]

    2. No [ go to 16a]

16a. Do you use the internet on a regular basis at some other location, like a library, work,
or a friend’s house?

a. yes

b. no [terminate]

16. Besides reading email, what are typical activities you do on the computer, smart phone, or tablet? [e.g., gaming/entertainment; reading the news; shopping/banking; graphic design/digital pictures; programming/word processing, etc.]
    
    

_________________________________________________________

17. With what type of device do you typically access the internet?

1. Smart phone

2. Laptop computer (skip to 21)

3. Desktop computer (skip to 21)

4. Tablet

5. Other: ______________________(skip to 22)

18. What kind of smart phone or tablet do you use?

1. iPhone/iPad

2. Android

3. Other smartphone: ______________________

4. Other tablet: ________________________

All recruits MUST be articulate, able to answer open ended questions, and able to provide explanations for their responses.

19. Have you participated in a research study or focus group in the past 6 months? (RECRUIT NONE.)

1. Yes (Thank and terminate.)

2. No (Continue)

Based on your answers, we would like to invite you to participate in an interview (to discuss your child’s care and education or your education and work credentials). You will receive $60 as a thank you for completing the interview. The interview is about 90 minutes in length and will be conducted (at the U.S. Census Bureau Headquarters or at a local Montgomery County Library [If uses a desktop primarily for the Internet]). [If they use a Smart phone, ask] So that we can see how you use the online survey we would like for you to bring your smart phone with you to conduct the testing. [If they are in the laptop condition tell them] We will provide a laptop on site for you to use.

We are conducting interviews on __________ and ___________. Are you available to participate in this research study?

	CIRCLE
Yes	1	 (CONTINUE AND GIVE DIRECTIONS)
No	2	 (THANK AND TERMINATE)

RECRUITER: Check off all of the criteria this adult meets in order to categorize the interviews we will conduct.

Interview Criteria:

Parent of K-12 child	Parent of homeschool child (K-12 equivalent)	Parent of child < 6 NOT enrolled in school	Work experience	Certificate	Certification/License	General Population	Not eligible for topical (65 or older – no eligible children) (Q5 = Yes)

Attachment 3 – Protocol Materials

NHES:2017 Usability Testing

Table of Contents

Formative Testing Protocol 6

Practice Think Aloud 7

Eye tracking 7

Interview – Starting with Invitation letter 7

Interruption – Log out 8

FAQ Interruption 8

Satisfaction Questions 8

Debriefing on Individual Screens 9

Additional Debriefing Questions 9

Finishing up session 9

Summative Testing Protocol 12

Eye tracking 13

Interview – Starting with Invitation letter 13

Satisfaction Questions 13

Debriefing on Individual Screens 13

Additional Debriefing Questions 14

Finishing up session 14

Background and Demographics Questionnaire 17

2. What is the highest grade or year of school you have completed? 17

3. Are you male or female? 17

4. Are you of Hispanic, Latino, or Spanish origin? 17

5. What is your race? Choose one or more races. 17

Mobile Experience Questionnaire 18

Questionnaire for User Interaction Satisfaction (QUIS) 19

Formative Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Demographic Questionnaire

• Mobile Use Questionnaire

• Invitation Letter with Username

• QUIS Survey

• Laptop or Mobile Phone

• Incentive

• Payment Receipt

• Digital Recorder and Extra Batteries

• Pens and Paper

• Guest Wireless Username and Password

• Extra Equipment and Charging Cords

INTERVIEWER FORMATIVE TESTING SCRIPT

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult to use are various web survey products. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

Thank you.

OK great.

I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices.

[Have participant fill out the background and demographic questionnaire and the mobile experience questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you.

Today you will be helping us evaluate the design of a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. You will do this by completing it using [your smartphone/this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they do the think aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?

• What are you including?]

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

Eye tracking

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

Interview – Starting with Invitation letter

[Hand participant the mailing materials (invitation letter with their user name/ID info. Make certain to write down the PIN or any information that user may need later in survey]

This is an example of mailing materials you would receive at your home if you were selected for the survey. Imagine that you received this at your home address. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.

Laptop/Mobile: Please go ahead and do what the letter says to get started with the survey. Remember that I am interested in what you have to say as you go about answering the survey questions.

Interruption – Log out

[STOP Respondent after they entered initial screener part demographics for themselves and ask them to pretend that they had to leave the instrument to do something else.]

Now I’d like you to imagine that you have to leave this right where you’re at as something has come up. For example, imagine that you just remembered you need to pick up something from the grocery store. What would you do?

[Once they are exited, then ask them to resume.]

Okay now I’d like you to imagine that you are ready to continue working on the survey. What would you do?

[Interviewer instruction: You may either text them or type the correct URL for them if they had difficulty in the start of the session. Note what participant does if they don’t have their PIN Give the respondent their PIN if they didn’t have it]

FAQ Interruption

[Interruption for FAQ or Help. After they have started into the first topical survey]

Okay now I would like you to imagine you had a question about this survey. You wanted to know how much time it will take to complete this survey. What would you do?

OR

Okay now I would like you to imagine you had a question about this survey. You wanted to learn more about why the survey is being conducted. What would you do?

[First wait to see if user can find the answer on their own. If it appears user does not see/think of using the FAQ can probe, such as, “Is there any other way you could find this information?” “Is there any place on the website that would give you think information.”]

[Once they find the information, see if they can return to the survey without assistance.]

[After final SUBMIT is clicked, have participant answer satisfaction questions.]

Satisfaction Questions

Now I would like you to answer these satisfaction questions.

[Open the QUIS on the laptop and have participant answer the questions.]

Okay great. Now the next task is on the laptop. We are going to go over some of the screens you saw.

Debriefing on Individual Screens

[Open the ppt screenshots Ask the probes using the power point slide deck with screen shots. Show respondents the screen.]

Do you have any comments on this screen?

[If there were issues with it:] Tell me more about your experience with this screen.

[On screens where there were instructions –]

Did you read the instructions? What did they mean to you?

Additional Debriefing Questions

[If participant marked any QUIS responses either on the low or high end, probe:]

Tell me what you were thinking about when you marked this question a XXX.

Overall, what was your impression of the survey?

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session

[Give the respondent incentive envelope [$60 incentive], and thank them for their time and effort. Collect materials and escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $60 as a thank you for your participation.

Interview Log Sheet

P#___

TA:______________________

Date: _____________________

Location: Headquarters / Other :________________________

Device: _____________________

WiFi or Dataplan

Browser type used:_____________________________________

Other browsers on device:________________________________

Accessing URL

Used search / typed in URL

# attempts (when they hit enter, or when they chose a website from a search):___

# of incorrect sites accessed:____________

#spelling errors made:______________________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Login: # attempts with submit: ______________

# of self-corrections before submit: ____________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Re-entry: : ______________

Problems Y / N : Describe

Used PIN Y / N

TA assist: Y / N

Problems Y / N

Email

Email provided Y / N Comments:

Spontaneous comments on instruction (circle): None Confidentiality Other: (Describe)

Roster

# hhld members:

Any errors:

Spontaneous comments:

Demographics:

Description of household (circle):

Adult with certifications & or special licenses /

Child age is than 5 years old /

Children 5-18 in Public School /

Children 5-18 in Home schooling

Spontaneous comments:

Any edits triggered: Y / N

If Y: what was outcome:

Spontaneous comment s

Other Comments

Any spontaneous comments on questions that respondent says are “sensitive.”

Summative Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Demographic Questionnaire

• Mobile Use Questionnaire

• Invitation Letter with Username

• QUIS Survey

• Laptop or Mobile Phone

• Incentive

• Payment Receipt

• Digital Recorder and Extra Batteries

• Pens and Paper

• Guest Wireless Username and Password

• Extra Equipment and Charging Cords

INTERVIEWER SUMMATIVE TESTING SCRIPT

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult to use are various web survey products. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the tape or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

Ok great.

I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices.

[Have participant fill out the background and demographic questionnaire and the mobile experience questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you.

Today you will be helping us evaluate the design of a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. You will do this by completing it using [your smartphone / this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. I appreciate your help so we can make the online survey work well for everyone.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

Eye tracking

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

Interview – Starting with Invitation letter

[Hand participant the mailing materials (invitation letter with their user name/ID info. Make certain to write down the PIN or any information that user may need later in survey]

This is an example of mailing materials you would receive at your home if you were selected for the survey. Imagine that you received this at your home address. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.

Laptop/Mobile: Please go ahead and do what the letter says to get started with the survey.

[After final SUBMIT is clicked, have participant answer satisfaction questions.]

Satisfaction Questions

Now I would like you to answer these satisfaction questions.

[Open the QUIS on the laptop and have participant answer the questions.]

Okay great. Now the next task is on the laptop. We are going to go over some of the screens you saw.

Debriefing on Individual Screens

[Open the ppt screenshots – any that were noticeably problematic for participant. Ask the probes using the power point slide deck with screen shots. Show respondents the screen.]

Do you have any comments on this screen?

[If there were issues with it:] tell me more about your experience with this screen.

[On screens where there were instructions –]

Did you read the instructions? What did they mean to you?

Additional Debriefing Questions

[If participant marked any QUIS responses either on the low or high end, probe:]

Tell me what you were thinking about when you marked this question a XXX.

Overall, what was your impression of the survey?

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session

[Give the respondent incentive envelope [$60 incentive], and thank them for their time and effort. Collect materials and escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $60 as a thank you for your participation.

Interview Log Sheet

P#___

TA:______________________

Date: _____________________

Location: Headquarters / Other :________________________

Device: _____________________

WiFi or Dataplan

Browser type used:_____________________________________

Other browsers on device:________________________________

Accessing URL

Used search / typed in URL

# attempts (when they hit enter, or when they chose a website from a search):___

# of incorrect sites accessed:____________

#spelling errors made:______________________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Login: # attempts with submit: ______________

# of self-corrections before submit: ____________

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Re-entry: : ______________

Problems Y / N : Describe

Used PIN Y / N

TA assist: Y / N

Problems Y / N

Email

Email provided Y / N Comments:

Spontaneous comments on instruction (circle): None Confidentiality Other: (Describe)

Roster

# hhld members:

Any errors:

Spontaneous comments:

Demographics:

Description of household (circle):

Adult with certifications & or special licenses /

Child age is than 5 years old /

Children 5-18 in Public School /

Children 5-18 in Home schooling

Spontaneous comments:

Any edits triggered: Y / N

If Y: what was outcome:

Spontaneous comment s

Other Comments

Any spontaneous comments on questions that respondent says are “sensitive.”

Background and Demographics Questionnaire

1. In what month and year were you born? _________________ __________________

month year

2. What is the highest grade or year of school you have completed?

Less than high school

Completed high school

Some college, no degree (indicate area of study: ____________________________________)

Associate’s degree (AA/AS) (indicate area of study: __________________________________)

Bachelor’s Degree (BA/BS) (indicate area of study: __________________________________)

Post-Bach degree (indicate degree and area of study: ________________________________)

3. Are you male or female?

Male

Female

4. Are you of Hispanic, Latino, or Spanish origin?

Yes

No

5. What is your race? Choose one or more races.

White

Black or African American

American Indian or Alaska Native

Asian

Native Hawaiian or Other Pacific Islander

Mobile Experience Questionnaire

Instructions: Please answer the following questions about your mobile phone or tablet use experiences.

1. Do you own a smartphone (such as iPhone, Android, Blackberry) or tablet (such as iPad, Galaxy, etc.)?

O Yes, smartphone

O Yes, tablet

O Yes, smartphone and tablet

O No, neither

2. Thinking abou the smartphone that you use most frequently, how long have you owned that smartphone?

_____ year _____ month

3. How long (in years and / or months) have you ever used a smartphone?

_____ year _____ month

4. On average, how many hours per week do you use your smartphone to access the Internet?

_______ hours per week

5. How comfotable are you using your smartphone to use the Internet?

O Very comfortable

O Comfortable

O Neither comfortable nor uncomfortable

O Uncomfortable

O Very uncomfortable

6. Do you have a desktop or laptop computer?

O Yes

O No

(If yes to Q6a)

6a. What types of things do you do on your mobile device that you don’t do on your desktop or laptop?

7. During the last year, how many times did you complete a survey using your smartphone?

________ times

8. Not including email, how comfortable are you with providing personal information on Internet forms/surveys using your smartphone?
   
   
   

O Very comfortable

O Comfortable

O Neither comfortable nor uncomfortable

O Uncomfortable

O Very uncomfortable

Questionnaire for User Interaction Satisfaction (QUIS)

Please Note: The questionnaire will be electronic with word labels on either end of 7 radio buttons. See image below for an idea of what the final format will look like.

1. Overall experience with the survey

Unpleasant to Pleasant

2. The survey screens are visually pleasing

Never to Always

3. The survey screens look professional

Never to Always

4. Characters (words/text) on the screen

Barely legible to Very legible

5. The survey questions follow a logical sequence

Never to Always

6. Words used in the survey questions and answers

Hard to understand to Easy to understand

7. Coming up with answers to the questions

Never bothered me to Always bothered me

8. Instructions for correcting errors

Confusing to Clear

9. Entering your answers into the survey

Difficult to Easy

10. You knew what to do if you needed help answering the question

Never to Always

11. Additional Comments__________________________________________________________________

26