2017 National Survey of College Graduates
Thank you for your time today. My name is XX and I work with the Human Factors and Usability group at the Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.
Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.
[Hand consent form; give time to read and sign; sign own name and date, start recording.]
Thank you.
Ok great.
Today there are about 4 different tasks we would like for you to do.
The first task is to complete this questionnaire. It collects background information about you.
[Have participant fill out the demographic questionnaire—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]
Thank you.
Today you will be helping us evaluate the design of the 2017 National Survey of College Graduates by completing it using [your smartphone/ your tablet/ this laptop].
If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.
Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.
Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?
Great that’s what I want you to do throughout our session. If you fall silent during the session, I will remind you to think aloud by saying something like “Keep talking.” I don’t mean to interrupt your train of thought, but I simply want to remind you that we are interested in what you are thinking.
At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.
[Eye tracking]
We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To calibrate your eyes, please follow the dot with your eyes.
[Calibrate the participants’ eyes]
[Hand participant the mailing materials]
This is an example of mailing materials you would receive at your home if you were selected for the 2017 National Survey of College Graduates. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your address.
That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.
You may think aloud as you look over the materials too.
Protocol:
STOP Respondent half way through the survey and ask them to pretend that they had to leave the instrument to do something else.
Once they are exited, then ask them to resume. You may either text them or type the correct URL for them if they had difficulty in the first round, especially for smartphones.
After final SUBMIT is clicked, have participant answer satisfaction questions.
The next task is for you to answer a short questionnaire about your experience completing the survey you just finished.
Debriefing: Ask all the probes using a power point slide with those screens. We will skip the screens that the participant did not see.
The next task is to go over each of the screens you saw.
Each screen will have the following probe:
Do you have any comments on this screen?
(If we observed issues)
Please tell me more about your experience as you answered the question on this screen. (If the person forgets, say something like, “I thought I saw you /heard you say… , please tell me more about what you were thinking.”
NOTES PAGES
P#___
Educational Characteristics of Participant:___________________________________
TA:______________________
Date: _____________________
Location: Headquarters / Other :________________________
Device: _____________________
WiFi or Dataplan
Browser type used:_____________________________________
Other browsers on device:________________________________
Accessing URL
Used search / typed in URL
No problem/Problems
What happened?
Login:
No problem/Problems
What happened?
Re-entry:
No problem/Problems
What happened?
Problem questions:
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Erica Olmsted Hawala (CENSUS/CSM FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |