Protocol

NSP Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Demographic Questionnaire

• Mobile Use Questionnaire

• Mailing materials

• QUIS Survey location

• Laptop or Mobile Phone

• Incentive

• Payment Receipt

• Pens and Paper

• Extra Equipment –e.g., microphone for laptop studies, charging cords

• Location of extra UserIDs if using

Protocol

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today. In our group, we evaluate how easy or difficult it is to use web surveys. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes out to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

Thank you.

Start Recording (Camtasia)

OK great.

I have two questionnaires that I’d like you to fill out. They collect information about you and how often you use different electronic devices.

[Have participant fill out the background and demographic questionnaire and the mobile experience questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder. Make sure they hit the SUBMIT button. C:\Projects\NSP ]

Thank you.

Today you will be helping us evaluate the design of a survey that we are working on for the U.S. Department of Health and Human Services’ (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA). You will be helping us evaluate a survey they are working on. You will do this by completing it using [your smartphone/this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants that work with us on this survey to help improve it. I did not create the survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the survey work well for everyone.

[Think aloud] I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Because thinking aloud is not something that we do every day, let’s do a practice before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they do the think aloud explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?]

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

Eye tracking – if we decide to do it for the 5 laptop cases

[TOBII ET set up with bar on laptop]

While you are performing the task, we will record where you look on the screen. In order to accurately record where your eyes look, we would like you to do a simple task that will allow the computer to find your eyes. To get the best possible data, first please situate yourself in a position that is comfortable for you. On the next screen, you will see a red dot moving across the screen. Please follow the red ball with your eyes as it moves and keep your eyes on it as it pauses in each corner of the screen.

Ready?

[Calibrate the P’s eyes, check the calibration quality, repeat calibration until the calibration quality is satisfactory – e.g., green markers tightly on the gray balls]

Interview – Starting with Invitation letter

[Hand participant the mailing materials (invitation letter with their user name/ID info. Make certain to write down the PIN or any information that user may need later in survey]

This is an example of mailing materials you would receive at your home if you were selected for the survey. Imagine that you received this at your home address. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.

Laptop/Mobile: Please go ahead and do what the letter says to get started with the survey. Remember that I am interested in what you have to say as you go about answering the survey questions.

[Note: you will need to redirect them to the correct URL. Don’t redirect them until after they attempt to write in the correct URL because we want to see if they notice where to look on letter for URL and also whether they type it in or do a search or what… The correct URL for the study is:

https://insert it here ]

Interruption – Log out vignette

[STOP Respondent after the mid point of the survey. Ask them to pretend that they had to leave the instrument to do something else.]

Imagine you had to go somewhere and were not able to finish answering the survey questions at this time. What would you do?

[Once they are exited, then ask them to resume.]

Okay now I’d like you to imagine that you are ready to continue working on the survey. What would you do?

[Interviewer instruction: You may either text them or type the correct URL for them if they had difficulty in the start of the session. Note what participant does if they don’t have their PIN Give the respondent their PIN if they didn’t have it]

FAQ Interruption

[Interruption for FAQ or Help. After they have completed aprox ¾ the questionnaire]

Okay now I would like you to imagine you had a question about this survey. You wanted to learn more about why the survey is being conducted. What would you do?

You have some concerns about whether the information you enter into the survey will be kept safe. What would you do to learn more about how the information will be kept secure?

[First wait to see if user can find the answer on their own. If it appears user does not see/think of using the FAQ can probe, such as, “Is there any other way you could find this information?” “Is there any place on the website that would give you think information.”]

[Once they find the information, see if they can return to the survey without assistance.]

[After final SUBMIT is clicked, have participant answer satisfaction questions.]

Satisfaction Questions

Now I would like you to answer these satisfaction questions.

[Open the QUIS on the laptop and have participant answer the questions.]

Okay great. Now the next task is on the laptop. We are going to go over some of the screens you saw.

Vignettes

1. Log out and log back in – during session

2. Question about the FAQ – during session

3. Seeing different (less frequented pathing) – after SATQ

3.1 Section C:

If currently providing clinical care: How many years have you been providing direct clinical care? (If more than 1 year, then continue). We’d like to ask you about your clinical care experience a year ago. Please answer the following questions about your clinical care position(s) a year ago.

[TA will present screens for the following questions: C4, C5, C6, C21, C28, C29, C30]

3.2 Section D:

We want you to imagine that you are currently working in the medical field but as a professor of psychiatry at a local university running experiments, and teaching students. You are no longer working directly with patients and have not worked in a clinical setting in the past 10 years.

[TA opens up survey to section A3]

Okay so I’d like you to answer this set of questions with those made up characteristics I just gave you.

Debriefing on Individual Screens

[Open the ppt screenshots Ask the probes using the power point slide deck with screen shots. Show respondents the screen.]

Do you have any comments on this screen?

[If there were issues with it:] Tell me more about your experience with this screen.

[On screens where there were instructions –]

Did you read the instructions? What did they mean to you?

Additional Debriefing Questions

[If participant marked any QUIS responses either on the low or high end, probe:]

Tell me what you were thinking about when you marked this question a XXX.

Tell me a little about your work environment. Would you be the person to open the mail? Would you be the person to answer the survey? Is there anyone in your office that you would have had answer the survey?

Overall, what was your impression of the survey?

Is there anything else you’d like to mention that we haven’t talked about?

Finishing up session

[Give the respondent incentive envelope [$100 incentive], and thank them for their time and effort. Collect materials and escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $100 as a thank you for your participation.

5