Usability Testing under the National Crime Victimization Survey (NCVS) Redesign

Attachment 2

NCVS Redesign

Usability Testing Protocol

INTRODUCTION

Thank you for taking the time to help us today. The session will take approximately an hour and a half. I’ll give you a little background on what we’ll be doing today.

Westat is working with the Bureau of Justice Statistics, which is part of the U.S. Department of Justice. We are interested in how people think about different experiences that may happen to them to help us test questions that may be used on the National Crime Victimization Survey. The NCVS is administered every year nationwide and collects information on people’s experiences with crime and victimization. (YOUTH: Anyone age 12 or older is eligible to be selected for this national survey.) The goal of this interview is to get your feedback on the online survey to make sure it is easy to understand and navigate and that the questions make sense to you.

Today, I am going to ask you to go through the whole questionnaire as a survey respondent. (REMOTE INTERVIEWS: In a moment, I’m going to pass control over to you.) I am going to watch you as you do this and will take some notes as you go through the survey. Every now and then, I will stop you and we’ll talk about what you are seeing on the screen or how you decided on your answer to the question. If anything is confusing for you, or if you’re not sure how to answer a question or are unsure what to do next, just tell me.

This is a research project and your participation is voluntary. You can skip any question and you can stop at any point. We would very much appreciate your permission to (IN PERSON: audio record/REMOTE: audio and video record) this conversation. The recording will be used for note-taking purposes only and may be (listened to/watched) only by members of the project or the BJS staff. It will be destroyed when the project is over. When we are finished, we will give you (ADULTS: 60/YOUTH: 40) dollars to offset the cost of your participation.

INFORMED CONSENT

ADULTS: HAND THE CONSENT FORM TO PARTICIPANT, ANSWER ANY QUESTIONS, AND OBTAIN CONSENT BEFORE CONTINUING. This form contains more information about the research and your rights in this interview. Please read it over and let me know if you have any questions. (ANSWER QUESTIONS) If you are willing to take part in the study, please sign both copies. One copy is for you, and I will hold onto the other.

YOUTH: HAND THE ASSENT FORM TO PARTICIPANT, ANSWER ANY QUESTIONS, AND OBTAIN VERBAL ASSENT BEFORE CONTINUING. This form has more information about the research and your rights in this interview. Let me know if you have any questions. (ANSWER QUESTIONS) If you are willing to take part in the study, please check off the box at the bottom of the form and I will sign at the bottom. One copy is for you, and I will hold onto the other.

[START RECORDER AND GET ORAL PERMISSION TO RECORD.] It is [DATE AND TIME], do I have your permission to (audio/video) record this conversation? ~~~~Thank you.

Do you have any questions before we get started? I am going to hand over control of the computer to you and will watch as you answer the questions. Every now and then we will stop to talk about the questions and the web site.

PART A. ISSUE LOG

Any time the respondent has issues with navigation or questions, record the question number and a few words about the problem on the Issue Log. Use the Code Key to determine what probes to ask about the issue.

PART B: QUESTION-SPECIFIC PROBES

Person Characteristics I

P1

What did you think about the way this screen looks?
How did you decide whether to answer in months, years, or both?

Police Ask-All/Community Measures

Probe to get R’s reaction to the grid format.

What did you think about the way this screen looks?
How easy or hard was it to answer the questions on this screen? What made it easy/difficult?

Screener

6 month calendar

When R first encounters calendar:

Did you notice the calendar?
What did you think about this calendar?
How did the calendar help you, if at all, to figure out if the incident happened in the past 6 months?
When would you use it?
What would you change, if anything?

Periodically throughout as the calendar pops up, note how respondent is using the calendar. Probe why/why not using.

Did you use the calendar to help you remember if this incident happened in the past 6 months? (Probe to understand why/why not)
Was it helpful to show the 6-month calendar again on this screen? (Explain)

Incident counting probes (How many times did ____ happen in the past 6 months?)

When R first encounters the incident counting probe:

Do you have any feedback on this screen before we move on?

De-duping probes (Is this incident part of any other incident you have already mentioned?)

When R first encounters the de-duping probe:

In your own words, tell me what this question is asking.
How did you decide how to answer to this question?

Interleaving

Ask when R first encounters interleaving questions:

What did you think about being asked some follow-up questions after you reported an incident?
IF ANSWERED YES TO ANY: Just to confirm, did this happen as a part of the incident you just mentioned [POINT TO TEXT AT TOP OF PAGE]?
Did you read the text at the top of the page? In your own words, what are these follow-up questions asking you to do?

Transition from screener to CIR – CIR 1

What do you think about this page?
IF MORE THAN ONE INCIDENT: Does this page tell you what you need to know about the incident you are going to be asked about?
IF MORE THAN ONE BULLET POINT APPEARS ON SCREEN: Thinking about what happened to you in this incident, what do you think of the way we’ve ordered this list?
IF NEEDED: Is there a better order that would make more sense to you? (EXPLAIN)
Do you have any other feedback on this page before we move on?

Display text for offender definition:

YOUTH: Have you heard of the word “offender” before?
How do you feel about using the word “offender” to describe the person or persons who did this to you?
IF NEEDED: What word would you use to describe the person or persons who did this to you?

CRIME INCIDENT REPORT – CIR

If R answered about more than one incident in screener, note in issue log if he/she switches to another incident during the CIR or answers about multiple incidents at the same time.

SV1

What did you think about the look of this page with the yes/no questions?
How easy or hard was it to answer the questions in the grid? What made it easy/hard?

Module A - Attacks

Ask after respondent finishes module:

What did you think of this series of questions?
What did you think about the order of the questions?
How well did these questions describe what happened to you?
Were there any questions that were hard or frustrating to answer?

Module CI – Injuries

CI1

What do you think this question is asking?
What kinds of injuries do you think it means to include? Exclude?
How did you decide whether to answer yes or no?

CI2/CI2a

How did you decide what answers to mark?
How easy or hard was it to find categories that fit the types of injuries you had?

CI6

What do you think this question is asking?
What kinds of medical or dental professionals were you thinking about?

CI7

What do you think this question is asking?
How did you decide on your answer?

CI10

What do you think this question is asking?
How did you decide on your answer?

CI11

How easy or hard was it to answer this question?
How sure are you of your answer?

Module T – After T2a/T2b

How well did these options describe what was stolen from you?
Were there any responses missing that you were looking for?

Location Series

Ask after respondent finishes Location Series:

How well do these questions describe where the incident happened?
Is there anything else about the location of the incident that was not covered?
Were there any questions that were difficult or frustrating to answer?

Victim Offender Relationship: VO19

What did you think about the look of this screen?
How easy or hard was it to answer the questions on this screen? What made it easy/hard?

Self Protection: SP2

How easy or hard was it to answer this question?
What do you think about this list of options?
Is there anything missing from the list?

Hate Crimes: HC1

Can you tell me what this definition of hate crimes means in your own words?
(IF YES) How did you decide on your answer to this question?

Police Involvement PI2a

How easy or hard was it to answer this question?
What do you think about this list of options?
Is there anything missing from the list?

Consequences – Socio Emotional Series

CS1

What do you think is meant by “serious problems”?
(IF YES) What made you decide that your problems were serious?

CS2

(IF YES) How did you decide on your answer to this question?

CS4

(IF YES TO ANY) How easy or hard was it to decide on your answers to these questions?
Were any of the categories confusing or unfamiliar to you?

CS8

How easy or hard was it to answer this question?
How sure are you of your answer?

Consequences – Economic Series

CE1/CE4/CE5/CE7/CE13/CE18

How easy or hard was it to answer this question?
How sure are you of your answer?

CE16

How did you decide how to answer this question?
How sure are you of your answer?

Victim Services: VS2

How easy or hard was it to answer this question?
What do you think about this list of options?
Is there anything missing from the list?
Is anything on this list confusing?

Narrative

How easy or hard was it to answer this question?
What did you think about having to type in your story?
Do you think what you typed in is the whole story or are there pieces you left out? (If so, why did you decide to leave out those pieces?)

DEBRIEFING

General

Now that you’ve worked through the questionnaire, what are your overall thoughts on it? What did you like about the survey? What didn’t you like about the survey?
Were there any features or things about the web survey design that you particularly liked?
Were there any features or things about the web survey design that you didn’t like?
What did you think about the way the questions looked on the screens, in terms of the font, the font size, and being able to find the buttons to click for your answers?
How about the background color?
What did you think about the screens that were arranged in grids, with rows of questions for you to answer?
Were there any web design features that you felt were missing that would have helped you in completing the questionnaire?
Did you have any problems with going from screen to screen?
Do you have any feedback for me on the amount of scrolling up and down that you had to do on screens?
Do you have any feedback for me on the amount of clicking you had to do on the screens?

If respondent had more than one incident that happened to them:

In the 2nd half of the survey, you answered questions about ONE of the incidents that happened to you in the past 6 months. Was it clear that you were ONLY supposed to be answering about that incident?
(IF UNCLEAR, PROBE TO FIND OUT WHAT CONFUSED THEM)

Burn line

While you were answering about the specific incident that happened to you, did you happen to notice the reminder at the top of the screens to make sure you know which incident you were being asked about?
How did it help you, if at all?

Do you have any other thoughts for me on your experiences completing the survey?

Thank you! (END RECORDING, SEND RESPONDENT BACK TO HOST OR PROVIDE INCENTIVE)

NCVS Redesign – Usability Testing Protocol Wave 1 Page 4