Usability and Cognitive Findings

The Bureau of Justice Statistics (BJS) is experimenting with the use of Interactive Voice Response (IVR) as a complementary mode of data collection for the National Crime Victimization Survey (NCVS). As part of the NCVS core methodology, an IVR mode could be incorporated as part of a multi-mode design within the rotating panel design. As a self-administered, computerized mode, the IVR could yield better information on sensitive events, such as domestic violence and sexual assault while at the same time minimizing interviewer effects. For instance, after the initial in-person interview, respondents could be asked to call into an 800 number to complete the survey in subsequent contacts. An IVR could also be used as a supplemental survey to generate local area estimates. Previously, agencies have relied on mail or telephone surveys to conduct local area victimization surveys. IVR could provide a way to increase both the quality and efficiency of these surveys.

In preparation for field testing, a series of usability tests are being conducted. The first stage will focus on the NCVS crime screener (NCVS 1). The second stage will implement findings from the first stage and also test the detailed incident report form (NCVS 2). The third stage will be a micro dry run incorporating findings from the first two stages. The focus of this report is on the results of the first stage.

The goal of the usability test is to identify and correct features of the IVR system that would discourage response and/or lead to measurement error. More specifically, the usability testing is designed to:

Identify questions that may be too long, or too cognitively burdensome for respondents. Questions that are too long or cognitively burdensome may be difficult for respondent to retain in memory when formulating their answers, or they may be unable to map their answer to the response options.
Ensure that respondents are given instructions when needed. Respondents using touch-tone data entry (TDE) will need instruction on how to execute their answers at the time of response. These instructions should not be offered too early where respondents would not remember. Conversely too much instruction increases the length of the instrument and burden possibly also increasing break-offs.
Identify how respondents interact with an IVR system. The motivation when participating in a voluntary survey request is different from the motivation when accessing a bank account through IVR system. Respondents will have low tolerance for a complex system. Recognizing and adapting to respondent expectations are important to motivate response.
Identify how respondents handle problems. Respondents need to be able to correct errors in entry, difficulty with offering a response, or decisions not to respond. The error handling features of the IVR system should be tested to see if they satisfy respondent expectations or place too much burden on the respondent. For questions that are prone to error this may mean identifying the places where burden would be increased by verifying responses before continuing.

In the remainder of this report the methods, results and primary recommendations resulting from the usability tests are described in more detail.

Methodology

2

Two separate modes for collecting responses were tested. The first mode used Dual Tone Multiple Frequency (DTMF), or ‘keypad’ entry, for respondents to provide answers to each question. With this entry each question was followed by instructions on which keys on the telephone keypad corresponded to the response options. The respondent used the telephone keypad to enter the response. The second mode used Speech-to-Text (STT), or ‘speech’ entry, for respondents to provide answers. With this entry mode respondents spoke their answers.

The IVR version of the NCVS included key adaptations to facilitate administration. This generally consisted of breaking up single questions with multiple response options into multiple questions. It also involved cutting the length of some questions. NCVS-1 was designed with a “short cue” approach which administers multiple examples of possible events and situations to assist in the recall process. Respondents to an IVR tend to be less patient, as indicated by a higher breakoff rate (e.g., Tourangeau, 2002; Dillman, et al., 2009). The modifications were intended to address this concern. Second, some of the questions in NCVS include response options that would be considered too complex for a speech to text system. In the process of decomposing questions, to the extent possible, questions were formatted to allow for responses of “yes” or “no”. The benefit to this format is that it permits consistency throughout the IVR instrument. An advantage for DTMF (keypad) respondents is that respondents are not required to continually learn the appropriate keys that correspond to the response options. For STT (speech) yes and no responses are simple utterances easily recognizable by a speech to text system. In fact, many of the crime screening questions on NCVS-1 are already in this form. The re-formatting of questions was primarily required for the demographic questions.

Westat conducted the testing for the DTMF IVR from April 27 through June 6. Testing of the STT IVR was conducted from June 6 through June 15. Three survey methodologists conducted the cognitive testing. The cognitive testing was carried out at Westat’s cognitive and usability testing facility in Rockville, MD.

2.1 Protocol

The protocol (Appendix A) was designed so that information about the respondent’s interaction with the IVR would be collected. The protocol provided the interviewers with ways of probing potentially problematic questions. Respondents were initially provided the invitation letter and asked to review it as they would at home. Interviewers conducted a short debriefing on the letter asking them to comment on the length and any issues they found confusing or problematic. Once the letter was discussed, respondents were instructed to call the IVR system and complete the interview. While the interview was being completed, the interviewer listened on another telephone extension to the interview and observed the respondent’s behavior. The interviewer was instructed to observe and take notes of any potential issues or problems. In addition the interviewers also made note of instances when the respondent:

Pressed an incorrect key / provided invalid incorrect response
Backed-up (or attempted to do so)
Timed out (no response given by respondent)
Appeared confused
Used the help function

After completing the interview, respondents were asked questions about their experience and probed on any questions that appeared to be problematic. Interviewers also probed on any other issues or perceived difficulties the respondent experienced, or that were observed by the interviewer.

Respondents were then asked to call the IVR system a second time. During this second pass, the respondent was given a crime scenario to use when completing the IVR. The respondent was to imagine that the event described in the scenario had happened to him/her and to fill out the survey accordingly. While the respondent was going through the instrument, the interviewer asked if he/she understood how to access the ‘help’ functions. Once the IVR was completed, the interviewer debriefed the respondent on the use of the help system. This debriefing ended by getting an overall assessment of the IVR system from the respondent.

The interviews were conducted at the cognitive and usability testing laboratory facility in Rockville. At the start of each interview session, the interviewer briefed the respondent by reading scripted text that described the purpose of the project and the procedures that would be used during the interview. The respondent was also asked to sign a research consent form that briefly summarized the main points explained to the respondent verbally. See the Appendices for the study protocols and the IVR interview.

Each session was summarized by reviewing the notes and audio recording. The summaries from each interview were reviewed so that all responses and notable issues could be considered in the analysis. In addition, the audio recordings were reviewed to count the number of times the respondent interrupted the IVR when responding.

2.2 Participant Recruitment

A total of 20 individuals were recruited to participate in the testing. Due to the need to identify an uncommon population group (recent victims of crime), multiple methods were used to find potential respondents. Westat placed an ad on Craigslist, put up flyers, and placed newspaper advertisements in the Montgomery County, Prince George’s County an/d Frederick County Gazette to identify people. The following characteristics were listed as criteria for inclusion in the study:

Over age 18;
Experienced a victimization within the past 12 months.

Westat employees were not eligible to participate as a respondent, but their friends and family members were eligible.

The ads provided a phone number to call and an email address if interested in volunteering for the study. These volunteers were contacted by a Westat employee by telephone who administered a series of questions to determine eligibility, as well as to record the potential participant’s demographic characteristics. Those who met the screening criteria were scheduled for a usability interview at Westat facilities. Table 1 shows selected demographic characteristics of the participants.

Table 1. Demographic Characteristics of Stage 1 Participants

Demographic Characteristics		Number of DTMF Participants	Number of STT Participants
All participants		10	10
Age	18 - 29	2	3
	30 - 39	3	3
	40 - 49	1	2
	50-59	4	1
	60+	0	1
Sex	Female	7	3
	Male	3	7
Race	White	6	3
	Black	4	6
	Hispanic	0	0
	Asian	0	1
	Other	0	0
Education	Less than high school	0	1
	High school/GED	5	2
	Some college, or college degree	5	7
Victimizations	Theft from person	5¹	5
Reported at	Theft from vehicle	3	1
Recruitment	Theft from home	0	1
	Assault	1	2
	Threatened assault	1	0
	Burglary	1	2

¹One respondent reported more than one victimization.

Findings and Recommendations

3

In this section, the results from the 20 interviews are summarized. The first section discusses the invitation letter, with the remainder of the section discussing the results of the IVR. Recommendations are presented along with the results.

3.1 Invitation Letter

As discussed in Section 2, each respondent was provided the survey invitation letter for the IVR version of NCVS. After reading the letter, respondents were probed about the content of the letter, with specific probing on their understanding of instruction for selecting the respondent within the household.

Finding

Generally, most respondents skimmed the letter. Most did not express any initial confusion about any particular part of the letter. However, when asked who the letter requested to complete the survey, 10 of the 20 respondents either did not notice or ignored the instruction to select the household member with the next birthday. When specifically asked about the next birthday instruction, 18 respondents understood what the instruction was asking but generally reported some level of confusion about the instruction. Many of these respondents reported needing to re-read the instruction and reported not understanding why the instruction was necessary. The quotes below illustrate this point:

“If he is at least 18 what does it matter who has the next birthday?”
“I’m just thinking of the reasoning behind that, but I guess that’s the easiest way to pick one of the members of the household I guess for some reason.”

The remaining two respondents did not understand what was meant by the person with the next birthday. One respondent thought the letter was asking for the youngest person who is at least 18, while the other thought the instruction was asking for a legal adult.

The sentence in the letter that provides the instruction to select the person with the next birthday reads:

“Please ask the person who lives at this address who is at least 18 years old and who has the next birthday to complete the automated survey by calling the number below.”

Some of the confusion related to reference to both someone who is 18 and to the person with the next birthday. Respondents tended to think it was either someone who was 18 or the person with the next birthday. We believe this confusion also stems from respondent’s lack of understanding of why one would select someone with the next birthday. This confusion also came out in the focus groups on the advance materials. Something that may have added to the confusion was that respondents in the usability lab were expecting to take the survey. Consequently, referral to a procedure that really didn’t apply to the present situation was not expected.

While not specifically probed, only one out of the 20 respondents appeared to have any difficulty identifying the telephone number and access ID for the IVR system. This was evident from the successful completion of this task when asked to complete the IVR portion of the protocol.

Recommendation

Reword the instruction asking for the adult with the next birthday to clarify and increase saliency. Our suggestion is:

“Please have the adult with the next birthday complete the automated survey using the number below. This will help us represent adults across all age groups.”

The last sentence is included to provide more rationale for the selection of the person with the next birthday.

3.2 General Reactions to the IVR

The average time to complete the IVR was about ten minutes (10:02). The amount of time did not differ by DTMF (keypad) and STT (speech) modes. Timings were calculated from point where the system accepted the entered ID to the end of the interview.

After each respondent completed the NCVS IVR they were asked about their general reactions on what they did or how the system performed. They were also asked what worked particularly well or not so well for them and whether this is something they would take part in if they received this invitation at home. The discussion below combines the responses to all of these questions, as well as our own observations of any difficulties respondents had when going through the system.

Finding

Generally speaking, respondents thought the IVR was understandable and that they could get through it without major issues. This was especially the case for the respondents to the DTMF methodology where it was easier to input responses (see discussion below). Some of the positive comments about the interview included:

“Just common questions, was nothing hard you had to think about.”
“The questions were pretty clear.”
“Basic…how pretty much all surveys are.”
“I thought it was easy. It asked good questions and told me what to press.”
“Pretty straightforward questions.”
“It was pretty good…pretty much direct. Some of the questions pertained to my experience.”
“I thought the survey was interesting, I think it was to the point.”
“It is not unlike any other voice automated system.”
“She [IVR system voice] asked the right questions
I think it’s interesting because….you can express more feelings. If you’re talking to a person, you’re going to get the vibes from the person. You don’t know if you’re going to convey all the information.

Respondents said they would generally be willing to participate if they had received the invitation at home. A few respondents said that it would depend upon the invitation letter or envelope. Of the three respondents suggesting participation is dependent upon the letter or envelope, one stated it would depend on what the envelope looked like (in terms of marking, logo, sponsorship). For usability testing only plain white envelopes were used. For the other two respondents one wanted a letter that looked more official (bigger/flashier logo), while the other wanted more detail on when the survey is being conducted.

Respondents seemed to like the tone and quality of the voice. Several respondents commented that it was clear and was easily understood.

Many of the negative comments were directly related to problems identified with the IVR system. One issue was problems the system had recognizing verbal responses in STT mode. This lead to frustration with the system overall. Similarly, there were problems with the timing of the open ended question on “What happened”. Both of these issues are discussed more below.

A few respondents were dissatisfied with the pace of the reading and the repetitive nature of the questions. A few individuals thought it was too slow, especially in certain spots (e.g., when entering the ID). Related to this, there were a few comments on the repetitive nature of the “yes/no” format. These individuals did not like answering questions that were not particularly relevant to them. The race and education questions were the primary examples of this. For education, respondents have a tendency to want to answer the question with their degree or highest grade, rather than waiting until the right categories are read. For the race question, respondents have to provide an answer to all of the categories. For the crime questions, one respondent felt the questions were too detailed and asked for too much information. This person would have preferred a single open-ended question.

Recommendation

There are no recommendations for immediate changes based on these comments. Recommendations specifically addressing difficulty with the IVR system are covered later in the report where observed difficulties are discussed. Longer term, we will want to consider whether we should quicken the pace of speaking on the IVR recordings. We will re-visit this issue once the second stage of usability testing is completed.

3.3 Incident Reporting

All respondents to the NCVS IVR usability testing had reported at least one victimization within the last 12 months (the specified recall period for NCVS IVR). This allowed the interviewers to observe how all of the victimization questions were working.

Findings

During the IVR interview several respondents reported the same incident for more than one question. This occurred for eight of the 20 respondents (4 – DTMF; 4 – STT). During probing several reasons for over-reporting were uncovered. A majority of respondents appeared not to hear or notice the instruction to exclude victimizations already reported. Once an incident is reported, the screening questions insert the phrase “other than any incidents already mentioned” at the beginning of each subsequent item. Some of the respondents did not hear or process this statement.

The other respondents cued in on the examples read in each item and responded ‘yes’ to items that described their incident. For example, one respondent felt that the crime screening questions were approaching different aspects of the victimization to gather detail about the victimization. Another respondent stated that when hearing later descriptions of crimes that his victimization was a better fit in a later question and reported it there.

Unlike incidents of over-reporting, only two respondents failed to report a victimization. In one case, the respondent reported a victimization, but during debriefing an additional victimization was uncovered that was not reported. The incident was pick pocketing while at a metro station. The respondent said she forgot about the event until the debriefing. A second respondent did not report any victimizations. When asked about this during the debriefing, he explained that he had been victimized within the last few years, but it was not within the last 12 months.

Respondents also offered comments relating to their perception of the NCVS questions. Specifically these comments related to perceived redundancy or length of the interviews and tended to differ by entry mode. For respondents assigned to the DTMF mode 5 of the ten respondents offered comments relating to perceptions of redundancy. For respondents assigned to the STT mode only one respondent offered similar comments. The following are some quotes:

“like 1000 different questions after I just said no-nothing else, other than the one incident I just reported.”
“This was too monotonous”
“It came back to it [victimization] three times, the same crime.”
“I thought it was kind of redundant, some of the same questions are asked over and over again.”
“…you can move on, but when it keeps repeating it got a little boring.”

Differences in reports between the DTMF and STT mode input be attributed to the repetition of the instruction for which keys to press in the DTMF mode. Also two of the DTMF mode respondents mentioned the series of questions asking about race or education when offering comments relating to redundancy (see discussion in section 3.2).

Recommendation

Modify the introduction to address perceived redundancy, explaining it as something that will improve the information that is collected on the survey. This modification would also add a statement to report victimizations only once. Example text is provided below:

“Next, you’re going to hear some examples that will give you an idea of the kinds of crimes this study covers. Please {press one for yes / say yes} when you think of anything, otherwise, please wait until the end of the list to answer. Include crimes even if it doesn’t fit the example you hear. Some questions may seem to cover the same types of crimes, but we want to provide you as much assistance as possible in recalling victimizations. Please report each victimization that you have experienced only once.”

In addition, we propose adding a question that is asked whenever a respondent answers ‘yes’ to more than one screener question. This question will ask whether the incident has already been reported:

“Is this victimization the same as what you reported earlier?”

If the answer to this is ‘yes’, the respondent will go to the next victimization question. They will not be asked “how many time” or to describe the incident.

3.4 Specific Difficulties with the IVR

Specific difficulties with the NCVS IVR were identified through interviewer observation and review of the audio recordings. The number of problems differed between the DTMF and STT modes. More instances of difficulty were observed in the STT mode.

Table 2 Count of behaviors observed for DTMF and STT IVR modes.

Behavior	DTMF	STT
Pressing an incorrect key / providing invalid incorrect response	2	15
System did not understand response	N/A	59
Respondent attempted to use telephone keypad	N/A	5
Uses of the help function / request for help
Occurrence or attempts to back-up	1	1
Occurrence or attempts to have a question repeated	1	1
Incidents of time-out (no response given by respondent)	2	8
Interruption of open-ended verbal recording	16	8
Respondent Initiated Interruptions
Interrupted the Question	58	9
Interrupted Response Instruction	156	17

The IVR system was programmed to allow for up to two contiguous errors before terminating the interview. Example of errors would be ‘time out’ where no response is given (or detected by the system), or invalid responses (incorrect keypad entries, or incorrect / unrecognizable speech responses). While the system would terminate the call if there were two contiguous errors, respondents could have several errors or instances of difficulty as they would always be allowed another attempt on the same question.

Finding

Pressing an incorrect key. Incidents of pressing an incorrect key were uncommon and specific to the DTMF mode. This behavior occurred twice for two separate respondents. In the first case the respondent pressed the pound (#) key instead of the star (*) key. The star key is used to indicate the end of verbal open-ended recordings, while the pound key is used to access help (in DTMF only). The second respondent was similar to the first, where the respondent pressed ‘3’ instead of the star key.

Recommendation: There are no recommendations for changes as this is not significant problem.

Providing an invalid or incorrect response. This behavior occurred 15 times across multiple respondents. Twelve of these incidents were associated with the marital status question in the STT mode. The marital status question was different from others since it offered five response categories instead of yes or no, as shown below:

“What is your current marital status? Are you married, widowed, divorced, separated, or never married?”

In 11 instances respondents said “single” before they heard the instruction on how to respond (in this case likely ‘never married’).

Recommendation: The marital status question works well in DTMF mode, but not for STT mode. Reword the question for the STT mode so the responses are part of the question and not read after the question. Example text is provided below:

“Are you currently married, widowed, divorced, separated, or never married?”

System did not understand response. This system difficulty occurred 59 times and was restricted to the STT mode only. For 19 of these incidents it was associated with attempts to enter the ID number. For one respondent in particular it took six attempts before the system recognized and accepted her ID.

For the remaining incidents, the respondent gave a response that attempted to answer the question with a valid response, but the system did not recognize it. In some cases, the respondent used some type of disfluency or extraneous utterance. For example respondents would say “um yes” or “um no”. In other instances, the respondent might have used a different word – e.g., ‘none’ instead of ‘zero’. In still other instances, there were cases where the respondent said “yes” or “no’, but the system did not recognize the response.

Recommendation: We recommend three changes to the system. One, for ID entry we suggest using DTMF entry only. This would match some respondent expectations and eliminate speech recognition difficulties with an eight digit ID. Second, we will expand the response dictionary (e.g., for numeric questions add ‘once’ and ‘twice’ as valid responses for ‘one’ or ‘two’). This will address those instances where respondents would commonly use slightly different utterances when answering. The third recommendation is to adjust the confidence level of speech input. Setting the confidence too high results in rejection of legitimate responses. Setting it too low risks accepting responses that are not correct. The setting for Phase 1 may have been too high. We will make an adjustment for Phase 2 and conduct more extensive tests on the ability of the system to recognize soft utterances, speech with background noise or accents.

Respondent attempted to use the telephone keypad. This behavior was specific to the STT mode and was generally related to instances where the IVR system did not understand the speech response. This occurred five times and was a recourse for respondents when the system would not recognize, for example, the spoken ID number.

Recommendation: There is no recommendation for addressing this as it is addressed above.

Use of help function / request for help. Instances of this behavior were uncommon on both DTMF and STT modes. For the DTMF mode one respondent accessed help to have a question repeated. Another respondent wanted to back-up when she was cut-off while giving a verbal open-ended response, but did not know how to back-up through accessing ‘help’. For the STT mode one respondent attempted to access help to have a question repeated and another attempted to access help to change an incorrect response (response to number of cars owned in past 12 months).

Recommendation: There are no recommendations for these observations. Recommendations directly addressing help are discussed later in the report.

Incidents of time-out (no response given by respondent). These behaviors occurred for different reasons in the DTMF and STT modes. This occurred twice in the DTMF mode because the respondent appeared to need time to think about the answer. In one case in particular the respondent counted the number of times she had moved in the past five years. Time-outs or nonresponse in the STT mode were generally related to moments of confusion due to other problems discussed elsewhere. These occurred eight times in the STT mode. One example of this was when a respondent had difficulty providing her age, the IVR system was recognizing her response as ‘5’ instead of ‘25’. After several attempts the respondent did not know what to do. In other cases the time-out occurred after a verbal open-ended recording where the respondent was cut-off by the IVR system. The respondent was unsure what to do since they had not finished their verbal report.

Recommendation: There are no recommendations for changes at this time. It would be possible to allow more time for a response. However very few time-outs occurred. Of those that did occur, only a few occurred because the respondent did not have enough time to think of his/her answer. Allowing for more time for a response would slow the interview down and have a negative effect on respondent perceptions of the burden of the interview. For example, the question on the number of moves allows for a two digit response (how many times moved in last five years). It is likely most responses will be less than two digits. The IVR system waits the full allotted time to see if another digit may be entered. Increasing the time to respond would also increase this wait time after a response is provided.

Interruption of open-ended verbal report. Respondents were asked to describe, in their own words, what happened for each victimization that was reported. The system recorded this description. The IVR system instructed the respondent to press the star (*) key once they were finished describing the incident. The IVR system allowed for a maximum recording length of 10 seconds before continuing to the next question. Respondents were interrupted by the system continuing with the next question a total of 24 times (16: DTMF; 8: STT). In one case for DTMF this occurred five times for one respondent.

In addition to the time being too short, some respondents were not prepared to provide a verbal description of the incident. Up until this question, they had been asked to either press a ‘yes/no’ button or give a one-word answer. Providing a narrative of what happened is a different cognitive task.

Some of the duplicative reporting may have been caused by the interruption of the verbal report by the system. In particular one respondent stated she was unable to “tell her story” because the system had cut her off.

Recommendation: Increase the current time limit to 30 seconds to allow more time for respondents to offer descriptions of their victimization. In addition, add in text for this question to provide respondents some time to prepare how they want to describe the incident:

“Please briefly describe the incident in your own words. Please give a brief description of what happened, including any details such as where it happened, when it happened, who was involved or any other details that might be important to you.”

Respondent Initiated Interruptions. These include instances where the respondent gave a response either before the system had completed reading the question, or before the system had completed reading the instruction on how to respond.

Interrupting the question may be problematic if respondents are prematurely cutting off the question and not fully processing the information necessary to provide a proper response. The NCVS-1 is structured to aid recall of events by reading multiple cues. If respondents cut the cueing short, they may not recall as much. On the other hand, if respondents become frustrated because of having to listen to questions they view as unnecessary, response quality may be jeopardized. In total (across all questions and interviews) respondents interrupted the playing of a question 67 times. This differed by mode, with the DTMF having 58 interruptions and the STT mode only 9. To put this in perspective, there are about 48 responses requested in each IVR interview. The number of interrupted questions, therefore, is approximately 6% of all questions asked across all 20 respondents.

There were many more interruptions in the DTMF mode. Most of these interruptions (44 of 58) were with 3 respondents. The interruptions occurred primarily within the crime screener. The interruptions generally occurred after the respondent heard the first part of the question, which indicated the general type of crime that was being asked about. For example, the question on motor vehicle theft includes the phrase:

“….was the vehicle stolen or used without permission?”

One of the three respondents answered this question after hearing”… was the vehicle stolen or…”. This cut off the last part of the question on whether the vehicle was used without permission. A similar pattern occurred for the personal violence questions, which are generally introduced with a phrase like “were you attacked or threatened….”.

Of the 3 respondents that accounted for most of the interruptions, one had said that he was, by nature, someone who is not very patient with surveys like this. He would have preferred an interviewer ask him questions and would not want it to last much more than 5 minutes. He was one of the few individuals who said that he would not likely do this if he received it at home. Another one the 3 persons who interrupted frequently had not experienced any victimizations over the past 12 months. He said that he listened to enough of the question to determine that it didn’t apply to him.¹

There were very few interruptions for the remaining respondents. Only 9 occurred for the STT respondents. It isn’t clear what accounts for the difference between the DTMF and the STT mode. It might be that the DTMF happened to have the three individuals that tend to interrupt. It may also be something different about the interaction that occurs for the DTMF and the STT modes of entry. One possibility is that STT respondents are reluctant to interrupt for fear that their answer will not be understood by the machine. There may be a tendency to wait for silence to make sure the computer is ready to receive the information.

The other type of interruption was during the response instruction. This happened 173 times. It occurred primarily when the DTMF gave the instruction to ‘press 1 for yes and press 2 for no’ (156 times). Respondents quickly picked up which keys mapped to yes and no (1 and 2 respectively). Seven of the 17 interruptions of the STT interruptions occurred at the marital status question (discussed above).

Recommendation: For the DTMF, we recommend reducing how often the instruction is provided by the system once the respondents appear to understand the response process. Otherwise, the interruptions of the instructions do not pose a serious threat to data quality and, in the case of interrupting the yes/no response categories, should be encouraged since it shortens the interview.

Interrupting the question may lead to respondents failing to process the full intent of the item. This type of problem did not occur for the vast majority of the respondents (17 of the 20). Our general assessment is that the three respondents that interrupted frequently would also interrupt as part of an interviewer-administered instrument.

If this becomes a concern in later rounds of testing, there are several possible remedies. The easiest to implement is to not accept answers until the question is completely read. This risks alienating respondents who either have something to report or are very impatient with the interview. For this reason, we are hesitant to implement this until there is more evidence that it is an issue. A second remedy would be to shorten the screening questions. This would involve eliminating more examples and cues from individual items. This would have to be done in close consultation with BJS.

3.5 Question Specific Results

Respondents did not demonstrate any difficulty with understanding the questions in the IVR instrument. Several questions were probed, but respondents overall reported a clear understanding of the questions. Two potential issues uncovered during probing are provided below.

“How many times..” One respondent assigned to the DTMF mode noted that one of the follow up questions was confusing. The question was ‘How many times did this incident happen?’ The respondent explained: “That question [is] confusing…[How many times] that I’ve had my purse stolen?…In how much of a period? In the past year?...I’ve had it stolen many times in my lifetime but just once in the past year...” She apparently did not carry over the reference period that was noted in the previous screening question. Despite being confused, the respondent did report for the past 12 months.

Recommendation. There are no recommended changes based on this one case. There was some concern that external telescoping might be an issue, given the length of the reference period and that it is unbounded. The NCVS-1 screener does not repeat the reference period at every question. In the above example, the respondent reported an incident that occurred in the last 12 months. During the debriefings, there were several respondents who said they did not report an incident because it occurred more than 12 months ago. This is some evidence that respondents are paying attention to the reference period. We will continue to monitor this in the next rounds of testing.

Alternate Question Versions. An alternate version of the first victimization question was included for the usability test. The purpose was to investigate whether a longer version of this question elicited different reactions in terms of perceptions of length and interruptions. The two versions are shown below. Version 1 is a longer version with all examples provided in one question. Version 2 uses examples distributed across three questions.

Version 1:

In the last 12 months, that is since {DATE} was something belonging to YOU stolen, such as: things you carry, like luggage, a wallet, purse, briefcase, or books; clothing, jewelry, or cellphones; a bicycle, or sports equipment; things in your home, like a TV, stereo, or tools; things outside your home, such as, a garden hose, or lawn furniture; things belonging to children in the household, or things from a vehicle, such as, a package, groceries, camera, or CD’s.

Version 2:

In the last 12 months, that is since {DATE} was something belonging to YOU stolen, such as, things you carry, like luggage, a wallet, purse, briefcase, or books; or things like clothing, jewelry, or a cellphone?

In the last 12 months, was something belonging to YOU stolen, such as, a bicycle or sports equipment, or things in your home like a TV, stereo, or tools; or things from a vehicle, such as a package, camera, or CD’s?

In the last 12 months, was something belonging to YOU stolen, such as, things outside your home, such as, a garden hose or lawn furniture; or things belonging to children in the household?

Of respondents assigned to the DTMF response mode, four received version 1, while the remaining six received version 2. Three respondents for each version (six total) interrupted to respond. For version 1, one respondent interrupted the question, while the other two interrupted the instruction to respond. For version 2, all three interruptions were during the response instruction. The interruption of the question for version 1 was an affirmative response (i.e. it was to report a victimization).

In the STT mode, three respondents were assigned to version 1, while the other seven received version 2. For Version 1, one respondent interrupted the question. As with the DTMF, this was a report of a victimization.

Recommendation. These data does not provide information to recommend one version over the other. Respondents were expected and instructed to interrupt either version of the question if they experienced a victimization that fit the examples. Both question interruptions were affirmative interruptions reporting a victimization. No respondent interrupted the question with a negative response. We will continue the two versions in the second round of testing to collect more data.

3.6 IVR System Help

Once respondents had completed the IVR and were debriefed, they were asked to go through the IVR again. The purpose of this was to have respondents use IVR system help. The help feature allowed respondents to have a question repeated or be able to back up one question in order to change an answer. When going through a second time, respondents were given a scenario describing a victimization and asked to respond as if this had occurred to them. Respondents were told they would be interrupted to test a function of the IVR. Respondents were not told what was being testing.

Finding

System help could be accessed by pressing the pound or hash key (#) in DTMF mode, and by saying ‘help’ in speech mode. The wording of the instruction differed slightly by entry mode:

DTMF

“Before I get started if at any time during this call you need help, for example to change an answer or replay a question press the pound key.”

STT

“Before I get started if at any time during this call you need help, for example to change an answer or replay a question say HELP.”

For respondents assigned to the DTMF (keypad) response mode, six respondents remembered what the IVR instruction was for requesting help. Of this six, one had actually used help during the initial pass through the IVR. Of the four respondents who did not know how to access help, one pressed the incorrect key, but knew to press a key to access help (pressed star instead of pound). The other three did not know what to do.

When respondents in the DTMF response mode were asked to use the help function by having a question repeated, no respondent displayed any difficulty using or navigating the help menu.

The help system was queried for 6 of the 10 STT respondents. Four of the six were able to recall what the IVR system instructed them to do for help. However, one of these four said she was not sure and guessed what to do. During the testing, we found that the help system for the STT did not work properly. Consequently, we could not actually watch respondents perform different functions (e.g., backing up a question) using the help instruction. This will be tested more thoroughly during the second phase of testing.

Recommendation. We do not recommend making any changes to the placement or wording of the help function at this time. We will monitor its use during Phase 2 to collect more data on whether the instruction needs to be changed or provided more frequently to users.

Appendix A

Usability Testing Protocol

NCVS-1 Debriefing Interview

PRESENT LETTER TO RESPONDENT AND ASK THEM TO GO THROUGH IT AS THEY WOULD AT HOME.

DISCUSS LETTER WITH RESPONDENT – FOCUS ON IF THEY NOTICED AND UNDERSTOOD RESPONDENT SELECTION

BEGIN IVR – FIRST PASS

MAKE NOTE OF:

ANY POINTS OF CONFUSION
ANY POINTS OF FRUSTRATION
ANY ERRORS (WRONG KEY/TIMEOUT)
ANY TIME HELP IS INITIATED IN THE IVR

ONCE IVR COMPLETED

We have some questions that we want to ask you, but before we do that, we would like you to tell us in your own words about what you just did – explain the process, what you did, how it worked, what kinds of questions you answered, etc.

REVIEW SECTIONS WHERE THERE WERE NOTICEABLE DIFFICULTIES WHILE GOING THROUGH THE INSTRUMENT.

REVIEW OTHER SECTIONS, NOT COVERED BY DIFFICULTIES, ASKING:

[PROBE AS NECESSARY TO GET THE RESPONDENT TALKING ABOUT HIS/HER EXPERIENCE OF THE IVR.]

You just said that you ~~~. Can you say more about that?

You mentioned the ~~~~ ; how did that work for you?

IF IT IS NOT CLEAR RESPONDENT UNDERSTOOD QUESTION, FOLLOW-UP WITH:

In this question, you were asked ~~~~~~. Can you tell me in your own words what this question means to you?

In this question, what does ~~~~~~ mean to you?

Cover the following questions (NOTE RESPONSES DURING IVR INTERVIEW):

HCR0400 – What was the TOTAL number of cars, vans, trucks, motorcycles, or other motor vehicles owned by you or any other member of this household during the last 12 months? Include those you no longer own.

HCR0550– During the last 12 months, other than any incidents already mentioned, were you attacked or threatened or did you have something stolen from you at home or near a friend’s, relative’s, or neighbor’s home?

HCR0590– Were you attacked or threatened or did you have something stolen from you in places such as a mall, or restaurant, or other places you go for entertainment or recreation such as a party, theater, or gym?

HCR0700– Other than any incidents already mentioned, has anyone attacked or threatened you with a weapon, such as a gun or a knife, scissors or anything that could be thrown, such as a rock or a bottle?

HCR0740– Other than any incidents already mentioned, has anyone attacked or threatened you with rape, attempted rape or any other sexual attack?

HCR0850– People often don’t think of incidents committed by someone they know. Other than any incidents already mentioned, did you have something stolen from you OR were you attacked or threatened by someone at work or school, a neighbor or friend, a relative or family member, or any other person you’ve met or known?

HCR0950 – Incidents involving forced or unwanted sexual acts are often difficult to talk about. Other than any incidents already mentioned, have you been forced or coerced to engage in unwanted sexual activity by someone you didn’t know before, a casual acquaintance, or someone you know well?

ONLY IF HCR1050 = 1 (YES)

HCR1070 – Were you attacked or threatened, or was something stolen or an attempt was made to steal something that belonged to you or another household member?

ONCE COMPLETING THE INSTRUMENT, ASK (IF NOT ALREADY DISCUSSED)

Can you tell us whether you thought it was interesting or not so interesting?

Was there anything that you particularly liked or disliked about the IVR?

Was there anything particular that worked well for you or did not work well for you?

If you got a call like this at home, is this something you would choose to do or choose not to do?

BEGIN IVR – SECOND PASS – PROVIDE CRIME SCENARIO

TELL RESPONDENT YOU WILL INTERRUPT TO ASK THEM HOW THEY WOULD HANDLE A FEATURE OF THE IVR

AFTER RESPONDENT ANSWERS FOR SCENARIO – INTERRUPT AND ASK:

What would you do if you needed to hear this question again?

IF RESPONDENT DID NOT KNOW HOW TO ACCESS HELP, TELL THEM TO PRESS ‘#’ TO ACCESS HELP.

PROBE ON HOW THEY FEEL THE HELP MENU WORKED FOR THEM – COULD THEY UNDERSTAND IT.

Do you have any other thoughts or comment about the IVR?

Appendix B

Consent Form

Informed Consent Script - NCVS

Thank you for your interest in helping us test some systems for a study we are conducting for the Bureau of Justice Statistics (BJS). BJS collects data on the amount of crime and victimization that is occurring in the US. BJS collects this information with a survey, called the National Crime Victimization Survey (NCVS). Westat is assisting BJS in redesigning the NCVS. We are testing an Interactive Voice Response system for the survey. IVR is a computer system that will ask you questions over the telephone and ask that you give your answers over the telephone.

Today we will ask you to go through the IVR version of this survey. The questions on the survey will ask about crimes that you may or may not have experienced. This is a research project and your participation is voluntary. You can skip any question and you can stop at any point. Everything covered today will be treated as confidential. Your name will never appear in survey data or any of the results described in the report.

The IVR survey will take about 30 minutes to complete and we then would like to talk to you about your experience with this system. We expect the total amount of time this will take will not be more than 60 minutes. As a token of our appreciation, we have $40 cash for you.

If you have any questions about this study, you can call Pat Dean Brick, at 301-301-517-4196. We would also like to audio record both your interaction with the computerized system and the conversation that follows. Do we have your permission to record?

_____________________________________________ ___________________

Signature Date

Appendix C

Crime Scenarios

Scenario 1

In January, your next-door neighbor had a party that was very loud and disruptive. You went to your neighbor to complain. Your neighbor called you a whiner and punched you in the face.

Scenario 2

In January, someone broke into your car and stole your car radio/CD player and your GPS. Your car was parked in your driveway. You reported it to your insurance company and to the police.

Scenario 3

In January, you were at a service counter trying to rent a car. You put your cell phone and sun glasses on the counter right by where you were standing. When you turned around to go, you saw that your sun glasses and your cell phone were no longer there.

Scenario 4

In January, you were on a week-long business trip. When you arrived home, you saw that your front door had been broken open. When you walked in, a young kid ran up to you, knocked you down and ran out of the house. You discovered that about $5,000 worth of electronic equipment was missing and $1000 in cash. You immediately reported it to the police. You did not suffer any injuries from being knocked down.

Appendix D

Invitation Letter

Dear Maryland Resident:

Your household has been selected for a study being conducted by the U.S. Bureau of Justice Statistics (BJS), an agency within the U.S. Department of Justice. BJS is the primary source of information on crime and victimization. For example, in 2009 BJS found there were approximately 20 million victimizations and less than half of these were reported to the police. ^²

Your participation in this study will help BJS find the best ways to collect data like this and help the government monitor and control crime in communities like yours

Please ask the person who lives at this address who is at least 18 years old and who has the next birthday to complete the automated survey by calling the number below. Once calling, please enter the unique survey ID:

1-888-715-8721 ID: 90004613

Even if the person selected to participate has not experienced a crime, it is important that your household is represented on the survey. As a token of our appreciation, we will send $20 to your household once the survey is completed.

Completing this survey is voluntary. In order to include people with a wide variety of experiences in Maryland, we obtained your address from one of two sources: a) a sample of all addresses in the U.S. or b) a sample of addresses in Maryland where someone had reported a crime. All the information provided is confidential by federal law (42 USC 3789g). If you do not want to participate on the study, please call 1-888-251-1500 and provide the ID number listed above. We will not contact you again.

Westat, a private research firm, is conducting this study for us. If you have any questions, please call 1-800-251-1500.

We thank you in advance for helping with this study.

Sincerely,

Michael Rand

Director, Victimization Unit

US Bureau of Justice Statistics

http://bjs.ojp.usdoj.gov/

Some Frequently Asked Questions about the Crime Victimization Study

Q: What is the study about? What kind of questions will you be asking?

A: The study concerns victimizations you may have experienced. This includes those incidents that may not have been reported to law enforcement. For example, we will ask you about any crimes you may have experienced or witnessed and some general details about the incident. These may include what happened, if anything was stolen, if anyone was injured, or if police were notified.

Q: How will the study results be used? What will be done with my information?

A: This is a pilot study that will help the U.S. Bureau of Justice Statistics improve the way they collect information about characteristics of criminal victimization including those not reported to law enforcement.

Q: How did you get my address?

A: Your address came from one of two different sources. One was a random selection of addresses among all addresses in your community. The second source was a random selection of addresses that had reported a crime to the local police within the last 12 months.

Q: Why should I take part in this study?

A: Crime and the victims of crime are important issues for our Nation. A greater understanding of crime can be used in crime prevention measures. Groups that represent victims also need hard data on how crimes are perpetrated.

You represent thousands of other households like yours, and you cannot be replaced. Your answers and opinions are very important to the success of this study, as you represent others who share your knowledge and experiences.

Q: Do I have to do this?

A: Your participation is voluntary, and you may refuse to answer any questions. You can stop participating at any point. However, your participation is very important to the success of this study and will help the government improve the way they monitor crime in communities like yours.

Q: Will my answers to the survey be kept confidential?

A: Yes. Your individual answers will not be revealed to anyone but the researchers in a way that identifies you or your household. Your individual answers are protected by federal law and cannot be shared with anyone outside the individuals working on this project.

Q: How long will it take to answer the questions?

A: About 15 to 20 minutes.

Q: Who is sponsoring the study? Is this study approved by the Federal Government?

A: The study is sponsored by the U.S. Bureau of Justice Statistics, which an agency within the U.S. Justice Department. The study has been approved by the Office of Management and Budget (OMB), the office that reviews all federally-sponsored surveys. The OMB approval number assigned to this study is XXXX-XXXX.

Q: I called the number, but it only goes to an automated system.

A: We are testing the use of a new method for collecting information for this survey that offers you additional privacy when answering the survey questions. If you are having difficulty accessing the survey or it is not recognizing your responses, please call 1-800-251-1500 for additional assistance.

Appendix E

IVR SCREENER

IVR-NCVS-SCREENER_1/24/2011:

[THIS SECTION IS FOR INBOUND CALLS ONLY.]

IBN0100 INBOUND INTRODUCTION

IBN0101 – “Thank you for calling the National Crime Victimization Survey. If you received a letter inviting you to participate in this survey please have the ID number from the invitation letter ready. You must be an 18 year-old adult living in the household the invitation letter was sent to participate in this survey. Do you have your ID number ready?”

YES – GO TO IBN0120

NO – GO TO IBN0110

IBN0110 LOCATE ID ON INVITATION LETTER

IBN0111 – “The ID number is eight digits long and located near the top of the letter you received. Are you able to find your ID?”

YES – GO TO IBN0120

NO – GO TO IBN0150

IBN0120 ENTER ID TO ACCESS QUESTIONNAIRE

IBIN0121 – “Please tell me your ID number.

|__|__|__|__|__|__|__|__| ID NUMBER

IBN0160 CONFIRM ID ENTRY

IBN0161 – “Just to confirm, the ID you entered is {ID NUMEBR}. Is this correct?”

YES – GO TO IBN0200

NO – GO TO IBN0120

IBN0200 SAMPLING VERIFICATION

IBN0201 – “The invitation letter for this survey instructed the person with the next birthday to call to complete this survey. Are you the household member whose birthday is next?”

IBN0210 COLLECT NAME FOR RECONTACT

IBN0211 – “Just in case you need to call back to complete your survey, please tell me your first name. The system will begin recording, when you are finished, press the star key.”

|__|__|__|__|__|__|__|__| NAME

IBN0220 NUMBER OF PEOPLE IN HOUSEHOLD

IBN0221 – “Before we get started, I would like to ask you a couple questions about who lives in your household. What is the total number of people who live at this address?” Include everyone who lives at this address, including people who are not related to you and any young children or babies.”

|__|__| TOTAL HH MEMBERS

IBN0230 NUMBER OF PEOPLE IN HOUSEHOLD 18 AND OLDER

IBN0231 – “What is the total number of people at this address who are age 18 or older? Include only people age 18 or older who live at this address, including people who are not related to you.”

|__|__| HH MEMBERS 18+

GO TO HHD0100 – PAGE 3

IBN0320 CONTACT SCRIPT FOR INTERRUPTING INTERVIEW DUE TO RESPONDENT DIFFICULTY

IBN0321 – “Our system appears to be experiencing some technical difficulties. Please call at another time to complete your survey. Good-bye.”

END CONTACT PROCEDURES

HHD0100 RESPONDENT HELP INTRODUCTION

HHD0101 – “The next questions are about you. Before I get started if at anytime during this call you need help, for example to change an answer or replay a question say HELP.”

HHD0110 AGE OF HOUSEHOLD RESPONDENT

HHD0111 – “What is your age?”

|__|__| AGE

HHD0140 RESPONDENT MARITAL STATUS

HHD0141 – “What is your current marital status? Are you married, widowed, divorced, separated, or never married?”

MARRIED
WIDOWED
DIVORCED
SEPARATED
NEVER MARRIED

HHD0150 RESPONDENT GENDER

HHD0151 – “Are you male or female?”

MALE

FEMALE

HHD0160 RESPONDENT EDUCATION – HIGH SCHOOL

HHD0161 – “These next questions ask about your highest grade or level of school completed. Did you graduate high school or receive a GED?”

YES – GO TO HHD0170

NO – GO TO HHD0210

HHD0170 RESPONDENT EDUCATION – TWO YEAR DEGREE

HHD0171 – “Have you completed at least 2 years of college or an associate’s degree?”

YES – GO TO HHD0180

NO – GO TO HHD0210

HHD0180 RESPONDENT EDUCATION – FOUR YEAR DEGREE

HHD0181 – “Have you completed a four-year college degree?”

YES – GO TO HHD0190

NO – GO TO HHD0210

HHD0190 RESPONDENT EDUCATION – SOME GRADUATE EDUCATION

HHD0191 – “Have you attended graduate or professional school?”

YES – GO TO HHD0200

NO – GO TO HHD0210

HHD0200 RESPONDENT EDUCATION – GRADUATE/PROFESSIONAL DEGREE

HHD0201 – “Have you completed a graduate or professional degree?”

HHD0210 CURRENTLY ENROLLED IN SCHOOL OR TAKING CLASSES

HHD0211 – “Are you currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?”

HHD0220 RESPONDENT ETHNICITY

HHD0221 – “Are you of Hispanic, Latino, or Spanish origin?”

YES – HISPANIC OR LATINO

NO – NOT HISPANIC OR LATINO

HHD0230 RESPONDENT RACE - WHITE

HHD0231 – “What is your race? Please select one or more. Are you White?

HHD0240 RESPONDENT RACE - BLACK

HHD0241 – “Are you Black or African American?

HHD0250 RESPONDENT RACE – AMERICAN INDIAN/ALASKA NATIVE

HHD0251 – “Are you American Indian or Alaska Native?”

HHD0260 RESPONDENT RACE – ASIAN

HHD0261 – “Are you Asian?”

HHD0270 RESPONDENT RACE – NATIVE HAWAIIAN OR PACIFIC ISLANDER

HHD0271 – “Are you Native Hawaiian or other Pacific Islander?”

HHD0280 RESPONDENT RACE – OTHER

HHD0281 – “Are you some other race?”

YES – GO TO HHD0170

NO – GO TO HHD0210

HHD0290 TENURE – IS HOME OWNED

HHD0291 – “Is this house owned or being bought by someone in this household?”

YES – GO TO HHD0320

NO – GO TO HHD0300

HHD0300 TENURE – IS HOME RENTED

HHD0301 – “Is this house rented by someone in this household?”

YES – GO TO HHD0320

NO – GO TO HHD0310

HHD0310 TENURE – IS HOME OCCUPIED BY SOME OTHER ARRANGEMENT

HHD0311 – “Is this house occupied by some other arrangement?”

HHD0320 HOW MANY YEARS LIVED AT THIS ADDRESS

HHD0321 – “How many years have you lived at this address?” If less than 1 year say zero.”

|__|__| - NUMBER OF YEARS

IF 5 YEARS OR LESS GO TO HHD0330

IF MORE THAN 5 YEARS GO TO HCR0075

HHD0330 HOW MANY TIMES MOVED IN LAST 5 YEARS

HHD0331 – “Altogether, how many times have you moved in the last 5 years, that is, since {CURRENT MONTH/YEAR – 12 MONTHS}.”

|__|__| YEARS

GO TO HCR0075 – NEXT PAGE

HCR0075 HH SCREENER INTRODUCTION CRIME CONTENT

HCR0076 – “Next, you’re going to hear some examples that will give you an idea of the kinds of crimes this study covers. Please say ‘yes’ when you think of anything, otherwise, please wait until the end of the list to answer. Include any crimes even if it doesn’t fit the example you hear.”

GO TO BOX HCR01

BOX HCR01

CHECK ITEM:

QUESTION EXPERIMENT (THEFT)

IF VERSION 1 GO TO HCR0100

IF VERSION 2 GO TO HCR0110 – NEXT PAGE

HCR0100 WAS SOMETHING BELONGING TO R STOLEN 1

HCR0101 – “In the last 12 months, that is since {CURRENT MONTH/YEAR – 12 MONTHS}, was something belonging to YOU stolen, such as:

things you carry, like luggage, a wallet, purse, briefcase, or books;
clothing, jewelry, or cellphones;
a bicycle, or sports equipment;
things in your home, like a TV, stereo, or tools;
things outside your home, such as , a garden hose, or lawn furniture;
things belonging to children in the household, or
things from a vehicle, such as, a package, groceries, camera, or CD’s.”

YES – GO TO HCR0200 – PAGE 9

NO – GO TO HCR0150 – NEXT PAGE

HCR0110 WAS SOMETHING BELONGING TO R STOLEN 1

HCR0111 – “In the last 12 months, that is since {CURRENT MONTH/YEAR – 12 MONTHS}, was something belonging to YOU stolen, such as, things you carry, like luggage, a wallet, purse, briefcase, or books; or thinks like clothing, jewelry, or a cellphone?”

YES – GO TO HCR0200 – NEXT PAGE

NO – GO TO HCR0120

HCR0120 WAS SOMETHING BELONGING TO R STOLEN 2

HCR0121 – “In the last 12 months, was something belonging to YOU stolen, such as, a bicycle or sports equipment, or things in your home like a TV, stereo, or tools; or things from a vehicle, such as a package, camera, or CD’s?”

YES – GO TO HCR0200 – NEXT PAGE

NO – GO TO HCR0140

HCR0140 WAS SOMETHING BELONGING TO R STOLEN 3

HCR0141 – “In the last 12 months, was something belonging to YOU stolen, such as, things outside your home, such as, a garden hose or lawn furniture; or things belonging to children in the household?”

YES – GO TO HCR0200 – NEXT PAGE

NO – GO TO HCR0150

HCR0150 ATTEMPTED THEFT FROM R

HCR0151 – “In the last 12 months, did anyone attempt to steal anything belonging to you?”

YES – GO TO HCR0200 – NEXT PAGE

NO – GO TO HCR0300 – PAGE 10

HCR0200 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0201 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0220 RECORDING OF INCIDENT

HCR0221– “Please describe what happened during this incident.”

HCR0222 – “I will begin recording, when you are finished, press the star key.”

HCR0300 ILLEGAL ENTRY OF HOME

HCR0301 – “Other than any incidents already mentioned, has anyone broken in or attempted to break into your home?”

YES – GO TO HCR0360

NO – GO TO HCR0320

HCR0320 ILLEGAL ENTRY OF GARAGE OR SHED

HCR0321 – “Has anyone illegally gotten in or tried to get into a garage, shed, or storage room?”

YES – GO TO HCR0360

NO – GO TO HCR0340

HCR0340 ILLEGAL ENTRY OF HOTEL/VACATION HOME

HCR0321 – “Has anyone illegally gotten in or tried to get into a hotel or motel room or vacation home where you were staying?”

YES – GO TO HCR0360

NO – GO TO HCR0400 – NEXT PAGE

HCR0360 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0361– “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0380 RECORDING OF INCIDENT

HCR0381 – “Please describe what happened during this incident.”

HCR0382 – “I will begin recording, when you are finished, press the star key.”

HCR0400 TOTAL NUMBER OF MOTOR VEHICLES

HCR0401 – “What was the TOTAL number of cars, vans, trucks, motorcycles, or other motor vehicles owned by you or any other member of this household during the last 12 months? Include those you no longer own.”

0 (ZERO) – GO TO HCR0550 – NEXT PAGE

1 OR MORE – GO TO HCR0420

HCR0420 WAS VEHICLE STOLEN OR USED WITHOUT PREMISSION

HCR0421 – “During the last 12 months, other than any incidents already mentioned, {was the vehicle / were the vehicles} stolen or used without permission?”

YES – GO TO HCR0480

NO – GO TO HCR0440

HCR0440 WERE PARTS STOLEN FROM VEHICLE

HCR0441 – “Did anyone steal any parts such as a tire, car stereo, hubcap, or battery, or steal any gas from a vehicle?”

YES – GO TO HCR0480

NO – GO TO HCR0460

HCR0460 WAS THERE AN ATTEMPT TO STEAL VEHICLE OR PARTS

HCR0461 – “Did anyone attempt to steal any vehicle or parts attached to a vehicle?”

YES – GO TO HCR0480

NO – GO TO HCR0550 – NEXT PAGE

HCR0480 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0481 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0500 RECORDING OF INCIDENT

HCR0501 – “Please describe what happened during this incident.”

HCR0502 – “I will begin recording, when you are finished, press the star key.”

HCR0550 ATTACKS AT HOME / FRIEND’S / NEIGHBOR’S

HCR0551 – “During the last 12 months, other than any incidents already mentioned, were you attacked or threatened or did you have something stolen from you at home or near a friend’s, relative’s, or neighbor’s home?”

YES – GO TO HCR0650 – NEXT PAGE

NO – GO TO HCR0570

HCR0570 ATTACKS AT WORK OR SCHOOL

HCR0571 – “Were you attacked or threatened or did you have something stolen from you at work or school?”

YES – GO TO HCR0650 – NEXT PAGE

NO – GO TO HCR0590

HCR0590 ATTACKS AT PLACES OF ENTERTAINMENT

HCR0591 – “Were you attacked or threatened or did you have something stolen from you in places such as a mall, or restaurant, or other places you go for entertainment or recreation such as a party, theater, or gym?”

YES – GO TO HCR0650 – NEXT PAGE

NO – GO TO HCR0610

HCR0610 ATTACKS AT ON STREET OR IN VEHICLE

HCR0611 – “Were you attacked or threatened or did you have something stolen from you on the street, or while riding in any vehicle?”

YES – GO TO HCR0650 – NEXT PAGE

NO – GO TO HCR0630

HCR0630 ATTEMPTED ATTACHS OR ATTEMPTED THEFTS

HCR0631 – “Did anyone attempt to attack or attempt to steal anything belonging to you from any of these or other places?”

YES – GO TO HCR0650 – NEXT PAGE

NO – GO TO HCR0700 – PAGE 14

HCR0650 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0651 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0670 RECORDING OF INCIDENT

HCR0671 – “Please describe what happened during this incident.”

HCR0672 – “I will begin recording, when you are finished, press the star key.”

HCR0700 ATTACKS WITH WEAPONS

HCR0701 – “Other than any incidents already mentioned, has anyone attacked or threatened you with a weapon, such as a gun or a knife, scissors or anything that could be thrown, such as a rock or a bottle?”

YES – GO TO HCR0760

NO – GO TO HCR0720

HCR0720 ATTACKS WITH WEAPONS

HCR0721 – “Has anyone made any face-to-face threats or attacked or threatened you by grabbing, punching, or choking you?”

YES – GO TO HCR0760

NO – GO TO HCR0740

HCR0740 ATTACKS WITH THREAT OF RAPE

HCR0741 – “Has anyone attacked or threatened you with rape, attempted rape or any other sexual attack?”

YES – GO TO HCR0760

NO – GO TO HCR0850 – NEXT PAGE

HCR0760 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0761 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0780 RECORDING OF INCIDENT

HCR0781 – “Please describe what happened during this incident.”

HCR0782 – “I will begin recording, when you are finished, press the star key.”

HCR0850 ATTACKS/THEFTS BY KNOWN OFFENDER

HCR0851 – “People often don’t think of incidents committed by someone they know. Other than any incidents already mentioned, did you have something stolen from you OR were you attacked or threatened by someone at work or school, a neighbor or friend, a relative or family member, or any other person you’ve met or known?”

YES – GO TO HCR0870

NO – GO TO HCR0950 – NEXT PAGE

HCR0870 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0871 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0890 RECORDING OF INCIDENT

HCR0891 – “Please describe what happened during this incident.”

HCR0892 – “I will begin recording, when you are finished, press the star key.”

HCR0950 HAS R BEEN SEXUALLY ASSAULTED

HCR0951 – “Incidents involving forced or unwanted sexual acts are often difficult to talk about.”

HCR0952 – “Other than any incidents already mentioned,”

HCR0953 – “have you been forced or coerced to engage in unwanted sexual activity by someone you didn’t know before, a casual acquaintance, or someone you know well?”

YES – GO TO HCR0970

NO – GO TO HCR1050 – NEXT PAGE

HCR0970 HOW MANY TIMES DID INCIDENT HAPPEN

HCR0971 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR0990 RECORDING OF INCIDENT

HCR0991 – “Please describe what happened during this incident.”

HCR0992 – “I will begin recording, when you are finished, press the star key.”

HCR1050 DID R CALL POLICE TO REPORT SOMETHING

HCR1051 – “During the last 12 months,”

HCR1052 – “other than any incidents already mentioned,”

HCR1053 – “did you call the police to report something that happened to you which you thought was a crime?”

YES – GO TO HCR1070

NO – GO TO HCR1150 – NEXT PAGE

HCR1070 DID ATTACK, THREAT, OR THEFT OCCUR FROM POLICE CALL

HCR1071 – “Were you attacked or threatened, or was something stolen or an attempt was made to steal something that belonged to you or another household member?”

YES – GO TO HCR1090

NO – GO TO HCR1150 – NEXT PAGE

HCR1090 HOW MANY TIMES DID INCIDENT HAPPEN

HCR1091 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR1110 RECORDING OF INCIDENT

HCR1111 – “Please describe what happened during this incident.”

HCR1112 – “I will begin recording, when you are finished, press the star key.”

HCR1150 DID SOMETHING HAPPEN, BUT DID NOT CALL POLICE

HCR1151 – “During the last 12 months,”

HCR1152 – “other than any incidents already mentioned,”

HCR1153 – “did anything which you thought was a crime happen to you, but you did not report to the police?”

YES – GO TO HCR1170

NO – GO TO HCR4010 (END)

HCR1170 DID ATTACK, THREAT, OR THEFT OCCUR DURING INCIDENT

HCR1171 – “Were you attacked or threatened, or was something stolen or an attempt was made to steal something that belonged to you or another household member?”

YES – GO TO HCR1190

NO – GO TO HCR4010 (END)

HCR1190 HOW MANY TIMES DID INCIDENT HAPPEN

HCR1191 – “How many times did this type of incident happen?”

|__|__| NUMBER OF TIMES

HCR2110 RECORDING OF INCIDENT

HCR2111 – “Please describe what happened during this incident.”

HCR2112 – “I will begin recording, when you are finished, press the star key.”

HCR4010 STANDARD END OF CONTACT SCRIPT

HCR4011 – “Thank you for answering these questions. You have completed the interview.”

HCR4012 – “Thanks for your time. Good-bye.”

1 This person was mistakenly recruited because he had experienced a victimization more than 12 months prior to the interview.

2 Truman, J.I. and M. Rand (2009) “Criminal Victimization, 2009” Bureau of Justice Statistics, Office of Justice Programs, U.S. Department of Justice, NCJ 231327

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Behavioral Risk Factor Surveillance System (BRFSS)
Author	Jennifer Crafts
File Modified	0000-00-00
File Created	2021-02-03

Usability and Cognitive Findings

Attachment 16 Usability and Cognitive Findings.docx

Research to support the National Crime Victimization Survey (NCVS)