Project Summary

Attachment 1

Project Summary

Development and Testing an Interactive Voice Response Questionnaire

for the National Crime Victimization Survey

Project Summary

This project will assess the feasibility of using Interactive Voice Response (IVR) as a mode of data collection for the National Crime Victimization Survey (NCVS). One of the primary motivations for the redesign of the NCVS is reduction of costs by moving to modes that are less expensive than in-person interviews. With respect to alternative, less expensive, modes of collection (e.g., telephone, web, paper self-administered), the IVR has several advantages. One is that it can be used by the vast majority of the population. It is estimated that only around 1% of the population does not have access to a telephone (Blumberg and Luke, 2009). Unlike a web survey, therefore, most individuals can use an IVR. A second positive feature of IVR is that is does not require extensive reading ability. The survey relies entirely on speech to communicate. This differs from web and a paper-mail survey, where reading is a necessity. Third, the IVR is a self-administered mode. This should increase the likelihood respondents will report sensitive information (Bloom, 2008; Couper, et al., 2004; Krueter, et al., 2008; Turner, Miller, Smith, Cooley, and Rogers, 1996). Combine these advantages with lower costs, an IVR offers some promise for the NCVS.

The proposed project will address the feasibility of using an IVR by examining six research questions.

1 To what extent does the NCVS interview need to be modified to be acceptable for IVR?

The NCVS includes items that take advantage of interviewer feedback and judgment. Developing an IVR version of this instrument will require significant adaptation. This project will develop an IVR version of the NCVS that abbreviates the information that is collected, while at the same time gathering what is needed for classification of the event into the type of crime classification needed for estimation.

2. What is the best mode to initiate contact with sampled respondents – mail or telephone?

Application of an IVR to the NCVS might take one of two forms. An outbound model might be used with a telephone interviewer. The initial contact and items on household characteristics would be administered by the interviewer and the IVR would be used to administer all or some of the victimization items. A second method is to contact respondents by mail and ask them to call into the IVR system to take the survey. This project will compare these two methods along several different dimensions, including response rate, completeness of data and user satisfaction.

3. Is it possible to effectively encourage sampled households to complete the interview in a self-administered mode?

One of the drawbacks of an IVR is motivating respondents to complete the survey. The project will test several ways to improve response to the survey request. One method will use an insert that contains a short message that motivates the respondent to participate. This approach was found to be very successful in a recent mail survey of Veterans (Han, et al., 2010). The second method will use incentives.

4: Are there differences in respondent acceptance between speech IVR and touchtone data entry (TDE) IVR?

The project will compare the use of speech and TDE IVR applications. It is not clear from the usability literature on which of these methods of data entry is best for a general population survey. This project will assess the advantages and disadvantages of these two approaches with respect to outcomes such as the response rate, user satisfaction and the collection of different types of victimization outcomes.

5: Does IVR lead to different victimization rates from a telephone interview?

The project will compare the victimization rates for telephone and IVR modes of interviewing. Based simply on the mode of communication, the IVR offers more privacy and anonymity than the telephone. On the other hand, a telephone interviewer may provide useful prompts, definitions and clarifications that assist the respondent when retrieving information from memory and formulating responses. Although limited by sample size, the project will compare victimization rates for different types of crimes.

6: Is there a difference in victimization rates for Speech and IVR modes of entry?

The mode of data entry may be related to the perceived anonymity and privacy of the response. Speaking a response, even if it is only a “yes” or “no” may be perceived as less private than entering the information onto a keypad. The sample sizes will be limited with respect to testing this hypothesis (as with hypothesis 5 above). However, it may be possible to detect large effect for the more common type of crimes.

Methods

To investigate the above research questions, the study will develop an IVR version of the NCVS interview and conduct a field test which compares two different versions of the IVR (touchtone data entry vs speech recognition) and a telephone (CATI) interview. In addition, the study will compare several different methods to increase participation on the survey, including incentives and the use of targeted communication material included in the mailouts.

Developing the IVR

The overall goal for development of the IVR instrument is to ensure that it functions in a way that encourages and aids accurate response. The IVR should be designed to place minimum burden on respondents and assist them in navigating through the questionnaire.

The goals of the IVR development will be to: 1) Identify questions that may be too long, or to cognitively burdensome for respondents, 2) Ensure that respondents are given instruction when they need it, 3) Identify how respondents interact with an automated IVR system, and 4) Identify how respondents handle problems. The usability testing will consist of a variety of different protocols delivered across a total of 50 respondents. The testing will be divided into three major components. The first is “Wizard of Oz”, where a person will mimic the functionality of the IVR. This step allows the designers to assess any major problems before the IVR is actually programmed. The second component is to test the programmed system with respondents. These will be two rounds of usability tests, with the first round examining the screener and the second round examining the entire instrument (screener and detailed incident form) . The third component is a very small test which administers the survey to 10 respondents under actual survey conditions (i.e., in their own homes, without being directly observed by project staff).

Developing Contact Materials

An important element of motivating individuals to participate on the survey is legitimizing the survey (Dillman, et al., 2009). The project will develop communication material that will be used as the advance and cover letters for both the telephone and mail contacts. As part of this development, the project will conduct 2-3 focus groups to review different versions of the contact material. Different approaches to communicating the message of the survey will be shown to the different groups in order to focus and format the material in ways that communicate both the legitimacy and importance of the NCVS. These materials will also be used to provide respondents with specific instructions for the IVR methodologies, such as how to access the survey. The groups will provide feedback on different approaches to communicating these instructions.

Field Test

There will be five experimental factors that will be part of the design for the field test (Table 1). One factor will compare the methods of initial contact and recruitment. One portion of the sample will be contacted by a telephone interviewer and another portion will be contacted by mail. Within each of these methods of contact, there will be experimental comparisons. For the telephone contact, the mode of interview will vary between IVR with speech recognition (STT or speech to text), IVR with touch tone data entry (TDE) and a telephone interview. For the mail contact, two experimental factors will be tested, both designed to encourage respondent participation. One will be an incentive experiment testing whether a promise of $20 would increase response. The second is the use of an insert designed to encourage participation.

Table 1. Experimental Design

Telephone Contact (n=3000)	Mail Contact – IVR with STT (n=10,000)*
1a IVR – Speech to Text (n=1000)	2. Promised Incentive
	a. $0
1b IVR – Touch Tone Data Entry (n-1000)	b. $20
1c Telephone Interviewer (n=1000)	3. Use of an insert
	a. Yes
	b. No

* The promised incentive and insert will be fully crossed. Each combination of incentive and insert will have n=2500.

The sample for the study will be drawn from two different sources. One will rely on getting permission from a police department to use records of victims that had reported a crime to the police. It would be desirable to make a direct connection between the police records and interviewed respondents to assist in validation of the interviews. However, if this is not feasible for human subject concerns, records would be used to seed the sample in order to increase the likelihood that respondents would report a victimization. The second source of data will be the USPS delivery sequence file (DSF). This frame contains all addresses the US Postal Service delivers mail. A sample will be designed which oversamples areas with high crime rates, as indicated by local police information.