All Other Supporting Documentation

Appendix B. Summary of Mail Questionnaire and Testing

Development and Cognitive Testing of
Self-administered NCVS Companion Survey Instruments

In the first phase of research of the NCVS Companion Survey (CS), we attempted to use the existing NCVS instruments in a computer-assisted telephone interview (CATI) environment using an address-based sample (ABS), with the goal of producing small area estimates (SAEs) through blending with data from the core NCVS. The pilot study in the Chicago MSA also included a mail screener, primarily to obtain telephone numbers for addresses where a number could not be obtained from our sample vendor. Based on the results of this research we have concluded that it is extremely difficult, if not impossible, to replicate NCVS estimates of victimization rates using a low-cost data collection approach. The NCVS is a large and complex survey, with many potential sources of relative bias compared with low-cost alternative data collection approaches, including nonresponse, mode effects, and house effects in data collection and processing. It does not seem feasible to control for all of these differences in a low-cost vehicle, regardless of the sample design or data collection mode(s). NCVS estimates of victimization rates are very sensitive to many of these factors, so estimates may change substantially when even small deviations occur in the survey process. Truman and Planty (2012) describe how the victimization estimates in the core NCVS changed when the sample size increased and new interviewers were needed; Rand (2008) reviews some effects when the sampled geographic areas changed and the data collection software was revised. Complete details on the initial phase of research are provided in the report of the Pilot research findings.

In the next phase of research we have developed a low-cost, self-administered approach that can support sub-national estimates of crime victimization, using a less complex instrument than the current NCVS. The goal is to create a survey that would parallel some NCVS and Uniform Crime Report (UCR) victimization estimates over time or across jurisdictions, rather than replicate either of these sources, and that could be used to assess the impact of local initiatives over time. Among the positive findings from the Chicago pilot test were (1) relatively high response to the mail screener and (2) good agreement between the mail screener and the CATI interview on whether the household had experienced a crime in the previous 12 months. Building on these positive results, the questionnaires we have developed are to be administered by mail, relying on a household informant to report experiences with crime for all adult household members. The questionnaires also include “non-crime” opinion questions on neighborhood safety and police performance. We adapted probes to identify crimes and more detailed questions about crime characteristics from the NVCS. The two primary changes in the survey design are (1) allowing proxy response for adults other than the household member completing the survey and (2) simplifying and reducing the length of the questionnaire to increase reliability and reduce burden. Both of these changes are potentially problematic.

Proxy response. While household-level response rates for the Chicago CATI pilot test were unacceptably low, response for sampled adults other than the household informant was much lower still. A mail questionnaire seems unlikely even to do as well as the pilot in obtaining self-responses from multiple adults in a household. For a mail questionnaire to succeed, it must allow proxy response. Previous research on the NCVS has questioned the accuracy and completeness of proxy response (Lehnen and Skogan, 1981). One of the major questions for testing a mail NCVS Companion Survey is whether error attributable to proxy reporting will be within acceptable limits. Concern about proxy reporting in the traditional NCVS incident-level design has led us to develop a person-level questionnaire oriented around individuals rather than incidents of crime, as well as an incident-level version more similar to the current NCVS.

Proxy response is likely to result in significant bias in estimates of domestic violence, including intimate partner violence. On the one hand, household respondents may be more likely to report their own such victimizations than they would be in the NCVS; self-administered questionnaires often elicit more reports of sensitive subjects than do interviewer-administered questionnaires. On the other hand, reports of such victimizations for other adults in the household may be less likely than in the NCVS, since the respondent may not be aware of such incidents or may even be the offender.

Simplifying the instruments. One of the factors making the NCVS-2 (Incident Report) in particular long and complex is the need to collect sufficient information to classify crimes. The Census classification process is also complex, and difficult to replicate. To reduce the length of the questionnaire, and to make the classification process more replicable and reliable, we have reduced the number of type-of-crime (TOC) codes that we plan to use for estimation. One of the major questions for testing a self-administered Companion Survey is whether it can reliably support crime classification at this higher level, or perhaps at a yet higher level.

Estimation. Both of the mail questionnaires will support victimization estimates at the household and person levels as discussed in the final section of this document. Household-level estimates will also include attitudes towards neighborhood safety and police performance as well as the proportion of households experiencing crimes by type during a 12-month reference period. Both instruments will also support similar estimates of victimization at the person level for personal crime. All of these victimization estimates may be compared with those from the core NCVS, and most estimates from the two test versions may be compared with each other. Only the incident-level instrument will support estimates of the number of incidents by type, which may be compared with the NCVS. In theory, either instrument could support blended estimates with the NCVS core or provide correlates for model-based small area estimates using NCVS core data.

Development process. Westat project staff developed draft versions of both the incident-level and person-level instruments. These were reviewed by BJS and by outside experts in questionnaire design. After revisions based on these reviews, each instrument was subjected to several rounds of cognitive testing, with revisions as needed after each round. The following sections describe the rationale for and development of these instruments in more detail, and also present testing results.

Incident-level Approach

Overview

Previous experience with developing a self-administered of the NCVS had modeled on the incident approach used in the in-person interview. It is our understanding that these attempts met with only modest success. For example, the redesign conducted in the 1980s included an attempt to develop a NCVS self-administered paper instrument, more recently, there was an attempt to develop a version as part of a recent BJS study looking at different modes of interviewing. Neither of these attempts successfully duplicated the complexities of the entire NCVS in-person interview. Given the results of these earlier efforts, Westat focused the development effort of the paper instrument on capturing critical elements of the NCVS in-person instrument as opposed to attempting to duplicate the NCVS in-person instrument.

As the form name indicates, the incident-level approach Westat has developed focuses on collecting information about each reported victimization. One advantage of this approach is that an incident involving multiple household members is reported only once. This helps to minimize respondent burden and allows for more in-depth questions about the incident. A potential disadvantage to this approach is that it requires the household respondent to match household members (in cases of incident presence) to each incident. Another disadvantage is that the mail format limits the number of incidents that may be reported for a given household.

Design

Self-administration: The incident approach form is designed to be a self-administered paper questionnaire mailed to households. This is different from the current NCVS interviewer-administered approach. Self-administration influences the design of the form; the questions and navigational flow through the instrument must be easily understood by respondents with a wide range of education and cognitive skills. We simplified the instrument by first creating separate questionnaire sections based on broad victimization types: violent crimes (formerly personal), theft and break-ins, and other crimes. The first two sections were natural groupings that largely map to high-level classification used in BJS reports; we also reduced the number of questions required to classify victimizations by focusing on a higher level. The third section includes crimes (e.g., vandalism and identity theft) not typically included in BJS reports, but which victims may feel are important and that might be reported in other sections.

Household/Proxy reporting: This form is designed to be completed by one household respondent, who is first asked to list up to four adults (18+) who live in the household. The respondent is then to complete the form for all incidents that occurred to the household members they listed. For each violent crime we ask which household members were affected by (i.e., were present for) the incident. Responding for multiple household members is potentially the most difficult aspect of the questionnaire. Respondents can vary in how they list or identify other household members. Over several rounds of testing, we have revised how information about household members is collected; we now ask for two identifiers: the person’s name or nickname and the person number from the household roster in the first part of the questionnaire.

Number of incidents: Another factor influencing the design of the questionnaire is the number of incidents that may be reported. Generally, the number of incidents per person is expected to be low, but a few persons will have multiple or repeated victimizations over the 12-month recall period. Since this is a paper questionnaire, it must accommodate respondents who have multiple victimizations. On the other hand, including space for detailed reporting of many incidents would make the questionnaire daunting and difficult in which to navigate. To address each of these concerns, the incident level questionnaire is designed to collect different levels of detail for multiple incidents. For violent crimes, full detail is collected for the two most recent incidents. Abbreviated detail (when, who was affected, and a summary of the incident) is collected for an additional two incidents (next most recent). If there are more than four incidents a final question collects a count of remaining incidents. For theft and break-ins, detail is collected for the four most recent incidents with a final question collecting a count of additional incidents. For other victimizations (vandalism, identity theft) only counts of incidents are collected.

Open-ended responses: Open-ended questions are used sparingly within the incident form. For each violent crime or theft/break-in one open-ended question asks for a summary of the incident. These summaries are important as they may collect clarifying or confirmatory information that will be used in coding the type of crime and whether the incident meets the NCVS definition of a crime. This information may include who was injured, the type of injury, where the incident occurred, what was stolen, etc.

Cognitive Testing

We tested three versions of the incident-level self-administered questionnaire, with revisions after the first two rounds of testing as shown in Table 1. There were 3 interviews in Round 1, 7 in Round 2, and 8 in Round 3. Two notable difficulties test subjects encountered across rounds were (1) reporting incidents in the appropriate section of the questionnaire (e.g., reporting thefts as violent crimes) and (2) reporting one incident both as an actual victimization and as an attempt. The first difficulty can be problematic since the detailed questions focus on a particular victimization type. Review of responses in coding may still permit appropriate high-level classification of the victimization, but some detail may not be collected.

Table 1. Changes to person-level questionnaire after cognitive testing, by round

Round 1	For household members’ names in the roster, ‘Name’ was changed to ‘First Name’ to reinforce that only the first name is needed.
	Changed ‘Personal Crime’ to ‘Violent Crime’. This was the result of a broad interpretation of personal crimes. The intent was to define and collect in this section victimizations where the respondent was confronted (i.e., present) during the incident.
	Added the word ‘Section’ to each heading title to clarify where respondents need to skip to begin the next section of the questionnaire.
	For violent crime incidents, added a question asking if the victim was confronted by the offender to establish presence for violent crimes.
Round 2	Revised visual design of household roster: shading to separate each person listed was affecting how respondents completed the household roster, so shading was removed and other visual cues were added to identify each person.
	Changed ‘Person X’ to ‘Adult X’ to clarify who should be listed, i.e., only adults 18 years of age and older.
	Revised question asking about ownership of residence (tenure) to be consistent with the person-level instrument.
	Revised introductory text for section A (violent crimes), as respondents did not appear to be reading all of the text.
	Added a screening question to ask about attempted sexual victimizations because respondents reported both attempted and committed attacks in the same question.
	Revised negatively worded skip instructions (e.g., if you said no to all versus if you said yes to any), which were causing some confusion.
	Revised question asking to which household member(s) the incident happened to allow up to four household members to be listed and to include the household members’ first names.
	Increased the size of the summary incident description open-ended response field and added instruction for what should be reported, as most responses provided during this round of testing were brief.
	Revised wording household income question to be consistent with person-level instrument.
Round 3	No changes were made based on this round of testing.

Person-Level Approach

Background

Another feasible way of capturing important elements of the NCVS in-person instrument is by taking a person-level approach in which a household respondent reports crime victimization for him or herself and other adult household members. In this approach, questions focus on whether each person has had a particular victimization or not in the 12-month reference period. Incident-level information is largely not collected, and each household member is reported on separately.

One hypothesis is that a household respondent will be more likely to report for the other household members when presented with a person-by-person reporting format. Thus, there would be less underreporting for other in the person-by-person approach. This potential advantage is why we have developed the person-level approach to compare with the simplified incident-level approach. The disadvantage of this approach is not being able to develop incident-level victimization estimates.

An initial challenge in developing the person-level approach was developing a structure that encouraged accurate reporting of household-level events. We tested several different formats for the reporting of household-level victimizations (property crimes). Initially, the household- level incidents were broken out and reported separately before the person-by-person reporting. In the next round of testing, the household-level questions were integrated into the person-by-person questions. In a later round of testing, the household-level questions were again broken out and asked before the person-by-person questions. Thus, the person-level questionnaire went through more significant changes from initial development through testing than did the incident-level form.

Version 1 of the Person-Level Instrument

The initial structure of the Version 1 Person-Level Instrument was laid out in the following sequence:

1. Engaging questions about the overall safety of the respondent’s community. These questions serve as a warm-up for the respondent and also provide information that may be of interest to local jurisdictions.

2. Questions about the respondent’s satisfaction with their local police department.

3. Household demographics (own or rent home; tenure at current residence; and household income).

4. An 8-question series on household-level crimes, specifically: break-ins, attempted break-ins, vehicle theft, and whether the police were notified about any property crimes.

5. A small matrix enumerating household by name/initials, age, and gender.

6. Person-by-person questions about personal crime victimization.

When Version 1 was tested with 2 respondents, it became apparent that the structure did not lend itself easily to reporting violent crimes. Respondents were confused about where to report their victimizations. Furthermore, some of the instructions did not reflect the way the respondents thought about crime. To address these issues, a Version 2 was developed. Additionally, the section heading “Crimes Against Persons” had no meaning to respondents, which contributed to the overall level of confusion.

Version 2 of the Person-Level Instrument

A major change in the Version 2 Instrument was integrating the household-level victimizations into the person-by-person sections. This structure allowed respondents more quickly to identify where they were to report their experiences. Version 2 of the Person-Level Instrument was laid out in the following sequence:

1. Engaging questions about the overall safety of the respondent’s community (no change).

2. Satisfaction with the local police department (no change).

3. The household composition section was simplified to a box where the respondent reported the number of adults age 18 and older and renamed “Adults in this Household”; the name/age/gender matrix was omitted. Some of the instructions previously located in the “Crimes Against Persons” section were simplified and included in this section.

4. Household-level victimization questions were integrated into the person-level questions, and thus were repeated for each adult in the household up to four adults in total. The section header “Crimes Against Persons” was dropped, and the combined household/person victimization sections were renamed “Adult 1,” “Adult 2,” etc.

5. Household demographics were moved to the very end of the instrument.

To test the major change of the integration of the household level victimizations into the person-by-person series of questions, Westat conducted a small (3 subjects) cognitive test to determine whether adults living in multiple-adult households could actually report the household-level victimizations for the other adults in the household. This small test was designed as a preliminary evaluation of the ability and willingness of the proxy respondent to report for the other household members on a person-by-person format. To test this one particular aspect, we recruited adults who lived in multiple-adult households and had experienced a break-in. The limited sample of respondents in this test was able to report the break-in for the other household members, so it was decided to continue developing Version 2 for further testing on respondents with a broader range of characteristics and victimizations.

After the small test on multiple-adult households, we simplified the language used in both the instructions and the questions throughout the questionnaire. The language used in Version 2 was appropriate for a grade 12 or above reading level; our goal was a grade 6-8 reading level. The major changes to the language of Version 2 are described below. Many other small changes are not included in this description.

1. The instructions for the section “Adults in this Household” were shortened and simplified; important pieces of information were presented in bullets.

2. The instructions before the person-by-person questions were both simplified and shortened; verbiage was deleted and key instructions were presented in bulleted form.

3. The language in the questions about “unwanted forced sexual activity” was simplified, shortened and made less explicit.

4. The questions about “other things that were stolen,” vandalism, and identity theft were simplified and shortened.

The Simplified Version 2 was then tested on a larger and more diverse sample (7 subjects) than in the previous testing rounds. The changes made after this round of testing produced a Version 3 Person-Level Instrument.

Version 3 of the Person-Level Instrument

The following changes were made after testing the Simplified Version 2 questionnaire:

1. Questions 1, 5, and 7a of the engaging questions in the “Safety” section were changed from Yes/No to a frequency scale because the Yes/No option did not provide enough variation for respondents. For example, one respondent said that the Yes/No forced him to make an extreme statement about his neighborhood that left him feeling bad.

2. The household-level victimization questions were removed from the person-by-person victimization series and placed in between the sections on the “Local Police Department” and “Adults in this Household” which asks for the number of adults living in the household. Thus the household-level victimization questions were placed before the person-by-person victimization questions.

3. A number of questions were added to the household-level victimization section to add detail to support type-of-crime classification and support additional estimates:

1. Number of times the home was broken into;

2. Whether anyone was at home during the break-in;

3. Whether a household member saw the offender;

4. Date of the most recent break-in;

5. Whether any break-ins were reported to the police;

6. Whether the household experienced a break-in attempt and, if so, the number; and

7. Whether a break-in attempt was reported to the police during the reference period.

4. The instructions for the personal crime victimization series were shortened and the emphasis of the instructions changed to stress the reporting of any type of victimization other than what was already reported.

5. The response space for the three open-ended incident narratives was reformatted to be clearer for respondents.

6. In the personal crime series, a question was added for each type of crime asking whether this type of crime had been reported to the police.

The Version 3 Instrument was evaluated in another round of cognitive testing. The 8 respondents represented a relatively diverse group with respect to victimizations and living arrangements. Respondents made some minor errors, but none indicating any further changes were needed before the pretest of the two mail questionnaires.

Estimates and Comparisons

A primary goal of the CS is to produce estimates of crime victimization that would be correlated with, if not directly comparable to, those from other sources, including the core NCVS. As described in the introduction to this document, the set of type-of-crime categories for which estimates may be possible will of necessity be at a higher level than for the core NCVS. As we shall see, the types of estimates the incident and person instruments, if successful, will support overlap considerably but are somewhat different from each other.

Violent Crime

Table 2 presents the set of violent crime victimization estimates possible from the incident-level instrument. It is presented as counts of victimizations; other supported estimates include counts of persons or households (with the limitation that only adults are included) and prevalence or incidence rates by person or household. With this approach, purse snatching and pocket picking would be treated as thefts under property crimes. Note also that threatened violent crimes are not included in Table 2 (with the exception of attempted robbery?).

An overall limitation of the instrument is its treatment of households that have experienced multiple incidents. The questionnaire currently allows full reporting of the two most recent violent crime incidents, very partial reporting of two more, and simply a total of any remaining incidents. For property crimes, the form allows full reporting of the four most recent incidents and a count of any remaining ones in the reference period. Depending on the treatment of the partial reports and summary numbers in processing, estimates may be biased either up or down by this constraint.

Table 2. Violent crime victimization estimates potentially supported by incident-level CS questionnaire

		NCVS TOC’s1	Number of victimizations	Domestic violence	Intimate partner violence	Reported to police
	All Violent Crimes					
1	Serious Violent					
1A	Rape & Sexual Assault					
1A1	Completed Rape	1				
1A2	Attempted Rape	2				
1A3	Sexual Attack	3, 4, 15				
1B	Robbery					
1B1	Robbery with injury	5, 6				
1B2	Robbery without injury	7				
1C	Aggravated assault	11, 14				
2	Simple Assault					
2A	Threatened Assault	12, 13, 20				
2B	Assault (no weapon/injury)	17				

1. The questionnaire does not support coding of TOCs 8, 9, and 10 (attempted robbery); such incidents may be reported and coded as aggravated or simple assault. The questionnaire also does not support coding of TOCs 16 (unwanted sexual contact without force) and 18 (verbal threat of rape).

Table 3 shows a similar set of estimates that may be supported by the person-level instrument, but these estimates are only of persons (or households) touched by a particular type of crime. At this point in development, the rows are slightly different between the incident-level and person-level form: the person-level form does not distinguish between rape and other sexual assault, or between attempted and threatened sexual assault, while the incident-level form does not ask about threatened rape or sexual assault. Further, for the domestic violence, intimate partner violence, and police report columns, the person-level form does not allow as much disaggregation as the incident-level form because one person may be reporting on multiple types of violent crime covered by the set of questions leading up to these items.

Several changes were made to the person-level instrument after the cognitive testing to help align the questions with the checked cells in Table 3.

Table 3. Violent crime victimization estimates potentially supported by person-level CS questionnaire

		NCVS TOC’s1	Number of persons	Domestic violence	Intimate partner violence	Reported to police
	All Violent Crimes		2	2	2	2
1	Serious Violent		2
1A	Rape & Sexual Assault		2	2	2	2
1A1	Forced Rape/Sexual Assault	1,2,3,4,15				
1A2	Atmp’d/Threat’d Sexual Assault	2, 19				
1B	Robbery		2	3	3	3
1B1	Robbery with injury	5, 6	
1B2	Robbery without injury	7	
1C	Aggravated assault	11, 14	
2	Simple Assault		2
2A	Threatened Assault	12, 13, 20				
2B	Assault (no weapon/injury)	17	

1. As with Table 2, the questionnaire does not support coding of TOC’s 8, 9, 10, 16, and 18.

2. Total is not necessarily the sum of subcategories

3. Cannot reliably distinguish among robbery, aggravated assault, and simple assault for these items

Property Crime

As with violent crime, the incident-level instrument collects information about a limited number of incidents (4), and then a count of the remaining incidents in the reference period. Table 4 shows the types of estimates that the incident-level instrument may support, given that caveat. After cognitive testing, a question on the dollar value of completed property crimes was added to the incident-level instrument. At this time, we have not established cutoffs for the dollar value variable. The incident-level instrument also supports household-level incidence and prevalence estimates of property crimes.

The person-level instrument will only support household prevalence estimates, and for a much more limited set of property crime classifications, as shown in Table 5. As currently designed, it collects no information about attempted property crimes and does not distinguish burglary by whether it involved forcible entry or not. We note that larceny is collected both at the household and person level in the current instrument; this may lead to some double counting of dollar amounts.

Table 4. Property crime victimization estimates potentially supported by incident-level CS questionnaire

		Number of victimizations	Dollar Value			Reported to police
			Cat 1	Cat 2	Cat 3
	All Property Crimes					
1	Household burglary (Total)					
1A	Completed					
1A1	Forcible entry					
1A2	Unlawful entry without force					
1B	Attempted forcible entry					
2	Motor vehicle theft (Total)					
2A	Completed					
2B	Attempted					
3	Theft (Total)					
3A	Completed					
3B	Attempted					

Table 5. Property crime victimization estimates potentially supported by person-level CS questionnaire

		Number of households	Dollar Value			Reported to police
			Cat 1	Cat 2	Cat 3
	All Property Crimes					
1	Household burglary (Total)					
1A	Completed					
1A1	Forcible entry
1A2	Unlawful entry without force
1B	Attempted forcible entry
2	Motor vehicle theft (Total)
2A	Completed					
2B	Attempted
3	Theft (Total)
3A	Completed					
3B	Attempted

Appendix D. Pre-Test Cover Letter

Month xx, 2014

Dear Resident:

Please complete the enclosed survey from the U.S. Department of Justice, Bureau of Justice Statistics. This survey is about your household’s experiences with crime and neighborhood safety. Results from this survey will be used to better understand the needs of American communities.

Your address is part of a random sample of addresses in the United States. This is part of a scientific study and your answers represent not only you and your household, but also hundreds of households like yours. For this reason, your voluntary cooperation is very important. Some households will be called for a brief follow-up interview later.

The information you provide will be used for statistical purposes only and may not be disclosed, or used, in identifiable form for any other purpose as required by law (Title 42, U.S. Code, Sections 3789g). Your responses will be combined with those of others to produce statistical summaries about crime and safety.

Answers to the most frequently asked survey questions are included on the back of this letter. Section 3732 of the Justice Systems Improvement Act of 1979 authorizes the Bureau of Justice Statistics to conduct this survey. If you would like further information, you can contact our survey support at 1-xxx-xxx-xxx or you can visit the BJS website at www.bjs.gov/ncvspilot.cfm.

Crime and safety affect all Americans. Thank you for your generous cooperation. The U.S. Bureau of Justice Statistics appreciates your help in this very important survey.

Sincerely,

Director, Bureau of Justice Statistics

Office of Justice Programs

U.S. Department of Justice

Commonly Asked Questions

What is the U.S. Bureau of Justice Statistics Survey of Crime and Safety?

The U.S. Bureau of Justice (BJS) Survey of Crime and Safety is a survey of households to obtain information about communities.

Who is the sponsor of this study?

The survey is sponsored by the Bureau of Justice Statistics (BJS), U.S. Department of Justice (DOJ). The survey is conducted under the authority of Title 42, United States Code, Section 3732. To learn more about BJS, you can visit them on the web at www.bjs.gov/ncvspilot.cfm.

How long will it take to complete this survey?

We anticipate that most households will be able to complete the mailed survey in about xx-xx minutes. Some households may be contacted later for a more detailed survey.

Am I required to complete this survey?

Participation is voluntary and there are no penalties for refusing to answer. However, your household was randomly selected for this scientific sample survey, and you cannot be replaced with another household. Your cooperation is extremely important to help ensure the completeness and accuracy of this much-needed information.

Who will use this information?

Results from this survey (and similar surveys conducted by The U.S. Department of Justice) will be used to better understand crime and safety in U.S. neighborhoods.

Who can I call with questions?

If you would like further information, you can contact Westat at 1-xxx-xxx-xxxx.

How was my household chosen for this study?

Households were selected at random from all residential addresses. By selecting households randomly, we will be able to create scientific estimates about households in your neighborhood. It’s important to participate, so that we have an accurate picture of all communities.

How do I know you’ll keep my information confidential?

The information you provide will be used for statistical purposes only and may not be disclosed,

or used, in identifiable form for any other purpose as required by law (Title 42, U.S. Code, Sections 3789g). Your responses will be combined with those of others to produce statistical summaries about crime and safety. After the study is completed, identifying information - your address and phone number - are destroyed.

Appendix D. Pre-Test Debriefing- Interview Protocol

PRETEST DEBRIEFING PROTOCOL

Date:_______________ Time_____ ID #:_____Interviewer Initials:_______________

1. Introduction

Thank you for taking the time to help us out today. The session will take approximately 45 minutes.

Westat is working on this project for the Bureau of Justice Statistics. We are helping them with a mail survey designed to collect information about Crime Victimization in the U.S.

This call is a followup to a mail survey completed by your household. We want to discuss the survey and your answers to it so that we can help improve the survey.

Today, I’ll ask you to discuss the experiences your household reported in the survey. I’ll also ask you some questions about your answers and how you arrived at your answers. We need to make sure that the survey is easy to use and that the questions are clear.

This is a research project and your participation is voluntary. You can skip any question and you can stop at any point. We would very much appreciate your permission to audio record this conversation. The recording will be used for note-taking purposes only and will be destroyed when the project is over. When we are finished, I would like to confirm your address so we can mail a check for 40 dollars in gratitude for your assistance. There are no right or wrong answers – we are interested in everything you have to say and we encourage you to speak openly about the questions and your answers.

2. Consent Process

Do you have any questions before we get started?

[Start recorder and get oral permission to record.] It is [DATE AND TIME], do I have your permission to audio record this conversation? ~~~~ Thank you.

For questions about your rights and welfare in this study, you may call the Institutional Review Board at Westat at 301-610-8828.

3a. Debriefing Content

• Confirm that debriefing respondent is same as the mail survey respondent.

• Probe how they decided who should complete the survey (for multiple adult households). Was it the person who usually gets/opens the mail, or was it the person who was victimized (victimized the most)?

• Probe how well the respondent recalls the survey.

  • Does the respondent recall how easy/difficult to complete (can they comment on usability)

  • Do they remember how they answered the questions

  • (if applicable) did they talk with other adults in the HH about it

  • Did they have any concerns about doing the survey

• For victimizations to other household members, did they consult other household members for information or not (this may be what you meant in the last bullet below).

• Ask the respondent to describe any crimes that occurred in the past 12 months (in analysis we can compare to the paper responses)

  • Probe in order to obtain enough information to code the crime(s)

 Violent and/or property crime  Presence of HHMs

 Weapons  Injury, hospitalization

 Police notification  Property recovered; value of loss

 Offender seen, known

• Probe to collect information on any other crimes (make sure those recorded in the paper survey are covered, and also probe to see if any additional crimes not reported – may want them to think about small, nuisance crimes, like theft of spare change from a car, theft of a gas cap, or vandalism to inexpensive items, like a garden hose).

• Probe crimes experienced by any other adults in HH (if applicable)

• Ask if any children in HH – anything stolen that belonged to children in the past year?

• Probe any response inconsistencies seen in the paper survey (based on your prior review of the paper survey)

• Multiple adult households – researcher will probe how proxy respondent answered for the other adults in the household.

  • How confident were they in answering

  • Do they think the other(s) would tell them about crimes

Appendix E. Pre-test Debriefing- Recruiting and Scheduling

PRETEST DEBRIEFING – RECRUITING & SCHEDULING

Hello, your household recently completed a mail survey from the Department of Justice about your household’s experiences with crime in the past year. We would like to followup with your household by phone. Our goal is to make improvements to the survey, and so we would like to discuss your household’s experience with the survey questionnaire.

We believe that the person who completed the survey is:

INCIDENT LEVEL INFO

• Named _____________ (or has the initials __________ )

• Gender ____

• Race ethnicity _____________________

• Age ______________

PERSON LEVEL INFO

• Age ______________

• Gender ____

• Race ethnicity _____________________

We would like to schedule an appointment to speak with this person to enlist their help in improving the survey.

[ONCE IDENTIFY THE MAIL SURVEY RESPONDENT REPEAT THE INTRO, THEN ATTEMPT TO SCHEDULE]

We’d like to schedule a telephone interview with you. Let me read you the days and times I have available, and you can tell me what would be best for you. This will take about 45 minutes of your time and we will pay you $40 for your help. May I please have your full name and the best phone number to contact you for the interview? We will send you a reminder before the call – we can call with the reminder or we can send an email reminder. [IF RESPONDENT PREFERS EMAIL REMINDER THEN COLLECT EMAIL ADDRESS]

Full Name: |______________________________________________________________|

Best phone: |________________________________________________________________|

E-mail address: |___________________________________________________________|

If you have to cancel your interview, please call us so that we can re-schedule. You can reach me at: [Recruiter’s Phone Number].

Appendix G - Research Questions for the Pretest

1. Will the response rate be acceptable?

The goal is to have a mail-based approach that yields an approximate response rate of 50 percent. If we find lower rates from the pretest then we may consider improvements to the methods before heading into the field test. Improvements might include additional incentives, additional mailings, or telephone prompts (for example). If pretest response rates are too low (25 percent or lower) this may be an indication that our operational approach is not viable.

2. Will the level of response error be tolerable?

We will investigate respondent adherence to skip patterns, response logic, level of item missingness, and reporting for adults 2-4 to help determine whether one or both of the instruments lack viability. This review will also include an investigation of open-ended responses to determine whether respondents are able to provide sufficient detail about crimes for coding purposes.

The debriefing interviews will also be used to investigate this research question. We will explore how the proxy respondents handle responses for other household members. We will also explore how respondents used the instrument, whether there were omissions, and whether the appropriate details were recorded in the mail survey.

3. Will the victimization rates be credible?

Because the survey instrument and procedures differ from those of the NCVS, we do not expect that the estimates of victimization rates will accord exactly with those from the NCVS. Based on previous research, including the Chicago pilot test, we anticipate that victimization rates estimated from the pretest would be somewhat higher than those from the NCVS. Should the rates be lower or very much higher, we would attempt to determine the reason(s) and modify the instruments or procedures for the field test.

However, for the survey to be useful in measuring change within an area over time, or for comparing areas, victimization rates from the survey should be positively correlated with those from the NCVS. The pretest sample size will not be sufficient for evaluating the consistency of the survey victimization rates across jurisdictions, but the pretest will be used to refine the procedure so that the main field test can be used to answer this research question.

4. How will placement of the non-crime questions affect response rates and estimates?

While non-crime questions are of substantive interest, particularly to local jurisdictions, another potential benefit of including such questions is to engage non-victims and increase overall survey response. This benefit would seem more likely if the non-crime questions were asked early in the instrument. We will compare response rates from the two treatments to test this hypothesis. Another concern is that there may be a context effect associated with the non-crime questions, i.e., respondents may answer the victimization questions differently after answering the non-crime questions. Similarly, when the victimization questions are asked first, they may affect respondents’ answers to the non-crime questions. We will compare estimates of victimization and of respondent attitudes in the two treatments (early and late placement) to assess context effects.

31