OMB Qs and BJS As document

Methodological Research to Support the Redesign of the National Crime Victimization Survey: An Examination of a Twelve-Month Reference Period

Phase III- Cognitive Interviews

GMS Award 2008-BJ-CX-K071

OMB Passback

1. Please provide additional detail on how NORC will determine when to end cognitive interviews if before the maximum of 100?

2. In the Survey of Crime Victimization Call Scripts, why the note to conclude the call if the respondent indicates that they would not like to have their interview recorded? According NORC’s memo to OMB, priority will be given to respondents who agree to recording but that interviews will also be scheduled with those who do not want to be recorded.

3. How will recordings of the cognitive interviews be analyzed and used?

4. Please provide additional detail on how NORC will draw on respondent answers to specific scripted probes (as well as unscripted probes in general), respondent debriefing, and recordings to compare the memory aid conditions, differences in respondent thought processes, mind-sets across the treatments, etc.

5. The race question included in Survey of Crime Victimization Screening Questions does not conform to the OMB standard. Please replace with the following:

What is your race?  Please select one or more. 

White   

Black or African American  

Asian  

Native Hawaiian or other Pacific Islander  

American Indian or Alaska Native

Responses to OMB Passback

1. Please provide additional detail on how NORC will determine when to end cognitive interviews if before the maximum of 100?

The cognitive interviews will bring to light problems that exist with the ECP and EHC. Some of these problems will be ones that we anticipated based on our review of the instruments and initial testing through mock interviews with other survey professionals at NORC. Other unanticipated issues are likely to be revealed by the cognitive interviews as well.

The cognitive testing will be conducted iteratively. That is, after an initial set of interviews, the data will be examined. We will address the major issues that have emerged and prepare revised materials for further cognitive testing. After an additional set of cognitive interviews, the data will be examined again and the materials will be revised again as needed.

The cognitive testing will be considered completed when the instruments meet our goals (questions and response categories are understood by respondents as NORC intends, interviewers have no problems administering, all issues are addressed and the instruments are ready for the field test).

NORC will examine a number of factors in determining when to end the cognitive interviews: respondent understanding of the task/questions, respondent burden, interview timings, incorporating feedback from interviewers/respondents on problems with the instruments. Some specific questions that will guide the cognitive testing include:

• Do respondents have any difficulty comprehending the materials?

• Are there any ECP questions that can be improved, clarified?

• Are there any additional ECP questions that should be included?

• Do respondents understand the EHC task?

• Are respondents able to provide personal events to record on the calendar?

• Are the probes for eliciting personal events (to be used if the respondent cannot provide events spontaneously) effective or do they need revision?

• How burdensome is the EHC task and has burden been reduced as much as possible?

• Can respondents provide approximate dates for personal events?

• Has all relevant feedback from respondents and cognitive interviewers been incorporated?

• Is the timing of the instruments within the appropriate parameters?

Certainly, any survey instrument will continue to improve with additional testing. After the main goals of testing the ECP and EHC have been met, continued testing to reach 100 completed interviews would potentially yield further refinements. The decision on whether to end the cognitive interviews before 100 completes will be made in conjunction with BJS depending on the additional refinements that might be made and schedule/budget considerations.

2. In the Survey of Crime Victimization Call Scripts, why the note to conclude the call if the respondent indicates that they would not like to have their interview recorded? According NORC’s memo to OMB, priority will be given to respondents who agree to recording but that interviews will also be scheduled with those who do not want to be recorded.

NORC updated Step 2-Call in Script to remove the note to conclude the call for respondents who do not wish to have the interview recorded. These changes are reflected in the Step 2-Call in Script attachment.

3. How will recordings of the cognitive interviews be analyzed and used?

At the beginning of the cognitive interview, during the consent process, we will ask respondents for permission to record. Based on NORC’s prior experience, we expect that most respondents will agree to be recorded. During the interviews, interviewers will take notes regarding respondent comments, problems with the instrument, their own observations. After the interview, interviewers will write more detailed summaries of their findings. The recordings of the cognitive interviews serve as a backup for the cognitive interviewing team to refer to in analyzing and summarizing what has been learned from the interviews. Review of selected portions of the recordings will allow the interviewers to enhance their understanding of specific issues that arise during the interviews and to share with the rest of the research team any interview portions of particular interest. The degree to which interviewers will need to refer to the recordings depends on the complexity of the issues that arise during the cognitive interviews.

Given the different requirements of the ECP and EHC interviews, the use of the recordings in analyzing the data from each of these types of interviews will differ. The ECP is a more conventional survey instrument, with scripted questions and response categories for most of the questions. A number of the questions in the ECP were adapted from other surveys and all the questions have been tested in mock interviews with other survey professionals at NORC. During the cognitive interviews, we will examine how the ECP works with the general population. Issues of question understanding, burden, and timing will be examined. In addition, we will incorporate comments and suggestions from respondents and the interviewers into the instrument (in consultation with BJS) and retest the revised instruments. The recordings will be useful for reviewing selected portions of interviews to fully understand and document any issues that arise. However, it is unlikely that extensive review will be needed for these interviews.

The EHC interviews are more complex because of their open-ended nature. The respondent-interviewer interaction will be guided by scripted instructions on how to complete the EHC and some scripted probes designed to elicit the respondent’s landmark events, but much of the interview will be unscripted and each EHC interview will be different. Further, the EHC has been used in surveys to collect specific types of information that are somewhat different from crime events, such as the start and stop dates of employment, schooling and other continuous events. The EHC is being used in the current survey in a novel way, to prompt recall of discrete events and specifically of crime incidents. Therefore, analysis of the EHC interviews will be somewhat more detailed and unanticipated issues may be more likely to arise with the EHC than the ECP. The recordings of the EHC will be reviewed to capture detailed information such as the following:

• Personal landmarks

  • What types of landmarks do respondents name? How many landmarks can they name? Which types of landmarks are more commonly named and in what order to respondents name them? What level of probing is needed to elicit landmarks? What questions do respondents have about the EHC task?

• Effectiveness of the interviewer instructions and the scripted probes

  • Do the instructions effectively guide the interviewer through completion of the EHC? Do the scripted probes need revising? Which parts of the interview are most challenging for interviewers to administer?

3. Please provide additional detail on how NORC will draw on respondent answers to specific scripted probes (as well as unscripted probes in general), respondent debriefing, and recordings to compare the memory aid conditions, differences in respondent thought processes, mind-sets across the treatments, etc.

By examining respondent answers to scripted and unscripted probes and debriefing questions, we will learn what aspects of the ECP and EHC instruments are problematic. Data across all respondents will be summarized with descriptive measures. For example, for each question or scripted probe, we will examine the frequency with which respondents:

• ask for clarification or repetition of a question

• provide a response that is unexpected or seems incorrect

• say they “don’t know” how to answer

• take longer than expected to answer

• make other comments or suggestions about the question or probe

• indicate that the interview is burdensome, uninteresting, etc.

We will attempt to obtain a sample that includes a broad demographic range of age, gender, education level, level of crime in respondent neighborhood and other variables; the data analysis will include focus on issues that arise within selected respondent groups. We will examine the data by such demographic factors to assess whether ECP and EHC materials are appropriate for respondents of different characteristics and whether problems with the instruments exist for some subsets of respondents but not others.

Since the ECP and EHC are quite different in their structure and in what they require respondents to do, respondent reaction to each is likely to be quite different. It is important to note that The ECP and EHC are being evaluated to improve the effectiveness of each memory aid, but we will not be able to compare them based on cognitive interview data to determine which is better at improving recall of crime incidents. This is in part because each memory aid will undergo revision and improvement throughout the cognitive testing and also because the overall sample size for the cognitive interviews is small. The field test planned for Phase IV, which includes a larger sample size and random assignment of respondents to each memory aid condition, will provide the data for comparing the ECP and EHC to each other and to a control condition in which no memory aids will be presented.

5. The race question included in Survey of Crime Victimization Screening Questions does not conform to the OMB standard. Please replace with the following:

What is your race?  Please select one or more. 

White   

Black or African American  

Asian  

Native Hawaiian or other Pacific Islander  

American Indian or Alaska Native

NORC revised the Step 3-Screening Questions to adhere to OMB standards. These changes are reflected in the Step 3-Screening Questions attachment.

Responses to SCV OMB Passback-August 27, 2010 Page 5