Background. In December 2010 and January of 2011, NORC conducted cognitive testing of the Enhanced Contextual Priming (ECP) memory aid. The memory aid is a separate module that precedes the NCVS crime screener and incident report(s). This section presents the methodology and results from the cognitive interviews.
Interview procedures. All interviews were conducted at NORC’s Chicago Loop office. The interview began with the informed consent process. Respondents then completed the ECP memory aid and proceeded to the NCVS-1 crime screener and, if crimes were reported in the screener, the NCVS-2 modified incident report. Cognitive probing was included in the ECP portion of the interview. At the end of the crime screener, the interviewer determined which of the elicited crimes had been reported to the police. In completing incident reports, interviewers gave priority to those crimes that were reported to police, and asked about these events in order of mention in the crime screener. If time permitted, additional incident reports on significant crimes not reported to the police were completed as well, up to a total of three incident reports.
At the conclusion of the interview, the respondent was paid $40. Then the interviewer asked the respondent for permission to obtain any police reports that may have been filed. The cognitive testing was conducted iteratively. After each round of testing, NORC analyzed the data, discussed the findings with BJS, and consulted with BJS on revisions to the instruments before the next round began.
Of the 38 cognitive interviews, 10 were completed in Round 1, 9 in Round 2, and 19 in Round 3; the mean age of these respondents was 39.6 years. Most of the respondents identified themselves as Black/African American (87%) and 11% (n=4) identified as Hispanic. The majority of respondents were male (61%). Fewer than half (34%) of respondents reported being employed either part-time or full-time. Most respondents (84%) reported that they had at least a high school education; 2% indicated that they had a BA/BS or graduate degree. In Round 1, potential participants had to have experienced a crime in the last 12 months to be eligible to participate. This approach was taken to ensure that the full instrument, both screener and incident report, would be tested. To test the materials with a broader pool of respondents, the recruitment criteria were expanded; for Rounds 2 and 3 respondents did not need to have experienced a crime to be eligible to participate.
Enhanced Contextual Priming Cognitive Interviews
Table 1 shows the ECP instrument that was developed in Phase 2. This instrument was the starting point for the cognitive testing. Respondents were asked questions about feelings of safety, places they go, and trust in people.
Table 1: Preliminary ECP
Preliminary Q1: Is there any area right around your home – that is, within a mile – where you would be afraid to walk alone at night? YES NO
Preliminary Q2: How about at home at night – do you feel safe and secure, or not? YES NO
Preliminary Q3: Crimes can happen in many different locations. To help remind you of crime incidents that may have happened, let’s begin with some questions about the places you have been. Thinking about the last 12 months, that is, since <REFERENCE MONTH> 1st, 2009, where do you go on a regular basis?
Preliminary Q4: Have you been away from home for at least one night in the last 12 months, that is, since <REFERENCE MONTH> 1st, 2009? YES NO
Preliminary Q5: Did you go… [CHOOSE ALL THAT APPLY.] On a vacation? To visit family and friends? On a business trip? For another reason? (Please specify: _____________________________)
Preliminary Q6: How safe do you feel when you are not at home? Please think about the places you mentioned that you go on a regular basis, such as [FILL IN FIRST 3 ITEMS FROM QUESTION 3]. Do you feel very safe, fairly safe, a bit unsafe, or very unsafe?
Preliminary Q7: Remember, crimes can be committed by people we know well, by acquaintances, or by strangers. Let us think about different people you know or happen to meet and how much you trust them. Using a scale of 1 to 5 where 1 means ‘Cannot be trusted at all’ and 5 means ‘Can be trusted a lot’, how much do you trust each of the following groups of people:
Strangers? People in your neighborhood? People you work with or go to school with? People in your family?
|
Results of three rounds of ECP cognitive testing. Round 1 cognitive testing revealed issues with some of the questions (detailed below). Revisions to the ECP were implemented and tested in Round 2. Results of Round 2 testing suggested that the revisions to the instrument were effective. No additional revisions to the ECP were implemented prior to Round 3. Round 3 testing focused on obtaining accurate timing of the instrument (without the intrusion of cognitive probes) and confirming with further evidence that the revised instrument was working well. Key changes to the instrument are:
Preliminary Question 3: Overall, respondents could list places they go and their explanations seemed to suggest they understood the question. Respondents generally named four or five places that they go. However, the Round 1 data suggested the possibility that some respondents thought the question referred to places they go regularly on their free time, for recreation. This was explored in Rounds 2 and 3 through additional cognitive probing. Further testing explored an alternate wording of the question (“where do you go as part of your normal routine”) and examined whether respondents were interpreting the question as referring to where they go for recreation. This additional testing revealed that the “regular basis” wording produces a broader range of responses than “normal routine” wording. Further, respondents were not restricting their responses to places they go for recreation.
Preliminary Question 5: The response choices for this question were not working. One respondent did not wait for the list to be read; one did not choose all the options that applied to him. Also, the same trip may be for more than one of the listed purposes, but the “choose all that apply” instruction was not clear to interviewers or to respondents. In Round 2 we revised and expanded this question. To promote more thinking about the places respondents went, we added questions on the number of trips taken, destination, number of nights away, lodging, and feelings of safety. Cognitive testing in Rounds 2 and 3 indicated that respondents understood the revised questions and were able to answer them.
Preliminary Question 6: This requires a summary judgment, which can be difficult. It would be cognitively easier for respondents to report on feelings of safety for each separate place they go on a regular basis, rather than attempt to judge their overall feelings of safety. Asking about each place separately would also encourage deeper processing about each of those places. These changes were implemented in Round 2. Respondents were asked to make a separate judgment about how safe they felt at up to three places they go on a regular basis. Testing in Rounds 2 and 3 revealed that this revision to the question worked well.
Preliminary Question 7: The scale needed to be repeated to a couple of respondents. One person wanted labels for all scale points, not just the endpoints. In Round 2 each part of the question was rephrased into a statement about trust and the scale was changed to four-point agreement scale. The new scale seemed easier for respondents to use. The requests for clarification and repetition of the question were greatly reduced in further rounds of testing.
The Round 2 ECP is shown in Table 2. Round 2 testing revealed no new information that warranted changes to the ECP. Therefore, the ECP questions were not modified prior to Round 3 testing. The cognitive interview findings suggest that the ECP continued to work well. The questions were clear and respondents could provide appropriate answers.
Table 2: Final ECP
Introduction: I am going to ask you some questions about crimes that may have happened to you in the last 12 months, that is, since <REFERENCE MONTH> 1st, 2009. Before we talk about these crimes, let's think about your feelings of safety at home, the places you go, and your trust in the people you meet.
Q1: Is there any area right around your home – that is, within a mile – where you would be afraid to walk alone at night? YN
Q2: How about at home at night – do you feel safe and secure, or not? YN
Q3: Crimes can happen in many different locations. To help remind you of crime incidents that may have happened, let’s begin with some questions about the places you have been.
Thinking about the last 12 months, that is, since <REFERENCE MONTH> 1st, 2009, where do you go on a regular basis?
Q4: You mentioned that you go to [Q3, PLACE 1]. When you go there, would you say you feel very safe, fairly safe, a bit unsafe, or very unsafe? b. How about [Q3, PLACE 2]? How safe do you feel there? [IF NECESSARY, READ: would you say you feel very safe, fairly safe, a bit unsafe, or very unsafe?] c. How about [Q3, PLACE 3]? How safe do you feel there? [IF NECESSARY, READ: would you say you feel very safe, fairly safe, a bit unsafe, or very unsafe?]
Q5: Have you been away from home for at least one night in the last 12 months, that is, since <REFERENCE MONTH> 1st, 2009? YN How many trips away from home did you take? ___________ TRIPS
Q6: What different places did you go? Q6b: How many nights did you stay in [DESTINATION]? Q6c: During your time there, what type of lodging did you stay in? [CHOOSE ALL THAT APPLY.]
Private home Hotel, motel, B&B, resort Condo, cabin, vacation home Camper, trailer, RV, tent/campsite Other
Q6d: While you were in [DESTINATION], would you say you felt very safe, fairly safe, a bit unsafe, or very unsafe?
Q7: Crimes can be committed by people we know well, by acquaintances, or by strangers. I’m going to read you some statements about different people you know or happen to meet and how much you trust them. Please tell me how much you agree or disagree with each statement.
|
Note: Introduction Questions 1 and 2 are taken from the General Social Survey (http://www.norc.uchicago.edu/GSS+Website/). Introduction Question 6 was adapted from the 2001 National Household Travel Survey (http://nhts.ornl.gov/). Introduction Question 7 was adapted from the Canadian General Social Survey module on crime victimization (http://www.statcan.gc.ca/cgi-bin/imdb/p2SV.pl?Function=getSurvey&SDDS=4504&lang=en&db=imdb&adm=8&dis=2).
Administration time for the ECP. The timings for the Round 3 ECP are summarized in Table 3. Cognitive probing that was interspersed throughout the ECP precluded accurate timings of the length of the instrument in Rounds 1 and 2. The timings presented are from the beginning of the Introduction statement through the last Introduction Question on trust in others. When the timings that were outliers are excluded, the ECP averaged about five minutes in length.1 Timing is nearly six minutes when the outliers are included.
Table 3: Timings for Round 3 ECP
|
Number of Interviews |
Mean Interview Length in Minutes (s.d.) |
Range of Interview Length in Minutes
|
Round 3 |
19 |
5.0 (1.2) |
3.5 – 7.5 |
Note: Mean and range of interview times excludes outliers, interview timings that
were more than one standard deviation longer than the mean.
Confidence in Dating of Crime Events
During the crime screener portion of Round 3, respondents in the ECP condition were asked to say when each reported incident occurred. Dating of events could be specific to the month, more specific (such as to the day), or less specific (such as within several months). Interviewers recorded the descriptions verbatim; respondents were then asked to say how they arrived at their answer, and to rate how confident they were in the accuracy of their dating of the incident.
Respondents reported that they were “Very confident” of most of the event dates (79%).Overall, 89% of dated events were initially specified to at least their month of occurrence.
Respondents’ explanations for how they arrived at their datings of crime events drew on a fairly small and consistent set of factors. Respondents were most likely to cite elements from episodic memory to support their knowledge; these included details of the event such as “It was taken by a family member; he had come to visit,” and causes or consequences of the event, such as “Because I had a temporary phone for like two months because I didn't have the hundred dollars to pay the deductible to replace my phone.” Respondents were also likely to date crimes with reference to events of personal importance, such as “It happened during the move” or “I have not been to the pool since then.” Other factors frequently cited include the weather at the time or the fact that the incident was very recent.
Summary and Conclusions from the Cognitive Interviews
The results from the cognitive testing of the ECP confirmed that respondents could complete this task successfully. Further, the testing yielded insights into their thought processes as they attempt to determine the dates that crime events occurred. The testing informed several modifications to the instrument:
A question on the respondent’s trips away from home was expanded to delve into detail such as where the respondent went and feelings of safety at those destinations.
A question on how safe respondents feel at the places they go to on a regular basis was expanded to ask separately for each of three places they go.
The response categories for a question on trust in people was changed from a 5-point scale with only the ends labeled (“can be trusted a lot” to “cannot be trusted at all”) to a four-point agreement scale, which respondents found easier to use.
The ECP has two advantages. First, the ECP is brief and easy to administer in a standardized manner, and second, the ECP provides valuable data that can be used in further analysis for the NCVS. The data generated by the ECP memory aid increases the contextual data of the NCVS importantly makes use of the non-victims that are administered the screening questions.
1 Mean interview length and standard deviation was first calculated including all ECP interviews in Round 3. Three interviews in which the mean length was more than one standard deviation from the mean were identified. The mean interview length and standard deviation were recalculated excluding those cases; these data are presented in Table 3. With the three outliers included, mean interview length in minutes would have been 5.8 (s.d. =2.2).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Feel Good Inc. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |