Justification Memo

Date:	December 6, 2010
To:	Michael Rand and Shannan Catalano
From:	David Cantor and Pat Dean Brick
Subject:	Behavior Coding and Comprehension probes

OMB has asked for elaboration on two points for the usability testing. One is to specify the behavior coding scheme to be used. The second is to provide additional probes to assess comprehension. Below each of these issues is addressed.

Behavior coding scheme

As noted in the analysis plan, the evaluation will rely partly on observations of issues respondents have when answering particular questions. The behaviors that would be coded would be the following:

Did respondent press an incorrect key?

Did respondent back up?

Did respondent time out?

Did respondent look confused or frustrated?

Did respondent use the ‘help’ function?

Counts of each activity, by question, will be cumulated and used as an indicator of possible issues for the item.

Comprehension probes

Our intent is to use the open-ended probing that we have in the debriefing as a way to get at comprehension issues. In combination with reactions from the respondent (e.g., looks of confusion), actions taken (e.g., time outs; backing up) and our own knowledge of the correct answer, we will administer these open ended probes to get the respondent to describe their logic for the answer they provided. The open-ended probes provided in the OMB package were intended for this purpose. Based on responses to probes like

“You just said that you ~~~. Can you say more about that?”

We expect the respondent will provide their logic for their answer. Use of open-ended probes like these will usually provide the information that is needed to assess comprehension. The interviewers will be trained to probe to get a complete picture. These are deliberately set as open-ended because of the possible contamination effects of using more specific probes which can lead to misleading data.¹

If the open-ended probes do not yield enough information on comprehension, we will administer more targeted probes. This revised protocol is shown in Table 1, with the text in italics being the added for the follow-up probes.

As an example of how this might work, question HC005A asks about the location of the incident. If the interviewer needs to use the additional probes, they would ask:

What does “in your own yard, sidewalk, driveway, carport, or unenclosed porch?” mean to you?”

As noted above, our plan is to apply these follow-up probes for any responses that meet one of the three criteria noted above (expressions of confusion, time outs or backing up and inaccurate responses). Time permitting, additional questions will be targeted based on our a-priori suspicions that respondents may have problems with the understanding them (Table 2). These problems might arise because the questions were altered in some way when converting the NCVS to an IVR mode or the questions themselves may not fully translate from interviewer-administered to IVR. For example, the NCVS-2 has a number of open-ended questions with relatively long lists which the interviewer can code based on the respondents answer (e.g., location of incident; types of property stolen; types of injuries). The IVR requires the respondent to self-classify themselves into one of the categories. This involves going through a series of questions, asking about each particular possibility (e.g., did it happen inside the home? Did it happen at school?, etc..). We will be particularly interested in how respondents comprehend these types of items.

In order to insure that the study collects enough information about the incident prior to the testing, we would like to expand the questions that will be asked on the screener used to recruit respondents. This is a replacement for Attachment 3 in the original package. We are adding 5 questions to the screener related to the details of the victimization event. Adding these questions will insure the interviewer has all of the requisite information about the event prior to the interview. Adding these questions increases our estimate of the administration time for the recruitment screener from 1.5 minutes to 3 minutes (goes from 7 to 12 questions). This increases the estimate of burden for the request from 5 to 10 hours.

Table 1. Revised debriefing probes for the NCVS-1 and NCVS-2 instruments*

We have some questions that we want to ask you, but before we do that, we would like you to tell us in your own words about what you just did – explain the process, what you did, how it worked, what kinds of questions you answered, etc.

REVIEW SECTIONS WHERE THERE WERE NOTICEABLE DIFFICULTIES WHILE GOING THROUGH THE INSTRUMENT.

REVIEW OTHER SECTIONS, NOT COVERED BY DIFFICULTIES.

[PROBE AS NECESSARY TO GET THE RESPONDENT TALKING ABOUT HIS/HER EXPERIENCE OF THE IVR.]

You just said that you ~~~. Can you say more about that?

You mentioned the ~~~~ ; how did that work for you?

[IF IT IS NOT CLEAR IF RESPONDENT UNDERSTOOD QUESTION, FOLLOW-UP WITH THE PROBE]

In this question, you were asked ~~~~~~~~~. Can you tell me in your own words what this means to you?

In this question, what does ~~~~~~~ mean to you?

*Additional probes in Italics

Table 2. Items targeted for probing

NCVS-1 (Screening instrument)

HC003 – Number of vehicles owned

HC005A, HC005C – threatened or something stolen

HC006A, HC006C – attacked or threatened with other violence

HC007A – Incidents by someone known to respondent.

HC008A – Sexual assault incidents

HC009B – Personal attack or threat

NCVS-2 (Detailed Incident Form)

EC001 – Anchors the respondent to what was said on the screener

EC001B – Asking about series crimes (only those that report more than one incident)

EC002A – Asks if the incident occurred outside of a town, city or village

EC003A to EC003H – Initial definitions of where the incident occurred

EC012 – Did offender have a right to be there?

EC016A, B, C – Evidence of tampering on window, door or something else

EC17A, B – Presence of respondent and other people during the incident

EC018 – Presence of a weapon of some type

EC027, EC028, EC027B, EC029 – Definition of rape, sexual assault that is not rape

EC044A to EC045G – Property that was taken

EC046A to EC046C – Who did the property belong to?

1 Conrad, F. G. and Blair, J. (2009). Sources of error in cognitive interviews. Public Opinion Quarterly, 73, 32-55.