OMB Passback Questions for 2011 PPCS

TO: Shelly Martinez

Office of Statistical and Science Policy

Office of Management and Budget

FROM: Christine Eith

Bureau of Justice Statistics

DATE: June 23, 2011

RE: Passback questions for 2011 PPCS

BJS received a total of 7 passback questions seeking additional information on the request for OMB clearance for the 2011 Police Public Contact Survey. Below please find the BJS responses to these questions. These changes have been integrated into a revised justification and questionnsire.

1.) We are quite concerned that the 2008 PPCS data are not yet out and are not familiar with the “classification” problem briefly mentioned in the SS.  Please provide more information about this problem, including how it was detected, how it is being addressed, and an updated release date.

The 2008 collection encountered significant classification error, due to use by interviewers of an “other specify” response category:

Persons who had more than one contact were only asked about their most recent contact that year. Respondents were provided a list of six specific reasons for having contact with police: (1) traffic accident, (2) traffic stop, (3) reporting a crime, (4) police provided assistance, (5) police investgating crime, and (6) suspected of something by police.

A seventh category allowed interviewers to record reasons that did not fall into any of the six specific reasons. Among the 9,549 respondents who had contact with police, 674 reported the reason for contact under this nonspecific category.

Following the data collection, BJS examined these write-in responses to determine if any descriptions fit one of the six categories and, when possible, coded the field into an existing category. This review uncovered 302 responses that indicated the reason for contact was related to a traffic stop.

Some of these records lacked sufficient detail to determine whether respondent was a driver or passenger in a vehicle that had been pulled over by police or was reporting some other type of contact that was connected to a traffic stop (e.g., paying a speeding ticket). Stopped drivers who were not originally classified under the specific category during the interview did not receive the additional questions about the traffic stop.

To address this problem, BJS instructed the Census Bureau to conduct follow-up interviews with 122 respondents (in the outgoing rotation) to seek clarification on the nature of their contact with police. Ninety respondents agreed to complete the follow-up interview. Of the completed interviews, 79 respondents reported that their contact with police occurred as a driver in a traffic stop and completed the additional set of questions regarding the traffic stops.

The information from the 79 respondents was added to the original file. The Methodology section of the 2008 report will contain a discussion of the misclassification and potential impact on the estimates.

The problem of misclassified cases was identified in April, 2009 when BJS received a preliminary data file. BJS met with Census on May 6, 2009 to discuss how to deal with the problem of these 302 misclassified stops. BJS and Census agreed that the best approach was to follow up with all of the outgoing sample respondents and ask them the series of questions that were missed in the original interview based on the misclassification. These follow-up interviews were scheduled to begin on July 1, 2009 and end on December 31, 2009. BJS made the decision to cut the follow-up operations at the end of September 2009 as it was decided that the extra 3 months of collection was not worth the small number of cases that would be received. The final 2008 data file, including the follow-up interviews, was received at BJS on October 26, 2009. BJS began processing the data in November, and have experienced some delays due to changing leadership in the organization. Once the Director was in place, the process has moved more quickly towards completion.

This issue has been resolved in the 2008 collection instrument, that will be used in the 2011 collection, with an additional direction to the field interviewer to check with the respondent if the reason for the most recent “other specify” response was traffic related. In the 2011 instrument, the opportunity for the field interviewer to misclassify was eliminated. In the revised screeners, individuals will be tracked directly into a specific set of topical questions based on their direct response to a series of possible contacts. When a respondent indicates a contact for something not addressed in 9 prior screener questions, the FR is directed to reiterate the 9 previous types of contact and repondents are looped through the appropriate items.

The 2008 report, Contacts Between the Police and Public, 2008, was delayed but is now scheduled for release on July 26, 2011.

2.) Please clarify if the 25% of the sample receiving the 2008 instrument in CAPI.  If so, please describe what BJS has done to ensure that any mode effects in moving from PAPI and CATI to all CAPI for the PPCS are understood and minimized.

The 2008 PPCS was collected in a single CAPI modality. The 2011 PPCS collection will use the same procedures (for the 15% of the sample that receives the 2008 instrument and for the 85% that receives the revised 2011 instrument). There are no expected mode effects between the 2008 and 2011 collections that could impact change estimates.

The paper and pencil modality (PAPI) was elminated in January 2006 (after the 2005 PPCS collection). Centralized CATI was eliminated in July 2007 (before the 2008 PPCS). Since July 2007, approximately a third of the NCVS interviews are conducted face-to-face in the sampled households (including all first interviews, all replacement households and all households requiring personal contact to obtain a response). This proportion has remained nearly constant since 2005.

While centralized CATI was eliminated (due to cost consideration), decentralized CATI (from the FR’s home) was continued. Approximately two-thirds of NCVS interviews are collected using this mode. BJS does not intend to change the relative use of these modes for the 2011 PPCS.

Overall, 16.9% of repondents reported a police contact in 2008, down from 19.1% in 2005. While the possibility of a mode effect exists for the PPCS 2005 and 2008 estimates, the impact is likely small. The shift from PAPI to CAPI (or decentralized CATI) did not change the proportion receiving a face-to-face contact. As a proportion of all contacts, traffic stops increased from 41.0% in 2008 to 44.1% in 2011.

3.) SS A 16 is supposed to cover project schedule and publication and analysis plans.  As written, it covers neither.  Please resubmit with additional detail.  In particular, we need to understand what will be published and when, particularly given that there are no publicly available results from three years ago.  We also need to understand how you will present changes in prevalence estimates occasioned by a changing definition of contact to include non-face-to-face contact.

The project schedule for the 2011 Police Public Contact Survey is as follows:

• Data collection for the 2011 PPCS is expected to run from July 1, 2011 through December 31, 2011.

• BJS expects to receive only one preliminary data snapshot, a quarter-3 file covering data collection from July and August, which is expected around the end of September 2011. This file will be used to assist Census and BJS in developing variable and value labes for each variable and to run intial checks on the data formatting to ensure that there are no errors in the structure of the data file.

• The complete public use data file is expected to be delivered to BJS by April 16, 2012. Once received, thhe data file will be checked for any errors and any necessary corrections will be made prior to data analysis.

• The final report, Contacts Between the Police and Public, 2011, is expected to be released by December 2012.

The 2011 analysis plan will expand upon the 2008 anlaysis:

In the 2011 report, the prevalence estimates will be provided in 2 ways. The first statistic will be equivalent to the 2008 estimates, including only face-to-face contacts with the police and excluding those classified as street stops. The second statistic will present the prevalence estimate including the face-to-face street stops and the non-face-to-face contacts for a complete prevalence of all contacts with police in 2011. There will then be an explanation provided to readers regarding the difference between the two measures, and how with improved measurement, BJS is able to capture a more complete prevalence estimate of contact with police.

BJS has the ability to analyze the data in two groups- a new prevalence measure with the new types of contact and a second prevalence measure that is comparable to previous measures in 2005 and 2008-to test comparability between the two collections to test actual change versus a change in measurement. The data will be analyzed separately to explain the changes in the prevalence measure when the non face-to-face contacts and street stops are included. This discussion will contain both the separate and the combined measure, as it is important to explain to users the difference in the two measures, and how the addition of the two additional types of contacts is improved over previous measures and how the prevalence of contact will be measured in future PPCS collections.

Additional analyses examining the change in contact rates using the two methodologies will be provided in the Methodology section of the 2011 report. A section in the methodology dedicated to explaining the changes in measurement in 2011 compared to past collection, and the results of the analysis comparing revisions between 2008 and 2011 within the split sample of the 2011 collection (See discussion in question #4, below).

4.) Related, please provide an analysis plans for the experiment cited above.

The goal of the split-sample experiment is to determine if the changes in the 2011 collection instrument, namely the improved screeners, have an impact on the prevalence measures (principally, percent reporting a police contact and percent reporting a traffic stop). An important part of the analysis of the 2011 PPCS data is a comparison with past PPCS data. BJS wants to avoid any break in series while introducing needed improvements to measurement.

Based on a power analysis completed by the Census Bureau, the PPCS collection agent, the plan is to split the sample so that 85% of the sample will receive the revised 2011 collection instrument and 15% will receive the 2008 collection instrument. (Note that the initial OMB clearance submission estimated a 75/25 percent allocation.)

The plan is to randomize the assignment of these instruments by forming reduction groups (of households) and then randomly assigning the questionnaires to each group. The formation of reduction groups (or subsamples) allows a reduction in the sample size that leaves a representative sample after the reduction. While the number of subsamples is somewhat arbitrary, a useful attribute is that it is prime to the number of rotation groups (six) so that reductions have a uniform effect across rotations.

For NCVS, there are typically four households in a hit string. All households in a hit string have the same reduction group number.

• For the unit, area, and group quarters (GQ) frames, hit strings are sorted and then sequentially assigned a reduction group code from 1 through 101.

• For the permit frame, a random start is generated for each stratification PSU and permit frame hits are assigned a reduction group code from 1 through 101 following a specific, nonsequential pattern. (This method of assigning reduction group code is used to improve balancing of reduction groups because of small permit sample sizes in some PSUs and the uncertainty about which PSUs will actually have permit samples over the life of the design.)

At the time of the determination of the split sample, all reduction groups will be in the sample. This permits a split in the sample to be achieved in the same way the Census Bureau would reduce the sample and maintain representative samples. Each reduction group will represent slightly less than one percent of the NCVS households. To obtain an 85/15 percent split, 15 of the 101 reductions groups, using a random start and systematic selection, will receive the old 2008 questionnaire and the rest will receive the new 2011 questionnaire.  All persons in the household will receive the same questionnaire type.

The expect levels of power from this split sample are presented in Table 1 below.  Using the 2008 data, Census simulated the proposed design for 2011 and calculated the police contact rate for each sample split and its respective standard error. The results show that the 85/15 percent split should be expected to detect a difference of about 7.5 percent, and a  75/25 percent split should detect a 6.7% difference at a 95% confidence level. For BJS’s purpose, the power from the 85/15 split was judged to be sufficient.

Table 1: Estimated Effects of Different Sample Splits

PPCS	0.25	0.75		0.15	0.85
Rate	17.24	16.81		16.50	16.99
CV	0.03	0.02		0.03	0.02
SE	0.47	0.34		0.56	0.32
VAR	0.22	0.11		0.31	0.10
VAR (diff)	0.33			0.42
SE (diff)	0.57			0.65
95% CI	1.13			1.27
% diff detect=CI/new rate	6.70			7.47

The 85/15 percent split will allow BJS to study the measurement effects between the 2008 and 2011 instruments using the same methodology and to determine if any observed change in the level of contact between 2008 and 2011 is true change or a result of changes to the questionaire. The key measures that will be analyzed are the overall contact rate and the proportion reporting a traffic stop contact.

With respect to the overall contact rate, the 1999 PPCS found that an estmated 20.9% of U.S. residents age 16 or older experienced face-to-face contact at least once with a police officer during the year. About the same percentage of contact between police and U.S. residents (21.1%) was identified when the PPCS was conducted 3 years later in 2002. An estimated 19.1% of U.S. residents had face-to-face contact with police in 2005 (which was a statistically significant decrease from the rates in 1999 and 2002). In 2008, the percentage of residents who experienced contact with police further decreased to 16.9%.

With changes to the 2011 questionnaire, it is expected that this prevalence will increase due to the change in measurement; however, this increase may be offset by the continuation of the general decline in police public contact from 1999 to 2008. BJS intends to decompose the change from 2008 to 2011 into component parts: (1) change due to new measurement and (2) change in actual levels of contact. To do so, BJS will examine the split sample in 2011 by calculating rates based on the new and old survey questionnaires. Any statistically significant difference will then be further decomposed to understand the underlying nature of the change (based on type of contact). This can only be done with a split sample.

Based on the comparison of the split samples, BJS intends to calculate a 2011 estimate adjusted for methodolgocial changes and compositional/definitional differences. The adjusted 2011 rate will then be based on a combination of the results from the old questionnaire and new questionnaire adjusted for the measurement effect. It is this adjusted rate in comparison with the 2008 PPCS rate that will be used as the measure of actual change in police contact. Similar analytical work is expected for measuring change in the proportion reporting a traffic stop. Further comparisons for subgroups will be examined, but the overall power to detect differences is based on the overall contact rate in 2008.

5.) Since no questionnaire change was identified to address the issue, what is BJS’s plan for dealing with the cognitive lab finding that individuals do not seem able to conceptualize a single police contact when there are multiple for a single incident?

Respondent recall is always a concern when asking for information of past experience. The PPCS has historically shown that of those respondents who have had contact with the police over the past year, approximately three quarters of the respondents have had only one contact. The issue of respondent recall – keeping the respondents’ focus on one event – has been addressed in the 2011 instrument with the addition of directions and reminders that will be read by the interviewer.

The revisions to the 2011 PPCS instrument have been focused on reinforcing the bounding period - by adding detail in the questions to assist the respondent with his or her recall about a specific event. With each set of type of contact questions, the interviewer will read an introduction to the respondent that this is related to the most recent contact. This introduction and interviewer note can be found on pages 3, 13, and 22 of the survey instrument in Attachment 7 of the OMB package. In addition, the CAPI instrument has been designed to provide automatic fills so that the interviewer is able to remind the respondent about the type of contact they reported as their most recent.

6.) How do the race of the police officer questions proposed compare to those used in the past?  How do these categories mesh with the ongoing research on race of the offender?  Why is the two question format used rather than a combined format?

BJS intends to revise the race of officer measure for the 2011 PPCS collection, specifically questions 13,15, 40 and 42 of the questionaire. The question will provide each respondent 5 separate racial categories (i.e., White, Black or African American, American Indian or Alaska Native, Asian, and Native Hawaiian or other Pacific Islander) to identify the race of the officer(s) with whom they had contact, and allow for the respondent to mark all races that apply. The 2011 collection instrument will also include a separate question prior to the race question to obtain the Hispanic or Latino origin of the officer. (See attached questions, below.)

Our intent is to follow OMB guidance. In NCVS, whether it is the race or ethnicity of offenders or police officers, we have found that there is considerable difficulty for some respondents to recall exact racial/ethnic categories. In our view, asking race as a separate question with distinct categories preceeded by a question specifically asking about Hispanic origin is preferable to asking a single question with many combined response categories. This approach often accounts for Hispanic origin as an ethnicity rather than a race, and allows for the respondent to provide racial identification independently. If a respondent is not capable of recalling the Hispanic origin, but able to recall the race of the office, the two-question design is a cognitively simpler task and may provide higher quality data. Such a respondent can provide a valid response on the race question and a “don’t know” on the Hispanic origin question.

At this time, the NCVS collects information on the race of the offender somewhat differently. It is BJS’s intention to adopt the same 2 question format with Hispanic origin followed by the full 5 racial categories starting in January 2012.

Questions regarding Race and Hispanic/Latino origin of officer(s) -

If there was a single officer present at the contact-

Was the police officer of Hispanic or Latino origin?

1. Yes

2. No

3. Don’t know

What was the race of the police officer? You may select one or more. (Read each category.)

1. White

2. Black or African American

3. American Indian or Alaska Native

4. Asian

5. Native Hawaiian or other Pacific Islander

6. Don’t know

If there were multiple officers present during the contact –

Was one or more of the police officers of Hispanic or Latino origin?

1. Yes

2. No

3. Don’t know

Were one or more of the police officers: (Read each category) You may select one or more for each officer.

1. White

2. Black or African American

3. American Indian or Alaska Native

4. Asian

5. Native Hawaiian or other Pacific Islander

6. Don’t know

7.) Please provide the rationale for interviewing age 16+ rather than 12+ for this module.

The survey items are designed to address specifc issues that can be directly related to one’s age – in particular being stopped by police while driving a vehilce. The 1996 pretest suggested that data collected from juveniles between the age of 12 and 15 did not have a significant impact on the key outcome variables: prevalence of contact and use of force. As a result, it was decided at that time to exclude youth under age 16.

In 2008, the majority of respondents who reported having interaction with police stated the contact was traffic related. In an estimated 44% of these stops, the respondent was the driver. Since persons under the age of 16 are not eligible for a drivers’ license in most states, inclusion in the PPCS would likely show that almost all of the 12 to 15 year olds involved in the traffic stops were passengers.

The Police Public Contact Survey was designed to be administered to an adult population. There would be a signficant investment of resources to revise the collection for 2011 to be appropriate for a younger population, including a revised questionaire and additional cognitive tests for the younger age group. In addition, there would be signficant issues with comparability with the past 10 years of data that have been limited to a sample of respondents age 16 and older.

In combination with the higher data collection costs and lower response rates among persons in this age group, we believe that it is prudent to continue to exclude persons under age 16 from the PPCS.