Justification Memo

December 7, 2011

BJS Request for OMB Clearance for Field Test under the National Crime Victimization Survey (NCVS) Redesign Generic Clearance, OMB Number 1121-0325.

Question

Because the power of the test related to the impact of ECP on victimization rates is low, are there other areas that the study will explore that would have a greater probability for identifying a statistically significant finding related to the utility of adding ECP questions?

Response

The following research question has been added to the ECP project. Does the use of enhanced contextual priming increase response rates to the crime screener by increasing respondent engagement in the survey?

In addition to aiding the recall of crime incidents, the ECP may also increase respondent engagement with the survey both for respondents who have experienced victimizations and those who have not. The ECP questions touch upon issues that respondents may have experienced in their everyday lives, whether they have experienced a crime or not. Respondents’ reports from the cognitive interviews suggested that the majority of respondents, victims and non-victims alike, found the ECP questions interesting. Many respondents commented that the ECP section contained questions that made them think about their own lives or their neighborhoods. One respondent stated that these questions “cause you to reflect on things that you kind of block out of your mind” while another specifically stated that “the questions about safety were interesting because they made me think about my own area.”  It is important to keep in mind that the respondents volunteered to participate in the survey (by responding to advertisements on a crime or neighborhood safety survey). However, the overwhelmingly positive reaction of the participants suggests that many field test respondents will also find the ECP questions to be relevant and of interest.

Evidence from the literature suggests that respondent interest in the survey influences decisions to participate and to complete a survey (e.g., Groves, Singer, &Corning, 2000; Groves, Presser, & Dipko, 2004; Galesic, 2006). In a web survey on unemployment (Galesic, 2006), level of interest in the survey questions along with the perception that the survey is not burdensome increased the likelihood that respondents completed the survey. This study further suggested that respondents who completed the survey yielded higher quality data with regard to open-ended answers than those who dropped out before the end. Groves et al. (2000) found that respondents who were motivated by a sense that community involvement were more likely to agree to do a survey (and less likely to be influenced by an incentive) than those who had a lesser sense of community involvement. They found response rates of 58% vs. 43%, respectively, for the high vs. low community involvement samples, a difference of 15%. In another study, Groves et al. (2004) compared response rates depending on whether the survey topic was one of interest to the respondent. Samples of teachers, new parents, adults age 65 and older, and political contributors were given one of four survey introductions (e.g., education, Medicare, etc.), one of which was assumed to be of greater interest given the respondent’s characteristics. Groves et al. (2004) found that response rates were up to 14% higher when the survey topic was of interest to the respondent than when it was not.

By highlighting issues of interest and importance to respondents, the ECP may increase respondent engagement in the survey. In turn, this engagement may yield fewer break-offs and higher response rates for the crime screener when it follows the ECP. As part of the field test we will examine the hypothesis that response rates to the crime screener will be higher in the ECP as compared to the Control condition.

A number of articles on the effects of the NCS attitude supplement on crime rates were examined to explore the effect of the attitude supplement on response rates (Gibson, Shapiro, Murphy, & Stanko, 1978; Murphy, 1976; Cowan, Murphy, & Wiener, 1978; Kalton & Schuman, 1982). None of these articles provided a breakdown of response rates for households or respondents based on whether they received the attitude supplement. However, the more recent work on respondent engagement provides data that allow for an estimate of the effects of increased engagement on response rates. The findings of Groves et al. (2000) and Groves et al. (2004) suggest that respondent interest in a survey enhances response rates, with observed increases up to 15%.

We expect the CASRO response rate in the Control condition will be 25%. We expect that the added survey interest provided by the ECP could increase the response rate in that condition by a few percentage points. Since the Groves et al. (2000, 2004) studies suggest increases of up to 15%, we make a conservative estimate of a 5% increase in response rates with the addition of the ECP, for a CASRO response rate of 30%. The design of the field test is shown in Table 1.

Table 1: Field Test Design

Mode	Memory Aid
	6-Month Control (No Memory Aid)	Enhanced Contextual Priming
Telephone/CATI	1,000	1,000

Burden Hours for Field Test

The burden hours requested for the field test is 1,339 hours. Approximately 2,000 respondents will be interviewed, and the approximate time required for each respondent is .28 hour on average. These estimates are derived from analyzing the cognitive interviewing data (Attachment 1 – Findings from Cognitive Interviewing). The findings from the cognitive interviews show that the average administration time for the ECP questions was approximately 5 minutes (.08 hour), with an average of 7.7 minutes (.13 hour) to answer the screener questions. Respondents averaged a completed incident report in 7 minutes (.12 hour). The consent process at the beginning and the additional demographic questions at the end of the interview are estimated to add 5 minutes (.04 hour consent and .04 hour, respectively) to the interview. No more than 4 incident reports per respondent will be collected during the field test. Hence, in Table 2 we assume an average of two incident reports for each respondent who reports victimization. For more detailed information on the sample, please refer to Table 6.

Table 2: Administration Times

	Reporting Method	Number of Respondents	Average Time (hours)	Total Burden (hours)
From RDD sample size of 26,646 numbers, 45% (11,991) will be resolved before CATI and 55% (14,655) will be worked. Estimated 1 minute of respondent burden (.017 hour) per number.
	CATI telephone numbers dialed	14,655	.017	249
Of 14,655 numbers worked in CATI, 45% (6,595) will be Working Residential Numbers (WRN). Estimated 5 minutes of respondent burden (.08 hour). Of the 6,595 working WRN, an estimated 80% (5,276) of households contact with an eligible adult will be made; in 20% (1,319) of households, no contact with an eligible adult will be made.
	CATI-WRN dialed for interview	6,595	.08	528
Of the 5,276 household contacts, an estimated 2,000 of the contacted eligible adults will agree to be interviewed. For 2,000 completed cases, time to administer consent is estimated to be 2.5 minutes (.04 hour).
	CATI-Consent	2,000	.04	80
Half of the 2,000 completed cases will be in the Control Condition and half in the ECP Treatment condition. Control cases will complete the crime screener in about 7.7 minutes (.13 hour). Treatment cases will complete the ECP module in 5 minutes (.08 hour), followed by the crime screener (.13 hour), for a total of .21 hour.
	CATI- Screener-Control	1,000	.13	130
	CATI-Screener-ECP Treatment	1,000	.21	210
Based on 2009 NCVS estimated crime rate of 127.4 per 1,000, in the Control condition an estimated 127 respondents will complete one or more incident reports. In the ECP Treatment condition, an estimated 144 respondents will complete one or more incident reports based on the predicted improvement found in earlier research. The estimated burden per incident report is 7 minutes (.12 hour). For burden calculation, we have assumed an average of two incident reports for those respondents who indicate one or more crimes in the crime screener (14 minutes, or.23 hour).
	CATI-Incident report-Control	127	.23	29
	CATI-Incident Report- ECP Treatment	144	.23	33
We have estimated a respondent burden of 2.5 minutes (.04 hour) to complete the demographic questions.
	CATI-Demographics	2,000	.04	80
Total				1,339 hours

Power and Precision Analysis

In calculating power and precision for the field test, we examined data from Cowan et al. (1978) on the effects of an attitude supplement on estimates of crime victimization. Half of the sample of respondents received an attitude supplement on crime prior to answering the NCVS screener and half completed the crime screener without the supplement. Table 3 summarizes the effects of the attitude supplement on crime rates. As can be seen in the table, the crime rate in the 13 cities sampled is somewhat higher than the national estimates from the NCS/NCVS. The relative change in crime estimates varies from 22.25% for violent crime and 12.69% for property crime.

Table 3: Crime rates with and without an attitude supplement

	No attitude supplement	With attitude supplement	Relative change (%)
Violent crime	48.27	59.01	22.25
Personal theft	97.56	112.70	15.52
Property crime	447.68	504.49	12.69

Note: Data from Cowan et al. (1978). The “no attitude supplement” condition is considered the reference point in determining relative change. Table 3 represents victimization rate per 1,000 persons or per 1,000 households.

We assumed the lowest level of relative change observed when the attitude supplement is included before the crime screener (12.69%, rounded up to 13%). Taking the lowest relative change observed will provide the most conservative assumption of the effect of the ECP on crime rates with respect to the sample size. The following assumptions were made:

• Following standard statistical assumptions, precision will be held constant at 95% for the power calculation.

• Number of completed cases will be 1,000 per condition (2,000 total): The remaining budget for the project supports the completion of 2,000 cases. The projected number of completed cases will be distributed evenly across the Control and Treatment conditions.

• The use of ECP will increase reporting of crime over the Control condition (one-sided test): Based on the work of Cowan et al. (1978), we expect that the ECP will increase crime reporting. The interest in the use of the ECP centers around its facilitative effects on reporting; our hypothesis reflects this expectation that the ECP will increase reporting.

• The 2009 NCVS property crime rate (127.4 per 1,000) will be adopted as the 6M Control value: The NCVS data provide the best estimate of what the crime rate will be in the Control condition of the field test. The Control condition reflects the current design of the NCVS, using the same reference period and crime screener. Therefore, the crime rate observed in the Control condition is expected to approximate that found in the NCVS.

• The increase in crime rates with the ECP will be 13%, the observed relative change in property crime rates when the attitude supplement was added to the NCS: This estimate of the relative change in crime rates that we expect to observe is based on the work of Cowan et al. (1978), in which the effect of an attitude supplement on the crime rates was examined. Although an increase in reporting is expected for violent crime and personal theft as well, we focus on the property crime rates because the incidence of these crimes is higher than for other crimes. Given the small sample size for the field test, it will be more difficult to observe changes in the rate of more rare crime events.

• The increase in response rates will be from 25% to 30%, a relative change of 20%, when the ECP is administered.

Analyses were conducted to determine the power and precision of 1,000 completed cases in the Control and ECP conditions. Precision is the width of the interval with which we try to estimate the true population value; precision estimates increase with larger sample sizes. Also, with a larger percent change between the Control and treatment conditions, we would have a higher likelihood of detecting the change as significant. Table 4 presents the confidence level for detecting 13% change in crime rates between the Control and treatment values for 1,000 cases per cell; this table also shows the confidence level for detecting 20% change in response rates between the Control and treatment conditions. This table tells us that 85% (for crime rates) or 99% (for response rates) of the time, we would observe that the percent change between Control and Treatment condition is greater than 0% (that is, that the direction of the effect is positive). If we wished to observe that the difference is greater than, say, 10%, our confidence would be much lower. Only 59% of the time would we observe that the percent change between Control and Treatment is greater than 10%.

Table 4: Confidence level

Hypothesis	Cell size	Proportion Control	Proportion Treatment	% change	S.E. of % change	Confidence level (%, one-tail)
Crime rates	1000	0.1274	0.1440	13	0.1278	85
Response rates	1000	0.25	0.30	20	0.0876	99

The power of a statistical test is its probability of correctly rejecting a null hypothesis; it tells us the likelihood that we will be able to detect a difference when the difference exists between groups. Power analysis is often done prior to a study in order to understand the likelihood of making a Type II error (failing to detect a difference that exists). Table 5 presents the power of the experiment to detect 13% change in crime rates or 20% change in response rates between the Control and treatment values for 1,000 cases per cell.

Table 5: Power

Hypothesis	Cell size	Proportion Control	Proportion Treatment	% Change	Power (%)
Crime rates	1000	0.1274	0.1440	13	29
Response rates	1000	0.25	0.30	20	80

Although the power and precision of the field test is not at an optimal level, the experiment is still capable of providing valuable information in two key areas. First, we expect to see the direction of the effect, that is, whether including the ECP affects recall and response rates, even though ascertaining statistical significance will be difficult. More importantly, the addition of attitudinal questions in the NCVS has not been examined since the 1970's. At various times in the history of the survey, BJS has considered incorporating attitudinal and behavioral questions as a method of utilizing the non-victims that are screened for crime thereby increasing the analytical value of the survey by generating valuable data by which to examine victimization. This study will provide initial findings on how respondents receive these questions and whether administering these questions in a CATI environment is viable.

Sample Design

Note that a test of the ECP vs. Control requires two comparable samples (not necessarily two samples that represent the general population) to determine whether the ECP elicits greater recall of crimes and higher response rates.  However, an RDD sample for the field test is desirable because it allows us to obtain the best evidence as to how the ECP memory aid is received by the general population. 

The target completed case goal for the field test will be 2,000 completed CATI interviews. As Table 6 indicates, the starting Random Digit Dial (RDD) sample size for the field test will be 26,646 cases. The “rate” column indicates the percentage of cases expected to be worked in the telephone center (that is, not identified through pre-screening as non-residential or non-working numbers), the resolution rate, working residential number (WRN) rate, rate of contact with household (HH) adult, and response rate. For reference, the terms NORC uses when discussing the sample and response rates are:

• Random Digit Dial (RDD): A method of selecting numbers by generating them at random;

• Resolution Rate: The expected rate of numbers that will be identified as either a residential number, a business number, or a non-working number;

• Working Residential Number (WRN) rate: The expected rate of telephone numbers that can be identified as valid numbers associated with a household;

• Rate of contact with household (HH) adult: The amount of households where NORC expects to make contact with an adult member of the household; and

• Council of American Survey Research Organizations (CASRO) Response rate: The CASRO response rate is the product of the resolution rate, contact with HH adult rate, and the CATI screener response rate. It takes into account the cases presumed eligible (but unresolved) in calculation of response rate.

We expect that 45% of numbers in the sample will be resolved before CATI. Of the remaining 55% of numbers that are called, we expect 90% of the numbers to be resolved as business or residential. From the resolved numbers, 50% will be identified as WRN. We expect to make contact with an adult in 80% of those households (that is, an adult who can speak for the household) and to complete the NCVS crime screener with about 35% of those households in the Control condition and about 42% of households in the ECP condition. The different NCVS crime screener response rates for the Control and ECP conditions shown in Table 6 are based on our hypothesis that respondent engagement will be higher in the ECP condition and that this engagement will increase cooperation. Because a lower percentage of contacted adults are expected to complete the crime screener in the Control as compared to the ECP condition, more cases will be attempted in the Control condition to arrive at 1,000 completes. The eligibility rate is set at 100% since there is no other screening criterion for participation other than that we speak to the most knowledgeable adult. On the basis of these assumptions, we estimate that we will start with a sample size of 26,646 to arrive at 2,000 completed cases.

The rates that NORC assumes for resolution of phone numbers before CATI, resolution rate, working residential numbers, and response are based on NORC’s broad experience with RDD studies such as the National Immunization Survey (NIS), Racial and Ethnic Approaches to Community Health (REACH), and Trends in US Public Awareness of Racial and Ethnic Disparities in Health (OMH).

Table 6: Estimated sample size and rates

	Rate	Sample
RDD Sample		26646
Phone numbers resolved before CATI	0.45	(11991)
Total phone numbers called	0.55	14655
Resolved phone numbers—Resolution rate	0.90	13190
Households identified—Working Residential Number Rate	0.50	6595
Contact with HH adult	0.8	5276
Eligibility rate Cases assigned to Control Cases assigned to ECP	1.00	(2878) (2398)
CATI Screener Response—Control CATI Screener Response—ECP	0.3475 0.4170	1000 1000
Total Completes		2000
CASRO response rate Control ECP	25% 30%

Works Cited

Cowan, C.D., Murphy, L.R., & Wiener, J. (1978). “Effects of supplemental questions on victimization estimates from the National Crime Survey.” Proceedings of the American Statistical Association, (pp. 277-282).

Galesic, M. (2006). Dropouts on the web: Effects of interest and burden experienced during an online survey. Journal of Official Statistics, 22(2), 313-328.

Gibson, C. O., Shapiro, G. M., Murphy, L. R., & Stanko, G. J. (1978). Interaction of Survey Questions as it Related to Interviewer-Respondent Bias. Proceedings of the American Statistical Association, (pp. 251-256).

Groves, R. M., Presser, S., & Dipko, S. (2004). The Role of Topic Interest in Survey Participation Decisions. Public Opinion Quarterly, 68(1), 2-31.

Groves, R.M., Singer, E., & Corning, A. (2000). “Leverage-saliency Theory of Survey Participation: Description and Illustration.” Public Opinion Quarterly, 64, 299–308.

Kalton, G., & Schuman, H. (1982). The Effect of the Question on Survey Responses: A Review. Journal of the Royal Statistical Society. Series A (General), 145(1), 42-73.

Murphy, L. R. (1976). Effect of Attitude Supplement on NCS—Cities Sample Victimization Data. Washington, D.C.: Bureau of the Census.

7