Section B Eval Community Oriented Enforcement 06222016

SECTION B

INFORMATION COLLECTION

SUPPORTING STATEMENT

Evaluation of Community-Oriented Enforcement Demonstrations

B) Collections of Information Employing Statistical Methods

NHTSA is seeking approval to gather information on changes in public support for enforcement in program and control (comparison) areas as part of the evaluation of the agency’s community-oriented enforcement demonstrations, Building Community Support for Impaired Driving Enforcement and Building Community Support for Seat Belt Enforcement.

As seen in Table 1, the total estimated respondent contacts for this proposed data collection is 21,216. This includes 16,416 respondents to complete the survey screener (i.e., Form 1321) and 2,400 respondents to complete each of the surveys (i.e., Forms 1322 and 1325).

Table 1. Total Contacts by Form

Form	Program/Control	Pre-Program	Mid-Program	Post-Program	Total Contacts
Form 1321 Screener	Form 1322 Program	1,368	1,368	1,368	4,104
	Form 1322 Control	1,368	1,368	1,368	4,104
	Form 1325 Program	1,368	1,368	1,368	4,104
	Form 1325 Control	1,368	1,368	1,368	4,104
	Total 1321	5,472	5,472	5,472	16,416
Form 1322 Impaired Driving Program Survey	Program	400	400	400	1,200
	Control	400	400	400	1,200
	Total 1322	800	800	800	2,400
Form 1325 Seat Belt Program Survey	Program	400	400	400	1,200
	Control	400	400	400	1,200
	Total 1325	800	800	800	2,400
Grand Total Contacts	Forms 1321, 1322, and 1325	7,072	7,072	7,072	21,216

As seen in Table 2, NHTSA estimates a 45% response rate^1,2 and a 65% eligibility rate.³ In order to reach 400 completed surveys per period (i.e., pre, mid, and post program), NHTSA estimates that 1,368 respondents will need to be screened. Of those screened, it is estimated that 889 will be eligible to participate, and of those who are eligible, it is estimated that 400 will participate and complete the survey.

Table 2. Breakdown of Eligibility and Response Rate Estimates

	Rate Estimate	Contacts
Contacts Participate and Complete Survey	45%	400
Contacts Eligible	65%	889
Total Contacts Approached	--	1,368

As typical with program evaluations of this nature, survey administration will follow a nonequivalent control group design. The intention of this design is to measure the effect of a program by taking a yardstick measurement pre, mid, and post program to determine change. With strong consistency in measurement protocol across measurement periods and similarity across the program and control samples to limit extraneous influences on the results, this design can produce a non-biased and reliable indication of change.

Utility of this evaluation does not necessitate administering to a probability based sample and generating representative results. The objective of the evaluation is to take a reliable measure of change in public support for enforcement, not to estimate what represents the community as a whole.

The design has strong consistency in measurement and sampling protocol across measurement periods and includes built in methods for addressing any inherent differences between the program and control samples. The nonequivalent control group design is susceptible to the threat of internal validity, as the group of respondents in the program and control areas may have been characteristically different prior to the intervention, possibly contributing to differences in response independent of the intervention effect. Ideally with nonequivalent control group designs, the two groups (program and control) would be characteristically similar prior to the intervention, just differ by intervention exposure, to produce results that reflect the influence of the intervention. However, with some programs, the evaluator has less control over site selection and must adapt to the realities of the situation, including any differences in the program and control samples. For the current project, any inherent differences across the program and control samples will be accounted for by weighting the data to the population to address any biases in the sample.

While nonequivalent control group design lacks random assignment, inherent characteristics of the sample universe and rigorous protocols can reduce bias in the sample. All licensed drivers must visit the Department of Motor Vehicles (DMV) office periodically for license renewal. The date of license renewal is based on birthday; therefore, it is reasonable to treat customers renewing their license on a given day as a random sample of the population of all available respondents (i.e., licensed drivers in area). In most cases, all visitors to the DMV office location will be approached. In cases where the office is too busy to approach all customers, data collectors will apply a systematic sampling interval to approach either every 2^nd or 4^th visitor, depending on the estimated office traffic flow and target sample for the day.

Completed survey forms will be delivered to the Contractor where a data entry person will enter survey response data into Microsoft Access to allow for analysis. ANOVA, F-tests, t-tests, and logistic regression will be used to determine whether there are any statistically significant changes that can be attributed to the program activity.

B.1 Describe the potential respondent universe and any sampling or other respondent selection method to be used.

The potential respondent universe is comprised of licensed drivers aged 18 years and older visiting the Department of Motor Vehicles (DMV) offices in the program and control site locations. From this universe, the new data collection will contact a total of 21,216 drivers.

As seen in Table 1, each measurement period will have a sample of 400 completed surveys. A power analysis indicated that for a population of 100,000, a sample of 383 respondents would be sufficient to achieve a 95% confidence interval and 5% margin of error. Because the programs will be conducted in sites with between 75,000 and 200,000 residents, the power analysis indicated that a sample size of 400 completed surveys for each measurement period would be sufficient for the proposed data collection.

The program and control sites for these programs have not been selected as of the time of this request. The site selection process is currently underway using criteria for selection that places the sites within the following parameters:

• Community with population between 75,000 and 200,000

• Local government and law enforcement interested in the project

• Seat belt use below the national average, unrestrained fatalities above the national average, and lower levels of seat belt enforcement (Seat belt program only)

• Impaired driving related fatality crashes above the national average (Alcohol-impaired-driving program only)

The research team will select control areas that are demographically similar to the program areas, which will help control for response differences due to the population rather than exposure to the intervention. Also, the research team will find control areas that are in a different media market to help reduce program bleed into the control area.

The data will be weighted to reflect the demographic makeup of each geographic location. The weighting process for this study entails two major steps. In the first step, target population benchmarks will be created for computation of weight factors. For this purpose we will rely on public data sources such as Current Population Survey (CPS) or American Community Survey (ASC) as well as commercial sources such as Claritas to obtain demographic profiles of adults in each geographic location. In the second step, an iterative proportional fitting procedure will be used to balance the composition of respondents in each location to their respective demographic profiles obtained during the first step. It is anticipated that weight adjustments will include characteristics such as gender, age, race, and ethnicity.

After site selection takes place, the research team will locate all of the Department of Motor Vehicles (DMV) offices in the sites to select the DMV offices for data collection. The research team will consider if the DMV Offices are sufficiently spread out across the participating areas. The distribution of the offices will help achieve external validity of the surveys because it will be more regionally inclusive and help alleviate the effect of regional differences on responses.

After site selection takes place, the research team will contact the DMV offices in the program and control sites to inquire about the magnitude of typical office traffic. The DMV reports of traffic will also help guide office selection because high traffic volume locations will maximize the sample and provide further evidence of local dependence on these offices for services.

B.2. Describe the procedures for the collection of information

The surveys will be administered in-person to licensed drivers aged 18 years and older at Department of Motor Vehicles (DMV) offices in the program and control areas for both programs. Data collectors will employ a multi- step process to survey respondents: (1) interception, (2) determine eligibility, (3) recruitment, and (4) completion of questionnaire. When office traffic to too heavy to sample all visitors, the research team will select participants using a systematic sampling interval by sampling every 2^nd or 4^th visitor, depending on the target sample for the day.

Upon approaching a potential participant, the screening interviewer will introduce him or herself and give a brief explanation of the study following a pre-determined script for this initial contact. Following the initial interception, the interviewer will verbally administer the survey screener (i.e., Form 1321) to the participant to determine if they are eligible. The objective of the screening questions is to determine if the approached respondent is a licensed driver who is aged 18 years or older. The screening interviewer will review each screening question with the participant. Based on this conversation, the interviewer will determine eligibility.

Regardless of the eligibility determination, the interviewer will enter a “disposition” code onto the survey screener (i.e., Form 1321) to indicate the results of the screening. Examples of disposition codes are:

• Ineligible

• Refusal

• Other (specify)

Once the eligibility of the driver has been determined, the interviewer will endeavor to recruit eligible participants to complete the questionnaire. In general, this will not be a scripted dialog, but the team member will cover key elements, which include additional details on the study, and an estimated time for completion. If the screening interviewer is successful in recruiting the driver, he or she will hand the survey to complete the remaining survey questions.

There will be two versions of the survey (Forms 1322 and 1325). The first version of the survey (Form 1322) will be used for the alcohol-impaired-driving program, Building Community Support for Impaired Driving Enforcement. The second version of the survey (Form 1325) will be used for the seat belt program, Building Community Support for Seat Belt Enforcement. Two unique forms are needed because the surveys will ask questions specific to the subject matter of each program. For example, the alcohol-impaired-driving program survey will ask about support of alcohol-impaired-driving enforcement, and the seat belt program will ask about support of seat belt enforcement.

B.3) Describe methods to maximize response rates and to deal with issues of non-response.

Participation in this study is voluntary. Several methods will be utilized to maximize response rates, including:

• Administering the survey at DMV offices where potential respondents will already be waiting for service, and are more likely to have extra time to complete the survey; and

• Providing a Spanish-language translation of the awareness survey questionnaire to minimize language barriers to participation.

NHTSA does not expect to address non-response bias in this context but will be weighting the data to the population to address any biases in the sample.

B.4. Describe tests for procedures or methods to be undertaken

As part of the study design, the Contractor will refine the study procedures by pilot testing the screener and two survey instruments (i.e., Forms 1321, 1322, and 1325) with nine participants. The pilot study will allow the Contractor to conduct an assessment of the overall comprehension of the individual survey questions. Testing the instructions and questionnaires prior to implementing the study will provide the Contractor with the opportunity to make slight wording changes when needed that will improve overall comprehension without changing the intent or direction of the questions.

While NHTSA must account for this possibility, NHTSA foresees minimal changes to result from the pilot testing because it has consulted with in-house experts on survey development and adopted some questions from validated surveys used in previous efforts. 

B.5) Provide the name and telephone number of individuals consulted on statistical aspects of the design.

The following individuals have reviewed technical and statistical aspects of procedures that will be used to conduct the intercept surveys (listed alphabetically):

Mary Byrd, M.A. Office of Behavioral Safety Research (NTI-132)

(202) 366-5595 National Highway Traffic Safety Administration

[email protected] 1200 New Jersey Ave SE, W46-466

Washington DC, 29590

Arthur Goodwin, MA The University of North Carolina

919-843-5038 Highway Safety Research Center

[email protected] 730 Martin Luther King Jr. Blvd., Suite 300

Campus Box 3430

Chapel Hill, NC 27599-3430

Loren Staplin, PhD. TransAnalytics LLC

(215) 538-3820 ext 102 336 West Broad St.

[email protected] Quakertown, PA 18951

1 Ellis, C. S., Evans, B., Santiago, G. M., & Reed, L. M. (2007, May). Surveying International Travelers: An Argument for Intercept Interviewing. Presented at American Association for Public Opinion Research Conference, Anaheim, CA. Retrieved from http://www.rti.org/sites/default/files/resources/aapor07_ellis_paper.pdf

2 Duke, Joshua M., Ilvento, Thomas W. (2005) A Conjoint Analysis of Public Preferences for

Agricultural Land Preservation. Agricultural and Resource Economics Review. Retrieved from https://www.researchgate.net/publication/4902860_A_Conjoint_Analysis_of_Public_Preferences_for_Agricultural_Land_Preservation

3 The eligibility rate was calculated using Federal Highway Administration 2014 United States licensing data. Retrieved from http://www.fhwa.dot.gov/policyinformation/statistics/2014/. To be eligible, a respondent must be a licensed driver and 18 years old or older. In 2014, 67% of the US population had a driver’s license, and 1.4% of the licensed drivers were under 18 years old. Based upon these statistics, NHTSA estimates that about 65% of respondents will be eligible to participate (i.e., 67% - 1.4% = 65%).

6