Section B Eval Community Oriented Enforcement 04052017

SECTION B

INFORMATION COLLECTION

SUPPORTING STATEMENT

Evaluation of Community-Oriented Enforcement Demonstrations

B) Collections of Information Employing Statistical Methods

NHTSA is seeking approval to gather information on changes in public support for enforcement in program and control (comparison) areas as part of the evaluation of the agency’s community-oriented enforcement programs, Building Community Support for Impaired Driving Enforcement and Building Community Support for Seat Belt Enforcement.

The program and control sites for each enforcement demonstration program:

Building Community Support for Impaired Driving Enforcement

Program: Joplin, Missouri

Control: Cape Girardeau, Missouri

Building Community Support for Seat Belt Enforcement

Program: Norman, Oklahoma

Control: Broken Arrow, Oklahoma

As seen in Table 1, the total estimated respondent contacts for this proposed data collection is 21,216. This includes 16,416 respondents to complete the survey screener (i.e., Form 1321) and 2,400 respondents to complete each of the surveys (i.e., Forms 1322 and 1325).

Table 1. Total Contacts by Form

Form	Program/Control	Pre-Program	Mid-Program	Post-Program	Total Contacts
Form 1321 Screener	Form 1322 Program Joplin, MO	1,368	1,368	1,368	4,104
	Form 1322 Control Cape Girardeau, MO	1,368	1,368	1,368	4,104
	Form 1325 Program Norman, OK	1,368	1,368	1,368	4,104
	Form 1325 Control Broken Arrow, OK	1,368	1,368	1,368	4,104
	Total 1321	5,472	5,472	5,472	16,416
Form 1322 Impaired Driving Program Survey	Program Joplin, MO	400	400	400	1,200
	Control Cape Girardeau, MO	400	400	400	1,200
	Total 1322	800	800	800	2,400
Form 1325 Seat Belt Program Survey	Program Norman, OK	400	400	400	1,200
	Control Broken Arrow, OK	400	400	400	1,200
	Total 1325	800	800	800	2,400
Grand Total Contacts	Forms 1321, 1322, and 1325	7,072	7,072	7,072	21,216

As seen in Table 2, NHTSA estimates a 45% response rate^1,2 and a 65% eligibility rate.³ In order to reach 400 completed surveys per period (i.e., pre, mid, and post program), NHTSA estimates that 1,368 respondents will need to be screened. Of those screened, it is estimated that 889 will be eligible to participate, and of those who are eligible, it is estimated that 400 will participate and complete the survey.

Table 2. Breakdown of Eligibility and Response Rate Estimates

	Rate Estimate	Contacts
Contacts Participate and Complete Survey	45%	400
Contacts Eligible	65%	889
Total Contacts Approached	--	1,368

As typical with program evaluations of this nature, survey administration will follow a nonequivalent control group design. The intention of this design is to measure the effect of a program by taking a yardstick measurement pre, mid, and post program to determine change. With strong consistency in measurement protocol across measurement periods and similarity across the program and control samples to limit extraneous influences on the results, this design can produce a non-biased and reliable indication of change.

Utility of this evaluation does not necessitate administering to a probability based sample and generating representative results. The objective of the evaluation is to take a reliable measure of change in public support for enforcement, not to estimate what represents the community as a whole.

The design has strong consistency in measurement and sampling protocol across measurement periods and includes built in methods for addressing any inherent differences between the program and control samples. The nonequivalent control group design is susceptible to the threat of internal validity, as the group of respondents in the program and control areas may have been characteristically different prior to the intervention, possibly contributing to differences in response independent of the intervention effect. Ideally with nonequivalent control group designs, the two groups (program and control) would be characteristically similar prior to the intervention, just differ by intervention exposure, to produce results that reflect the influence of the intervention. However, with some programs, the evaluator has less control over site selection and must adapt to the realities of the situation, including any differences in the program and control samples. For the current project, the program and controls sites are demographically similar (Table 3); however, any inherent differences across the program and control samples will be accounted for by weighting the data to the population to address any biases in the sample.

While nonequivalent control group design lacks random assignment, inherent characteristics of the sample universe and rigorous protocols can reduce bias in the sample. Data collection sites will be selected to represent both variety of venue and geographical location within the site. Department of Motor Vehicles (DMV) offices, municipal buildings (e.g., post office, library, city hall), automobile service centers, and shopping centers will be included throughout the site. All licensed drivers must visit the Department of Motor Vehicles (DMV) office periodically for license renewal. The date of license renewal is based on birthday; therefore, it is reasonable to treat customers renewing their license on a given day as a random sample of the population of all available respondents (i.e., licensed drivers in area). In addition, licensed drivers may be more likely to visit automobile service centers, and they may need to do so on a regular basis for vehicle maintenance and inspections required by the State. To broaden the representativeness of the sample, municipal buildings and shopping centers will also be included. In most cases, all visitors to the selected locations will be approached. In cases where there is too much traffic to approach all people, data collectors will apply a systematic sampling interval to approach every 2^nd person.

Completed survey forms will be delivered to the Contractor where a data entry person will enter survey response data into Microsoft Access to allow for analysis. ANOVA, F-tests, t-tests, and logistic regression will be used to determine whether there are any statistically significant changes that can be attributed to the program activity.

B.1 Describe the potential respondent universe and any sampling or other respondent selection method to be used.

The potential respondent universe is comprised of licensed drivers aged 18 years and older visiting the Department of Motor Vehicles (DMV) offices, municipal buildings (e.g., post office, library, city hall), automobile service centers, and shopping centers in the program sites (Joplin, Missouri and Norman, Oklahoma) and control sites (Cape Girardeau, Missouri and Broken Arrow, Oklahoma). From this universe, the new data collection will contact a total of 21,216 drivers.

The program sites were selected based on the following guidelines:

• Community with population between 75,000 and 200,000

• Local government and law enforcement interested in the project

• Seat belt use below the national average, unrestrained fatalities above the national average, and lower levels of seat belt enforcement (Seat belt program only)

• Impaired driving related fatality crashes above the national average (Alcohol-impaired-driving program only)

Joplin, Missouri and Norman, Oklahoma were found to meet all of the selection guidelines with the exception of Joplin, Missouri having a slightly smaller population (Table 3). The selection team decided to proceed with Joplin because their population was only slightly lower than the guideline and the site demonstrated interest in participation.

Cape Girardeau, Missouri and Broken Arrow, Oklahoma were selected as the control sites. They were found to be demographically similar to the program areas to help control for response differences due to the population rather than exposure to the intervention. Also, the research team selected these sites because they have different media markets than the program sites to help reduce program bleed into the control areas (Table 3).

Table 3. Program and Control Site Characteristics

Enforcement Demo/ State	Program/ Control	City	Media Market	Pop. Est.4	LEA5 Sworn Officers	White4	Black or African American4	Hispanic or Latino4	Median Household Income4
Impaired Driving Enforcement, Missouri	Program	Joplin	Joplin	51,042	112	87.9%	3.8%	3.5%	$38,169
	Control	Cape Girardeau	Paducah-Cape Girardeau- Harrisburg-Mt. Vernon	77,606	158	88.3%	7.6%	2.2%	$46,050
Seat Belt Enforcement, Oklahoma	Program	Norman	Oklahoma City	117,353	180	80.4%	4.1%	7.1%	$51,491
	Control	Broken Arrow	Tulsa	103,437	130	78%	4.2%	7.2%	$67,131

As seen in Table 1, each measurement period will have a sample of 400 completed surveys. A power analysis indicated that for a population of 100,000, a sample of 383 respondents would be sufficient to achieve a 95% confidence interval and 5% margin of error. Because the programs will be conducted in sites with between about 50,000 and 120,000 residents, the power analysis indicated that a sample size of 400 completed surveys for each measurement period would be sufficient for the proposed data collection.

The data will be weighted to reflect the demographic makeup of each geographic location. The weighting process for this study entails two major steps. In the first step, target population benchmarks will be created for computation of weight factors. For this purpose, we will rely on public data sources such as Current Population Survey (CPS) or American Community Survey (ASC) as well as commercial sources such as Claritas to obtain demographic profiles of adults in each geographic location. In the second step, an iterative proportional fitting procedure will be used to balance the composition of respondents in each location to their respective demographic profiles obtained during the first step. It is anticipated that weight adjustments will include characteristics such as gender, age, race, and ethnicity.

The research team has located all of the Department of Motor Vehicles (DMV) offices, municipal buildings (e.g., post office, library, city hall), automobile service centers, and shopping centers in the program and control sites to select the locations for data collection. The research team will select a total of eight locations in each site to administer the survey, including two DMVs, two municipal buildings, two automobile service centers, and two shopping centers. To reach a total of 400 completed surveys per measurement period, 50 surveys will be completed at each of the eight sites (8 * 50 = 400) for each measurement period. The eight locations will be sufficiently spread out across the participating areas. The distribution of locations will help achieve external validity of the surveys because it will be more regionally inclusive and help alleviate the effect of regional differences on responses.

When selecting the eight data collection locations, the research team will contact the Department of Motor Vehicles (DMV) offices, municipal buildings (e.g., post office, library, city hall), automobile service centers, and shopping centers in the program and control sites to inquire about the magnitude of typical office traffic and operating hours, as well as to gain permission to administer surveys on the premises. The reported traffic will also help guide selection because high traffic volume locations will maximize the sample and provide further evidence of local dependence on these locations.

B.2. Describe the procedures for the collection of information

The surveys will be administered in-person to licensed drivers aged 18 years and older at Department of Motor Vehicles (DMV) offices, municipal buildings (e.g., post office, library, city hall), automobile service centers, and shopping centers in the program and control areas for both enforcement programs. Surveys will be administered Monday – Sunday and at varying hours of the day depending on the hours/days of operation and traffic flow at the various data collection locations. Data collectors will employ a multi- step process to survey respondents: (1) interception, (2) determine eligibility, (3) recruitment, and (4) completion of questionnaire. When traffic is too heavy to sample all people, the research team will select participants using a systematic sampling interval by sampling every 2^nd person.

Upon approaching a potential participant, the screening interviewer will introduce him or herself and give a brief explanation of the study following a pre-determined script for this initial contact. Following the initial interception, the interviewer will verbally administer the survey screener (i.e., Form 1321) to the participant to determine if they are eligible. The objective of the screening questions is to determine if the approached respondent is a licensed driver who is aged 18 years or older. The screening interviewer will review each screening question with the participant. Based on this conversation, the interviewer will determine eligibility.

Regardless of the eligibility determination, the interviewer will enter a “disposition” code onto the survey screener (i.e., Form 1321) to indicate the results of the screening. Examples of disposition codes are:

• Ineligible

• Refusal

• Other (specify)

Once the eligibility of the driver has been determined, the interviewer will endeavor to recruit eligible participants to complete the questionnaire. In general, this will not be a scripted dialog, but the team member will cover key elements, which include additional details on the study, and an estimated time for completion. If the screening interviewer is successful in recruiting the driver, he or she will hand the survey to complete the remaining survey questions.

There will be two versions of the survey (Forms 1322 and 1325). The first version of the survey (Form 1322) will be used for the alcohol-impaired-driving program, Building Community Support for Impaired Driving Enforcement. The second version of the survey (Form 1325) will be used for the seat belt program, Building Community Support for Seat Belt Enforcement. Two unique forms are needed because the surveys will ask questions specific to the subject matter of each program. For example, the alcohol-impaired-driving program survey will ask about support of alcohol-impaired-driving enforcement, and the seat belt program will ask about support of seat belt enforcement.

B.3. Describe methods to maximize response rates and to deal with issues of non-response.

Participation in this study is voluntary. Several methods will be utilized to maximize response rates, including:

• Administering the survey at locations (e.g., DMVs and automobile service centers) where potential respondents will already be waiting for service, and are more likely to have extra time to complete the survey; and

• Providing a Spanish-language translation of the awareness survey questionnaire to minimize language barriers to participation.

NHTSA does not expect to address non-response bias in this context but will be weighting the data to the population to address any biases in the sample.

B.4. Describe tests for procedures or methods to be undertaken

As part of the study design, the Contractor will refine the study procedures by pilot testing the screener and two survey instruments (i.e., Forms 1321, 1322, and 1325) with nine participants. The pilot study will allow the Contractor to conduct an assessment of the overall comprehension of the individual survey questions. Testing the instructions and questionnaires prior to implementing the study will provide the Contractor with the opportunity to make slight wording changes when needed that will improve overall comprehension without changing the intent or direction of the questions.

While NHTSA must account for this possibility, NHTSA foresees minimal changes to result from the pilot testing because it has consulted with in-house experts on survey development and adopted some questions from validated surveys used in previous efforts. 

B.5) Provide the name and telephone number of individuals consulted on statistical aspects of the design.

The following individuals have reviewed technical and statistical aspects of procedures that will be used to conduct the intercept surveys (listed alphabetically):

Mary Byrd, M.A. Office of Behavioral Safety Research (NTI-132)

(202) 366-5595 National Highway Traffic Safety Administration

[email protected] 1200 New Jersey Ave SE, W46-466

Washington DC, 29590

Arthur Goodwin, MA The University of North Carolina

919-843-5038 Highway Safety Research Center

[email protected] 730 Martin Luther King Jr. Blvd., Suite 300

Campus Box 3430

Chapel Hill, NC 27599-3430

Loren Staplin, PhD. TransAnalytics LLC

(215) 538-3820 ext 102 336 West Broad St.

[email protected] Quakertown, PA 18951

1 Ellis, C. S., Evans, B., Santiago, G. M., & Reed, L. M. (2007, May). Surveying International Travelers: An Argument for Intercept Interviewing. Presented at American Association for Public Opinion Research Conference, Anaheim, CA. Retrieved from http://www.rti.org/sites/default/files/resources/aapor07_ellis_paper.pdf

2 Duke, Joshua M., Ilvento, Thomas W. (2005) A Conjoint Analysis of Public Preferences for

Agricultural Land Preservation. Agricultural and Resource Economics Review. Retrieved from https://www.researchgate.net/publication/4902860_A_Conjoint_Analysis_of_Public_Preferences_for_Agricultural_Land_Preservation

3 The eligibility rate was calculated using Federal Highway Administration 2014 United States licensing data. Retrieved from http://www.fhwa.dot.gov/policyinformation/statistics/2014/. To be eligible, a respondent must be a licensed driver and 18 years old or older. In 2014, 67% of the US population had a driver’s license, and 1.4% of the licensed drivers were under 18 years old. Based upon these statistics, NHTSA estimates that about 65% of respondents will be eligible to participate (i.e., 67% - 1.4% = 65%).

4 2011 – 2015 American Community Survey 5-Year Estimates. https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

5 Law Enforcement Agency.

8