Evaluation of Impaired Riding Interventions
Supporting Statement for Information Collection Request
TABLE OF CONTENTS
SUPPORTING STATEMENT
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Describe the potential respondent universe and any sampling or other respondent
selection method to be used................................................................................................1
2. Describe the procedures for the collection of information.................................................4
3. Describe methods to maximize response rates...................................................................6
4. Describe any tests of procedures or methods to be undertaken..........................................6
5. Provide the name and telephone number of individuals consulted on statistical
aspects of the design...........................................................................................................6
B. Collections of Information Employing Statistical Methods
The proposed information collection will employ statistical methods to analyze the data collected from respondents. The following sections describe the procedures for respondent sampling and data tabulation.
B.1. Describe the potential respondent universe and any sampling or other respondent selection to be used.
NHTSA plans to use intercept surveys to examine the attitudes, awareness, and self-reported behaviors of motorcycle riders associated with intervention programs designed to reduce alcohol-impaired motorcycle riding. Motorcycle riders will be interviewed in up to four program locations and in up to three appropriate comparison locations during the periods two weeks before and after the planned impaired riding interventions. NHTSA awarded a contract to M. Davis and Company, Inc. (MDAC), in September 2011 to manage the program locations where the intervention programs will occur. NHTSA and MDAC will solicit applications from locations to implement the interventions in the Fall of 2011, and anticipate selecting the program locations in January 2012.
In-person interviews will occur at sites within the program and comparison locations where large numbers of motorcycle riders congregate, such as at motorcycle-oriented bars and restaurants, motorcycle dealerships and accessory shops, and at motorcycle trade shows and events. Since a very small percentage of Americans are motorcycle riders, targeting motorcycle riders using random-digit dialing to landline phones or with a cell phone sample is difficult, and using intercept surveys is an efficient way to obtain responses from motorcycle riders within a limited geographic area. In this sense, the potential respondent universe is a sample of motorcycle riders that frequent the sites in the program and comparison locations where interviews will occur, in the two weeks before and after the interventions.
Sampling will be conducted at two separate levels. The first level is the site within each program and comparison area, and the second level is motorcycle rider at each site.
Site Sampling: The first stage of sampling (site) will be accomplished through joint effort by the program management contractor (MDAC), the evaluation contractor (Pacific Institute for Research and Evaluation [PIRE]), and the NHTSA research team. The goal of this phase of sampling is to ensure that the types of sites at which interviews will occur are representative of sites frequented by motorcycle riders in the study locations, and to ensure that the number of riders sampled at each site type is comparable for administrations of the survey that will be compared to one another (e.g., that site types will be matched for the pre-test and post-test within a program location).
A sampling frame will be constructed that contains all of the eligible sites within each program and comparison location likely to provide access to motorcycle riders. The types of sites that will be eligible will vary by location, but may include: bars and restaurants frequented by motorcycle riders, motorcycle dealerships and accessory shops, and motorcycle trade shows and events. Representatives from the evaluation contractor, PIRE, will visit the program and comparison locations prior to selecting survey sites to visit the eligible sites and to assess motorcycle traffic at the sites within the sampling frame.
Potential sites within the sampling frame will be classified based on type of establishment (e.g., bar, store that sells motorcycle-related goods). Sites within the establishment categories will be selected so that the number of riders interviewed at each type of site will be matched in the pre-test and post-test survey administrations. Because NHTSA anticipates that program and comparison locations will be chosen in January 2012, eligible sites within these locations will not be identified until that time.
Site selection will be influenced by several other factors, including those relating to safety of the data collectors and candidate motorcycle riders, site or community cooperation, and efficiency of data collection (i.e., volume of motorcycle riders). Permission will be gained from candidate site property managers or owners. For sites under community jurisdiction, government and law enforcement agencies will be notified.
Rider Sampling: The second stage of sampling (rider) will be conducted on site by the data collection researchers. Data collectors stationed at each selected site will be responsible for approaching and holding the initial contact with potential participants. Initially, interviewers will intercept every 3rd person passing the position where the interviewer is stationed (e.g., individuals leaving a bar or motorcycle event), with the goal of obtaining 10 completed interviews per hour. However, this protocol may be revised based upon the actual sampling conditions at each site.
We cannot report the demographics of the potential respondent universe of motorcycle riders in the program and comparison locations because the program and control locations have not yet been selected. Table 1 presents age distributions of American motorcycle riders (i.e., people who reported riding a motorcycle in the past month) from the 2009 National Household Travel Survey1, and corresponding estimates of the number of riders in the sample that will fall into each age range. When program and comparison locations are selected, similar demographic tables can be created based on local motorcycle registration and licensing records.
TABLE 1. Expected Population and Sample Distribution by Age |
||
|
Percentage of Motorcycle Riders |
Expected Distribution of Sample (n) |
Total |
|
14,000 |
< 21 |
2.4% |
336 |
21-35 |
19.2% |
2,688 |
36-45 |
29.6% |
4,144 |
46-55 |
27.6% |
3,864 |
56-65 |
17.6% |
2,464 |
66+ |
3.6% |
504 |
Of individuals who reported riding a motorcycle in the past month in the 2009 National Household Travel Survey, 89.4% were male and 10.6% were female1. If similar gender proportions were found in the sample for this study, 12,516 male riders and 1,484 female riders would be interviewed.
Response Rates:
Response rates in past in-person intercept interview studies have been high. Billheimer2 performed in-person interviews with motorcycle riders in sites where riders congregate, such as at motorcycle trade shows, with an 88% response rate. Similar response rates were found in prior intercept interview studies of college students on their drinking behavior (88% of students approached3, 92% of party hosts approached4), of electronic music dance event attendees on drug use (82 to 90%)5, and of young adults returning to the US after drinking in Mexico on their drinking behavior (90%)6. Some of these studies provided small monetary incentives of $5-20 to respondents in order to achieve these high response rates4,5, which is similar to the incentive we plan to offer to respondents to ensure a comparable response rate.
Total Sampling Needs:
Overall, the total sample needs for the interviews are 14,000 respondents over the course of one year. Since the interviews are estimated to be 5 minutes in length, the estimated annual time is 1,166.67 hours. The breakdown of the total number of respondents appears in the tables below. Program locations will be selected in part based on evidence that they have high motorcycle traffic and an impaired riding problem. Thus, program locations and appropriate comparison locations will be chosen that have a high enough volume of motorcycle riders to conduct the number of interviews necessary.
Table 2. Sample Size |
|||||
|
1st Wave |
2nd Wave |
|
||
|
Pre |
Post |
Pre |
Post |
Total |
Program Location #1 |
500 |
500 |
500 |
500 |
2,000 |
Program Location #2 |
500 |
500 |
500 |
500 |
2,000 |
Program Location #3 |
500 |
500 |
500 |
500 |
2,000 |
Program Location #4 |
500 |
500 |
500 |
500 |
2,000 |
Comparison Location #1 |
500 |
500 |
500 |
500 |
2,000 |
Comparison Location #2 |
500 |
500 |
500 |
500 |
2,000 |
Comparison Location #3 |
500 |
500 |
500 |
500 |
2,000 |
Total |
3,500 |
3,500 |
3,500 |
3,500 |
14,000 |
|
Annual Sample |
Minutes |
Total Hours |
Total |
14,000 |
x 5 |
1,166.67 |
B.2. Describe the procedures for the collection of information.
The intercept surveys will be used in a Non-Equivalent Groups Design (NEGD), which is frequently used by NHTSA evaluators to evaluate high visibility enforcement campaigns, such as the Click It or Ticket seat belt campaign, the Over the Limit Under Arrest and Drive Sober or Get Pulled Over impaired driving campaigns, and the Phone In One Hand, Ticket In the Other distracted driving campaign. The design is structured like a pretest-posttest randomized experiment, but lacks random assignment. The intercept interviews will examine the changes that occur as a result of specific impaired riding interventions. The interviews will be conducted on a schedule corresponding with the fixed dates for the impaired riding mobilizations that will depend on the timing and sequencing of the components of each demonstration project.
Large numbers of motorcycle riders can attend rallies and other motorcycle events, and it is therefore possible that intervention activities in a location may occur in conjunction with one of these events. For example, signs publicizing impaired riding enforcement may be placed along routes that event attendees may travel to arrive at the event, and increased enforcement of impaired driving may occur on routes leading to the event and in the community surrounding the event during the week in which the event is held. Although NHTSA desires to use a pre-post interview design in all sites, this design may not be possible in locations where intervention activity occurs in conjunction with special events such as motorcycle rallies. Riders may travel long distances to attend these events, and thus it may not be feasible to conduct baseline interviews with these riders prior to the intervention because it would not be possible to locate these riders prior to their travel to the event.
If intervention activity in a program location occurs in conjunction with a special event for which it would not be possible to collect pre-event interview measures, interviews would follow a test-comparison design instead of a pre-post design. Interviews would be conducted in the program location during the intervention period, and would be compared to interviews conducted in an appropriate comparison location that did not experience an intervention during a comparable special event. The types of sites where riders are interviewed and the number of riders interviewed at each site type would be matched between the program and comparison location. In these cases, we will not be measuring the change in behavior, awareness, and perception before and after the event, but rather will be comparing a snapshot of riders at a location that did experience an intervention to a snapshot of riders at a location that did not.
Pre- and post- intervention responses from the program and comparison locations will be compared using Chi Square to determine if there are any significant differences that can be attributed to the impaired riding demonstration program activities. For interventions that occur during a special event and where a pretest-posttest design would not be possible, responses from the program location during the intervention will be compared to responses from a comparison location that did not experience an intervention collected during a similar special event.
A sample of N = 500 motorcycle riders will be recruited from each program and comparison location for each interview administration. Previous NHTSA research indicates that this sample size has sufficient power to detect pre-post changes in intervention awareness surveys.
The data collectors will employ a multi- step process to survey riders: (1) interception, (2) recruitment, (3) determining eligibility, and (4) completion of interview.
Data collectors stationed at each selected site will be responsible for approaching and holding the initial contact with potential participants. Initially, interviewers will intercept every 3rd person passing the position where the interviewer is stationed (e.g., individuals leaving a bar or motorcycle event), with the goal of obtaining 10 completed interviews per hour. However, this protocol may be revised based upon the actual sampling conditions at each site.
Upon approaching a potential participant, the screening interviewer will introduce him or herself and provide a brief explanation of the study. Following the initial interception the interviewer will administer a set of one or two screening questions (see Attachment C—Survey). The objectives of these screening questions are to determine eligibility. Each question will be read to the participant and the interviewer will record the participant’s responses on the questionnaire form. A rider will be eligible to complete the interview if (1) they have operated a motorcycle on the road in the past 30 days, and (2) there are no language barriers. For locations in which interventions occur in conjunction with a special event, a rider must also have attended the event to be eligible.
Thus, one screening question (“Have you operated a motorcycle on the road in the past 30 days?”) will be administered to riders in program locations in which the intervention does not occur in conjunction with a special event, and in corresponding comparison locations. Two screening questions (“Have you operated a motorcycle on the road in the past 30 days?” and “Are you attending [name of event]?”) will be administered to riders interviewed in locations in which the impaired riding intervention occurs in conjunction with a special event, and in corresponding comparison locations.
Regardless of the eligibility determination, a “disposition” code will be entered onto each form to indicate the results of the screening. Examples of disposition codes are:
Ineligible Due to Interview Responses
Eligible Due to Interview Responses
Refused
Ineligible Due to Language Barrier
Ineligible Due to Other (specify other)
Once the eligibility of the rider has been determined, the data collector will administer the remainder of the interview to eligible participants, or thank the respondent and end the interview with ineligible participants. The remainder of the interview will follow a pre-determined script (provided in Attachment C--Survey).
B.3. Describe methods to maximize response rates.
PIRE will hire interviewers who are also motorcycle riders, and who will wear motorcycle gear when conducting interviews. This should maximize response rates by making potential respondents comfortable with the interviewers. We also plan to give riders a small monetary incentive or a gift card worth approximately $10 for their participation. Other in-person intercept interview studies have offered similar incentives for participation to maximize response rates4,5.
NHTSA does not intend to assess non-response bias in this evaluation project. Generally, only a small proportion of non-respondents are willing to participate in a non-response follow up interview. We will track the number of refusals in each site and the rider’s approximate age and gender by observation. As noted in section B.1., response rates in past similar intercept interview studies with motorcycle riders and of drinking and drug use behavior have been above 80%.
B.4. Describe any tests of procedure or methods to be undertaken.
NHTSA will conduct a pretest of the interview methodology and questions with a small number of motorcycle riders. Any problems encountered during the pretests of the interviews will be resolved before they are put into the field.
B.5 Provide the names and telephone numbers of individuals consulted on statistical aspects of the design.
Jessica Cicchino, Ph.D.
Research Psychologist
National Highway Traffic Safety Administration
1200 New Jersey Ave SE, W46-491
Washington, DC 20590
(202) 366-2752
Alan Block, M. A.
Research Psychologist
National Highway Traffic Safety Administration
1200 New Jersey Ave SE, W46-499
Washington, DC 20590
(202) 366-6401
Maria Vegega, Ph.D.
Chief, Occupant Protection Division (former Chief, Behavioral Research Division)
National Highway Traffic Safety Administration
1200 New Jersey Ave SE, W44-302
Washington, DC 20590
(202) 366-2668
John Lacey, MPH
Center Director, Alcohol, Policy, and Safety Research Center
Pacific Institute for Research and Evaluation
11720 Beltsville Drive, Suite 900
Calverton, MD 20705
(301) 755-2779
Richard Blomberg, M.S.
President
Dunlap and Associates, Inc. (subcontractor to Pacific Institute for Research and Evaluation)
110 Lenox Avenue
Stamford, CT 06906
(203) 323-8464
1 U.S. Department of Transportation, Federal Highway Administration, 2009 National Household Travel Survey. URL: http://nhts.ornl.gov.
2 Billheimer, J. W. (1996). California Motorcyclist Safety Program, Program Effectiveness: Accident Evaluation. Final report to the California Highway Patrol.
3 Foss, R. D., Marchetti, L. J., & Holladay, K. A. (2001). Development and Evaluation of A Comprehensive Program to Reduce Drinking and Impaired Driving Among College Students. Publication No. DOT HS 809 396. Washington, DC: National Highway Traffic Safety Administration.
4 Clapp, J. D., Min, J. W., Shillington, A. M., Reed, M. B., & Croff, J. K. (2008). Person and environment predictors of blood alcohol concentrations: A multi-level study of college parties. Alcoholism: Clinical and Experimental Research, 32, 100-107.
5 Voas, R. B., Furr-Holden, D., Lauer, E., Bright, K., Johnson, M. B., & Miller, B. (2006). Portal surveys of time-out drinking locations: A tool for studying binge drinking and AOD use. Evaluation Review, 30, 44-65.
6 Lange, J. E., Lauer, E., & Voas, R. B. (1999). A survey of the San Diego-Tijuana cross-border binging: Methods and analysis. Evaluation Review, 23, 378-398.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jessica Cicchino |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |