PRA-2126NEW.OBMS.SS.PartB.071811

Department of Transportation

Office of the Chief Information Officer

SUPPORTING STATEMENT

Evaluating the Safety Benefits of an On-Board Monitoring System (OBMS) in Commercial Vehicle Operations: Independent Evaluation and Data Analysis

This Supporting Statement is developed to request the Office of Management and Budget’s (OMB) review and approval of a new information collection (IC) entitled, “Evaluating the Safety Benefits of an On-Board Monitoring System in Commercial Vehicle Operations: Independent Evaluation and Data Analysis.”

Part B. Collections of Information Employing Statistical Methods.

1. Describe potential respondent universe and any sampling selection method to be used.

Currently, FMCSA estimates that there are 514,000 active motor carrier companies in the U.S and approximately 4.2 million drivers with a commercial driver’s license. Within this potential respondent universe, three motor carrier fleets have been selected to participate in the FOT. These three fleets have a combined total of 11,600 drivers and 11,660 trucks. At each participating carrier, only one carrier terminal was selected to participate in the FOT. By focusing on just one terminal at each carrier, all the trucks at the selected carrier terminal can be outfitted with OBMS. The choice of carrier fleet and terminal was based on carrier willingness and commitment to undertake the FOT, fleet size, geographic location, type of operation (long haul or short haul), budget constraints, and carrier’s technical capacity to successfully implement the requirements of the FOT. The overall objective of the information collection request is to understand the participating fleets’ CMV drivers’ expectation, attitudes and acceptance of OBMSs.

The University of Washington will be acting as the independent evaluator for examining the effectiveness of OBMS, by assessing driver performance and acceptance during the 24-month study period. As part of the broader OBMS FOT study, the questionnaire portion of the FOT study seeks to assess CMV drivers’ acceptance of the onboard monitoring system in their trucks, and changes in attitudes over 18 months, covering the periods before, during, and after drivers receive coaching feedback. CMV drivers from the three participating carriers will be asked to participate in the questionnaire portion of study. Based on the driver attrition rate for each carrier, up to 500 drivers are estimated to participate in the OBMS FOT study. All drivers in the study will be asked to participate in the questionnaire portion of the study. These drivers will be randomly assigned into a comparison group (group 1), long-term feedback group (group 2) and short-term feedback group (group 3). The number of drivers to be randomly assigned to each group is as follows:

Group 1 drivers (no feedback: Drivers who are in Group 1 (30 devices, up to 125 drivers) will use the OBMS in their vehicle for an 18-month period without any feedback provided.

Group 2 drivers (long-term feedback): Drivers who are in Group 2 (210 devices, 250 drivers, long-term adaptation group) will experience a data collection period of an approximate 2-month baseline phase, 14-month intervention phase, and 2-month Withdrawal phase.

Group 3 drivers (short-term feedback): Drivers who are in Group 3 (30 devices, up to 125 drivers, short-term adaptation group) will experience data collection period of an approximate 2-month Baseline phase, 7-month Intervention phase, and up to 9-month Withdrawal phase.

Based on previous experiences, it is estimated that the response rate would be 50% for questionnaires and 80% for the exit interview. While it may be impossible to speculate whether non-respondents will have been less favorable, more favorable, or neutral toward OBMS, there is a potential for some non-response bias. To address this potential problem, driver characteristics and performance data from both respondents and non-respondents would be available from the carrier and will be used to analyze and characterize any potential bias problem.

2. Describe procedures for collecting information including statistical methodology for stratification and sample selection, estimation procedures, degree of accuracy needed, and less than annual periodic data cycles.

The purpose of this information collection is to assess the participating carriers’ CMV drivers’ expectations, attitudes and acceptance of an OBMS, as a part of a FOT study. Approximately 500 CMV drivers will be asked to provide responses to a total of four unique questionnaires administered six times over the course of 18 months. More specifically, all drivers regardless of study group will receive questionnaires at month 0 [start of study], and at the end of months 2, 4, 9, 16, and 18. At the end of the 18-month study, an exit interview will be conducted in-person to assess participants’ more detailed opinions on the OBMS functions, safety climate change due to implementation of the system.

Prior to administering the first questionnaire, drivers will be briefed about the purpose of the study and an informed consent will be obtained from those who decide to participate. Drivers will be asked to sign the consent form (Attachment K). Drivers who decline to participate will not be asked to sign the consent form and will not be contacted in the future. Drivers will be asked to complete the first questionnaire after the consent session.

The second questionnaire will be sent to drivers via their preferred method (e-mail/postal mail) to the address they provided in the consent session. If after one week, no responses are obtained from the driver, a follow-up letter with a replacement questionnaire will be sent. Telephone follow-up may also be conducted if no response is received after one week of the first follow up. A telephone follow up transcript will be prepared and the interviewer will be trained for the telephone calls. Another replacement questionnaire can be provided if necessary. When a response is received, a thank you letter will be sent to participants as well as information regarding the next questionnaire. The procedure for all other questionnaires will be the same. All introductory and follow-up letters and telephone follow-up reminders are attached as Attachment L.

At the end of the full study period, the research team will contact participants to arrange an in-person exit interview with a trained interviewer. Drivers who agree to participate in the interview will be asked to complete the post-study questionnaire before the interview.

Based on the expected response rates (50% for the questionnaires, and 80% for the in-person interviews), we anticipate that a total of 500 drivers across all three groups should be sufficient to assess significant differences. To verify, a power analysis was conducted to ensure that we have a large enough sample size to observe differences based on the variation expected in the data. Using G*Power 3.1 program (Faul, et al, 2007); we computed the power analysis for testing two independent proportions. The power analysis indicates that in order to achieve a power of 0.8 with an expected difference in proportions of 0.2, 69 drivers in Group 1 and 138 drivers in Group 2 are required. Similarly, in order to achieve a power of 0.75, 63 drivers in Group 1 and 126 drivers in Group 2 are required.  In addition, a power analysis was computed for a simple repeated measures study of six repeated observations (or six questionnaires) within a group.  With power (1- β) =0.81 and α=0.05, the results conclude that a sample size of 109 is needed to detect small effects [0.10]) and a sample size of 19 would be needed to detect medium effects (0.25). 

3. Describe methods to maximize response rate.

The drivers responding to the surveys are employees of the carriers participating in the FOT. Given that the drivers are responding to actual conditions in their work environments, they have a greater incentive to participate and respond to the survey questionnaires. However, because questionnaires will be administered six times over the course of 18 months, incentive payments will be made to ensure that respondents stay interested and continue to provide thoughtful responses through the end.

To maximize the response rate, participants will be paid $10 after completing each questionnaire. In addition, they will be compensated with $40 for completing the study and participating in the exit interview. Therefore, participants who complete all six administrations of the four unique questionnaires would receive $100. In addition, the following measures would be put in place to maximize the response rate:

• Each questionnaire response time would be kept to a maximum of 20 minutes and the questionnaire items would be worded so that each question is clear and unambiguous.

• Short instructions and figures will guide respondents through the questionnaire.

• All letters, including the first questionnaire announcement, reminder letters, and thank you letters, will be addressed to each individual respondent.

• Follow-up with respondents by sending reminders and replacement questionnaires after initial unresponsive mailing.

• After receipt of responses, send out thank you letters with incentive and a notice to respondents about the next upcoming questionnaire.

• Use pre-paid return envelope to simplify the mailing process for respondents.

4. Describe tests of procedures or methods. 

The questionnaire pilot test was conducted on a sample of four drivers from the three participating carriers. All drivers were approved by the research team and signed an Informed Consent approved by the University of Washington IRB. They completed the first questionnaire after the consent process. The completion time required by each driver was an average of 20 minutes. Participant feedback from the pilot study revealed some issues in the wording of a few of the questionnaires. These comments were considered and incorporated into newer questionnaires.

The data indicated that the survey took an average of 11 minutes to administer (interview time ranged from 10 – 15 minutes). Participant feedback from the pilot study identified issues in the wording of a few survey items and response format. Feedback from drivers was incorporated into survey design, and a revised survey was developed.

5. Provide the name and telephone number of individuals who were consulted on statistical aspects of the information collection and who will actually collect and/or analyze the information.

FMCSA is sponsoring this information collection. The FMCSA contact is:

Olu Ajayi

Research Division, Mathematical Statistician

Office of Analysis, Research, and Technology

Telephone: (202) 366-0440

Email: [email protected]

The University of Washington is responsible for designing the data collection, collecting and analyzing the data.

The University of Washington contacts are:

Dr. Linda Boyle, Associate Professor

Telephone: 206 616 0245

E-mail: [email protected]

Yiyun Peng, Graduate Research Assistant

Telephone: 206 616 0277

E-mail: [email protected]

Reference:

Faul, F., Erdfelder, E., Lang, A.G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191

4