Supporting Statement B - ADS Teams 2024-2-9 Clean

_{Department of Transportation}

Federal Motor Carrier Safety Administration

SUPPORTING STATEMENT PART B

Safety Impacts of Human-Automated Driving System (ADS) Team Driving Applications

INTRODUCTION

This is to request the Office of Management and Budget’s (OMB’s) review and approval of a new Federal Motor Carrier Safety Administration (FMCSA) information collection request (ICR) titled Safety Impacts of Human-Automated Driving System (ADS) Team Driving Applications.

Part B. Collections of Information Employing Statistical Methods

1. DESCRIBE POTENTIAL RESPONDENT UNIVERSE AND ANY SAMPLING SELECTION METHOD TO BE USED.

1.1 Respondent universe

The inclusion criteria for this study are that drivers:

1. possess a valid Class A or Class B commercial driver’s license (CDL),

2. not be prone to simulator/motion sickness,

3. be 21 years of age or older, and

4. be within one day drive of the Virginia Tech Transportation Institute (VTTI; as reported by the driver).

As of 2020, over 6.5 million commercial vehicle drivers were operating a large truck or bus in the United States.¹ Of these drivers, approximately 4.0 million have a valid CDL. Furthermore, it is estimated that approximately 1.9 million individuals hold a Class A CDL² and more than 645,000 individuals hold a Class B CDL.³ Approximately 20% or fewer of these drivers are expected to be prone to simulator/motion sickness.^4,5 A previous VTTI simulator study had evidence of simulator sickness in 5%–12% of participating drivers.⁶ Drivers “within one day drive” includes drivers living within a one day drive, as well as drivers who live farther than a one day drive but whose working drive route brings them near VTTI. This is the potential respondent universe.

1.2 Sampling selection method

We anticipate a convenience sample of 80 drivers to complete one study session. Multiple participant recruitment methods will be utilized:

1. VTTI will leverage its large database of Class A or Class B CDL drivers who have participated in previous research studies or have expressed interest in possible participation. A total of 959 Class A or Class B CDL drivers are currently in the system, which undergoes continuous updating. The database includes information on driver attributes, such as driver age, gender, and location.

2. Additionally, VTTI has existing relationships with numerous commercial fleets within a day’s drive of southwest Virginia. VTTI has conducted research on trucking for over 20 years. Many VTTI studies have involved working with local commercial fleets. Through these experiences, VTTI has developed positive working relationships with over 40 fleets within a four-hour drive of VTTI. These fleets have agreed to share VTTI research study information in their break rooms, next to time clocks, and via email.

3. Finally, VTTI will place recruitment ads in newsletters and on social media.

Through these methods, interested participants may be contacted by VTTI or may contact VTTI. We anticipate receiving a pool of potential participants, with the ability to include difficult to reach truck driver types such as females and minorities. However, there will be no required minimum number of female or minority drivers to be included in the study. VTTI will select the first 88 (to account for participant drop outs) drivers who express interest in study participation through the contact methods above and who meet the study inclusion criteria. VTTI anticipate 80 participants will complete the entire study.

The study is investigating four human-Automated Driving System (ADS) teaming use cases (each described in more detail in Section 2.1). At a high level, a teaming use case is defined as the operation of one or more commercial motor vehicles (CMVs) by at least one person in partnership with an ADS, each with a distinct role. A job role is one of the ways in which a human can oversee, operate, or control a CMV that has ADS capabilities. Three of the four teaming use cases have one job role in the teaming use case for a participant. One of the four teaming use cases has two job roles in the teaming use case for participants. Each participant will be assigned to one teaming use case and one job role. The final participant sample will contain 80 commercial motor vehicle (CMV) drivers with a valid Class A or Class B commercial driver’s license, resulting in 16 participants per job role type in each teaming use case (i.e., 16 participants each for three of the teaming use cases and 32 participants for one teaming use case). Selected participants will be assigned to the teaming use cases with consideration for balance by age and gender across the experimental conditions, depending on the distribution of these factors in the participant sample. To account for attrition and safeguard the analysis against partial or incomplete data, the participant sample recruited will be approximately 10% to 20% higher than the intended sample size. Participants will be randomly assigned to an experimental condition (driving team use case and driving schedule).

The sample size in the proposed study was chosen in consideration of the following study factors: 1) the study design includes approximately four transitions per teaming use case (an opportunity to assess up to 60 transitions per teaming use case); 2) the proposed shift length of 14 vehicle in motion hours will provide ample time to collect data on distracted behaviors, fatigue, and driving performance; and 3) collecting driving data using a simulator will give researchers more control over the driving environment compared to a naturalistic driving study.  

A power analysis was performed to determine the appropriate sample size. The power analysis provides an estimated sample size for developing a strong study with sufficient data to answer all research questions. Current published research focused on simulator studies and driver readiness and performance was reviewed to identify possible values to expect in the current study. A study by Zhang et al.⁷ included 22 professional truck drivers in a simulator study assessing driver behavior in a truck platooning scenario. Truck platooning requires drivers to monitor the vehicle and to be prepared to take over vehicle control. In the study, drivers received training and then operated the simulator under different trials and conditions. The conditions included features planned in the current study, such as opportunities for the driver to perform non-driving tasks. The researchers compared drivers’ perception reaction time (the time it takes for a driver to perceive a stimulus, cognitively process the situation, and decide on a response), movement reaction time (the time it takes the driver to perform the mitigation strategy), and the total reaction time (the total time for transfer of control from the vehicle to the driver). The authors found significant differences in reaction times among driver groups, ranging from 15% to 55%.⁷

The power analysis considered the need to detect differences in perception reaction time (PRT) for drivers monitoring (drivers not physically interacting with pedals or steering wheel but visually monitoring the roadway) and drivers not-monitoring (drivers not physically interacting with pedals or steering wheel, visually interacting with a tablet, but allowed to visually scan the roadway at their choice). The driver monitoring group had a PRT of 0.78 s (SD = 0.14) and the driver not-monitoring group had a PRT of 0.91 s (SD = 0.35).⁷ The power analysis used these PRT values as a baseline level for significant findings. The power analysis was performed in SAS 9.4. The power analysis assumptions included a desired power of 0.80 (industry standard), alpha or significance level of 0.05 (industry standard), group PRTs of 0.78 s and 0.91 s respectively, and an analysis of variance (ANOVA; the study analysis approach to assess reaction time will involve multiple covariates and control of multiple observations per driver). To obtain a power of 0.80, a sample size of 39 observations will be required to detect a statistically significant difference in PRT. Each driver will likely contribute multiple data points for certain variables. With 16 drivers in each teaming use case and role, and 2 to 4 transitions per driver, we expect between 32 and 64 transitions to measure reaction time. To ensure a sample large enough to answer all research questions, a sample of 16 drivers per teaming use case and role who complete the study protocols is recommended for recruitment. Figure 1 shows the power curve in the solid blue line, generated based on the above stated assumptions, as the estimated sample size changes from n = 28 to n = 68. The dashed orange line, which indicates power = 0.80, intersects with the power curve at approximately n = 39.

Figure 1. Estimated study sample size in terms of number of transitions captured for each teaming use case.

2. DESCRIBE PROCEDURES FOR COLLECTING INFORMATION, INCLUDING STATISTICAL METHODOLOGY FOR STRATIFICATION AND SAMPLE SELECTION, ESTIMATION PROCEDURES, DEGREE OF ACCURACY NEEDED, AND LESS THAN ANNUAL PERIODIC DATA CYCLES.

2.1 Procedures for collecting information

The following section describes the procedures for collecting information in terms of the study methodology. Following this information, additional sections will provide answers to the stratification, sample selection, estimation procedures, degree of accuracy needed, less than annual periodic data cycles, and analysis methodology.

The study includes data collection from a driving-simulator experiment. Participants in the driving-simulator experiment will provide survey data. The collected survey data will support the simulator experiment data. The survey data will be used in the assessment of driving performance data as covariates in the model (to control for certain variables, such as age, gender, and driving experience). Data on driver readiness and performance will be collected from the simulator experiment. The planned experiment is explained in full below. The final experimental design will be informed by results of the literature review and an assessment of industry practices. These results will support the simulator experimental design in terms of teaming use cases, job roles, and lengths of driving shifts (these terms are defined following this paragraph). These design components are specified in detail in the following sections, with expectations that these details are hypothetical and could be adjusted to best match industry practices. However, the plans for survey data collection and the data sources during the simulator experiment will remain as stated in this document.

The study design for the driving-simulator experiment is shown in Table 2. This design includes four human-Automated Driving System (ADS) teaming use cases where control of the vehicle transitions between job roles. As stated previously, a teaming use case is defined as the operation of one or more commercial motor vehicles (CMVs) by at least one person in partnership with an ADS, each with a distinct role, and a job role is one of the methods in which a human can oversee, operate, or control a CMV that has ADS capabilities. The study design includes data collection sessions that may resemble a CMV driver’s working day, with periods of driving, non-driving work, and breaks. During the study, participants will drive or monitor the CMV in their teaming use case and can choose to engage in driving-related and non-driving tasks at their own discretion. Drivers can engage in secondary tasks, but they will not be explicitly instructed to do so. Data collected from this study will be used to understand the impact of teaming use cases and job roles on driver performance, workload, and inattention.

In this study, a “driver” is a human in the driver seat that is in control of the vehicle. SAE J3016 defines a “[human] driver” as, “A user who performs in real time part or all of the dynamic driving task (DDT) and/or DDT fallback for a particular vehicle.” It goes on to define an “in-vehicle driver” as, “A driver who manually exercises in-vehicle braking, accelerating, steering, and transmission gear selection input devices in order to operate a vehicle.” The driver might go off-duty and enter the sleeper under teaming use case(s) relinquishing DDT to a remote driver or ADS.

A “remote driver” is defined as, “A driver who is not seated in a position to manually exercise in-vehicle braking, accelerating, steering, and transmission gear selection input devices (if any), but is able to operate the vehicle.” Remote driving is often colloquially referred to as “teleoperation.” SAE J3016 defines remote driving as, “Real-time performance of part or all of the DDT [dynamic driving task] and/or DDT fallback (including, real-time braking, steering, acceleration, and transmission shifting), by a remote driver.” As J3016 defines, the remote driver is responsible for object and event detection and response but could operate in cooperation with an ADS by overruling the ADS for motion control. Industry experts caution that remote driving should be limited to slower speeds (e.g., below 35 mph), especially in dense traffic.

J3016 defines a remote assistant as, “A human(s) who provides remote assistance to an ADS-equipped vehicle in driverless operation.” Remote assistance is defined as, “Event-driven provision, by a remotely located human (see J3016 3.31.5), of information or advice to an ADS-equipped vehicle in driverless operation in order to facilitate trip continuation when the ADS encounters a situation it cannot manage.” J3016 goes on to note that remote assistance does not include real-time DDT or fallback performance by a remote driver.

Table 2. Study design and sample size for hypothetical simulation test and evaluation design. 

Use Case No.	Teaming Use Case with Job Roles	Teaming Use Case Description	Study Design Description	Teaming Use Case Operation Time	Use Case Sample Size
1	Driver teaming with an ADS in same truck	Driving performed by the onboard driver in the driver seat, who is responsible for monitoring and directly controlling the vehicle while ADS is not in operation, and by ADS when ADS is operating	Driving team transitions after 1- and 2-hour blocks with responsibility for the DDT (e.g., driver for 1 hours, ADS for 2 hours, etc.; six block combinations to be randomly assigned across participants)	14 hours vehicle in motion split between team; 16 hours on duty	N = 16
2	Driver in Truck 1 teaming with an ADS in Truck 2	Driver in the driver seat of Truck 1 responsible for monitoring or directly controlling Truck 1, while being followed by Truck 2 which is fully operated by ADS	Driver operates truck, with scheduled rest or work-related breaks at 1- or 2-hour intervals	14 hours vehicle in motion, not split between team; 16 hours on duty	N = 16
3	Driver teaming with a Remote Assistant in same truck	Driver in the driver seat responsible for monitoring and directly controlling the vehicle when the ADS is not in operation, and a remote assistant responsible for monitoring and providing prompts to the vehicle when the ADS is operating	Driving team transitions after 1- and 2-hour blocks with responsibility for the DDT using same method described above	14 hours vehicle in motion split between team; 16 hours on duty	N = 32 (16 drivers, 16 remote assistants)
4	ADS teaming with a Remote Driver in same truck	Driving performed by remote driver outside of the vehicle, who is responsible for monitoring or directly controlling a vehicle with ADS while ADS is not in operation, and by ADS when ADS is operating	Driving team transitions after 1- and 2-hour blocks with responsibility for the DDT using same method described above	14 hours vehicle in motion split between team; 16 hours on duty	N = 16

The following hypothetical examples are intended to further explain the operational profile of the teaming use cases listed in the previous table. The example descriptions use the terms for the job roles included in each teaming use case. As this study will be completed in a driving simulator, the actual study will not include operation of fully operational CMVs. Where the term CMV is used below, a participant would be operating a driving simulator.

Use Case 1: Single CMV; Human Driver performing DDT or ADS performing DDT 

Hypothetical Example: A driver is on board the CMV in the driver seat and operates the manual controls to move the CMV from a freight distribution center to the highway. After entering the on-ramp of the highway, the driver operates the CMV for a period of time in heavy traffic on the highway. After this period ends and the traffic calms, the driver pulls the CMV off the highway onto the shoulder or onto a specially marked automation transition lane. The driver checks all ADS functions and enters the navigation destination. The driver activates the ADS, re-positions and restrains as necessary for the scenario, the ADS engages and begins operating the CMV on the highway, continuing to the destination point, after the driver has transitioned to a secure rest area in the CMV sleeper. The ADS may encounter challenging roadway conditions in the future that may require the ADS to pull the CMV off the highway onto a shoulder. After parking (in a minimal risk condition), the driver can return to the driving seat and operate the CMV during the current leg of the trip or the entire trip. This use case may be applicable and of interest to less-than-truckload operations. After continued ADS operation ends, the driver is on board the CMV and back in the driver seat and operates the manual controls to move the CMV from the highway to a freight distribution center.

Use Case 2: Two-CMV Platoon; Human Driver performing DDT in Truck 1, ADS performing DDT in Truck 2

Hypothetical Example: CMV-1 and CMV-2 have been moved and setup at a specially marked automation transition lane on the highway. A driver remains onboard CMV-1 in the driver seat and operates the manual controls to move CMV-1 from the highway automation transition lane onto the highway. CMV-2 is being operated by ADS after being staged in the highway automation transition lane and follows CMV-1 onto the highway. CMV-1 continues to be operated by a driver and CMV-2 continues to follow CMV-1 and be operated by the ADS until arriving at another automation transition lane waypoint. CMV-2 may need to separate and re-connect with CMV-1 occasionally due to traffic. The driver in CMV-1 will monitor the status of CMV-2 throughout the trip and may slow to a minimum safe speed to allow CMV-2 to re-connect when necessary. This scenario represents a platooning operation under consideration (past and present) by ADS developers.

Use Case 3: Single CMV; Human Driver performing DDT or ADS performing DDT combined with a Remote Assistant

Hypothetical Example: This example is similar to Use Case 1 and involves only one CMV; however, a Remote Assistant will be available at all times to monitor the ADS and to provide prompts to the ADS. When the ADS is operating the CMV and encounters a challenging roadway condition (e.g., dynamic construction zone), the Remote Assistant can support the ADS in making operational decisions. The Remote Assistant may allow a dynamic transition of the DDT to the driver prior to arriving at the challenging roadway section of highway and after a minimum period of time that allows the driver onboard the CMV and the Remote Assistant to verify situational awareness and takeover. This scenario may represent operations that are under consideration by several of the leading ADS developers as described in the interview task.  

Use Case 4: Single CMV; ADS performing DDT or Remote Driver performing the DDT

Hypothetical Example: This case involves only one CMV. Only the ADS is onboard the CMV while a Remote Driver is prepared during specific parts of the trip to perform the DDT instead of the CMV. The CMV is located at a distribution center or automation hub. The Remote Driver performs the DDT moving the CMV out of the distribution center and onto a highway ramp and onto the highway. As the CMV gets up to speed, approximately 25-35 mph, the Remote Driver is confirming the ADS is ready to perform the DDT for the CMV at higher speeds. The ADS may encounter challenging roadway conditions (e.g., construction zone) in the future that may require the ADS to pull the CMV off the highway onto a shoulder. The Remote Driver prepares to perform the DDT at reduced speeds to navigate the construction zone or other challenging conditions. If the CMV must operate at high speeds to safely operate on that specific section of highway, the Remote Driver will maintain control at low speeds interacting with onsite construction or law enforcement personnel to move the CMV to a safe parking location until a driver can be dispatched to the CMV to complete the current leg of the trip or the entire trip. This scenario represents the operational environments considered by at least one ADS developer.

The following assumptions of the study inform the team use case designs.  

1. The ADS will operate at SAE International level 4 within its operational design domain (ODD), which never requires the driver to take over the driving task while the CMV is in motion.  

2. The ADS shall be capable of coming to a safe stop independently when an equipment failure or un-solvable situation occurs, or the ADS-equipped CMV goes beyond its ODD.  

3. The driver will be allowed to move from the driver seat to the passenger seat or sleeper berth after the transition to ADS operation, per the below conditions.  

   1. The transition from driver to ADS operation will be completed while the vehicle is stationary. 

   2. The ADS-equipped CMV must not operate when the occupant is not properly positioned and restrained

4. Remote assistants and remote drivers will be considered on duty/driving when monitoring or controlling the ADS-equipped CMV.  

CMV drivers are limited in their working and driving hours according to Hours of Service (HOS) regulations. The data collection runs will include “working shifts” beyond current HOS regulations (participant on duty; vehicle in motion for 13 hours in a 16-hour duty period; vehicle in-motion duties split between driver, remote driver, and/or ADS roels). Data collected throughout the long working shifts will provide opportunities to assess teaming use cases in the following scenarios: 

1. CMV operation hours 1–6 as a proxy for short working shifts;

2. CMV operation hours 7–14 as a proxy for working shifts following current HOS regulations; and

3. CMV operation hours 15-16 as a proxy for working shifts beyond current HOS regulations.

The driving time conditions include data collection for periods of time in each teaming use case job role and to capture transitions between teaming use case job roles. A minimum of two transitions will be planned for each participant run. Participant runs will include combinations of 1- and 2-hour blocks per job role, with each run starting with the human driving and ending with the human driving for a total of 13 hours vehicle in motion in 16 hours of on-duty time. An example run (working shift) is shown in Team Use Case Example Run #1 of Figure 2 below. In this example run, the working shift includes 1 hour human driving, 2 hours ADS, and a break in hours 1-6; 2 hours driving, 1 hour ADS, a break, 1 hour driving, 2 hours ADS, a break, 1 hour driving, and 1 hour ADS in hours 7-14; and 1 hour ADS and 1 hour driving in hours 15-16. The design incorporates multiple transitions within each of the proxy working shift lengths. Data collected before and after these transitions will be essential to assess the impact of ADS and remote operation on driver performance. Additionally, within the remote driver scenarios, it will be possible to vary the number of trucks being monitored.

Figure 2. Examples of team use case experimental runs, with team roles switching after 1 and 2 consecutive hour periods.

The simulated drive will include various environmental and roadway conditions, which will be balanced across the study duration and over the participants. The selection of environmental and roadway conditions, to be based on current industry practice and operational design domains of ADS-equipped CMVs, will determine study timing and counterbalancing needs across participants. These scenarios may include dense traffic around rush hour, work zones, divided and undivided highways, slow traffic, and unexpected hazards, for example.

As stated earlier, data collected from this study will be used to understand the impact of driving team use cases and job roles on driver performance, workload, and inattention. To assess driver performance, workload, and inattention, the following data will be collected during the driving simulator portion:

• continuous data such as steering input, brake input, acceleration/deceleration, speed, stop sign/traffic light violations, major and minor crashes, curb strikes, near-crashes, and lane excursions to inform conclusions on driver performance during driving/assisting tasks and transitions;

• eye-tracking data to collect objective measures of the driver's attention, gaze direction, reaction time, and drowsiness to inform conclusions on engagement in the driving/assisting tasks, distraction from a task, and fatigue;

• psychomotor vigilance tests (PVT) data, collected between driving tasks, to inform conclusions on driver fatigue.

2.2 Statistical Methodology for Stratification and Sample Selection

The sampling selection will be purposive in nature. The research team will recruit CMV drivers using the following techniques: contact Class A or Class B CDL drivers from VTTI’s large database of drivers who have participated in previous research studies or have expressed interest in possible participation; contact commercial fleets within one day’s drive of VTTI and ask the fleets to disseminate recruitment announcements describing the study and providing contact information; place recruitment ads in industry-relevant newsletters and social media accounts.

The sample selection method will follow a non-random, convenience sampling plan. Drivers who express interest in the study and meet study inclusion criteria will be selected on a first-come basis. Through the recruitment methods, we anticipate receiving a pool of potential participants that includes females, minorities, and varying driving experience levels. Although CMV carriers will be selected in a nonrandom fashion, this sampling methodology should not produce any bias in any of the key findings to be generated from the study. This stems from the fact that the research is not intended to produce national representative point estimates for any metric, but rather to continue to better understand the potential impact of driving team use cases and ADS teaming technology on driver performance and behavior while operating a vehicle with these technologies.

To the extent possible, participants will be balanced by age and gender across the experimental conditions. Stratification by key variables, such as age, gender, or driving experience, will be included in the analysis approach. In the analysis models, drivers’ demographic characteristics, including age and gender, will be treated as blocking factors, as they may influence driving performance under the study conditions. Participants will be randomly assigned to an experimental condition (teaming use case and driving schedule).

2.3 Degree of Accuracy Needed

The study design includes 80 CMV drivers with a valid Class A or Class B commercial driver’s license. This sample contains 16 participants per job role in each teaming use case. Driving simulator studies allow the effective analysis of fatigue and driving behaviors based on wide ranges of participant sample size.⁸ The sample size in the proposed study was chosen in consideration of the following study factors: (1) the study design includes approximately two to four transitions per teaming use case (an opportunity to assess up to 64 transitions per teaming use case); (2) the proposed shift length of 14 vehicle in motion hours will provide ample time to collect data on distracted behaviors, fatigue, and driving performance; and (3) collecting driving data using a simulator will give researchers more control over the driving environment compared to a naturalistic driving study. The power analysis used a difference in PRT of approximately 15% between drivers in a vehicle monitoring state and in a not-monitoring state as a baseline level for significant findings.⁷ The power analysis assumptions include a goal power of 0.80 (industry standard), alpha or significance level of 0.05 (industry standard), a 15% difference in PRT, and an ANOVA (the study analysis approach to assess reaction time will involve multiple covariates and control of multiple observations per driver). Based on the power analysis, the recommended sample includes approximately 39 transitions per teaming use case. The inclusion of 16 drivers per teaming use case job role is expected to meet the accuracy needs for assessing driver performance and behavior under various study conditions. In addition, the power analysis uses a previous study with similar metrics to guide assumptions. If the proposed study were to find a larger difference in PRT or similar driver metrics, the power of the study would increase.

2.4 Less than Annual Periodic Data Cycles

The proposed study will include data collection from each participant at the time of each sub-study. The data collection is expected to be on less than annual periodic data cycles.

2.5 Analysis Methodology

The analysis methodology uses a multifaceted approach to address research questions on driver workload, fatigue, alertness, distraction, and safety critical event (SCE¹) rate. The principal statistical method for analyzing the data will include mixed-effect regression models to account for multiple, correlated data points from a single participant. The experimental design allows for comparisons of:

1. the same hour of vehicle in motion between teaming use cases (e.g., comparing driving hour 6 for teaming use case Driver + ADS in the same truck and driver in Truck 1 + ADS in Truck 2);

2. different hours in a shift within a teaming use case (e.g., comparing driving hours 2, 4, 6, 8, and 10 for driving team use case Driver + ADS in same truck); and

3. driver/remote assistant/driver roles at the same hour of vehicle in motion within a teaming use case (e.g., comparing driving hour 6 for driver and remote assistant/driver for teaming use case Driver + Remote Assistant/Driver in same truck).

Data collected in this study will include continuous data such as steering input, brake input, acceleration/deceleration, speed, stop sign/traffic light violations, major and minor crashes, curb strikes, near-crashes, and lane excursions; eye-tracking data to collect objective measures of the driver's attention, gaze direction, reaction time, and drowsiness; and PVT data, collected between driving tasks. These data will be assessed in various response variables to inform conclusions on driver performance, workload, inattention, and fatigue, as described below.

Continuous data will be collected through a video monitoring system that collects continuous video and simulator data during the driving scenarios. This system will be integrated to record data from the forward roadway simulation, the left-side and right-side simulations, a driver-facing camera, and an over-the-shoulder camera (when appropriate). Examples of driver performance variables and metrics from the continuous data include eye state; peak acceleration/deceleration; peak and average speed; standard deviation of lane position; and frequency and severity of traffic violations, crashes, curb strikes, near-crashes, lane excursions, hard braking. Inattention variables and metrics include eye state; total time eyes off forward roadway; average and maximum glance length; and frequency and duration of secondary task engagement.

Eye-tracking data will be used to assess driver workload, fatigue, alertness, distraction, and reaction time. These data will be described using summary statistics and advanced plotting techniques to visually compare drivers and remote drivers (distinct job roles) during in-vehicle driving, in-vehicle monitoring, and remote operation. Data will be assessed for periods of operation of the vehicle or transitions between job roles of a teaming use case. Metrics to measure fatigue and inattention through eye-tracking data include the following. One way to measure fatigue will be through observation of eye closures or other drowsy behaviors, using previously validated metrics like PERCLOS or Observer Rating of Drowsiness. Distraction or inattention will also be measured by time the eyes look off road or frequency of long glances away from the forward roadway (greater than 2-seconds in duration). Reaction time can be measured as the time between a roadway event and driver returning hands to steering wheel or feet to brake pedals. Reaction time will be plotted over driving time or compared between periods of transition driving and regular, non-transition periods of driving. Another option to measure fatigue will be through PVT data.

A generalized linear mixed model (GLMM) will be used to assess differences in average performance, fatigue, workload, alertness, distraction, and reaction time metrics between in-vehicle driving and remote driver driving operation types. A GLMM measures the relationship between a response variable and explanatory factors and in the transportation safety field, GLMMs are often used to analyze driver behavior and assess relationships between driving scenarios and behaviors.⁹ A GLMM is an extension of the generalized linear model, in which the linear predictor contains random effects in addition to the usual fixed effects. As participants will experience multiple transitions, it will be necessary to use a model that includes driver-specific random effects.

For each research question, individual models will be built for the driver readiness and performance measures and experimental condition variables. Explanatory factors included in each model will be determined by the research question. They can include teaming use case, vehicle driver, driving hour, and transition or non-transition period. The models will also control for correlated data from individual participants by including a random error term for a participant. General descriptions of anticipated models are:

1. For driver performance or readiness measures with binary response options (e.g., eye state), the model will be a mixed-effect logistic regression model.

2. For categorical variables with more than two response options (e.g., driver state), a multinomial logistic regression model will be used.

3. For continuous response variables (e.g., speed or following distance), linear mixed models will be used to test for differences between experimental conditions.

4. Longitudinal mixed models may also be used to assess fatigue and reaction time over time in a shift (Grove et al., 2016). Longitudinal mixed models can compare changes in a variable over time, while controlling for data from a single participant. The models can be used to assess the impact of time on task, under different operating conditions, and on fatigue or reaction time.

In addition, hourly rates of SCEs, including unintentional lane deviations (which are surrogates for fatigue and alertness), will be analyzed using a Poisson or negative binomial mixed-effect regression model. Poisson or negative binomial regression models are standard practice for the assessment of events over a unit of exposure in the field of transportation safety.¹⁰ Mixed-effect Poisson or negative binomial regression accounts for exposure time (in this case, driving hours) as well as correlations among observations from the same driver.¹¹ As participants will likely have multiple driving operation type transitions, non-transition breaks, and operating periods, it is necessary to use a model that includes driver-specific random effects. Output of these models will include odds ratio estimates of the increased likelihood of being involved in an SCE at the time of human-ADS transition. In addition, population attributable risk estimates will be calculated.

2.6 Considerations for the Study Design

The study design for the driving team use case assessments was created to best address the research questions within certain experiment bounds. The design maximizes data collection in scenarios of particular interest within the time constraints of a simulated driving session. The design also creates a simulated driving environment that closely resembles real-world driving. The design also controls for simulator experience as a possible confounding variable. However, it is important to also highlight design limitations. One such limitation is the incomplete randomized order of study factors. Orders of environmental or roadway conditions will be counterbalanced across participants but will be limited to orders that most resemble real-world driving (e.g., drivers will not cycle through night driving, day driving, and again night driving within a short period of time).

3. DESCRIBE METHODS TO MAXIMIZE RESPONSE RATE AND TO DEAL WITH THE ISSUES OF NON-RESPONSE.

3.1 Methods to Maximize Response Rate

Participants will be recruited from VTTI’s database of drivers who indicated interest in future studies and CMV drivers located within a day’s drive of VTTI headquarters in Blacksburg, Virginia. The database includes drivers who have previously completed a study with VTTI, and therefore have shown a commitment to study participation in past research. Considering the proximity of accessible, interested participants, VTTI expects to find drivers with a desire and ability to participate and complete the study. Drivers selected for the study will be reminded of the study participation date through phone calls and email the day before their scheduled participation. The phone calls and emails will include VTTI contact information, with multiple VTTI contact options.

The participants will be told their participation is voluntary, and they can terminate their participation at any point without prejudice or harm to them in any way. Participants will have the opportunity to have any questions answered prior to deciding to participate. This should also increase the likelihood that they complete the entire study.

The research team will offer incentives to promote interest in participating in the study and to improve retention over the study period. The proposed incentives have been reviewed and approved by the Virginia Tech Institutional Review Board. Drivers will receive an additional incentive if they complete the study session. Incentives will be distributed via a rechargeable debit card, which participants will receive at meeting.

In simulator studies, participants may feel simulator sickness and excuse themselves from further participation. To minimize these simulator side effects, and thereby maximize responses, verbal health checks will be given periodically to allow for breaks and rests. In the case of participants who do not respond, or choose to withdraw from the study, new participants will be recruited to fill their spots. Non-response data points will not be included in the analysis. However, any data points collected prior to a participant’s withdrawal will be included.

3.2 Methods to Deal with Issues of Non-Response

During data collection, there will be multiple opportunities to correct missing data to lessen instances of non-response. For example, driver questionnaires filled out at the study start will be reviewed by a researcher during the simulator portion of the study. If the questionnaire is not fully complete, participants can be asked at study completion if they would like to provide a response to missed questions or keep responses as originally submitted. During the simulator portion of the study, drivers will be presented with multiple opportunities designed to elicit a behavior and capture a driving performance metric (examples include several transitions between driving team use cases during the participation period or including multiple events during the simulated drive that would normally cause a driver to brake). This will provide the participant multiple opportunities to provide answers and data for study questions scheduled during the participation time window. Researchers will address any follow-up questions participants may have regarding the study question.

There will be several additional strategies to deal with non-response in the data. These include:

1. Generalize to the respondents only. This strategy avoids making erroneous inferences about the larger population.

2. Compare data in hand on respondents and nonrespondents. If data (e.g., gender, age, race) is available, the composition of respondents will be compared with that of nonrespondents to see if there are any differences. The presence of differences indicates response bias and that caution is necessary in making inferences.

4. DESCRIBE TESTS OF PROCEDURES OR METHODS TO BE UNDERTAKEN.

All data collection methods proposed in the current study have been assessed in prior VTTI studies. The demographic questions have been successfully used and tested in various prior VTTI studies.¹³ Alertness questions included in the demographics survey were cited from the well-validated Karolinska Sleepiness Scale.¹² The driver behavior questions pertaining to signs of aggressive driving on the roadway use the Dula Dangerous Driving index, a well-known metric for assessing driving characteristics.¹³ Moreover, the simulator sickness procedures and questionnaires are all materials administered in previous VTTI simulator studies to minimize participant risk of negative side effects.^6,14,15

The study will include a pilot test with a trained researcher with a valid CDL acting as a study participant. The pilot test will use the protocol drafted for the full experiment. During the pilot test, the researcher will operate the simulator as directed in the full tests. Results from the pilot test will be used to identify areas for improvement and refine study protocols for the full test runs.

5. PROVIDE NAME AND TELEPHONE NUMBER OF INDIVIDUALS WHO WERE CONSULTED ON STATISTICAL ASPECTS OF THE INFORMATION COLLECTION AND WHO WILL ACTUALLY COLLECT AND/OR ANALYZE THE INFORMATION.

Project Leads for this ICR:

Susan Soccolich

Statistician

Virginia Tech Transportation Institute

3500 Transportation Research Plaza

Blacksburg, VA 24061

(540) 231-1032

[email protected]

1 Safety-critical events, or SCEs, are moments during driving where the subject vehicle is involved in a crash, near-crash, crash-relevant conflict, or unintential lane deviation; each SCE type has a specific definition and validated kinematic threshold for vehicle-related metrics, such as longitudinal acceleration, swerve, and time-to-crash.

1 Federal Motor Carrier Safety Administration. (2021). 2021 pocket guide to large truck and bus statistics. Federal Motor Carrier Safety Administration. https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/2022-01/FMCSA%20Pocket%20Guide%202021.pdf

2 Zippia. (2021). CDL Class A Driver: Demographics and statistics in the US. https://www.zippia.com/cdl-class-a-driver-jobs/demographics/

3 Zippia. (2021). Class B driver demographics and statistics in the US. https://www.zippia.com/class-b-driver-jobs/demographics/

4 Nelson, E. T., Kidd, D. G., & Cades, D. M. (2010). Examining patterns of simulator sickness

during increased exposure to a motion-base driving simulator over Time. Journal of the Washington Academy of Sciences, 96(3), 1–14. http://www.jstor.org/stable/24536258

5 Classen, S., Hwangbo, S.W., Mason, J., Wersal, J., Rogers, J., & Sisiopiku, V.P. (2021). Older drivers’ motion

and simulator sickness before and after automated vehicle exposure. Safety, 7(26). https://doi.org/10.3390/safety7020026

6 Morgan, J. F., Tidwell, S. A., Medina, A., Blanco, M., Hickman, J. S., & Hanowski, R. J . (2011). Commercial motor vehicle driving simulator validation study (SimVal): Phase II (Report No. FMCSA-RRR-11-014). Washington, D.C.: United States Department of Transportation Federal Motor Carrier Safety Administration.

7 Zhang, B., Wilschut, E. S., Willemsen, D. M. C., & Martens, M. H. (2019). Transitions to manual control from highly automated driving in non-critical truck platooning scenarios. Transportation Research Part F: Traffic Psychology and Behaviour, 64, 84-97.

8 Soares, S., Ferreira, S., & Couto, A. (2020). Driving simulator experiments to study drowsiness: a systematic review. Traffic Injury Prevention, 21(1), 29-37.

9 Guo, F. (2019). Statistical methods for naturalistic driving studies. Annual Review of Statistics and Its Application, 6, 309–328.

10 Guo, F. (2019). Statistical methods for naturalistic driving studies. Annual Review of Statistics and Its Application, 6, 309–328.

11 Blanco, M., Hanowski, R. J., Olson, R. L., Morgan, J. F., Soccolich, S. A., & Wu, S.-C. (2011). The impact of driving, non-driving work, and rest breaks on driving performance in commercial vehicle operations. (DOT-HS-810594). Washington, D.C.: United States Department of Transportation Federal Motor Carrier Safety Administration. Retrieved from http://ntl.bts.gov/lib/51000/51300/51387/Work-Hours-HOS.pdf

12 Kaida, M., Takahashi, T., Åkerstedt, A., Nakata, Y., Otsuka, T., Haratani, K., et al. (2006). Validation of the Karolinska sleepiness scale against performance and EEG variables. Clinical Neurophysiology, 117, 1574–81.

13 Dula, C. S., & Ballard, M. E. (2003). Development and evaluation of a measure of dangerous, aggressive, negative emotional, and risky driving 1. Journal of Applied Social Psychology, 33(2), 263-282.

14 Robin, J. L., Knipling, R. R., Tidwell, S. A., McFann, J., Derrickson, M. L., & Antonik, C. (2005). FMCSA Commercial Truck Simulation Validation Study Phase I Pilot Test: Driving scenario definition and development. Driving Simulation Conference North America 2005, Orlando, FL.

15 Morgan, J. F., Tidwell, S. A., Blanco, M., Medina-Flinstch, A., & Hanowski, R. J. (2013). Commercial truck driver performance in emergency maneuvers and extreme roadway conditions presented in a driving simulator. Washington Academy of Sciences, 99, 25-37.

7