DRAFT_ENS_Sign_Justification_PartB_6 14 19_VolpeResponses

SUPPORTING JUSTIFICATION – Part B

ENS Sign Study; OMB No. XXXX-XXX

Form Number FRA F 6180.158

1. Description of sampling method to be used.

FRA will recruit between 100 and 120 drivers for the experiment.

A convenience sampling method will be utilized. The population of interest is individuals who have a basic automobile license (Class D). They will be recruited from the general public located in and around the Boston, MA geographical area. Participants will be recruited through flyers (e.g., on college campuses or in DMVs), emails to known organizations or groups who may be interested, as well as through online postings on recruitment websites. The reason for the range in drivers needed for the experiment is due to the possibility of poor or incomplete data collected from several participants. We will only need to collect full observations from 100 participants.

2. Description of procedures for information collection, including statistical methodology for stratification and sample selection.

2.1 Statistical Methodology:

The purpose is to explain the statistical methodology and, in particular, to explain the difference between conducting an experimental test of a hypothesis (what we are planning to do) and trying to generalize the results from a survey. It is important to discriminate between these two purposes because they lead to very different estimates of the sample size. (a) The number of subjects needed for testing a hypothesis (e.g., the means of two groups of drivers are equal) depends on the predicted effect size, probability of a Type 1 error, and probability of a Type II error. (b) The number of subjects needed for generalizing the results from a survey depends on the size of the population, the level of confidence, and the precision.

The research question being considered can be summarized by the following: are drivers aware of the ENS signage at highway rail crossings and where do they look to find it in situations where they might consider calling for assistance? The scenarios described below will help to better understand this question through a simulator-based study that gathers driver behavior and eye movement data. There are several hypothesis related to this general research question that the experimenters will be testing:

• Drivers approaching a malfunctioning crossing are unlikely to consider looking for the ENS sign or to consider calling the ENS sign number if they see the signage.

  • However, signage that is oriented perpendicular to the direction of traffic (i.e., forward-facing upon approach) will be most likely to be seen and have the most likely chance for a call.

• Drivers whose vehicles are stalled on a crossing are likely to look for the ENS sign and to consider calling the number for assistance.

  • Among stalled vehicle situations, signage oriented parallel to the roadway will be more likely to elicit eye glances and calls than signage oriented perpendicular to the roadway, as the latter relies on memory from prior instances.

Additionally, while not a formal hypothesis, the experimenters expect that conversations with participants after the simulator portion will help to reveal their level of familiarity with ENS signs in the real world. It is expected that the majority of participants are unlikely to be familiar with the signage and how the signage may be used.

For evaluating the hypotheses that we are testing, we need about 100 subjects, for generalizing the results from a survey we would need about 384 subjects. We have discussed in detail the actual computations required to obtain the sample size for the test of our hypothesis. Below, we point to the use of a calculator available on the internet which can be used to determine the sample size for the test of a hypothesis. This is the sample size we need. We also point to an online sample size calculator for computing the sample size to obtain a precise estimate of a population statistic.

1. Experiments: Testing the Difference in Proportions

We estimate that we will need 50 participants in each group to test the hypothesis that the recognition of perpendicular and parallel signs differ. We assume that the probability of a Type 1 error is 0.05, that the probability of a Type II error is 0.20 (power is equal to 0.80), and that the parallel sign group will detect only 10% of the signs posted whereas the perpendicular sign group will detect 30% of the signs posted. For the purposes of this study when we say that a participant detects the sign that means the participant looks at the sign for at least .5 seconds. It is also possible to use statistical calculators, like the one found at https://www.stat.ubc.ca/~rollin/stats/ssize/b2.html. While 3 of the four scenarios described in detail below will utilize a within subject design (participants will see both parallel and perpendicular signs), the final scenario (scenario 4) will utilize a between subject design (half participants will see a parallel sign and the other half will see a perpendicular sign). Due to this between subject design we will need to use 50 subjects for each of these groups for a total of 100 participants (50 participants in the parallel sign group and 50 participants in the perpendicular group).

2. Surveys: Precise Estimates of Proportions

Below, we show the results from a statistical calculator (https://www.surveysystem.com/sscalc.htm#one) that computes the sample size for surveys. Note that now we need to know the population size. We did not need to know this for the experimental test of the hypotheses (above). With a population of 220 million, a confidence interval of +/- 5, and a confidence level of 95%, we would need roughly 384 participants. We are not doing this type of research.

2.2 Procedures (All time calculations have been included in the burden estimates):

1. Each participant will arrive at the Volpe Center. Prior to commencement of any study procedures, the participant will read and sign a consent form, and then be assigned to an experimental condition.

2. Once the participants are familiar with the simulator, they will do a complete practice (training) run in the simulator world on the driving simulator (RTI). The practice run lasts approximately 10 minutes. This practice run will allow the participants to become familiar with the territory and the simulator controls.

3. After the practice run, participants will be allowed a 5-minute break, during which they can stretch, consume refreshments, and use the restroom facilities. When participants return, they will complete their first experimental run on the driving simulator. The experimental run will last approximately 40 minutes. Upon completion of all runs (training and experimental), participants will be allowed a final break.

4. After the simulator portion of the experiment, participants will fill out a brief demographic and usability questionnaire. This will take approximately 10 minutes. The questionnaire following the simulator experiment will help us gather preliminary information on the usability of the ENS Signs. This may be helpful in future experiments. The answers to this questionnaire are not expected to be generalizable to the population of drivers, and are solely to provide information to the research team that may inform future experimental sign placement and design.

Detailed Description of Simulation:

The simulator portion of this study will be designed to examine participants’ vehicle, glance, and report behaviors as they approach a variety of rail crossings, including some which are malfunctioning. A mock town (or system of roadways) will be developed with several rail crossings as well as other scenarios that a driver may expect to encounter driving through a real life town (these are discussed further below). Before starting the study, participants will be able to familiarize themselves with the various roadways, crossings, and locations through which they will be asked to navigate. This familiarization will occur without any train activations, traffic, or pedestrians – but rather simply to make participants aware of the road layouts and routes to locations of interest. They will also learn that there are several routes to the same location. Thus, if USDOT wants to determine whether drivers will violate a crossing warning system or take an alternative route, an alternative route will be available to them.

Once the experiment begins, the participants will be asked to drive through several scenarios. We expect each driver to experience five to seven scenarios, each lasting approximately five minutes. In each of these scenarios the driver will start driving from a different location within the simulated town and be asked to navigate to another location, necessarily going through several rail crossings and experiencing several other non-crossing traffic hazards. Each drive will be timed and drivers will be asked to get to the end point as quickly as possible.

Crossings throughout the scenarios will have the ENS sign in different orientations, likely split 50% parallel to the train tracks (Figure 1) and 50% perpendicular to the train tracks (Figure 1). The participants’ vehicle, glance and report behaviors will be recorded for each scenario to see how they react to each crossing with particular attention paid to behaviors at malfunctioning crossings. Experimenters will investigate: the driver behavior (vehicle actions), if they looked at the ENS (the glance behavior), and if they made any attempt to report the malfunction. The participant’s glance behaviors are recorded with a head mounted eye tracker which records a video of what the driver sees on the roadway ahead and overlays a marker on the exact location at which the driver is looking. The accuracy of the eye tracker is sufficient to determine whether the participant’s eyes actually did or did not fixate an ENS sign.

Figure 1. ENS Signs Parallel to Train Tracks Figure 2. ENS Signs Perpendicular to Train Tracks

It is common in a simulator experiment for the participant to ask questions of the experimenter, especially when the scenario presents the participant with a situation that the participant thinks may be an error. With a gate malfunction or stuck vehicle, some sort of dialog between the participant and experimenter is expected (e.g., “is this supposed to be happening?” or “what should I do?”). For this study, the experimenter may be allowed to converse with the participant if/when prompted and provide a response (from a limited set of experimenter responses) asking them what they would do in the real world. If the participant indicates that they would call the number on the ENS sign (or call 911 or any other number), the experimenter will tell them to imagine that the call was made and the issue was rectified (e.g., no train is coming and the issue is being resolved). All interactions between the participant and experimenter will be recorded and analyzed and experimenter responses would be limited to a finite set.

To ensure that participants understand that this type of interaction – and a choice to inform the experimenter that they would make a call – is possible, there will be a situation early in the experiment or practice where the driver experiences a flat tire. The vehicle will suddenly become immobile. The participant will ask the experimenter why the vehicle isn’t moving and the experimenter will inform them that their car got a flat tire and is stuck. This will lead to a dialog about what the participant would do in real life – and if they say they would call AAA or a friend, then the experimenter will tell them that the call was made and the vehicle was fixed. The goal of this scenario is to show participants that interaction with the experimenter is possible and that they can act as a surrogate to call for help in this world.

In each scenario the participant may experience the following:

Crossing Related:

• False Activation (or false partial activation): The driver approaches an active crossing that has been activated, but no train will ever come. In some cases, this may be a bells/lights only crossing. In other cases, the crossing may have gates, but only the gate for the opposing traffic is down – and the gate in front of the participant is still up (despite the bells/lights being active).

• Activation Failure: As the driver approaches an active crossing with gates, but before they are too close, a train arrives without the warning systems ever activating.

• Correct Activation: The warning systems at a crossing function as intended. Bells/lights activate (and in some cases the gates descend) 20 seconds prior to a train arrival.

• Crossing without Activation or Train: The warning systems don’t active and no train arrives – driver just traverses the crossing normally.

• Stalled Vehicle: In the final scenario tested, the driver will start out on the tracks and their vehicle will not be able to move. After a short while they may hear a train horn in the distance, but the crossing will not activate. The driver will have to communicate with the experimenter about what they would do in this situation.

[Notes about Stalled Vehicle Scenario:

• It is expected that a vehicle stalled on the tracks would be the situation where a driver is most likely to look for a number to call. As such, gathering data on if/where participants look and how they react in such a scenario is an important aspect to capture.

• Since a stuck vehicle is likely to be the situation where participants are most likely to look for a number to call, we wanted to only present this situation at the very end of the study. Presenting this situation earlier may result in participants being more aware of the signage than they otherwise would have been, and thus bias them to look for or mention the signage in other parts of the study. The experimenters believe that if a driver notices and mentions the ENS signage prior to the stalled vehicle scenario, then they would be extremely likely to mention it in the stalled vehicle scenario as well, but not necessarily the other way around, hence the need for this scenario to be last.

• One potential concern is that this type of stuck vehicle scenario may bias participants towards the parallel signage orientation (as this orientation is the only one visible when the vehicle is stuck.) While the experimenters understand this concern, and will be sure to document this fact, we believe that this bias for seeing the parallel sign in this scenario exists in reality as well. Drivers would likely rely on their memory of seeing that signage in prior instances to seek out the number to call in both this scenario and in real life. Drivers in real life are afforded the option to exit the vehicle and walk back to see the signage – something that is not available in this simulator – and therefore experimenters will seek to engage with participants should they indicate that they would exit the vehicle were this to happen to them in real life. This engagement would probe, but not lead participants, with questions such as, “After you exited the vehicle where would you walk?” and “What would you do once safely away from your vehicle?”]

Non-Crossing Related:

Note that not all of the situations below will be seen by every participant. Data will not be collected on driver behaviors around these other incidents, they are present to simply make the goals of the experiment less clear while the participant drives through each scenario. The experimenters will pilot these situations to identify a limited number of these decoy scenarios which will serve to take focus away from the crossing scenarios.

• Stop Light Malfunction: The driver will encounter an intersection (T intersection) where the traffic signals have stopped working and are simply off. Driver will have to decide when to go in traffic.

• Mid-Block Crosswalk with Pedestrian: The driver will encounter a mid-block crosswalk, potentially obstructed by a parked vehicle, with pedestrians crossing as they approach.

• Work Zone Impediments: The driver will encounter a work zone that partially infringes into their lane of travel. With oncoming traffic, the driver must determine the appropriate sized gap to circumvent the work zone safely. In some cases, we may place a police vehicle near this zone.

• Vehicle Breakdown: The driver’s vehicle, early in the first scenario, will become inoperable. The driver will have to communicate with the experimenter about what they would do in this situation.

Sign Clutter

• Signage Clutter: There will be many signs present throughout the study. The idea of presenting a wide variety of signs throughout the experiment is because this is often what driver experience on today’s roadways. Additionally, it will make the ENS signs stick out less abruptly when other signs are also present.

Each crossing throughout the experiment will have at least two ENS signs (one for each direction of traffic). Depending on the crossing those signs may be placed perpendicular to the roadway (e.g., on the mast below the crossbuck) or parallel to the roadway (e.g., on its own post facing the roadway next to the tracks). Drivers will experience both perpendicular and parallel signage orientations throughout, but only one orientation per crossing. Each driver will experience one perpendicular and one parallel orientation for the two malfunction types, or four total malfunction scenarios. Drivers will only experience one orientation for the final stuck on the tracks scenario, which will be counterbalanced across all participants.

One potential concern with this type of design is the internal validity of the study after repeated exposure to the test. If a driver mentions/responds to the ENS sign during the first gate malfunction, they are surely more likely to mention/respond to the ENS sign in all subsequent scenarios. However, due to the within subjects design of the study, subsequent gate malfunctions may present the participant with a different ENS sign orientation, thus collecting data about where they look to find the sign is still valid. Sign orientation in the real world is not necessarily consistent from crossing to crossing, thus knowledge can be still be gained from these subsequent exposures. Documentation of when the participant first mentions/responds to the ENS sign will be kept so that results can be analyzed separately prior to and subsequent to initial discovery of the ENS signage.

1.1Summary of Variables

To summarize what each participant will experience and what the experimenters will be controlling across the study. The following variables will be considered across the entire study – which is likely be contain between five and seven scenarios, each lasting at least 5 minutes.

Controlled variables for which vehicle, glance, and report behaviors will be analyzed:

• One false activation with parallel ENS sign orientation

• One false activation with perpendicular ENS sign orientation

• One activation failure with parallel ENS sign orientation

• One activation failure with perpendicular ENS sign orientation

• One stuck vehicle on the tracks with either parallel or perpendicular ENS sign orientation

Dependent variables to be studied (for each scenario described above):

• Driver behavior: How did the driver operate the vehicle in this situation?

• Glance behavior: Where did the driver look in this situation?

  • Using an Eye Tracker, we will determine:

    • Did the driver glance at the ENS sign and dwell on the sign for at least one and a half seconds (Binary outcome). A glance can occur at any time while the driver is approaching, stopped at, or progressing through the crossing.

      • Was the first glance at the ENS sign before, after, or without any experimenter interaction?

    • How long after encountering a malfunctioning crossing did it take the participant to glance towards the ENS sign (in seconds)?

      • In what order did the participant glance at the signage at the crossing (assuming they glance at more than one sign)? (i.e., Do they first glance towards the surrounding signs, such as a stop sign nearby, that are not the ENS sign and then later at the ENS sign?)

    • Does their glance return to the ENS sign multiple times/ how many glances towards the ENS sign are made after the first glance? A glance towards the ENS sign will only be counted if the dwell time is 1.5 seconds or greater.

• Report behavior: Did the driver mention the ENS? Did they indicate that they would make use of the information on the ENS sign?

Controlled variables which will not be analyzed:

• One stuck vehicle, not on the tracks (flat tire)

• At least two work zone scenarios

• At least four mid-block crosswalks – all of which will have a pedestrian and in some cases that pedestrian threat may materialize

• At least two malfunctioning traffic signals

• At least four normally functioning active crossings which are activated by an approaching train

• At least four normally functioning active crossings which are not activated by a train (no train materializes)

3. Description of methods to maximize response rate and to deal with non-response issues.

Compensation for each participant’s time and travel will be given as an incentive to participate in the experiment. Researchers will continue recruitment efforts until the expected sample size is achieved. If initial recruitment efforts are found to be insufficient, the research team will explore alternative online outlets for recruitment or may consider altering the compensation amount if response rate is exceedingly poor from the beginning. A previously OMB approved study titled “Driver Ability to Cope with Unintended Acceleration by Shifting Gears in a simulator Experiment” paid each of their 178 participants $75 for their time. This study lasted approximately one hour which is the amount of time we expect the current study to last.

4. Describe any tests for procedures or methods to be undertaken.

A pilot study with 5 or 6 drivers will be conducted to refine the data collection procedures and instruments. Because this pilot is designed solely to test the study methods and not for analysis of the data, we will select the pilot participants. This pilot study will include a complete experimental run on the simulator. The drivers will then provide the research team with informal feedback regarding the experimental design. The experimenters are seeking to specifically refine their understanding of how participants are likely to react to the malfunction scenarios. This includes:

• Do participants engage the experimenter when they encounter a malfunction – perhaps thinking that the simulator is not working as desired?

• This information may be used to make slight modifications to how the experimenter responds to these engagements with a goal of never leading the participant towards a certain behavior while still opening the possibility to mention a behavior that is limited in the simulator (e.g., using their phone or seeking an alternate route).

• Do the looking behaviors that we observe, and the definitions we have set to define a glance, meaningfully capture the drivers’ reports of seeing/reading the ENS signage?

• The experimenters have a plan for how to define glances at the ENS signage, however, the pilot participants will provide an opportunity to compare the participant feedback about what they did/saw with the eye movement data. Feedback from participants about if they saw/read the sign may help to refine the boundaries around which the experimenters consider the sign to be officially glanced at.

If the pilot testing were to identify concerns that would result in more substantial alterations and would require changes to the overall study design, the experimenters will seek to coordinate with OMB to amend supporting documents, as needed.

Limitations

This is primarily an exploratory study of the behaviors of drivers when approaching signs that can be placed in two different locations. Given this, the work includes several limitations that may affect the generalizability of the results. First, this is a convenience sample utilizing one hundred (100) drivers from the Boston, MA area, therefore, the results cannot be generalized to the entire population of drivers across the United States. Also, this study will be conducted in a simulator and hence the results cannot necessarily be directly generalized to the on road community without further study. FRA will not be making recommendations as a result of this study and FRA will be publishing the study independent of whatever results are obtained.

5. Provide name and phone number of individuals consulted on statistical aspects of study design and other persons who will collect/analyze information for agency.

Scott Gabree, Ph.D. (617) 494-2530. Volpe (Collect and Analyze)

Jared Young, (617) 494-2629. Volpe (Collect and Analyze)

Emma Ranalli, (617) 494-3021. Volpe (Collect and Analyze)

James Dahlem, (202) 493-0571. Federal Railroad Administration (Consulted)

9