Summary of Responses to OMB Comments

Summary of Changes to the Flexible Sleeper Berth Pilot Program Analysis Methodology Modifications

Summary of Comments Received from the Office of Management and Budget (OMB)

After reviewing the Flexible Sleeper Berth (FSB) Pilot Program Information Collection Request (ICR), OMB had several comments pertaining to the analysis methodology:

1. The lack of control group. Every participant would effectively be in the treatment group, so there would be a lack of data to compare against. It would not be possible to separate treatment effects from latent variables with the current study design.

2. The hypothesis described in Part B does not accurately reflect the goal of the study as described in Part A. The nulls appear to be mis-specified and do not flow logically from Part A.

3. The assumption that _I is a random effect over subjects with mean, =0, and variance, ²=₂ is not a reasonable assumption, which is the key to the error in the overall research design.

4. There are concerns with the assumptions made in the power analysis, and it does not address co-variance or repeated measures.

Summary of Original Analysis Methodology

The original analysis methodology included a within-and-between subjects analysis of drivers operating naturalistically under the flexible sleeper berth allowance, during which time they could choose to split their 10-hour rest period or take a consolidated 10-hours of rest, depending on what worked best for their schedule.

The original analysis plan specified the null hypotheses to compare whether participants received an equivalent amount of sleep when splitting their sleep as compared to consolidated daytime or nighttime sleep, as shown in Figure 1.

Figure 1. Original hypotheses tests for Flexible Sleeper Berth Pilot Program.

Several secondary hypotheses followed the same structure as the primary hypotheses, to test psychomotor vigilance test (PVT) lapses, safety critical events (SCEs), subjective sleepiness, and roadside violations.

The primary regression model was formulated as shown in Figure 2.

Figure 2. Original proposal for regression model analysis.

Summary of Analysis Methodology Modifications

The revised analysis methodology is described below:

1. A baseline period of two weeks will be included, where drivers will be operating only under the current HOS regulations (i.e., consolidated sleep periods). This will allow each driver to experience a control period of two weeks. Note that days where drivers have several days of consolidated sleep after the baseline period may be grouped with the baseline/control period, as appropriate; however, the two-week baseline will ensure sufficient data are collected on drivers operating under the current HOS regulations.

2. A single primary hypothesis test, will seek to determine whether there is statistical evidence that drivers perform worse under an HOS including flexible sleeper berth than current regulations (seen below in Figure 3).

3. Additional modeling will be used to determine the effects of other variables when comparing driver operations under the different HOS rules (current regulations versus the flexible sleep option).

The revised primary hypothesis is:

H_o: safety outcome rate | split sleep ≥ safety outcome rate | consolidated sleep

H_a: safety outcome rate | split sleep < safety outcome rate | consolidated sleep

Here, split sleep is defined as periods were the driver is operating under the flexible sleep schedule (i.e., shifts were the driver has utilized the spilt sleep provision) and consolidated sleep includes the baseline period and other periods where the driver operated under consolidated sleep.

In this situation, a failure to reject the null hypothesis would result in favorable findings for allowing flexibility in the sleeper berth regulations by finding no statistical evidence that drivers perform less safely under flexible sleep than they do under consolidate sleep (current HOS regulations).

The proposed model for looking at additional variables will use multiple regression modeling, which will follow the general structure of:

where:

• is the observed safety outcome rate for driver

• is an indicator variable for sleep type, . is the corresponding regression parameter.

• is an indicator variable for day and night, . is the corresponding regression parameter.

• is a variable placeholder for incorporating driver subsample of carrier size. is the corresponding regression parameter. is a random effect term to incorporate the correlation among observations from the same driver .

• is random error.

This modeling method will be used to estimate the effects of regression parameters on driver sleep and performance, including PVT lapses, SCE rates, subjective sleepiness, roadside violation rates, total sleep duration, and other items deemed appropriate during data review.

The minimum power expected by this revised analysis methodology (assuming two safety events per driver) would be 0.80 for a type I error threshold of  = 0.05. The five-driver pre-test showed an average of 16 contributions (vice two) over a 14-day period, which would lead to power exceeding 0.99 for drivers participating for a 90-day period. It is therefore reasonable to assume that power will be at least 0.80, but likely much higher.

Comments and Modifications on Supporting Statement A

Page	Section	Original Text	OMB Comment	Response & Modification
1	1	"Laboratory studies have demonstrated that a split sleep schedule, with the same total hours dedicated to rest divided between two periods, can result in as much or more total sleep time than a consolidated daytime sleep schedule."	Here total hours of sleep and alertness used interchangeably. Is three evidence to support this position, such as research showing that two sessions of five hours of sleep result in greater alertness than one session of eight hours of sleep?	This was an error of omission; changed to read: "Laboratory studies have demonstrated that a split sleep schedule, with the same total hours dedicated to rest divided between two periods, can result in as much or more total sleep time and improved driver alertness than a consolidated daytime sleep schedule.”
7	2.3	"The purpose of this pilot program is to demonstrate how regulatory flexibility related to the SB provision, in conjunction with optional FMP training, could be used to improve driver rest, alertness, and safety performance."	See Part B for comments on how to formalize this research goal.	FMCSA agrees with the OMB comment; the analysis methodology in part B has been revised to properly reflect our interest in comparing safety outcomes of the two regulatory options; additionally, "in conjunction with optional FMP training" has been removed from this sentence, as it is unclear until we determine how prevalent optional FMP training is to say whether we can fully analyze this component or not.
9	8	"While FMCSA understands the commentator’s frustration, our commitment to public safety requires scientific data and statistically significant findings before attempting to revise our current HOS regulations."	This response doesn’t seem to make much sense in this context. If the goal is to show that providing regulatory flexibility doesn’t significantly negatively impact alertness, then the absence of statistical significance might justify action by the agency. Also, significance hinges largely on sample size such that very large samples may determine differences to be statistically significant even if they have no practical meaning.	This should have read "statistically valid", not "statistically significant" per FMCSA's Pilot Program Regulations; updated to read: "While FMCSA understands the public commentator's comment, our commitment to public safety requires scientific data and statistically valid findings before attempting to revise our current HOS regulations. No data has ever been collected on driver fatigue levels under a split-sleep schedule. The Agency chose to remove the old split-sleep rule due to an NTSB recommendation. The agency now has the ability to ensure that a change in the rule will do no harm to a driver's overall sleep time and alertness, resulting in no detrimental effects to the current level of safety for both CMV drivers and the driving public."
11	9	"• $5/day for participation (for up to 90 days, or $450 total)."	Does this mean filling out all forms for the day?	This has been updated to reflect "$100 for agreeing to participate and signing the ICF and completing the background questionnaire" and an additional $100 per month for each month of participation. Drivers will need to remain compliant during the study as part of FMCSA's monitoring plan; drivers will be counseled if they are not compliant with study protocol and if they continue not being compliant they will be removed from the study.
11	9	"• $20 for participating in the study for the full 90 days."	Consider providing a bonus to respondents who participate for the full time span and meet a minimum requirement for completion of assigned tasks.	The payment section was incorrect and has been updated. Drivers will receive $200 for completing the entire 90 days of data collection; they will be monitored throughout the study for a certain level of compliance. Added in the following to clarify: "Compliance will be determined by daily actigraph wear time, PVT performance (e.g., frequently skipping tasks or excessively poor performance indicating lack of effort or distraction), and proper ELD usage. If non-compliance is observed in any of the data, the driver will be called; if the driver receives three calls regarding compliance, they will be withdrawn from the study. However, in cases of extreme non-compliance (no PVTs are taken for multiple consecutive days, the actigraph is removed for multiple days [but not broken], tampering with the SmartDrive system, etc.) drivers may be withdrawn immediately."
11	9	"• $50 for returning equipment at the end of the study."	Suggest making an effort to provide non-monetary incentives, like a hat, decal/bumper sticker, and copies of research reports as they’re published.	The research team feels that a monetary incentive would be much more effective than a non-monetary incentive, such as those suggested. Monetary incentives have worked quite well in the past in similar studies, and we expect that it will greatly increase participation and compliance throughout the pilot program.

Comments and Modifications on Supporting Statement B

Page	Section	Original Text	OMB Comment	Response & Modification
1	1	"While enrolled in this field study, drivers will be given the option to divide their required hours of rest time"	This effectively puts every respondent in the treatment group and no one in the control group. Some set of drivers should be required to operate under the current regulatory regime.	The statistical analysis methodology has been modified to address this comment; a two-week baseline period has been added, per driver, to serve as a control period where drivers are operating on the current HOS regulations for consolidated sleep.
2	2	"As drivers will inevitably leave their companies and thus exit the study"	What about attrition due to burden? Consider a completion bonus for say 80% of requested data submitted over the full 90 days.	This has been revised to note that the anticipated attrition rate covers both drivers leaving their company or voluntarily leaving the study. It has been edited to read: "As drivers will inevitably leave their companies (and thus exit the study), and some may choose to leave the study voluntarily, we expect an attrition rate of up to 20 percent and will recruit up to 240 drivers."
4	3.1	"Drivers will be free to choose whether to operate within the study-granted SB exemption or within the current HOS regulations during each duty period."	See the above comment. This protocol doesn’t include any assignment to conditions, and therefore it won’t be possible to separate any treatment effects from latent variables.	The statistical analysis methodology has been modified to address this comment; a two-week baseline period has been added, per driver, to serve as a control period where drivers are operating on the current HOS regulations for consolidated sleep.
4	3.1	"Each driver may contribute data to one, two, or all three of these categories."	This design ignores individual differences, cumulative effects of sleep deprivation, and it tests a different hypothesis than the one described in Part A. See the comment on the hypothesis tests below for a suggested reframing.	The study plan has been modified to address this comment. An initial baseline period is now included, during which drivers will operate under the current HOS regulations for a two-week period at the start of data collection. Following the baseline period, drivers have the opportunity to use current or flexible sleep as they want. Safety and sleep performance data will be collected during both the baseline and flexible-option period. Driver safety and sleep performance can be compared between the baseline and flexible-option period while controlling for driver differences. In addition, the two periods can be compared to determine how driver performance changes under real-world flexible option use (which may mean drivers use both flexible sleep option and the current regulation, depending on which one they deem most beneficial on any one individual duty day). This will give the research team the most accurate data on how drivers would perform if flexible sleeper berth time was allowed within FMCSA's HOS regulations.
4	3.1	"Our statistical methods (outlined in section 2.4) are robust and account for this potential imbalance."	No statistical methods will be able to account for the lack of a control group.	A two-week base period of drivers operating under the current HOS regulations (i.e., consolidated sleeper berth time only) has been added in to account for this. This design allows drivers to serve as their own control to mitigate individual driver differences that could confound a separate control vs treatment group.
4	3.1	"Our primary focus is on sleep duration"	Why not on measures of alertness? Focusing on sleep duration requires an extra intuitive leap to link the treatment to improved safety.	To better assess the relationship between a change in SB flexibility and safety, the primary focus has been revised to identifying safety outcomes, which will be collected from the OBMS. The analysis methods will be able to compare changes in safety outcome rates between the two periods, to see if a flexible sleep option increases, decreases, or maintains the safety outcome rate as compared to current HOS regulations. More detail has been included in Statement B to reflect this. Additional analyses will include variables also shown to be associated with alertness, sleep duration, etc. and will be used to better understand how the flexible sleep option affects driver performance.
4	3.1	"may also lead to equivalence, rather than advantageous change"	What does this mean?	This was not worded well; edited to read: "may also lead to equivalence, rather than an improvement in safety performance"
4	3.1	"double comparison (split sleep versus nighttime sleep and split sleep versus daytime sleep)."	This design ignores self-selection due to latent variables, correlation within individuals, and differences between individuals.	The study design has been adjusted to address this concern. Regression models will be used to analyze the data, including terms for driver differences. There will be self-selection in which rule they choose regarding flexible sleep versus consolidated sleep, which would be realistic if a regulatory change went into effect. The two-week baseline period will give enough data to have a comparison of current driver performance to performance when flexibility is introduced. Our goal is to better understand how having a flexible option may affect driver safety and we believe the new study design and analysis methods will best evaluate this given that some conditions cannot be fully controlled.
5	3.1	"Our Primary (Dual) Hypothesis Is:"	The more logical hypothesis, as described in Part A, would be that alertness|flexibility > alertness|no flexibility	This is an important and valid point. To assess how a flexible policy affects driver performance, the research team has revised the primary hypothesis statement to determine if there is statistical evidence of improved or equivalent safety when using the flexible sleeper berth option, which better aligns with FMCSA’s mission of reducing crashes and improving safety. Alertness and fatigue are contributing factors to safety outcomes, so alertness will be evaluated as an input to the overall safety outcomes.
6	3.1	"Figure 5. Fourth secondary (dual) hypotheses test."	The nulls all seem misspecified, given that the goal of the research as described in part A is to test whether a flexible policy performs significantly worse than the current policy.	The analysis methods have been modified to address this comment and re-align the study with FMCSA's mission of improving safety and reducing crashes and fatalities. We have limited the analysis to a primary hypothesis test, which aims to determine if there is statistical evidence that the flexible sleeper berth option would introduce safety risks for drivers and the general motoring public. The primary hypothesis is a one directional test, identifying whether flexible sleep performs worse than current regulations. Additionally, modeling will be used to see the effects on other significant variables, including PVT lapses (an indicator of possible fatigue), subjective sleepiness, roadside violations, and total sleep duration.
11	3.4	"let αi be a random effect over subjects with a mean = 0 and a variance of ω2"	This is key to the error in the research design. This is not a reasonable assumption.	The analysis methodology has been revised to address this comment.
14	3.5	"For two-sided testing within (and between) subjects, we find that the statistical power to be anticipated exceeds 99 percent."	This result raises some concerns about the assumptions being made in the power analysis. Does this assume random assignment to conditions? Where’s the covariance? How are repeated measures being treated?	Due to the revisions in the analysis methodology, the power analysis has been completely revised.
14	4	"4. DESCRIBE METHODS TO MAXIMIZE RESPONSE RATE AND TO DEAL WITH THE ISSUES OF NON-RESPONSE. "	See relevant comment on Part A.	This has been addressed in Part A to reflect comments received. Part B has been revised to match Part A as appropriate.

Additional Modifications Identified

Part	Page	Section	Original Text	Comment & Modification
A	1	1	"The aim of the Flexible Sleeper Berth Pilot Program is to demonstrate how HOS regulatory flexibility in conjunction with an optional fatigue management program (FMP) could be used to improve driver rest and alertness."	Removed "in conjunction with an optional fatigue management program (FMP)" as analysis in this aspect cannot be guaranteed since the FMP is optional.
A	2	2.1	"The research team will oversample and aim to enroll up to 240 CMV drivers, allowing for attrition, to attain the targeted distribution of drivers who participate for a minimum of two duty cycles."	Removed "who participate for a minimum of two duty cycles" as this is no longer applicable with the modeling strategy.
A	4	2.1.2	"Responses to the driver information form will be used for secondary data analyses and for enriching the public-use data set."	Added language to specify where the public-use data set will be located: "Responses to the driver information form will be used for secondary data analyses and for enriching the public-use data set to be provided through the FMCSA Data Repository and the FMCSA public website."
A	4	2.1.2	• Wrist actigraphy. • Smartphone applications. • Onboard monitoring systems (OBMSs). • Electronic logging devices (ELDs). • Weekly phone briefings.	Added an additional bullet for "Debriefing session." which was unintentionally omitted.
A	5	2.1.3.2	"Throughout their period of study enrollment, drivers will take the PVT-B three or four times daily, depending on provision use. On duty days in which drivers choose to be compliant with the current SB regulations, . . ."	Change to reflect baseline period: "Throughout their period of study enrollment, drivers will take the PVT-B three or four times daily, depending on provision use. During the two-week baseline period, and on duty days in which drivers choose to be compliant with the current SB regulations, . . ."
A	5	2.1.3.3	"Information collected will include a video during the event of the road in front of the truck and of the driver's face."	Updated to reflect additional details: "Information collected will include a 30-second epoch surrounding the trigger and will include video of the road in front of the truck and of the driver's face as well as audio data."
A	6	2.1.3.4	Electronic Logging Devices Section	Updated to reflect additional details, such as that this will be an app on a tablet, not a smartphone, which have been finalized since conclusion of the pre-test.
A	7	2.3	"Allowing split sleep will not increase or decrease available duty time, as the minimum rest requirement will remain unchanged. Instead, it will allow drivers the opportunity to sleep at times that best suit their needs."	The team decided this statement was misleading, as the 14-hours of on duty time may include hours of sleeper berth time increasing the actual window of time; therefore, it has been updated to read: "The minimum rest requirement will remain unchanged, but it will allow drivers the opportunity to sleep at times that best suit their needs."
A	8	5	Efforts to Minimize the Burden on Small Businesses	This section did not accurately reflect burden on small businesses and our effort to minimize those; it has been edited to reflect small business burden appropriately. Now reads: "Drivers from small carriers and owner-operators are two of the required samples. Recognizing that burden may impact these groups more than medium or lare carriers, steps have been taken to minimize the burden on these small business entities. The OBMS and engine diagnostic data are continuously collected, with no extra demand on the driver or carrier. The research team will work with drivers to determine when installation and de-installations are convenient for them to minimize work delays or loss of revenue. Carriers will only have to complete the application for participation, which is a minimal burden anticipated to take only one hour (maximum)."
Throughout			"the Commercial Driver's License Information System (CDLIS)"	This was an error in identifying the database; changed "Commercial Driver's License Information System (CDLIS)" to "Motor Carrier Management Information System (MCMIS)"
Throughout			Any text referring to "nighttime sleep duty periods" and/or "daytime sleep duty periods"	Changed to reflect "consolidated sleep" to remove the portion of analysis reflecting nighttime versus daytime sleep periods, which did not adequately reflect FMCSA's main goal of how current regulations (including consolidated daytime or nighttime sleep) would compare to the possible flexible sleep regulations

3