INFORMATION COLLECTION
SUPPORTING STATEMENT
Evaluation of Correct Child Restraint System Installations
B. Statistical Methods
NHTSA is seeking approval to conduct a quasi-experimental study using an incomplete factorial experimental design (2 x 4 x 4) to gather data regarding the proper installation of Child Restraint Systems (CRS). For this study, 150 participants (75 experienced CRS users and 75 novice users) will be recruited to participate. In addition to the one between subject independent variable (CRS experience),this study design includes two within subject independent variables: the child-size dummy the participant secures in the CRS (16-month old, 3-year-old, 6-year old, and 8 year-old) and vehicle type (sedan, sports utility vehicle (SUV), minivan, and pickup truck ). The participant will select the CRS type (rear facing only (previously known as infant), convertible, combination, and belt-positioning booster seats). Each participant will perform four installations. The order of installations (i.e., trials) for a given participant will be randomized with respect to the child dummy and vehicle types in such a way that control is achieved with respect to the number of observations per experimental factor dimension and level. The result will be 600 installations or trials, distributed evenly across the levels of each within subject independent variable, as follows:
Vehicle type: 150 observations per each of four vehicle types
Child doll’s age/weight/height: 150 observations per each of four child size-dummy types
This is an incomplete factorial design because all participants will not be run through every possible experimental condition; rather, the conditions will be assigned to participants and trials at random to achieve a balance across the 150 participants and 600 trials by randomizing the order of the list of experimental settings for the 150 participants in advance. In addition, the order of installations for a given participant will be randomized with respect to vehicle and child dummy type to preclude any effects of sequence and control for any learning or fatigue that might take place.
This is a quasi-experimental study because participants cannot be randomly assigned to the CRS experience condition; placement into the experienced or novice group is based upon a preexisting participant characteristic. However, for the within subject variables, participants will be randomly assigned to the various experimental conditions to control for the influence of extraneous variables on how the participants perform under the experimental conditions.
To gather information on participant performance and response associated with the various experimental conditions, the following dependent measures will be collected:
pre-install self-administered questionnaires, including:
a risk appraisal assessment tool specific motor vehicle crash and injury risks (including inaccurate CRS installations)
a measure of invincibility beliefs (invincibility beliefs index)
a demographics questionnaire
four CRS installation observations with an observer checklist to record performance
post-install questionnaire via interview using a series of ratings and open-ended questions to learn about:
description of how the participant used the CRS
problems that were encountered
errors identified
CRS type acceptability
User confidence in installation accuracy
Data Analysis Plan
Once the study data have been collected and reviewed, the questionnaire and observation data will be used to conduct logistic or multi-level models to identify experimental factors significantly influencing the probability of a successful choice of Child Restraint System (CRS), installation in vehicle, and securing of the child to vehicle. Independent variables will include all three experimental factors (experience of user, child dummy type being secured in the CRS, and vehicle type) and possible interactions. The analysis will relate CRS feature design, CRS manual features, vehicle features and manual design, and risk perception to measures of performance, ease of installation, presence of errors, and confidence related to the installation. Examples of basic measures of analysis for novice and experienced users include 1) measures of exposure to various child passenger safety seat information resources and how that relates to accurate and secure installations, as well as confidence ratings, and 2) measures of how risk appraisal and ease of use ratings relate to the participant’s confidence in their accurately and securely installing the CRS.
The potential respondent universe is comprised of all licensed drivers, with and without Child Restraint System (CRS) experience, in the greater Washington, DC area. From this universe, the new data collection will qualify 150 drivers by administering a recruitment eligibility screener to select 75 participants with CRS experience and an additional 75 “novice” CRS users. Experienced users are individuals who regularly care for a child under the age of 4 years, transport the child in a vehicle at least twice a week, have secured the child in a CRS a minimum of five times in the past 6 months, and have installed any type of CRS at least once in the past 12 months. Novice CRS users are individuals who do not regularly transport children and have not installed a CRS in the past 6 months.
NHTSA
expects a 50% response rate, with 150 of 300 screened potential
participants to be eligible for participation. The estimated
response rate is due to the proposed methods for increasing
participation and the special recruitment efforts planned to reach
potential respondents with CRS experience.
The sample size
of 150 was selected for 1) reasonable precision to detect change and
2) availability of resources to complete the experiment. More
specifically, we designed the study to detect a difference of 10% or
larger between the novice (i.e., 70%) and experienced (i.e., 80%)
child restraint system (CRS) user group installation success rates,
and a sample of 300 observations per group does this with a power of
78%. While we realize that this level of power is slightly below the
conventional 80% cutoff, we believe that this level of precision is
adequate to achieve the objectives of the project. Other planned
comparisons include within group comparisons for vehicle and child
dummy types such as successful installation rates for 6 year old
child dolls (i.e., 80%) compared to 8 year old child dolls (i.e.,
60%) with 150 observations each, which yields a power of 96% for
detecting a difference of 20% or larger.
Table 1. Precision for estimates of differences
Scenario |
P1 |
Q1 |
n1 |
P2 |
Q2 |
n2 |
D |
STE(D) |
CI+- |
Power |
Novice (P1) v/s experienced (P2) |
0.7 |
0.3 |
300 |
0.8 |
0.2 |
300 |
0.1 |
0.035119 |
0.068833 |
78% |
Child size - 6 year old (P1) vs 8 year old (P2) |
0.8 |
0.2 |
150 |
0.6 |
0.4 |
150 |
-0 |
0.05164 |
0.101214 |
96% |
Tables 2 - 4 present the precision offered by the total observations across the sample (600), the observations for comparing within groups of vehicle and child dummy types (300), and the total participant sample (150). The precision is lowest at 50%. The relative standard error (RSE) at 50% for the total sample size of 600 observations is 4.08% (Table 2), while the RSE for the group comparison is 5.77% (Table 3). The RSEs are in the vicinity of 5%, and below 10%, which are equal to or lower than RSEs sometimes used as basic sample design criteria.
Table 2. Precision offered by 600 observations for P ranging between 30-70 percent
P |
Q |
N |
Var |
Ste |
RSE |
LCI |
UCI |
30.00% |
70.00% |
600 |
0.00035 |
0.01871 |
6.24% |
26.33% |
33.67% |
40.00% |
60.00% |
600 |
0.00040 |
0.02000 |
5.00% |
36.08% |
43.92% |
50.00% |
50.00% |
600 |
0.00042 |
0.02041 |
4.08% |
46.00% |
54.00% |
60.00% |
40.00% |
600 |
0.00040 |
0.02000 |
3.33% |
56.08% |
63.92% |
70.00% |
30.00% |
600 |
0.00035 |
0.01871 |
2.67% |
66.33% |
73.67% |
Table 3. Precision offered by 300 observations for P ranging between 30-70 percent
P |
Q |
N |
Var |
Ste |
RSE |
LCI |
UCI |
30.00% |
70.00% |
300 |
0.00070 |
0.02646 |
8.82% |
24.81% |
35.19% |
40.00% |
60.00% |
300 |
0.00080 |
0.02828 |
7.07% |
34.46% |
45.54% |
50.00% |
50.00% |
300 |
0.00083 |
0.02887 |
5.77% |
44.34% |
55.66% |
60.00% |
40.00% |
300 |
0.00080 |
0.02828 |
4.71% |
54.46% |
65.54% |
70.00% |
30.00% |
300 |
0.00070 |
0.02646 |
3.78% |
64.81% |
75.19% |
Table 4. Precision offered by 150 observations for P ranging between 30-70 percent
P |
Q |
N |
Var |
Ste |
RSE |
LCI |
UCI |
30.00% |
70.00% |
150 |
0.00140 |
0.03742 |
12.47% |
22.67% |
37.33% |
40.00% |
60.00% |
150 |
0.00160 |
0.04000 |
10.00% |
32.16% |
47.84% |
50.00% |
50.00% |
150 |
0.00167 |
0.04082 |
8.16% |
42.00% |
58.00% |
60.00% |
40.00% |
150 |
0.00160 |
0.04000 |
6.67% |
52.16% |
67.84% |
70.00% |
30.00% |
150 |
0.00140 |
0.03742 |
5.35% |
62.67% |
77.33% |
Once potential respondents are identified, they will be administered a recruitment screening questionnaire to determine eligibility for participation in the study (Appendix E). The screening questionnaire collects information on age, sex, race, experience with child restraint seats, and seat belt use. Potential participants will be specifically asked if they have any experience installing a CRS, as well as how often they install them, asked to provide an indication of the last time they installed a CRS, which CRS types (make and model) they use, and if they ever participated in a seat check. The screener questionnaire will also ask potential participants if they are able to lift and carry 25 lbs and move in and out of a vehicle, so we can verify all scheduled participants can perform the task at hand. We also will ask if they have any medical conditions affecting dexterity or motor skills, as these may contribute to installation errors, but participants will not be excluded for this reason.
Each session will begin with the experimenter providing a brief introduction to the study, after which each participant will be required to sign a consent form. Prior to installing the CRSs, participants will complete a questionnaire assessing their risk appraisal specific to motor vehicle crash and injury risks (including inaccurate CRS installations). The questionnaire will be completed prior to installing the CRSs to minimize the influence of installing a child safety seat in several vehicles on some of their responses regarding risk appraisal of crash and injury (See Appendix F).
Once the survey has been completed, each participant will be asked to install a CRS for each of the four child-size dummies (16-month-old, 3-year-old, 6-year-old, and 8-year-old) into one of the four vehicle types(sedan, SUV, minivan, truck). Vehicle and CRS make and model will be assigned to each participant and trial in such a way that the assignment to participants and trials is random, yet achieves a balance across the 150 participants and 600 trials with respect to these dimensions and factors.
Participants will be provided with the age, height, and weight of the child, likely on a card hanging around the child size doll’s neck and also by verbal instructions from the researcher. Participants will be asked to select the appropriate CRS to install for the child’s age, height and weight. Participants will be provided with the owner’s manual of the vehicle as well as the instruction manual for the CRS and informed that they are available for their use. No verbal instructions on how to use the CRS features or vehicle features will be provided, unless the participant is simply unable to install the CRS within some criterion time (to be determined during pilot testing). Participants will then be asked to complete each installation by securing the CRS to the vehicle and securing the doll in the CRS.
The order of installations for a given participant will be randomized with respect to vehicle type and child dummy type to preclude any effects of sequence and control for any learning or fatigue that might take place. After each installation, various types of objective and subjective measures will be collected via an interviewer-administered questionnaire (See Appendix G). Together, these measures will describe how the participant used the CRS, what problems were encountered, errors identified, how acceptable the system was to the user, and the degree of confidence each participant exhibited with correctly installing the CRS to the vehicle and securing the child in the CRS. Participants will convey this information by responding to a series of ratings and open-ended questions regarding the ease of installation and challenges related to usability of the CRS, it’s manual, and the vehicle features and manual.
Following the installation and while the participant is being administered the post-install questionnaire, another researcher will examine the CRS in the vehicle for proper installation and record any installation errors observed. The researcher will look for specific types of installation errors with respect to each type of CRS, vehicle type, and child size. The data gathered will serve two purposes. First, NHTSA will be able to determine how successfully each participant was able to use the instruction manuals and various design features to correctly install the CRS. In addition, NHTSA will use the data to objectively evaluate different installation errors that relate to the physical design features on the CRS, CRS manual/vehicle manual, and vehicle features.
All data will be treated with sensitivity and security considerations commensurate with its level of confidential content. Security measures will include, but not be limited to:
Carefully controlled access to the Contractor’s password protected server location associated with the participant data via project director and system administrator authorization.
Identification of participants by a unique subject numbering convention that will be kept separate from the data files and used only for study administration personnel requirements.
Provision of password access only to key staff as needed.
Payment of the honorarium shall take place after the participant completes all four installations.
Participation in this study is voluntary. Several methods will be utilized to maximize response rates, including:
Completing recruiting and screening by telephone for potential participant convenience
Mailing a confirmation letter within 48 hours of the initial call, including an agenda, schedule, location of the data collection session, directions to the site, and a contact telephone number in case of cancellation or questions
Conducting reminder calls within 24 hours of the scheduled session to boost show rate and help alleviate schedule misunderstandings or forgetfulness.
Providing participants with a $75 incentive after completion of the study session to both encourage participation and session completion.
As part of the study design, the Contractor will refine the study procedures by pilot testing the protocol and instructions for all aspects of the study with 9 participants. The pilot study will allow the Contractor to have a more accurate assessment of the overall time it will take to complete the session as well as each task within the session. While the current estimate for completing the study session is 2 hours, it is possible that the pilot study results may indicate that up to 2 ½ hours are needed to complete the session. Therefore, NHTSA has used the longest session time possible (i.e., 2 ½ hours) to calculate the overall burden for the proposed information collection.
The pilot study will also allow the Contractor to assess the overall comprehension of the instructions and individual questionnaire items. Testing the instructions and questionnaires prior to implementing the study will provide the Contractors with the opportunity to make slight wording changes when needed that will improve overall comprehension without changing the intent of the direction or questions.
While NHTSA must account for this possibility, NHTSA foresees minimal changes to result from the pilot testing because it has consulted in-house with experts in the area of CRS, with the Contractor for the information collection, and with Ms. Emilie Crown, an expert in the area of CRS, on the proposed information collection effort. As a result, this study has been designed to complement current knowledge of proper CRS installation and child fit and to fill in key gaps in this knowledge base.
Mary T. Byrd, MA
Social Science Researcher
National Highway Traffic Safety Administration, NTI-132
1200 New Jersey Ave., S.E., West Bldg. 46-466
Washington, D.C. 20590
Phone: 202-366- 5595
Email: [email protected]
Doreen De Leonardis, Ph.D.
Senior Research Scientist
Westat
1600 Research Boulevard
Rockville, MD. 20850
Phone: 301-315-5963
Email: [email protected]
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | amanda.kelley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |