2130-NEW-RSI-001 WFD_SJ_Part B

Department of Transportation

Federal Railroad Administration

INFORMATION COLLECTION SUPPORTING STATEMENT B

Reviewing the Reviewers: How Restructuring Performance

Management Systems Can Increase Diversity in Rail.

OMB CONTROL NUMBER 2130-NEW

1. Description of sampling method to be used.

A random sample of rail stakeholders cannot be selected because no comprehensive list of rail stakeholders including those in executive and labor positions exists. Potential survey respondents will be identified from a pool of rail stakeholders using a snowball sampling methodology. KEA Technologies (KEA)’s project team received letters of support for this project from several individuals and organizations who work at various job levels at rail carriers, professional associations, and/or academic/research settings. The individuals who we received letters from should garner responses from the groups we are particularly interested in, which includes folks at different job levels, those who work for rail carriers directly versus those who do not, and those who work in adjacent positions to rail, such as researchers.

Through a snowball sampling method, those individuals and/or organizations will distribute the survey link to individuals in their networks. Snowball sampling has been proven to be an effective and efficient way to yield participation in studies and reach populations that are otherwise difficult to engage with (Knight, Hauschildt, Buchanan, Greene, & Clark, 2021; Leighton, Kardong-Edgren, Schneidereith, & Foisy-Doll, 2021; Naderifar, Goli, & Ghaljaie, 2017). We expect to include a note on which groups provided responses and which groups were either not reached or elected not to respond.

2. Description of procedures for information collection, including statistical methodology for stratification and sample selection.

KEA will contact the rail stakeholders who provided letters of support for this project to forward the web-based survey to potential respondents in their networks. KEA recently conducted a technical survey to evaluate symbology developed for a Head-Up Display sponsored by FRA using a similar sampling methodology and received 28 responses. We aim to contact 150 people and to collect at least 30 responses on the survey for the purposes of this project. We also aim to contact up to 30 individuals to conduct at least three interviews and at least one focus group. A focus group could have as many as 4 people included. Sim, Saunders, Waterfield, & Kingstone (2018) conducted a literature review examining appropriate sample size for conducting qualitative research and broadly found that between two and sixty participants is sufficient to uncover themes within a population and to reach data saturation. Reaching data saturation means that there is enough data that new themes are not being uncovered, the study could be replicated, and when further coding is no longer appropriate (Fusch & Ness, 2015). Whereas, researchers have found that small sample sizes are adequate for qualitative research in homogenous populations, others have found that for research using the grounded theory approach (meant to uncover themes in people’s interactions and experiences), samples sizes of 20-30 are typical (Boddy, 2016). Therefore, for qualitative research a sample size that is larger is not necessarily better and does not mean researchers will reach data saturation; instead, it is about the depth and quality of the data (Burmeister & Aitken, 2012). Both the survey and the interview/focus groups will be used to identify qualitative themes rather than proving effect size. Additionally, while our recruiting pool is very broad and we aim to reach as many groups as possible, part of this research effort will be understanding who chose to respond and who did not, and that will be noted in the report. The project team expects to receive limited responses from certain populations due to the nature of the demographic composition of the rail industry. For example, if only a small number of women respond to the survey, their data and the associated themes may be considered more heavily due to the limited number of women in the rail population at large. Accessibility to persons in a research population also must be considered (MOCĂNAȘU & Galati, 2020).

For the survey analysis we will use R, a language and environment for statistical computing, to do an analysis of responses stratified by different demographics collected such as job level, job type, gender, and racial/or ethnic background. We will create summary statistics in table and graphical form. Simple regression analysis may be attempted if we are able to stratify successfully by job level i.e., those in management positions versus those who are not or those who are employed by rail carriers versus those employed by rail associations. Classification will be attempted depending on the demographics of the whole pool of respondents. If there is potential risk of identifying a participant, we will aggregate without classification. Due to the nature of this research being mostly qualitative and focused on uncovering themes, we do not intend to weigh the data as we are not as concerned with effect size. The grounded theory approach may be utilized in coding themes from the interview and focus groups.If there are certain groups of people who are overly represented in the data versus others who are not, that will be reported as notable results. Further research or follow-on work may be needed to try to reach groups who were not reached during the initial outreach.

3. Description of methods to maximize response rate and to deal with non-response issues.

For this project, we have contacts who have provided letters of support for this project and have agreed to disseminate emails with the survey link to their professional networks. To deal with potential non-response issues, we will follow-up with our contacts after one week and then again after one month if we still have not received enough responses. We will follow-up as needed after that initial survey release period. Due to the nature of this research, if there are a significant number or specific group of people (e.g., women or other minority individuals) with low response rates on the survey, we will disclose that in the report because that is a notable finding.

4. Describe any test procedures for procedures or methods to be undertaken.

The survey, interview, and focus group questions have been reviewed by rail industry stakeholders and subject-matter experts to identify appropriateness and language for the intended target audience. Additionally, KEA pilot tested the survey with five different users and confirmed that it took approximately twenty minutes to complete the survey. Qualitative methodologies will be used to conduct semi-structured interviews and/or focus groups and will potentially use the grounded theory approach for analysis. KEA does not foresee any additional tests of procedures or methods to be undertaken in this study. The data collection and analysis methods and surveys that KEA plans to employ for this effort have already been used in similar studies that the project team has conducted for transit agencies and state departments of transportation. As mentioned above, KEA recently administered a technical survey to evaluate symbology developed for a Head-Up Display sponsored by the FRA using a similar sampling methodology.

5. Provide name and phone number of individuals consulted on statistical aspects of study design and other persons who will collect/analyze information for agency.

None consulted beyond original research proposer. The proposer for this research, KEA Technologies, Inc. will do all collection and analysis of data.

Kelly Ozdemir, Ph.D. 508-658-9425. KEA (Project manager)

Kianna Pirooz, MPH. 508-658-9710. KEA (Collect and Analyze)

Tim Allen, MS. 508-658-3679. KEA (Collect and Analyze)

Maura Campbell, BS. 508-658-9714. KEA (Collect and Analyze)

FRA point of contact for the study:

Shala Blue

[email protected]

US Department of Transportation

Federal Railroad Administration

Human Factors Division (RPD-34)

Washington, DC 20594

References:

Boddy, C. R. (2016). Sample size for qualitative research. Qualitative Market Research, 19(4), 426–432. https://doi.org/10.1108/QMR-06-2016-0053/FULL/XML

Burmeister, E., & Aitken, L. M. (2012). Sample size: How many is enough? Australian Critical Care, 25(4), 271–274. https://doi.org/10.1016/j.aucc.2012.07.002

Fusch, P. L., & Ness, L. R. (2015). Are We There Yet? Data Saturation in Qualitative Research, 455. Retrieved from https://scholarworks.waldenu.edu/facpubs/455

Knight, T., Hauschildt, S., Buchanan, B., Greene, A., & Clark, M. D. (2021). It Requires a Community to Raise a Deaf Adult: A Comparative Study. Open Journal of Social Sciences, 09(03), 77–95. https://doi.org/10.4236/jss.2021.93006

Leighton, K., Kardong-Edgren, S., Schneidereith, T., & Foisy-Doll, C. (2021). Using Social Media and Snowball Sampling as an Alternative Recruitment Strategy for Research. Clinical Simulation in Nursing, 55, 37–42. https://doi.org/10.1016/J.ECNS.2021.03.006

MOCĂNAȘU, D. R., & Galati, R. D. de J. U. of. (2020). DETERMINING THE SAMPLE SIZE IN QUALITATIVE RESEARCH. In International Multidisciplinary Scientific Conference on the Dialogue between Sciences & Arts, Religion & Education (Vol. 4, pp. 181–187). IFIASA (Ideas Forum International Academic and Scientific Association). https://doi.org/10.26520/mcdsare.2020.4.181-187

Naderifar, M., Goli, H., & Ghaljaie, F. (2017). Snowball Sampling: A Purposeful Method of Sampling in Qualitative Research, 14(3), 67670. https://doi.org/10.5812/sdme.67670



Page 1