Part B FRA WFD Survey

SUPPORTING JUSTIFICATION – Part B

Workforce Development Survey; OMB No. 2130-NEW

Form Number FRA F 6180.158

1. Description of sampling method to be used.

This study is specifically focused on deriving information related to workforce issues in the Railroad industry. The sampling (analysis) unit are the organizations involved in railroad industry workforce development, the railroad workforce and organizations impacting the railroad workforce. The observation units are officials in these organizations that have knowledge of the role that their organizations play in determining and impacting workforce issues. In view of the very qualitative nature and the specific requirement of the study, and the involvement of groups of different organizations, mixed sampling percepts are adopted in determining the sampling approach.

We adopt an expert sampling methodology. Expert sampling - a type of purposive sampling technique was chosen since the study needs knowledge from individuals that have particular expertise to respond to survey items. We developed stratified sampling approach to determine the subgroups within the target population.

The total population of the railroad industry is extensive, reaching across many classes of rail operations and including professional organizations and academia. Class I rail organizations employ thousands although the number of organizations is very small. In contrast Short Line and Regional rail is comprised of many organizations employ a small number of people. Also included in the survey strata are labor unions, associations and academia.

From the thousands of personnel associated with the railroad industry this survey will focus on a study population of those organizations that exercise or influence workforce development activities. Specifically, these study populations include organizations from:

• Class I Passenger Railroads

  • Large passenger railroads with annual carrier operating revenues of $433.2 million in 2011 dollars

• Class I Freight Railroads

  • Large freight railroads with annual carrier operating revenues of $433.2 million in 2011 dollars

• Short Line and Regional Railroads

  • Regional (Class II) with annual operating revenues greater than $37.4 million in 2011 dollars

  • Short-Line (Class III) with annual operating revenues less than $20 million in 1991 dollars

• Academia

  • Learning institutions with dedicated curriculum and training programs for the railroad industry

• Labor Unions

  • Union organizations with a specific focus on railroad workforce membership and railroad programs

• Professional Associations

  • Associations with a specific focus on railroad workforce membership and the railroad industry

These six groups of organizations define the strata for this study. From this strata, a sample of the organizations will be purposively selected to represent the railroad community. The survey population of interest was identified through working with management representatives and research to pinpoint organizations with workforce development expertise and experience within their organization. These organizations have been identified for their experience and expertise to provide the qualitative data needed. Short-line corporations (Class II and Class III) were included in the population under study.

The targeted organizations have the necessary experience and exposure to workforce development activities to provide valuable information regarding the efficiency and effectiveness of ongoing activities and provide input on opportunities and threats that should be addressed. FRA, therefore, believes this is the best sample from which to draw this qualitative data. We believe this group to be very knowledgeable about the railroad workforce issues that are under study and they will be responsive and cooperative.

The scope for this is survey includes capturing qualitative data to understand the current status of workforce development activities across defined parameters. These parameters or survey items were developed by FRA to encompass the wide range of processes that comprise the workforce development lifecycle. The below list represents the Survey Items for this survey.

• Workforce Population

• Aging Workforce

• Workforce Diversity

• Workforce Development Data

• Industry Image Considerations

• Training in the Industry

• Work/life Balance

• Technology Adoption

Using the study population as a starting point the survey team was able to develop the sampling frame using the parameters to define which organizations to include. The primary rule of inclusion used to determine the survey frame was organizations that were expected to have knowledge and experience with the survey items. Understanding that the survey groups vary significantly in size and total organizations represented, the survey will be fielded using stratification techniques to ensure each group is fairly represented. The sampling frame consists of an extensive list of organizations and subject matter experts from across the industry.

FRA intends to collect this data every two years so the data will reflect the current state of the rail industry as it relates to its workforce. The qualitative data FRA will collect via this survey is not intended to draw inferences to matters outside the design of the survey items. In this sense we are adopting a design based inference approach. The FRA has chosen an electronic survey instrument for data collection. A survey instrument will allow for the collection of consistent data across years and the data the FRA wishes to collect can best be collected with the use of the electronic survey instrument.

The FRA has carefully considered the benefits and disadvantages associated with an electronic data collection methodology. A web-based survey methodology may exclude individuals that may be less comfortable with technology. However, given the nature of work the preponderance of the survey population performs, technology challenges in taking the survey are not anticipated. For this data collection effort, FRA made several assumptions regarding sampling size:

• The sample population consists of organizations involved in workforce development activities from across the railroad industry, associations, unions and academia. Duties regularly require work with personnel to support the full workforce development lifecycle, ranging from recruiting and training to retention and succession planning.

• The sample population adequately represents all the strata comprising the population as determined by the expert opinion of the survey team.

• Data from the sample will be used for qualitative purposes only. No quantitative analyses will be conducted on sample data.

The survey does not include questions regarding age, years in craft or primary work region. The survey questions are designed for and targeted to a very specific population with workforce development expertise, in order to increase the quality of the responses. The demographics of the respondents are not considered relevant to the quality of the responses or the representation of the workforce development community.

The survey population includes representatives from railroad organizations, academia, labor unions and railroad associations. This broad spectrum approach is intended to ensure a comprehensive representation from all facets and interests in railroad workforce. Understanding that these entities have differing policies, goals and cultures, the survey questions are slightly tailored to align to their perspective.

Although any given group to be surveyed may represent a small percentage of the total population of railroad entities, survey sample sizes are consistently applied across all survey groups to present a fair representation of all groups.

It is anticipated that approximately 50% of those invited to participate in the survey will respond. This number is based on interaction with the stakeholder community and FRA’s previous experience in soliciting data from these groups. The calculated sample was adjusted to reflect the invitations needed to obtain the desired sample size given a 50% response rate.

If after the first year of this survey data collection effort it is determined that actual response rates are much higher or lower than anticipated, FRA will adjust its assumed response rates and sample sizes accordingly.

To address non-responsive groups, if any specific strata achieve less than a 30% response, the survey team is prepared to send the survey to additional organizations to increase the survey response rate for that survey group. This will ensure all organization groups are fairly represented in the response pool.

2. Description of procedures for information collection, including statistical methodology for stratification and sample selection.

Survey data will be collected once every two years. FRA has considered the use of less frequent data collection periods and determined data collection should occur every two years to support the publication of results to industry and to establish trends as appropriate. If proposed collection of information were not conducted or conducted less frequently, the ability for FRA to assess the status of the industry workforce requirements and challenges would be diminished.

Trustworthy qualitative data collected over a period of years is needed to ascertain the extent of the workforce problems and to inform the design and development of potentially effective programs, and to realign activities as needed.

The survey design limits the scope of the data collected to specific survey items and therefore limits the ability to draw inference beyond the scope of the workforce related topics being studied. This approach best fulfills the intent of the survey by providing specific information about targeted workforce related issues. However, in the future, the approach may be changed if a wider set of information is needed or the FRA desires to extrapolate inferences to other workforce issues not currently covered in the survey.

Using a purposive sampling technique (nonprobability approach), the survey team has developed the sample survey respondent organizations using the sample frame, based on their experience and influence within the railroad industry and the workforce development domain. The judgment of the research team was required to identify organizations that have subject matter expertise relative to the survey items to be studied.

The survey frame consists of the organizations from across the industry that the survey team deemed most likely to provide a valuable response. The frame consists of 318 organizations across the selected strata. These strata represent the railroad industries’ organizations with workforce management and/or development expertise. These are organizations that provide for and support workforce development activities for their respective agencies and organizations. Academia representatives provide the training and partnership with industry to identify skills gaps and to develop and deliver the education needed to support the railroad industry. Railroad associations focus on specific railroad issues and concerns while capturing and highlighting workforce related needs and trends. FRA, therefore, believes these strata are the best population from which to draw a sample for this data collection effort.

Of the 318 organizations shown in the table below a total of 91 invitations will be sent. Based on the size and span of the Class 1 organizations we will attempt to survey 100% of the population. Although the number of Short Line and Regional organizations is large, each organization is much smaller. Therefore, we believe a 25% survey will adequately represent the strata. The remaining survey strata (Labor Unions, Associations and Academia) will be surveyed at 50% to provide the necessary representation of that segment of the industry population. Based on previous experiences we believe that a 50% response rate will be achieved and therefore we conservatively estimate a minimum response from 45 organizations for this survey.

Survey Strata of Industry Population	Frame #’s per Strata	Derived Survey Invitations per Strata	Estimated Survey Responses per Strata
Class I Passenger	5	5	3
Class I Freight	10	10	5
Short Line & Regional	250	50	25
Labor Unions	15	7	3
Associations	20	10	5
Academia	18	9	4
Total	318	91	45

3. Description of methods to maximize response rate and to deal with non-response issues.

This qualitative survey data collection effort will be carried out in collaboration with the leadership of the relevant stakeholder groups. A standard introductory email will be created by FRA. This email will alert the survey sample that they will receive an invitation to participate in the survey. All potential participants will receive the same information.

The survey invitations have been designed to enhance response rates. Specifically, introductory emails sent to respondents will come from official email addresses. The introductory emails will explicitly state the purpose of the study and the estimated time to complete the survey. The emails will also stress that no special preparation is needed to participate. The survey has also been extensively reviewed to insure that the questions use simple and straightforward language. Estimated time to complete the 30 question survey is 20 minutes.

Participation will be encouraged by FRA leadership, indicating that the importance of the survey data in structuring workforce strategies and identifying future workforce development activities. By completing the survey, they will be able to influence how these interventions are designed and implemented. The opportunity to provide their input should provide adequate motivation for participation.

Prior to the issuing of survey invitations, a pre-notification of the survey will be released. This pre-notification will be available on the FRA’s website. This pre-notification will explain the purpose of the survey including: when data will be collected, how data will be collected, the type of information to be collected, and why this information is being requested. Pre-notification and survey invitations will contain contact information for FRA personnel who can be contacted for additional information about the survey.

The survey will be a web-based (Survey Monkey) data collection effort. Respondents will be invited to respond via an e-mail containing a hyperlink to an online survey instrument. FRA is asking for response by survey respondents within a few days of receipt of the information as e-mails tend to get lost and respondents may forget about the survey invitation. To alleviate forgetfulness, reminder notices will be sent, one week, two weeks, three weeks, one month, and 45 days after the initial invitation is sent. In an effort to increase response rates for this survey data collection effort, reminder email subjects will be modified. Respondents will be given the flexibility to respond at any time up to 60 days after the initial survey invitation is sent. These reminders should increase data collection for those groups who didn’t initially respond, increasing the number of respondents.

If after 45 days FRA finds a low response rate, time to respond will be extended out to 90 days and FRA will encourage those who have received an invitation to complete the survey within the new extended timeframe.

Individuals may be more comfortable completing a paper version of the survey. The email invitation to participate will also include information regarding how to request and complete a paper survey for those who are uncomfortable or unwilling to complete the survey online. A printable electronic copy of the survey will be provided. Those who receive an email invitation to complete the survey but would prefer to do so via paper will receive instructions on how to obtain a paper copy and where to send it upon completion.

Each question in the survey will also have a “prefer not to answer” option for those questions where the decision by the respondent not to respond was intentional. Analysis of non-responses will not be pursued.

4. Describe any test procedures for procedures or methods to be undertaken.

Pre-testing of the survey questions and survey tool functionality was conducted prior to ensure questions are clear and that the tool is designed to present questions as required.

The pre-testing process proceeded as follows:

• Key personnel at the FRA were consulted to elicit their input on what data collection is critical for the survey.

• Analysis of the most recent Modal Profile was conducted to inform and validate data collection goals.

• Survey questions were developed and modified to improve clarity and align to survey items and objectives.

• Knowledge derived from the above was augmented by workforce development experts to validate and confirm approach and target goals.

• The survey was developed and revised by a team consisting of contractor and FRA personnel.

• A draft final survey will be created in the survey tool (Survey Monkey) and tested by FRA personnel to insure proper functionality.

After the initial survey administration, FRA plans to review the survey with relevant stakeholders prior to subsequent deployment. FRA will discuss any changes to survey questions, sampling, or administration strategy as required with OMB and resubmit the survey information to OMB for approval as needed.

5. Provide name and phone number of individuals consulted on statistical aspects of study design and other persons who will collect/analyze information for agency.

Ramesh Rajagolapan, Ph.D. (Consultant)

Dennis Lasley, M.Ed., Lean Six Sigma Black Belt (Consultant)

Tosan Tutse-Tonwe (Consultant)

Eric Saa (Consultant)

Monique Stewart, FRA (Executive Champion)

 

7