SUPPORTING JUSTIFICATION – PART B
Information and Communications Technology (ICT) Needs Assessment
OMB Control Number 2130-New
B. Collections of Information Employing Statistical Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.
This needs assessment is focused on the active train, yard, and engine (TY&E) craft employees segment of the Class I railroad industry because it is the largest portion of the industry and the target of most industry educational outreach and knowledge dissemination efforts.
It is not realistic to survey the entire Class I TY&E employee population for several reasons. The effort required to perform a census far surpasses current budget constraints. While convenience sampling would likely be more cost effective, that method would not allow us to infer results to the rest of the industry employees because we would not be sure the sample would be representative of the industry (Crano & Brewer, 2002; Isaac & Michael, 1997). It is more practical to obtain a random sample of a portion of the industry and generalize their responses to the rest of that portion of the railroad industry.
Because all TY&E employees are represented by one of two labor unions, we intend to target the International Association of Sheet Metal, Air, Rail and Transportation Workers-Transportation Division (SMART-TD) and the Brotherhood of Locomotive Engineers and Trainmen (BLET). These two unions represent all TY&E employees working on Class I railroads, and therefore, coverage of the industry will be provided from within their ranks. Both union organizations maintain a database with the names, mailing addresses, date of birth, and date joined the union for all its members. Both the BLET and SMART-TD (will) have agreed to generate a random sample of their members in accordance with the method described herein.
From our preliminary discussions with union leaders, we estimate there are 85,000 Class I railroad TY&E, non-retired, craft employees comprise the combination of freight and passenger rail, making the potential respondent universe 85,000 Class I TY&E craft employees in the United States. Most U.S. TY&E employees – 45,000 – are members of the SMART Union. The remainder – 40,000 – are members of the BLET. This estimate is used to define sample size and stratification across the unions. Small adjustments (+/- 2,000) will have minor effects on sample size needs. These two groups provide a comprehensive universe of U.S. TY&E Class I employees.
Describe the procedures for the collection of information including statistical methodology for stratification and sample selection.
Confidence in generalizations is set to the industry standard of 95% and a 50/50 distribution of proportions is assumed because it is the most conservative and this study is not focused on testing any particular population differences.
Sampling error is directly related to acceptable margin of error and for this study we are setting this at +/- 5%. This choice is based on the balanced consideration of effort and value. Reducing the margin of error may add to the precision of our understanding of this population, but in turn would significantly increase the number of responses necessary to generalize with some validity (e.g., at 3% the required sample would go from 382 to 1,054. With the selection of a 5% margin of error, if we find that 59% of our sample strongly agrees with a statement we can say with some confidence that the population’s response would be between 54% and 64%. Given the descriptive nature of this study, this margin of error is acceptable.
This formula is used to compute sample size (Dillman, 2007, p.206):
n = (N) (p) (1-p) . Equation (1)
(N – 1) (B/C)2 + (p) (1-p)
n = sample size needed
N = population size
p = proportion of population differences
B = acceptable amount of sampling error
C = Z statistic associated with the confidence level; 1.96 for 95% level
Using an estimated 85,000 labor railroaders with a 5% margin of error, the following computation (Equation 2) estimates a sample size of 382.
n = 85,000 * .5 * (1-.5) . Equation (2)
(85,000 – 1) (.05/1.96)2 + (.5) * (1-.5)
Response Rate:
We use existing research outside the railroad community to inform our response rate calculation. Dillman (2007) has found that by applying some incentive and outreach efforts surveys can obtain 20-30% response rates. Therefore, and with no evidence to support the assumption that the railroad industry employees will be more or less likely to participate, we will oversample with the assumption of a 25% response rate. Attaining approximately 382 respondents will allow our results to be generalized to the population with a 5% margin of error. Therefore, an initial sample size of 1,528 (1,528*.25=382) is recommended.
Of the 85,000 members across the two unions, the SMART-TD represents 52.9% of that population and the BLET represents 47.1%. Therefore, the stratum defined for each union is designed to represent those percentages; see Table 1. The total number of potential participants is 1,528.
Sampling stratification for TY&E workers in the BLET and SMART-TD unions
Union |
Sampling Frame |
% of Frame |
Stratum N |
Total |
SMART-TD |
45,000 |
52.9% |
.529 x 1,528 |
808 |
BLET |
40,000 |
47.1% |
.471% x 1,528 |
720 |
Total |
85,000 |
100% |
100% |
1,528 |
Describe methods to maximize response rates and to deal with issues of non-response.
An announcement letter will be sent out in advance of the survey. Our intent is that the letter will contain union leadership signatories. This announcement is intended to raise the awareness of the forthcoming survey and explain the importance of their participation (Dillman, 2007, 2011). We are recommending that the announcement letter include union leaders’ signatures to convey the importance of the coming invitation.
One week after the announcement letter is sent, the survey with a cover letter will be sent. Postcard reminders will be sent two weeks after that and again two weeks later, to all non-respondents reminding them of the survey and its importance. Two weeks later, six weeks after the initial mailing, a final mailing will go out to those who have not yet completed the survey. This final mailing will include another copy of the survey to encourage their completion.
The announcement letter will explain the procedure and describe the benefits of the study to encourage individuals to participate. This letter, as well as an article we will write for inclusion in various train newsletters prior to the study, are intended to familiarize members with the effort and to legitimize the study.
The survey materials will be sent via first class mail with a personally addressed and dated letter printed on high quality paper. The package will contain a cover letter, the questionnaire, and a first-class postage paid envelope for return of the questionnaire.
At 2-week intervals, two follow-up reminder postcards will be sent to all recipients who have not returned their questionnaire and who have not indicated that they do not wish to participate.
The questionnaire is 8.5 x 11 in., printed on white paper with no questions on the cover page. The cover page contains only the title “Information and Communications Technology Questionnaire,” the FRA form number, the OMB control number, and the participant’s ID number. The required OMB statement, including the assigned OMB Control number and the confidentiality assurance statement, is on the inside of the cover page of the questionnaire. The questionnaire instrument will be visually pleasing and easy to read.
If the response rate for any key item falls below 70 percent, we will conduct a nonresponse bias analysis using standard analysis procedures (i.e., OMB, 2006). If the overall response rate for the study falls below 80 percent, we will conduct a nonresponse bias study. From the union membership databases, it is possible to determine the age and length of union membership of each TY&E employee. The nonresponse bias study will involve comparing the age distribution of the survey non-respondents with the age distribution of the respondents. The mean age for each group will be compared. For several reasons, age is an important characteristic for assessing potential bias. First, research has shown that use of technology is more prevalent with younger users. In addition, age is highly correlated with years of work experience and seniority. If a Z test shows that the respondents and non-respondents have similar age distributions, then we will be confident the respondents reflect the larger TY&E population.
Describe any tests of procedures or methods to be undertaken.
Experts in survey development and the railroad industry have reviewed and provided input into the current instrument. The survey items have been reviewed by four TY&E labor union leaders. A pilot study will be undertaken to improve wording of items and create a more readable instrument for this population, and if any item emerges as redundant or exceptionally problematic, it will be dropped.
For the pilot study, a convenience sample of 20–30 TY&E railroaders, selected by union leaders, will be contacted and asked to complete the survey. If they agree, the introductory packet described herein will be sent to them. Railroaders will be encouraged to provide input on the survey items on a 1-on-1 basis by calling the Volpe point of contact. Concurrently, a stakeholder panel of key industry representatives will be convened to review the instrument. This input will be incorporated into the instrument.
As a result of the pilot study, no substantive changes will be made to any items, nor will any additional items be added to the instrument. Therefore, OMB’s approval of the instrument for the ICT study should remain applicable, and no re-review would be warranted.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
Data collection and analysis will be led by the Volpe National Transportation Systems Center.
Heidi D. Howarth, Ph.D. (617) 494-2522
Kim Davies-Schrils, Ph.D. (617) 494-2583
With consultation on statistical aspects and needs assessment survey design from:
Michael A. Harnar, Ph.D., iBiz/CORA (909) 524-7800
Juna Z. Snow, Ph.D., iBiz/CORA (617) 440-4435
References
Crano, W. D., & Brewer, M. B. (2002). Principles and Methods of Social Research (2nd ed.). Mahwah, New Jersey: Lawrence Earlbaum Associates
Dillman, D. A. (2007). Mail and Internet Surveys: The tailored design method (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Dillman, D. A. (2011). Reconsidering Mail Survey Methods in an Internet World. American Statistical Association/American Association for Public Opinion Research Webinar, April 13, 2011. Downloaded April 6, 2014 from http://www.umb.edu/editor_uploads/images/u54/ASAAAPORwebinar4_11_11.pdf
Isaac, S., & Michael, W. B. (1995). Handbook in Research and Evaluation: For Education and the Behavioral Sciences (3rd ed.). San Diego, CA: Educational and Industrial Testing Services.
Office of Management and Budget (2006). Standards and guidelines for statistical surveys, September 2006.
1/6/2017, Original;
11/22/2017, 8/26/2018, Updated Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |