drfC3RSB

CONFIDENTIAL CLOSE CALL REPORTING SYSTEM

SUPPORTING JUSTIFICATION - PART B

1. DESCRIBE (INCLUDING A NUMERICAL ESTIMATE) THE POTENTIAL RESPONDENT UNIVERSE AND ANY SAMPLING OR OTHER RESPONDENT SELECTION METHOD TO BE USED. DATA ON THE NUMBER OF ENTITIES (E.G., ESTABLISHMENTS, STATE AND LOCAL GOVERNMENT UNITS, HOUSEHOLDS, OR PERSONS) IN THE UNIVERSE COVERED BY THE COLLECTION AND CORRESPONDING SAMPLE ARE TO BE PROVIDED IN TABULAR FORM FOR THE UNIVERSE AS A WHOLE AND FOR EACH OF THE STRATA IN THE PROPOSED SAMPLE. INDICATE EXPECTED RESPONSE RATES FOR THE COLLECTION AS A WHOLE. IF THE COLLECTION HAD BEEN CONDUCTED PREVIOUSLY, INCLUDE THE ACTUAL RESPONSE RATE ACHIEVED DURING THE LAST COLLECTION.

The interview element of the C³RS evaluation is designed to augment a larger-scale evaluation effort by providing in-depth insight from close observers (and people involved with) of C³RS. (Other data sets include corporate data and assessments of safety culture.) Two groups will be interviewed: 1) railroad workers, and 2) key stakeholders.

Railroad workers: Two factors preclude obtaining a formally representative sample. First, because of the labor intensive nature of interview data collection, only a small number of respondents can be reached. Second, reaching respondents requires close coordination with people’s working schedules, thus requiring seizing opportunities to talk to people, rather than imposing a systematic data collection plan. Thus, the focus of the recruiting effort will be for participants who are typical, rather than representative in a statistical sense.

Key stakeholders. Key stakeholders are people involved in C³RS (e.g., local labor leaders, local management), or close observers of C³RS (e.g., corporate management, national labor, FRA personnel, members of the C³RS steering committee, Volpe personnel involved in implementing C³RS). These interviews will take place monthly during C³RS implementation phases or when the program seems to be changing and less often during quieter periods. The focus will be on critical incidents that can help explain why C³RS is developing as it is, and what impact it is having. The population of key stakeholders is small. They include approximately four to six people at DOT, and four to six people at each participating railroad. (Depending on developments within C³RS, a few Steering Committee Members may occasionally be included.) Thus, these interviews will include most of the key stakeholder universe.

In tabular form, the interviewing process is as follows:

Location			Participation status	Eligible employees	# Interviewees
Railroad workers
	UP North Platte		Committed	Conductors, Engineers (BLET and UTU)	10 – 15 per interview round2
	CP Portage WI		In negotiation	tbd	tbd
	UP Commuter - Chicago		In negotiation	tbd	tbd
	Tbd		----	tbd	Tbd
Key stakeholders
	UP management and labor		Committed1	PRT members, corporate sponsors	4 - 63
	All other railroads4		tbd	tbd	Tbd
	FRA / Volpe personnel		Committed	4 - 65	4 – 6
1-	Participation in the evaluation is a condition of participation in the program.
2-	The precise number will depend on what management deems practical at the time of the site visit.
3-	The total number may vary depending on evolving opinions at the UP concerning the identity of crucial people.
4-	It is reasonable to assume the numbers will be about what they are for the UP, since the number of critical stakeholders does not scale with the size of the workforce. But the real situation cannot be estimated at present.
5-	This estimate is of the critical people at Volpe and FRA who are close observers of the C3RS program. Depending on the stage of implementation, we expect this number to vary over time.

2. DESCRIBE THE PROCEDURES FOR THE COLLECTION OF INFORMATION INCLUDING:

• STATISTICAL METHODOLOGY FOR STRATIFICATION AND SAMPLE SELECTION

• ESTIMATION PROCEDURE

• DEGREE OF ACCURACY NEEDED FOR THE PURPOSE DESCRIBED IN THE JUSTIFICATION

• UNUSUAL PROBLEMS REQUIRING SPECIALIZED SAMPLING PROCEDURES, AND

• ANY USE OF PERIODIC (LESS FREQUENT THAN ANNUAL) DATA COLLECTION CYCLES TO REDUCE BURDEN.

Interviews with employees concerning safety at the railroad will be aimed at identifying “typical” employees rather than a “representative sample” of employees because the number of interviews will always be too small to craft a statistically representative sample, as discussed in item 1 above. This is the inescapable cost of in-depth interviewing. Railroad workers will be identified by talking to managers and union officials at local sites who are in charge of implementing C³RS at their location. These people will be asked to use their judgment and local knowledge to nominate typical employees in particular categories (e.g., an engineer who has been with the railroad fewer than five years).

With respect to the worker interviews, accuracy in a statistical sense is not the goal. Rather, we are looking for insight than will help interpret the quantitative data we are collecting, and for a sense or how railroad workers perceive C³RS. Data from railroad workers will be collected once or twice per year, depending on how years break across the C³RS implementation schedule at participating sites. The questions will be the same, but the respondents will differ, thus minimizing the amount of time any given person will be interviewed.

With respect to the key stakeholders, over time we will be interviewing the entire universe. For all interviews, we expect to be accurate in the sense of careful data collection and recording, but as explained above, “statistical accuracy” is not an issue. Key stakeholders will be interviewed approximately every month early in the C³RS implementation process and less frequently as C³RS stabilizes in its implementation site. The precise schedule will depend on the smoothness or turbulence of implementation at particular locations, although every effort will be made to lengthen the time between interviews as much as possible, as soon as possible.

3. DESCRIBE METHODS TO MAXIMIZE RESPONSE RATES AND TO DEAL WITH ISSUES OF NON-RESPONSE. THE ACCURACY AND RELIABILITY OF INFORMATION COLLECTED MUST BE SHOWN TO BE ADEQUATE FOR INTENDED USES. FOR COLLECTIONS BASES ON SAMPLING, A SPECIAL JUSTIFICATION MUST BE PROVIDED FOR ANY COLLECTION THAT WILL NOT YIELD “RELIABLE” DATA THAT CAN BE GENERALIZED TO THE UNIVERSE STUDIED.

Local C³RS implementers are influential people with respect to the workforce, as they are either labor union leaders, or facility managers. We know from experience that their participation will succeed in identifying appropriate respondents. We also know from previous experience that by engaging these people, in recruiting respondents, that people will agree to be interviewed. The key stakeholder respondents are determined uniquely by their role. There is no pretense of their being representative or typical of any group. We need their insight because of who they are, and because of the unique perspective they have with respect to C³RS. These are people who are highly invested in C³RS, and will be very willing to cooperate with the trained evaluation team.

4. DESCRIBE ANY TESTS FOR PROCEDURES OR METHODS TO BE UNDERTAKEN. TESTING IS ENCOURAGED AS AN EFFECTIVE MEANS OF REFINING COLLECTIONS OF INFORMATION TO MINIMIZE BURDEN AND IMPROVE UTILITY. TESTS MUST BE APPROVED IF THEY CALL FOR ANSWERS TO IDENTICAL QUESTIONS FROM 10 OR MORE RESPONDENTS. A PROPOSED TEST OR SET OF TESTS MAY BE SUBMITTED FOR APPROVAL SEPARATELY OR IN COMBINATION WITH THE MAIN COLLECTION OF INFORMATION.

Data analysis will be comprised of content analysis of open-ended responses to questions. There will be no statistical analysis.

5. PROVIDE THE NAME AND TELEPHONE NUMBER OF INDIVIDUALS CONSULTED ON STATISTICAL ASPECTS OF THE DESIGN AND THE NAME OF THE AGENCY UNIT, CONTRACTOR(S), GRANTEES, OR OTHER PERSONS(S) WHO WILL ACTUALLY COLLECT AND/OR ANALYZE THE INFORMATION FOR THE AGENCY.

The evaluation is being carried out by personnel at the NewVectors division of TechTeam Government Solutions. The Study Director is Dr. Jonathan A. Morell, Ph.D. [email protected] (Tel. (734) 320-4668). In addition, personnel at the Volpe National Transportation Systems Center have been heavily involved in the development and execution of this evaluation. The Volpe contact is Joyce Ranney, Ph.D. [email protected] (Tel. (617) 494-2095).

3