Part B - FTA Program Evaluation for Processes and Outcomes FINAL

U.S. Department of Transportation

Federal Transit Administration (FTA)

Paperwork Reduction Act (PRA) – Part B Submission

FTA Program Evaluation for Processes and Outcomes

OMB Control No. 2132-New Information Collection

B. Collection of Information Employing Statistical Methods

1. Universe and respondent selection

• Target universe: the Federal Transit Administration’s (FTA) program participants, such as State Departments of Transportation (DOTs), Metropolitan Planning Organizations (MPOs), Transit Authorities, State and Local Government Units, and Indian Tribes. Subgrantees are not included in the target population.

• FTA will select programs for evaluation by considering factors such as their ability to undergo a program evaluation based on their level of data usage, program support, and program design. Programs will also be selected based on the importance of a program to the mission of the Department of Transportation (DOT) and FTA, the size of the program in terms of funding or populations served, and the extent to which the study would provide new and meaningful information to the program or populations served. FTA will also consider alignment with priority issue areas, including equity and sustainability.

• FTA programs (State of Good Repair Formula Program, Low- or No-Emission Bus Program, and Pilot Program for Transit Oriented Development Planning) were identified to represent the potential universe of FTA programs with relevant information collection opportunities such as surveys or focus groups. It is estimated that up to 4 personnel at each organization have applicable knowledge and will respond to surveys. All program participant organizations will receive an invitation to the focus groups, but only 50% of the personnel receiving surveys at each organization will be invited to a focus group. Therefore, 2 personnel from each organization will be invited to participate in a focus group. The purpose of having multiple respondents from each organization is to obtain more comprehensive feedback data and to increase the diversity of organizational perspectives.

• A representation of a year’s worth of work estimates 345 total program participant organizations in the population universe, and approximately 1,380, potential survey respondents and 690 focus group participants.

  • At a maximum, a census across defined program years will be conducted of program participants for surveys and focus groups; however, depending on the program, sampling methods may be required.

  • Participants are identified based on contact information provided to FTA’s grants management database (Transit Award Management System, or TrAMS) or other internal FTA data sources, such as FTA’s data reporting repository (National Transit Database, or NTD). While program participants may not interact directly with FTA’s services or staff (for example, bus operators and maintenance staff), individuals accessing TrAMS or NTD have made verified contact with FTA services. They have also been designated as points of contact by the program participant.

  • FTA may request program participant personnel confirm their involvement with the program and provide any additional points of contact within their organization who have interacted with the program and may be candidates for survey and focus group participation.

  • FTA will also leverage FTA’s 10 regional offices and Headquarters program offices, such as, but not limited to the Office of Planning and Environment (TPE), Office of Budget and Policy (TBP), Office of Project Management (TPM), Office of Transit Safety and Oversight (TSO), Office of Regional Services (TRS), and Office of Civil Rights (TCR), to identify and engage points of contact.

2. Procedures for collecting information

• The information collected through this clearance is necessary to illustrate the changes or benefits that occur because of program implementation. The information collection will assess the effectiveness of programs and potentially improve program processes, outputs, and enhance participant experiences. Each evaluation will differ in its scope and requirements, but stakeholder engagement will be a critical aspect of each evaluation. Methods of engagement may include a survey and/or a series of focus groups to understand program participant experiences in implementing eligible activities with program funds and impressions of the program.

• Duplication is not anticipated between the survey and focus group questions for a single program evaluation. Questions have been piloted to test their appropriateness for use in surveys versus focus groups, see section 4, Tests of Procedures, for more information.

Surveys

• One survey is anticipated per program, per year.

• The survey will be comprised of a mix of question types, including scale, select all that apply, multiple choice, and open-ended questions.

• Unique surveys will be developed for each program evaluation, based on stakeholder interests for answering a program’s overarching evaluation questions, by selecting questions from this comprehensive question bank. The question bank was developed so that the questions can be used in each program evaluation's survey without modification, thereby eliminating the need for multiple clearance packages.  

• Internally, program evaluation and subject matter experts will select the survey questions based on the questions’ alignment to the program evaluation requirements (to be determined during the evaluation planning phase of any single program evaluation). Questions will be selected based on relevant program characteristics, such as program type (e.g., formula or discretionary) or eligible activities (e.g., bus replacement; transit station rehabilitation).

• Surveys for each program evaluation may vary in length and number of questions but will align with the average time pilot testers took to complete the pilot survey. Based on the results of pilot testing, the average time among pilot testers was 19 minutes 28 seconds (approximately 0.33 hours) to complete each survey.

• Whenever possible, FTA will conduct surveys electronically and/or use online collaboration tools to reduce burden and improve efficiency. The technology will allow FTA to ask a mix of question types and automatically collect data. Surveys will include branching to decrease the number of unnecessary questions respondents must answer, therefore reducing burden. Branching questions direct survey respondents to the appropriate next question based on their answer to a previous question.

• The survey will be sent by an electronic mail message seeking participation. The message will include the appropriate hyperlink to the internet site that hosts the survey.

• FTA may collect identifiers such as name of program participant organization, type of award received, respective FTA region and state of the program participant organization, and role of the respondent at the participant organization. Contact information, including electronic mail address, may be requested to gauge interest in focus group participation but will not be mandatory. FTA will not collect individual IP addresses from survey responses.

• No individual names or identifiers will be distributed that link responses to individuals. The information participants provide will be kept confidential and will not be disclosed to anyone but the researchers conducting the study, except as otherwise required by law. 

Focus Groups

• One round of focus groups is expected per program, per year.

• When feasible, focus groups will be held virtually using video conferencing technology to reduce travel, cost, and burden requirements.

• Facilitation guides and focus group scripts will be developed to provide facilitators with a roadmap to guide the sessions and ensure they do not exceed preplanned time limits.

• Internally, program evaluation and subject matter experts will select the focus group questions based on the questions’ alignment to the program evaluation requirements (to be determined during the evaluation planning phase of any single program evaluation). Questions will be selected based on relevant program characteristics, such as program type (e.g., formula or discretionary) or eligible activities (e.g., bus replacement; transit station rehabilitation).

• Focus group questions will be selected to clarify and support interpretation of survey responses to aid in the most comprehensive data collection.

• Two personnel selected from the four survey participants will be invited to participate in a focus group. The sessions will allow program participants the opportunity to provide feedback from multiple roles within their organization.

• Sampling methods may be used to identify and select focus group participants. FTA may collect identifiers such as name of program participant organization, type of award received, respective FTA region and state of the program participant organization, and role of the respondent at the participant organization.

• No individual names or identifiers will be distributed that link responses to individuals. The information participants provide will be kept confidential and will not be disclosed to anyone but the researchers conducting the study, except as otherwise required by law.

3. Methods to maximize response

• An initial announcement of the program evaluation will be signed by FTA’s Executive Director or other FTA Leadership and sent by electronic mail to program participants. The announcement will inform program participants that a program evaluation is occurring and include a confirmation request for points of contact.

• Program evaluations may also be communicated and coordinated at existing FTA meetings where potential survey and focus group participants are in attendance.

Surveys

• FTA has set a goal to attain at least an 80% unit response rate and a 70% item response rate, aligned with the Office of Management and Budget (OMB) non-bias threshold described in the OMB Standards and Guidelines for Statistical Surveys.

• The survey window will be at least 8 weeks, providing ample opportunity to respond. An invitation to participate in the survey will be sent via electronic mail to program participant personnel at the start of the survey window. The invitation will be sent from a designated electronic mail account. The message will include a one-page document with a Frequently Asked Questions (FAQ) list intended to help program participant personnel complete the survey.

• The survey form will include detailed instructions and definitions of terms, including evaluation areas and response variables, helping users complete the survey and improving data quality. The questions are straightforward and do not require any computations or specialized data entry.

• FTA may host office hours virtually using video conferencing technology to allow program participant personnel to ask questions about the survey.

• Follow-up messages will be sent to non-responsive program participants via electronic mail two weeks after the survey window opens.

• A survey closing notification will be sent via electronic mail at the end of the survey window.

• FTA does not intend to offer any special incentives for participating in the survey.

• Because respondents are not required to provide their contact information, FTA cannot follow-up individually with non-respondents. FTA will monitor responses to identify potentially underrepresented population groups (e.g., transit agencies in Urbanized Zone Areas (UZAs) between 50,000 and 199,999 in population within FTA’s Region 5 territory) and note them in summary data presentations and reports.

Focus Groups

• As stated previously, 50% of survey respondents will be invited to participate in focus groups. Based on the methodology below, FTA expects at least 80% of those focus group invitees will participate. For example, among 200 survey respondents, 100 would be invited to participate in focus groups and at least 80 would participate.

• A question may be included in the survey requesting participants indicate whether they would like to participate in focus groups.

• Federal personnel at the regional level have contact information for grant recipients and will be able to assist in identifying the correct point of contact for distribution of invitations. Focus group invitations will be distributed to the same list of organizations as the survey, however, 2 contacts will be invited to participate in focus groups, compared to the 4 invited participate in surveys.

• An initial request for availability will be sent via electronic mail to coordinate around program participant personnel schedules. The request will be sent from a designated electronic mail account.

• Follow-up messages will be sent to non-responsive program participants via electronic mail two weeks after the initial request for availability.

• Once availability is collected, an electronic calendar invitation will be sent to focus group participants 3 weeks before the event.

• Multiple meeting times will be available to accommodate participant’s preferences and schedules.

• A reminder will be sent via electronic mail one week before the event.

• A follow-up message will be sent to focus group participants no more than one week after the event, summarizing key takeaways and providing any relevant materials.

• FTA does not intend to offer any special incentives for participating in the focus group.

• The questions are open-ended, straightforward, and do not require any computations or specialized data entry.

4. Tests of procedures

• In 2024, a small group of 6 program participant personnel were asked to participate in a pilot test of the question bank and provide feedback on whether the questions were appropriate and relevant to FTA programs and were not burdensome to respond to. The pilot surveys included a mix of question types, including scale, select all that apply, multiple choice, and open-ended questions. 45% of questions in the entire question bank were reviewed by pilot testers, including questions related to active awards and closed awards. No program evaluation would use all items from the question bank, only a subset of the bank will be relevant to any one program. Based on the results of pilot testing, the average time among pilot testers was 19 minutes 28 seconds (approximately 0.33 hours) to complete each survey.

• We anticipate all surveys will replicate the general structure and burden of the pilot test survey with a similar mix of question quantities and types, including scale, select all that apply, multiple choice, and open-ended questions. In partnership with FTA regional offices, as was the case during this initial pilot-test, all future surveys will be pilot tested to ensure they align with the average time presented in our burden calculations.

• Pilot survey feedback was collected via an online feedback form where pilot testers had the opportunity to provide written comments on each question and the survey overall. An optional meeting was also held to provide an opportunity for pilot testers to discuss their comments and suggestions with the program evaluation team. Comments were addressed to improve the quality and relevance of the questions to FTA programs.

• As set out in FTA's Policy Development and Approval Process (FA O 7001.1C), drafts of the survey and focus group questions were shared with key leadership and transit recipient representatives from across FTA at Policy Review Team and Policy Council. The full list of questions was reviewed by FTA representatives for appropriateness and relevance to FTA programs. Policy Review Team provided 3 comments on the draft question bank and Policy Council provided one comment. Comments were addressed to improve the quality of the questions.

5. Contact for statistical aspects and data collection

• Julianne Lee, FTA Office of Budget and Policy, 202-366-6597, [email protected]