Supporting Statement A - FTA Program Evaluation for Processes and Outcomes FINAL4

U.S. Department of Transportation

Federal Transit Administration (FTA)

Supporting Statement for Paperwork Reduction Act Submissions

FTA Program Evaluation for Processes and Outcomes

OMB Control No. 2132-New Information Collection)

Abstract:

This is a request to the Office of Management and Budget (OMB) to approve a new information collection for U.S. Department of Transportation (DOT) Federal Transit Administration (FTA) Program Evaluation for Processes and Outcomes. This is a new information collection request (ICR) which will enable FTA to collect vital stakeholder perspectives on FTA’s programs to conduct program evaluations in accordance with requirements of the Foundations for Evidence-Based Policymaking Act of 2018 (the Evidence Act).

SECTION A. JUSTIFICATION:

1. Explain the circumstances that make the collection of information necessary.

FTA’s mission is to improve America’s communities through public transportation. Since 1964, FTA has partnered with state and local governments to create and enhance public transportation systems, investing more than $13 billion annually to support and expand public transit services. FTA executes many programs in support of this critical mission.

The Foundations for Evidence-Based Policymaking Act of 2018¹ (the Evidence Act) requires federal agencies to develop evidence to support policymaking. Federal agencies, including FTA, must systematically collect and analyze data to assess the effectiveness and efficiency of their programs through evaluation. According to the Evidence Act, an evaluation involves a thorough examination of programs, policies, and organizations to gauge their impact. Evidence resulting from this examination is used to inform leaders about whether federal programs and activities are achieving their intended results, leading to improved use of data, and evidence-based decision making.

To effectively evaluate programs and policies for processes and outcomes, FTA must have the ability to collect data directly from program participants, such as State Departments of Transportation (DOTs), Metropolitan Planning Organizations (MPOs), Transit Agencies, State and Local Governments, and Indian Tribes, to understand their experiences and benefits of program participation. OMB’s M-19-23 Memorandum² emphasizes that engaging stakeholders enhances the utility of evaluation findings to improve program effectiveness. In alignment with OMB’s guidance, the U.S. Government Accountability Office (GAO) recommends engaging stakeholders early and often, tailoring engagement strategies based on needs and purpose. These practices are outlined in GAO-23-105460³, titled, “Evidence-Based Policymaking: Practices to Help Manage and Assess the Results of Federal Efforts.”

To ensure alignment with ongoing efforts, this ICR seeks to support the FTA in meeting the Evidence Act’s requirements and contribute to the broader framework of evidence-based decision-making. The information collected through this clearance is necessary to effectively evaluate FTA’s programs and inform future program improvements and policy decisions. Each evaluation will differ in its scope and requirements, but engagement of transit stakeholders external to FTA (e.g., grant participants, community partners) associated with FTA program implementation provides a critical perspective for each evaluated program. Data collection methods will include surveys and/or focus groups.

The survey portion of this information collection differs from other similar efforts, such as the FTA Stakeholder Survey (OMB Control No. 2132-0564), which targets grant recipients and inquiries about customer satisfaction with a range of FTA services, as well as familiarity with some recent FTA rulemakings. Current and previous information collections for grant applicants and grantees to provide routine program planning and metrics on plans and deliverables are separate from this collection. Rather, this information collection will allow FTA decision makers to understand challenges and barriers to program implementation, identify opportunities for improving program communications and outreach, and make stronger linkages between program progress and identified outcomes.

This information request supports DOT’s fifth strategic goal, Organizational Excellence. Specifically, this request relates to DOT’s Data-Driven Programs and Policies objective, which states “Develop and manage data systems and tools to provide objective, reliable, timely, and accessible data to support decision-making, transparency, and accountability.” This effort aligns with several strategies, including “Support informed decision-making by expanding U.S. DOT’s capacity for data collection, analysis, and sharing with stakeholders and the general public; Improve program evaluation processes to better quantify the outcomes of programs and policies and establish outcome-based performance measures for all major programs; and Identify and develop strategies to address data gaps to support safety, equity, and other priorities.” This ICR will allow FTA to collect information necessary to evaluate programs and inform future programmatic and policy decisions to improve outcomes, expand impact, and enhance participant experiences.

2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with all applicable Information Quality Guidelines.

Improving FTA programs requires ongoing assessment of program delivery and effectiveness. FTA will collect, analyze, and interpret information gathered through this clearance to identify strengths and weaknesses of FTA programs (i.e., formula, competitive) and make improvements based on feedback and data analysis. This information collection focuses on identifying program outcomes so that FTA can evaluate overall performance and make informed decisions about the program moving forward. This outcome- and implementation-focused information collection is distinct from other current FTA efforts related to program management which are more focused on regular reporting metrics (for example, monthly progress reporting or other required reporting). The solicitation of feedback may target areas such as: strategies employed to meet program requirements and goals, changes or benefits in communities and transit participants over time; changes in program implementation over time; additional activities undertaken outside of the program scope/timeframe; level of FTA involvement (e.g., technical assistance, provided resources) throughout the program; variation in proposed and actual timelines for activity completion; stakeholder engagement; resources and technology leveraged; partnerships and collaboration; and specific program outcomes including those related to equity, sustainability, innovation, and other FTA strategic goals and objectives.

Responses will be assessed to inform efforts to improve or maintain the quality of FTA’s programs. FTA will submit a report on the findings of the evaluations to the applicable internal FTA office for intended use in developing strategies for program improvement based on the results. The Office should collaborate to identify actionable steps and implement program improvement plans in response to the evaluation.

Information gathered will be used by FTA leaders to inform critical decisions about programs. If this information is not collected, vital feedback from program participants and stakeholders necessary to evaluate and improve FTA’s programs will be unavailable.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology.

Whenever possible, FTA will collect information electronically and/or use online collaboration tools to reduce burden and improve efficiency. Survey items may include branching to decrease the number of unnecessary questions respondents must answer, therefore reducing burden. Branching questions direct survey respondents to appropriate next questions based on their answer to a previous question. When feasible, focus groups will be held virtually using video conferencing technology to reduce travel, cost, and burden requirements.

4. Describe efforts to identify duplication.

As set out in FTA’s Policy Development and Approval Process (FA O 7001.1C), FTA shared drafts of the information collection tools with representatives from across the organization at Policy Review Team and Policy Council. FTA requested their review, and no potential duplication was identified. No similar data are gathered or maintained by FTA or are available from other sources known to FTA. There is no duplication.

5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.

Small businesses or other small entities, such as a government jurisdiction with a population of less than 50,000 or a small not-for-profit organization that is not dominant in its field, may be involved in these efforts. In these cases, FTA will minimize the burden on them for the information collections approved under this clearance by sampling, asking for readily available information, and using short, easy-to-complete information collection instruments. The information requested will be held to the minimum required for the intended use.

6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.

Without the opportunity to collect feedback through surveys or focus groups, FTA would be unable to fully meet the Evidence Act’s requirements. FTA could not effectively collect and incorporate stakeholder and program participant perspectives. This could limit the utility of FTA evaluations and the ability to recommend and implement effective future program improvements. Further, a lack of data directly from program participants would limit FTA’s ability to interpret benefits or changes that occur at the local level as a result of program implementation.

7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.

There are no special circumstances. The information collected is consistent with the guidelines in 5 CFR 1320.6.

8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the stakeholder outreach efforts with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. If you plan or have completed webinars, listening sessions, etc. provide date or targeted date.

A 60-day Federal Register notice was published on December 7, 2023, Vol. 88. No. 234 (pages 85357-85358), soliciting comments prior to submission to OMB. No comments were received. FTA also conducted outreach with potential respondents. In early 2024, a small group of six (6) program participants were asked to participate in a pilot test of the question bank and provide feedback on whether the questions were appropriate and relevant to FTA programs and were not burdensome to respond to. 

A 30-day Federal Register Notice was published on July 10, 2024 Vol. 89 No. 132 pages 56807-56808.

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

FTA will not provide payment or other forms of remuneration to respondents of its various forms of feedback collection.

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

FTA may collect identifiers such as name of program participant organization, type of award received, the respective FTA region and state of the program participant organization, and role of the respondent at the participant organization. Contact information, including electronic mail address, may be requested but will not be mandatory. No individual names or identifiers will be distributed that link responses to individuals. The information participants provide will be kept confidential and will not be disclosed to anyone but the investigators conducting the study, except as otherwise required by law.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

None of this information is of a sensitive nature.

12. Provide an estimate in hours of the burden of the collection of information. 

Respondents: State Departments of Transportation (DOTs), Metropolitan Planning Organizations (MPOs), Transit Authorities, States and Local Government Units, and Indian Tribes.

Estimated Annual Number of Respondents: 1,380 respondents

Estimated Annual Number of Responses: 2,070 responses (1,380 + 690)

Estimated Total Annual Burden Hours: 2,180 hours (455 + 1,725)

Estimated Total Cost: $110,591($23,082 + $87,509)

Frequency: Annual

Because this ICR covers all FTA programs that may be evaluated, the burden estimate methodology used is intended to represent all potential program evaluations over the period of this ICR. Three FTA programs were identified to represent programs across the potential universe of FTA programs that could be evaluated: the Pilot Program for Transit Oriented Development Planning, the Low- or No-Emissions Bus Program, and State of Good Repair Formula Program. These programs were selected by FTA program evaluation experts to reflect the broadest range of program characteristics, including the number of recipients, likelihood of being evaluated during the information collection request period, and general representativeness of FTA’s programs.

It is not anticipated that the same 3 programs would be evaluated each year but rather FTA could feasibly evaluate up to 3 distinct programs in a year. This logic allows for the possibility of evaluating only 1 program per year. Estimates for data burden reflect the maximum possible level of effort. Any FTA-selected program would only be evaluated once, not multiple times, within a 3-year period. FTA leadership will determine which programs to evaluate based on a variety of factors including policy priorities and likelihood of being evaluated during the information collection request period (based on FTA’s published evaluation plans).

These calculations are based on the total number of recent program participants (‘organizations’) for each of these three programs, i.e., the total number of organizations that could be involved in data collection (Table 12.1). Note: The timeframes presented in this table indicate any participants whose grants were active at any point during this time range.

Table 12.1

	Transit-Oriented Development (CY18 – Present)	State of Good Repair (CY20-Present)	Low or No Emission Vehicle Program (CY20-Present)	Total
Organizations	73	101	171	345

The subsequent calculations are based on maximum burden if all program participants within the selected time range were to actively participate in all applicable surveys and focus groups. This calculation represents a census, so all program participants are included (this calculation is based on no sampling). Therefore, these calculations are based on 100% participation rate.

It is estimated that 4 personnel at each organization have applicable knowledge and will respond to the survey. Therefore, the number of survey responses is calculated by multiplying 345 organizations x 4 personnel per organization = 1,380 survey responses.

All program participants will receive an invitation to the focus groups, but only half (50%) of the personnel receiving surveys at each organization will be invited to a focus group. Therefore, 2 personnel at each organization will be invited to participate in a focus group. The number of focus group responses is calculated by multiplying 345 organizations x 2 personnel per organization = 690 focus group responses. The total number of responses is calculated by adding 1,380 survey responses + 690 focus group responses = 2,070 responses (Table 12.2).

Table 12.2

Information Collection Tool	Total Organizations	Personnel Per Information Collection	Total Responses for 3 Programs
Surveys	345	4	1,380
Focus Groups	345	2	690
TOTAL			2,070

We conducted pilot testing of 31 survey questions representative of a mix of questions from the question bank to create an estimated burden time, which will apply to surveys for all three programs in our calculations. Pilot testers were program participants. The average time among pilot testers was 19 minutes 28 seconds (approximately 0.33 hours). Focus group participation and related activities will take 2.5 hours, accounting for pre- and post-focus group activities (e.g., coordination and logistics, preparing based on read-ahead materials, post meeting follow-up).

For survey information collection, the total annual hours burden is estimated by multiplying 1,380 responses x 0.33 hours per response = 455 hours. For the focus group information collection, the total annual hours burden is estimated by multiplying 690 responses x 2.5 hours per response = 1,725 hours. Therefore, the maximum total burden for FTA information collections will be 2,180 hours per year (455 survey hours + 1,725 focus group hours) (Table 12.3). Note: The total organizations have been included in Table 12.3 for continuity but are not part of the calculation.

Table 12.3

Information Collection Tool	Total Organizations	Total Responses for 3 Programs	Hours Burden per Response	Total Annual Hours Burden for 3 Programs
Surveys	345	1,380	0.33	455
Focus Groups	345	690	2.5	1,725
TOTAL		2,070		2,180

It is estimated that the occupation of the personnel that would be participating in the information collections on behalf of the organizations is Urban and Regional Planner (occupational code 19-3051) according to the U.S. Bureau of Labor Statistics (BLS). As of May 2022, their median salary is $39.63 per hour plus 28% fringe benefits which equates to $50.73 per hour.

Therefore, the total cost to respondents for the survey collection tool is $23,082 ($50.73 x 455 hours). The total cost to respondents for the focus groups is $87,509 ($50.73 x 1,725 hours). The total annual burden cost to respondents for all information collections is $110,591 ($23,082 + $87,509) (Table 12.4).

Table 12.4

Information Collection Tool	Hourly Wage	Total Annual Hours Burden for 3 Programs	Total Annual Cost
Survey	$50.73	455	$23,082
Focus Group	$50.73	1,725	$87,509
TOTAL		2,180	$110,591

13. Provide an estimate of the total annual recordkeeping/reporting cost burden to the respondents resulting from the collection (excluding the value of the burden hours in #12 above).

There is no additional cost beyond that shown in items 12 and 14.

14. Provide estimates of annualized cost to the Federal government.

The annualized cost to the federal government is $1,962,471. The cost is based on the 2024 OPM salary table and independent government cost estimate for contract support, and is calculated as follows:

Federal Program Evaluation Oversight

FTA’s Office of Budget and Policy will perform oversight functions for all evaluations. It is estimated that the team will include 3 staff in the Washington, DC metro area and use 50% of their time annually. This team is estimated to include a GS-15/step 3, GS-14/step 3, and GS-11/step 3 (Table 14.1). In total, the FTA evaluation oversight team’s time will cost the federal government $287,342 (1,040 hours x [$59.22 + $99.75 + $117.32]) (Table 14.2).

Table 14.1

Position	Hourly Wage (Incl. 40% Fringe Benefits)
GS-11/step 3	$42.30 ($59.22)
GS-14/step 3	$71.25 ($99.75)
GS-15/step 3	$83.8 ($117.32)

Table 14.2

Position	Hourly Wage Incl. 40% Fringe Benefits	Annual Hours (50% FTE)	Total Annual Cost
GS-11/step 3	$59.22	1040	$61,589
GS-14/step 3	$99.75	1040	$103,740
GS-15/step 3	$117.32	1040	$122,013
TOTAL			$287,342

Federal Program Staff Involvement

FTA staff operating each of the 3 programs being evaluated will also be engaged. We estimate that 2 program staff members will participate in the information collection for each program FTA evaluates: the FTA program manager would be a GS-14/step 3 and a program support staff member would be a GS-13/step 3, both located in the Washington, DC metro area (Table 14.3). It is estimated that each staff member will spend 54 hours on each program evaluation (Table 14.4). Therefore, the FTA program team’s time will cost the federal government $29,834 (3 program offices x 54 hours x [$84.41 + $99.75]) (Table 14.5).

Table 14.3

Position	Hourly Wage (Incl. 40% Fringe Benefits)
GS-13/step 3	$60.29 ($84.41)
GS-14/step 3	$71.25 ($99.75)

Table 14.4

Activity	Annual Number of Hours Per Program Staff Member
Data Collection Support	6
Providing Input, Feedback, Guidance on Evaluation Activities	12
Check-In Meetings and Ad Hoc Requests	36
TOTAL	54

Table 14.5

Position	Hourly Wage Incl. 40% Fringe Benefits	Program Offices (1 per evaluation)	Annual Hours	Total Annual Cost
GS-13/step 3	$84.41	3	54	$13,674
GS-14/step 3	$99.75	3	54	$16,160
TOTAL				$29,834

FTA Regional Support

FTA staff at the regional offices will also support information collections when they are involved with program participants and/or program administration. We estimate that 1 staff member, a GS-14/step 3, from each of the 10 FTA regional offices will participate. Because staff are likely located in FTA regional offices across the United States, their salaries are estimated using the locality pay area ‘Rest of U.S.’ (Table 14.6). It is estimated that each regional office staff member will spend 17 hours on each program evaluation. Estimates for cost burden reflect the maximum level of effort, FTA could feasibly evaluate up to 3 distinct programs per year (Table 14.7). Therefore, the FTA regional office staff’s time will cost the federal government $44,594 (10 regional staff x [3 programs x 17 hours] x $87.44) (Table 14.8; Table 14.9).

Table 14.6

Position	Hourly Wage (Incl. 40% Fringe Benefits)
GS-14/step 3	$62.46 ($87.44)

Table 14.7

Activity	Annual Number of Hours Per Regional Staff Member (1 Evaluation)	Annual Number of Hours Per Regional Staff Member (3 Evaluations)
Data Collection Support	3	9
Focus Groups	5	15
Point of Contact Identification	3	9
Check In Meetings and Ad Hoc Requests	6	18
TOTAL	17	51

Table 14.8

FTA Regions	Staff Per Region	Hours Per Regional Staff Member	Programs Evaluated
10	1	17	3

Table 14.9

Position	Hourly Wage Incl. 40% Fringe Benefits	FTA Regions	Annual Hours	Total Annual Cost
GS-14/step 3	$87.44	10	51	$44,594

Contractor Support

Additionally, the Federal government is likely to utilize contractor support for the information collection tasks. Based on existing and similar contracts, the contract(s) associated with each information collection is estimated to cost $533,567, or a total of $1,600,701 for 3 evaluations (Table 14.10).

Table 14.10

Task	Hours	Cost Per Evaluation	Total Annual Cost
Data Collection	3,310	$533,567	$1,600,701

Total Cost Summary

It is assumed that FTA will conduct no more than 3 evaluations per year, with one information collection (including surveys and focus groups) for each evaluation, and the team overseeing the evaluations’ time burden will remain constant at 50%. Therefore, the total annual cost to the Federal Government is estimated to be $1,962,471 ($287,342 + $29,834 + $44,594 + $1,600,701) (Table 14.11).

Table 14.11

Task	Annual Cost
Federal Program Evaluation Oversight	$287,342
Federal Program Staff Involvement	$29,834
FTA Regional Support	$44,594
Contractor Support	$1,600,701
TOTAL	$1,962,471

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14

This is a new collection of information. The summary provided at the beginning of question #12 has been updated since the publication of the 60-Day Federal Register Notice. The estimated annual respondents and number of responses have changed to reflect the current number of program participants for each representative FTA program as of April 2024. The estimated total burden was also updated to reflect this change in the estimated number of respondents, as well as a change to the estimated time per information collection based on the results of pilot testing.

16. For collections whose results will be published, outline the plans for tabulation and publication.

These will be historical analyses focused on finding patterns in relevant variables. This analysis may be made publicly available on FTA’s website as required in alignment with the Evidence Act requirements and/or program interests to disseminate the information to the public. All data will be de-identified and not attributable to any individual person.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

We are not requesting this. There is no reason not to display the expiration date of OMB approval.

18. Explain each exception to the certification statement identified in Item 19 of the OMB 83-I

No exceptions are stated. All activities comply with the requirements in 5 CFR 1320.9.

1 PUBL435.PS (govinfo.gov)

2 MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES M-19-23 (whitehouse.gov)

3 Evidence-Based Policymaking: Practices to Help Manage and Assess the Results of Federal Efforts | U.S. GAO