Supporting Statement A
Revision and Update of the Health Resources and Services Administration’s
Bureau of Health Professions (BHPr) Performance Data Collection
This request is for approval from the Office of Management and Budget (OMB) of revised data collection activities required for grant award applications, progress reports and annual performance reporting for the Health Resources and Services Administration’s (HRSA) Bureau of Health Professions (BHPr) programs. The proposed revision to data collection activities aim to achieve multiple objectives:
gather information on the activities and program participants supported with BHPr funding,
monitor grantee performance for oversight and program improvement activities, and
establish the foundation for measuring the impact of BHPr programs.
This data collection also helps HRSA/BHPr fulfill statutory reporting requirements for performance measurement. Currently BHPr is monitoring awards to 2,300 grantees in 50 programs. These programs award grants to health professions schools and training programs across the United States to develop, expand and enhance training and to strengthen the distribution of the health workforce. The programs are authorized by Titles III, VII, and VIII of the Public Health Service Act. The BHPr program reporting requirements are outlined in the Performance Report for Grants and Cooperative Agreements (PRGCA) and collected in the HRSA Web-based Enterprise system. This report was formerly called the Uniform Progress Report.
The Health Resources and Services Administration (HRSA), Bureau of Health Professions’ (BHPr) mission is to improve access to health care by providing national leadership in the development and distribution of a diverse, culturally competent health workforce that can adapt to the population’s changing health care needs while offering the highest quality care. The proposed BHPr annual program reporting requirements reflect the 2010 – 2015 Strategic Plan and goals of the Department of Health and Human Services (DHHS) as well as the Health Resources and Services Administration’s (HRSA) goal to strengthen the health care workforce. Building on changes to BHPr programs through the Affordable Care Act, the Bureau performance goals focus on five key outcomes:
Supply - increasing number of health professionals, particularly in key areas experiencing shortages,
Distribution - influencing the distribution of practitioners to practice in underserved and rural areas,
Diversity - encouraging the training of health professions from diverse backgrounds,
Infrastructure- supporting educational infrastructure, with a particular focus on providing new or expanded health profession training facilities and faculty support to sustain the facilities and;
Quality - enhancing the education experience with the longer term goals of improving the quality of health care and health outcomes.
The Affordable Care Act (P.L. 111-148) reauthorized most of the health professions programs and in some cases expanded eligibility and modified required or allowable activities. The Affordable Care Act also created some new health professions programs which have been implemented. These changes, along with a continuing interest in improved targeting of program investments, are driving efforts to improve accountability and oversight.
This application for OMB approval is a revision to OMB No. 0915-0061 which expires on December 31, 2012. BHPr is submitting this request for revision well in advance of the 2012 deadline to allow BHPr and our grantees to implement the necessary programmatic and IT changes to support pilot testing measures in advance of reporting academic year 2011-2012 activities.
The collection of performance metrics will provide data to establish the foundation to conduct evaluations to determine if programs meet our Bureau performance goals. The data describe:
Demographic characteristics, practice locations and program activities of BHPr grantees and trainees.
Specific program outputs to inform program decision making.
Across-program outputs to help measure BHPr program performance and alignment of programs with HRSA priorities to inform policymaking.
Revisions Made Since the Last Clearance
Revisions since the last OMB performance measure package include proposing to collect data at different points in time of the grant, for example, at time of application, at time of award, and annually. In response to feedback from OMB, metrics containing subjective data were eliminated. In addition, metrics are removed or consolidated with other elements to help ensure consistent data across programs. (See Attachment A)
BHPr has developed and/or revised application and notice of award forms for health professions programs. These new or revised forms help BHPr to:
verify applications for general eligibility and eligibility related to funding preferences;
make award determinations;
3) set baselines to help evaluate the effectiveness of our programs;
In order to make it easier for applicants to provide data, for the Objective Review Committee to identify the data, and to ensure that applicants submit comprehensive applications, BHPr has created program specific forms/templates in which to collect the data (e.g., tables, charts, etc.). Moreover, the templates call for more succinct responses in a consistent format which can be used to compare annual performance data. BHPr has always collected these types of data from applicants; however, the data were not collected with templates in all programs or the data did not focus on establishing baseline information and setting annual performance targets. Attachment B provides an overview of the metrics collected at the time of application.
Another key change in this data collection package is that the performance metrics are collected at three levels of annual data collection: individual-level, program-level and cross-cutting. The new individual-level data collection will strengthen the assessment of trainee demographics such as age, gender, race, ethnicity, and disadvantaged background. BHPr proposes to collect non-personally identifiable (e.g., de-identified) data at the individual trainee-level with an assigned unique identification by the grantee. A pilot test will be conducted with grantees in select cohorts using a unique identifier. The seven digit alphanumeric identification is not associated with a Social Security number of the individual trainees that is the beneficiary of the grant activities. Only self-identifying grantees reporting readiness to provide individual level data are included in the pilot. (Attachment C). The data collection will also provide descriptive data on workforce recruitment, participants’ training activities, retention, distribution (intended practice locations), and trainee characteristics such as disadvantaged background, racial and ethnic diversity. Descriptive metrics are reported in Congressional Justifications and other documents regarding BHPr programs.
Program-level data were revised to strengthen information unique to the grant objectives of the programs. These data incorporate accountability and are critical to reporting measureable outputs within program performance annually. Program level data are aggregate reporting on all the participants in the program.
At the third level of data collection, cross-cutting are data collected for a set of programs with similar activities or goals which can be aggregated into Bureau-level metrics. These cross-cutting metrics were developed to help lay the foundation to assess the broader BHPr goals, strategies, and outcomes. The revised data collection will enhance BHPr’s ability to assess its programs’ results related to cross-cutting initiatives. Attachment D is an example of collecting experiential and clinical training experiences across programs using consistent metrics. Attachment E provides a matrix of which program metrics correlate to Bureau Performance Measures.
The use of these three levels of data represents a fundamental shift from collecting majority program-specific metrics to majority cross-cutting metrics that will afford BHPr the opportunity to consistently collect annual data across programs that align with Bureau performance goals. The enhancements in requesting individual trainee data will strengthen quantitative reliability in data reporting, will provide annual qualitative data regarding trainee experiences, behavior, and intentions regarding practice, and will support further development of program evaluations.
Attachment F illustrates the linkage between BHPr performance goals and the three levels of annual data collection (individual-level, program-specific and cross-cutting). The dashed line represents a two phased plan. The measurement goals above the line reflect annual program measurement, and below the line represent the phase two program evaluation. Attachment G describes the Bureau’s preliminary evaluation strategy and how the performance data collected could be used to support future evaluations.
The proposed reporting requirements are fully automated through an established and tested HRSA Web-based Enterprise system that enables grantees to obtain and submit reports. The Enterprise system has been reconfigured to: 1) reduce grantee burden; 2) improve data validity, reliability, quality, and legislative requirements; 3) ensure a cost-effective data collection strategy, and 4) allow program staff to easily retrieve and analyze performance data as they monitor grantee progress.
The system incorporates new tools using the latest technology allowing the grantees the ability to upload spreadsheets or stream data. These new tools are intended to reduce the burden of manual data entry for some data collection forms thereby reducing grantee time and costs devoted to reporting. In fields that require manual entry, the system only permits grantees to enter information that is applicable to their particular programs. The tables are interlocked where data overlap, validations are built-in, calculations (e.g., ratios, rates, percentages, and totals) are automated and historical data are preserved so that only the annual data for the year in question needs to be newly entered.
Institutions that receive grant funding from HRSA for specific activities may report similar data to other organizations about their training programs. However, the grant-specific information requested for this reporting is not collected or reported elsewhere.
In efforts to streamline reporting requirements throughout the BHPr, many metrics are reported consistently across programs. This revision reduces data collection burden on grantees that may have more than one grant within HRSA. As noted earlier, the majority of metrics proposed for collection are crosscutting and are defined consistently for use in multiple programs. In addition to consistent Bureau reporting, BHPr has also consulted health professions experts in clarifying terminology and definitions among health professions. For example, the Bureau has adopted the Consensus Model for APRN Regulation to categorize advanced practice registered nurses. Therefore, any program reporting on advanced practice nursing trainees will use a drop down list of terms approved through the work of the APRN Consensus Work Group and the National Council of State Boards of Nursing APRN Advisory Committee.
This project does not have a significant impact on small businesses or other small entities.
Application information is required on a one-time basis to determine the eligibility of the applicant. Due to the highly competitive nature of the application process and short timeframes to conduct peer review, it is necessary to standardize information requested in the application process to facilitate eligibility determination. Information to be standardized within the forms includes a grantees’ historical context, HRSA priority and preference data, and institution trends in meeting agency objectives. The quantity, level of responses, and format of applicant responses currently varies tremendously. By providing uploaded forms, the revised format will reduce the burden to applicant reporting and provide information that can be consistently analyzed across programs.
Grantee performance reports must be submitted annually as a condition of receiving Federal funding. Reporting also satisfies GPRA annual reporting requirements and performance data for budget justification. If the performance metrics are not collected, BHPr will not be able to comply with requirements to report grantees’ activities and outcomes.
The proposed data collection is consistent with guidelines set forth in 5 CFR 1320.5(d) (2).
BHPr revised its performance reporting requirements based on further analysis of the agency’s information needs and its experience in collecting and analyzing data from prior years. All current grantees and the four BHPr Advisory Councils (i.e., Council on Graduate Medical Education, National Advisory Council on Nurse Education and Practice, Advisory Committee on Training in Primary Care Medicine and Dentistry, and Advisory Committee on Interdisciplinary, Community-Based Linkages) were invited to technical assistance webinars (Attachment H). Additional academic and professional organizations representing health professions were also provided information on the revision process and data collection instruments per request. BHPr hosted webinars for grantees and stakeholders to provide background information on the program reporting requirement process, importance of performance measures, the uses of the proposed performance metrics, timeline and due dates for grantee reporting. In addition to the above information each webinar was tailored to review data collection instruments for specific programs. The webinar allowed participants to ask questions and provide their views on the availability of data, frequency of collection, clarity of instructions and record keeping, disclosure or reporting format, and on the metrics to be recorded, disclosed, or reported. Participants were directed to provide additional comments through the federal register notice.
The public had the opportunity to comment during the 60- and 30- day public comment periods. A 60-day Federal Register Notice was published in the Federal Register on May 20, 2011, vol. 76, No. 98; pg. 29252 (Attachment I). There were 129 respondents to the federal register notice. The majority of comments received covered several topics. The number of comments received per topic is presented in Exhibit 1 below.
Exhibit 1. Thematic Responses Received from Public Comment Period |
|
Topics for clarification |
No. Comments Received |
Instructions for information collection instruments |
48 |
Additional elements to the data collection |
12 |
Definitions of various metrics |
77 |
Information and guidance about individual-level data collection, informed consent, and IRB approval |
72 |
Timeline for data collection |
25 |
Information on IT tools to reduce reporting burden |
85 |
Total comments received |
319 |
All comments were reviewed and considered in finalizing the data collection instruments and final guidance document. Based on public comments the following changes to the proposed annual data collection are summarized (See attachment J).
Clarified purpose and instructions for information collection instruments.
Established working group of experts in revising individual level data for future longitudinal studies.
Added program outcome metrics recommended by grantees.
Removed annual achievement level elements that grantees did not have direct records or data.
Delayed reporting of individual trainee data until the fourth quarter of the reporting cycle.
Clarified or developed definitions for metrics.
Pilot tested aggregate and individual trainee collection instruments with a subset of grantees (less than 10) to assess burden of data collection.
Clarified that IRB approval was not required for program evaluation.
Consultations:
Exhibit 2 presents the individuals that reviewed the instruments developed for the project:
Exhibit 2. Instrument Reviewers |
Physician Assistant Training in Primary Care Shenandoah University Dr. Anthony Miller, Principal Investigator Ph: (540) 545-7257 Email: [email protected] |
Academic Administrative Units in Primary Care Meharry Medical College Dr. Roger Zoorob -Principal Investigator Ph: (615) 327-6572 Email: [email protected] |
Primary Care Residency Expansion UMDNJ-New Jersey Medical School Dr. Theresa Liao, Principal Investigator Ph: (973) 926-6553 Email: [email protected] |
Preventive Medicine Residency The Research Foundation of SUNY-Stony Brook Dorothy Lane, MD, MPH Director, Preventive Medicine Residency Stony Brook, NY 11794-8036 Ph: (631) 444-2094 Email: [email protected] |
Public Health Training Centers (PHTC) Puerto Rico Florida PHTC Luz Maharaj , Principal Investigator Ph: (305) 348-5938 Email: [email protected] |
Area Health Education Centers (AHEC) National AHEC Organization (NAO) Committee on Research and Evaluation (CORE) Rob Trachtenberg, Executive Director, NAO Phone: 414-908-4953 x131 E-mail: [email protected] |
Diversity Programs Health Careers Opportunity Program (HCOP)/Centers of Excellence (COE): University of California, San Diego Sandra Daley Phone: 619-681-0642 Email: [email protected] |
Geriatrics Programs Janet C. Frank, DrPH Assistant Director for Academic Programs Multicampus Program in Geriatric Medicine & Gerontology 10945 Le Conte Avenue, Suite 2339 Los Angeles, CA 90095 Phone: 310.312.0531 Email: [email protected] |
Nurse Managed Health Clinics Bonita Ann Pilon, DSN, RN, BC, FAAN Senior
Associate Dean for Practice Vanderbilt
University, School of Nursing |
Other consultations in 2011 include:
Federation of Associations of Schools of the Health Professions (FASHP)
Primary Care Medicine working group 1: Faculty development, physician assistant, medical schools, residency program grantees
Primary Care Medicine working group 2: Faculty development, physician assistant, medical schools, residency, academic administrative units, and pre-doctoral program grantees
PHTC working group of grantees
Public Health Traineeship working group
AHEC grantee working group of grantees
HCOP- all grantee meeting working group
COE Program – all grantee meeting working group
Nurse Managed Health Clinic - all grantee meeting working group
Nurse Education, Practice and Retention - working group of grantees
Bureau of Primary Care-HRSA
HIV/AIDS Program - HRSA
Information collection and reporting issues were discussed during the consultations that lead BHPr to form additional working groups to resolve definition discrepancies and to propose information technology adjustments to reduce reporting burden.
No payments or gifts are to be provided to respondents.
The majority of data collected from the grantees is aggregate data. However, to support future longitudinal evaluation, individual-level data will be collected from a subset of programs reporting in the fourth quarter (beginning in April 2012) of their reporting cycle. BHPr data collection is supported by the following legislation:
Section 792 of the Public Health Service Act which authorizes HRSA to develop and maintain data on health professionals
Section 5103 of the Affordable Care Act which modified Section 761 of the Public Health Service Act requirements. Section 761 of the Public Health Service Act requires HRSA to establish a national internet registry of each grant awarded and a database to collect data from longitudinal evaluations.
The individual-level data will be matched outside of the web-based-system and then de-identified data will be stored back into the system. The data are protected following the Health Resources and Services Administration (HRSA) Management Policy and Procedures found in the HRSA Information System Security which meets or exceeds the DHHS policies, procedures, and practices to secure and protect the data from unauthorized access. Data will be kept private to the extent allowed by law.
This data collection is not considered research and is consistent with BHPr reporting requirements as a condition of grant award. BHPr does not anticipate that institutions will need to obtain IRB approval for this data collection. However, if necessary, grantees will obtain IRB approval from their respective institutions.
The DHHS requires that race and ethnicity be collected by all DHHS data collection instruments. The race and ethnicity data requested for this reporting is administrative data already collected by the grantees and does not require direct data collection from the trainee.
The number of FY 2012 annual performance respondents (2,300) has almost doubled (increased by 1,150) since the last OMB package. Annual performance respondents are based upon committed FY 2011 competing awards that use the PRGCA in FY 2011. The increase is primarily due to the number of new grantees (900) that would be subject to reporting. New grantees reporting in the system are either from new programs established in or funded from the Affordable Care Act, or from existing programs that are new to reporting in the Bureau-wide system. For example, programs like the Scholarships for Disadvantaged Students previously had separate data collection authorizations (OMB 0915-0149) will be consolidated into this data collection package.
Attachment C lists the programs in which each grantee must submit an annual performance report. A pretest of the application and performance forms was conducted with nine current grantees. Grantees were e-mailed the metrics and asked to estimate the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The burden estimate varies due to the variability in programs and grantee capacity. Results of the pretest show that program size, purpose, and complexity are factors for variance between programs. The number of trainees, grantee infrastructure, and grantee experience influence the variance among individual grantees within a program. Many grantees have the technological capacity to easily reformat their institution forms to easily submit trainee individual data, while other institutions anticipate more effort in gathering and submitting the data to HRSA.
Application forms are submitted on a one-time basis for competing grants. Application respondent estimates were based on an average of 2,500 applicants annually that applied for BHPR programs competed between FY 2009- FY 2011. The individual-level burden estimate ranges from 2 – 10 hours. The total of annualized estimate of burden for submitting application forms is 5,000 – 25,000 hours.
It is estimated that of the 2,500 applications received, 2,300 grantees are estimated to be awarded funding. Therefore, of the 2,300 anticipated FY 2012 annual performance respondents, 1,500 are estimated to submit program aggregate elements and program specific data, with an estimated 800 respondents anticipated to submit individual-level data.
The burden range estimate for submitting program aggregate elements and program specific data is 8.0-12 hours. Program respondent estimates were based on an average of 1,500 grantees. The range of annualized estimate of burden for submitting program aggregate and program specific data is 12,000 – 18,000 hours.
Twenty-two programs (800 grantees) will report individual trainee information. The individual-level burden estimate ranges from 2-8 hours. The range of annualized estimate of burden for submitting individual-level data is 1,600 – 6,400 hours.
Twenty dollars is a generally accepted wage rate for institution personnel responsible for completing the PRGCA. The estimated hour cost to all respondents to engage in all data request is a range of $515,640 - $1,126,321.
The estimated hour cost of one respondent that applies and is not funded at a rate of $20 per hour is an estimated annualized burden range of $40 – $400 per hour. The estimated hour cost of one respondent that applies, is awarded, and engages in all data collection required at a rate of $20 per hour is an estimated annualized burden range of $70 - $192 per hour.
Exhibit 3. Estimate of Annualized Hour Burden |
|||||
Instrument |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Burden Hours |
Application |
2500 |
1 |
2500 |
5 hrs |
12,500 |
PRGCA: Program Aggregate Data Collection |
1500 |
1 |
1500 |
8 hrs |
12,000 |
PRGCA: Individual-level Data Collection |
800 |
1 |
800 |
2 hrs |
1,600 |
Total |
|
|
4800 |
|
26,100 |
Exhibit 4. Estimated Annualized Burden Costs |
|||
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Institution personnel |
26,100 |
$20.00 |
$522,000.00 |
There are no capital and start-up costs to the respondents. Records used for this data collection are from exiting institution or grantee partner records.
Monitoring of the data base system is maintained within the Agency. Staff time is required (program staff and grants management personnel) to review the reports annually. Annual total cost of staff time is $138,448.00 as follows:
Program staff evaluates the grantee’s accomplishments on the program’s objectives and metrics. Experience indicates that this effort will require one full-time program staff at a GS 13 level for a total of $89,033.
Grants management staff evaluates the grantee’s accomplishments based on the budget requests of the project. It is estimated that this will require 33% of time for two staff at a GS 12 level for a total of $49,415.00.
The HRSA Web-based Enterprise system that enables grantees to obtain and submit reports will be redesigned for $2 million with annual IT maintenance of approximately $400,000. The HRSA Web-based Enterprise system was last updated in 2006 and the proposed upgrades will afford grantees with uploading and streaming features to reduce grantee burden as well as, provide project officers with analysis and administrative tools to increase program review efficiencies.
Program Change/Adjustment: BHPr is streamlining IT systems within the Bureau. Programs which historically maintained separate reporting systems are now integrated into the Bureau-wide system, hence increasing the number of grantees using the web-based reporting system.
There are currently 9,350 total burden hours approved by OMB for this activity. This request is for approval of roughly 26,100 burden hours, an increase of 16,750 hours. The increase is due to 1) the significant increase in the number of respondents for the academic year of 2011, 2) including the new and revised templates for applicants, and 3) additional performance information requested to meet new agency requirements or priorities.
However, this increase has been mitigated to a large extent by BHPr improving the electronic reporting processes by providing user-friendly templates. The new web-based reporting system features reduce the need for manual data entry thus reducing burden; includes the ability to pre-populate fields with previously entered data thus reducing date re-entry by the user; automates the calculation of total counts; and allows grantees the ability to upload data files, such as spreadsheets, to the web-based reporting system.
Application instructions and forms are revised to provide standard forms and clarity in the instructions in order to reduce the amount of data entry time in submitting.
BHPr is delaying the data reporting period for grantees submitting individual-level trainee data to April 1, 2012-June 30, 2012 instead of the beginning of reporting period. All reports are due on June 30, 2012, while the individual-level reporting is required over a shorter period of time. The delay in individual-level reporting allows BHPr to provide additional technical assistance to grantees on creating unique identification numbers and submitting individual trainee level data. This phased-in approach of collecting aggregated program data from July 1, 2011-June 30, 2012 and delaying the individual trainee data to a start date in April 2012 (fourth quarter) was well received by grantees and affords the grantees the opportunity to test the IT reporting tools before the performance report is mandatory.
There are no plans for publication beyond reporting results for compliance with the annual GPRA Report. This will include descriptive statistics and analysis of non-response/missing data on an annual basis. The results will be published in the HHS Online Performance Appendix as is required by GPRA. In addition, annual performance data including data related to the previous academic school year or previous project year may be included in the HRSA Online Performance Plan and Report and Budget submitted to Congress in February 2012. The reporting periods are identified for each table using the Electronic Handbooks (EHBs).
BHPr will display the expiration date for OMB approval for this data collection.
This information collection fully complies with the guidelines set forth in 5 CFR 1320.9. There are no exceptions to the certification statement.
This information provides a sample of grantee instructions and program reporting requirements. (See Attachment D)
Attachments
A - BHPr Data Collection Revisions
B - Summary of Revised Grant Application Forms
C - 2012 Programs Reporting in the Bureau Performance Management System
D - Sample Data Collection Instrument
E - BHPr Workforce Performance Measures
F - BHPr Performance Goals Model
G - BHPr Program Evaluation Strategy
H - Technical Assistance Webinar Schedule
I - 60-Day Federal Register Notice
J - 60-Day FRN Public Comments and Responses
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |