Supporting Statement A
Evaluation and Assessment of HRSA Teaching Health Centers
OMB Control No. 0915-0376
Extension
Terms of Clearance: None.
A. Justification
Circumstances Making the Collection of Information Necessary
The Teaching Health Center Graduate Medical Education (THCGME) program under Title III of the Public Health Service Act supports new and the expansion of existing primary care residency training programs in community-based settings. The primary goals of this program are to increase the production of primary care doctors who are well prepared to practice in community settings, particularly with underserved populations, and to improve the overall number and geographic distribution of primary care providers.
The legislation specifically requires that THC program award recipients report annually on the types of primary care residents trained, the number trained, the number who complete residency, the number who care for vulnerable populations, and any other information as deemed appropriate by the Secretary. The law states:
(1) ANNUAL REPORT. The report required under this paragraph for a qualified teaching health center for a fiscal year is a report that includes (in a form and manner specified by the Secretary) the following information for the residency academic year completed immediately prior to such fiscal year:
A) The types of primary care resident approved training programs that the qualified teaching health center provided for residents.
(B) The number of approved training positions for residents described in paragraph (4).
(C) The number of residents described in paragraph (4) who completed their residency training at the end of such residency academic year and care for vulnerable populations living in underserved areas.
(D) Other information as deemed appropriate by the Secretary.”
This program aims to increase the number of new primary care physicians and dentists trained in community-based settings who go on to practice in primary care and in rural and underserved settings. The THCGME program differs from traditional funding of GME programs, largely through Medicare, by requiring that the funding go directly to a community-based ambulatory patient care site. This program is significantly different than traditional Medicare GME which is paid largely to inpatient hospitals, and therefore the THCGME program is expected to incentivize a different model of training with the aim of producing primary care providers who are better trained to provide primary care and practice in community-based, often underserved, settings. The George Washington University (GW), through a competitive process, was awarded the Evaluation and Initial Assessment of HRSA Teaching Health Centers contract. The purpose of this contract is to conduct an assessment over a five-year period to better understand this model of community-based residency training and examine the outcomes of the THC program award recipients, in terms of production of primary care providers and providers who practice in underserved settings.
GW has been collecting information from THC residency program award recipients regarding the types of primary care residency programs and numbers trained, as well as information on curricular components that demonstrate community and primary care orientation. GW has also been directly gathering information from THC residents and alumni using a multi-staged survey format to establish a baseline census of the individuals who choose THC residency programs and determine the career outcomes of THC graduates. Research has shown that certain factors are associated with primary care and underserved career choices.1 At the individual level these factors include gender, age, rural background, student debt levels, and intent to practice in certain specialties or settings. Medical school level factors have also been shown to correlate with career outcomes, including primary care and rural experiences.2 Collecting this information is important both to assess whether THC residency programs are attracting those residents who are most likely to choose primary care and underserved careers, as well as to assess the relationship of these factors with future outcomes for THC residency programs.
Purpose and Use of Information Collection
Data Collection Instrument |
Subject (Recipient) |
Timeline |
|
Program manager |
Once for each program |
|
Matriculating resident |
Annual – July* |
|
Graduating resident |
Annual – June* |
|
Graduate physician (one year post graduation) |
Annual – June* |
* Surveys will be implemented annually with different cohorts of matriculating, graduating and alumni residents (academic years for residency programs run from July to June); surveys will be implemented by GW, under contract with HRSA, through the end of the 5-year evaluation contract and then provided as a resource for THC program award recipients to continue collecting information on their residents and graduates.
The Program Data Collection Tool provides information on residency programs that receive funding for THC residents. The Program Data Collection Tool will be administered one time to THC Program Directors. Annual updates are made on an as-needed basis. It will collect basic organizational and training characteristics of the programs (including program specialty, numbers trained, training sites, educational partners, and residency program financing), educational initiatives (particularly around training for changing health care delivery systems and community experiences), and health center characteristics (including current workforce and vacancies, clinical service provided by residents, and participation in workforce programs such as National Health Service Corps).
The tool addresses the following evaluation research questions:
What are the types of primary care resident approved training programs provided by the THC programs?
What number of approved training positions is being provided by the THC programs?
What advanced primary care delivery models (i.e. patient centered medical homes, inter-professional team-based care, quality improvement) are THC residents training in?
What community-based experiences are THC residents receiving?
What is the amount of training THC residents receive in the primary care setting?
See Appendix A for the Program Data Collection Tool. This information will be used to monitor program activities and inform program management within BHW. In addition, this information is critical to understand the characteristics of a new program and follow its progress over time.
Furthermore, three questionnaires have been developed for implementation with all THC residents at matriculation, graduation and one-year post-graduation: THC Matriculant Survey, THC Graduation Survey, and THC Alumni Survey. These three surveys will be administered to gather information on THC residents at different stages of their training and early clinical practice. The surveys are intended to identify the absolute number of primary care providers and providers that practice in underserved areas. This is a critical outcome measure for the THCGME program as THCs must be new or expanding residency programs. Therefore, THC graduates are new primary care providers above the number that would have been produced in the GME system prior to the THCGME program. The surveys also seek information about resident experience and satisfaction. As the THCGME program emphasizes a model of training that is different than the traditional hospital-based model, trainee satisfaction is an important outcome of the program. Program feedback questions also provide information on how well matched training is to the needs of primary care practice.
The tools address the following evaluation research questions:
What number and percent of the THC graduates practice in primary care (plan at graduation and one year after graduation)?
What number and percent of the THC graduates go on to practice in underserved settings (plan at graduation and one year after graduation)?
What are the characteristics of residents who choose THC programs, by demographics, intention to practice in primary care and intention to practice in rural and/or underserved areas?
Are there correlations between resident characteristics and the intention to practice primary care and/or practice in underserved settings?
How satisfied are residents with curriculum and enhanced programmatic features of the THC model?
How do residents suggest that the THC residency program can be improved?
The THC Matriculant Survey collects background information on THC residents to better understand the characteristics of individuals who apply and are accepted to THC programs (see Appendix B). The THC Matriculant Survey largely gathers demographic and background information of incoming residents. This information is the first opportunity to examine whether programs are recruiting residents consistent with the goal to produce physicians who will practice primary care in rural and underserved settings. The THC Graduation Survey gathers information on career plans (covered in an identifiable section), and on the quality of training received at the THC (covered in an anonymous section). Please see Appendix C for the THC Graduation Survey. The Graduation Survey is the first opportunity to assess plans to continue to practice in primary care and in underserved areas. The THC Alumni Survey collects information on actual career outcomes, including practice in primary care and in underserved settings following graduation (covered in an identifiable section), as well as feedback on the quality of training (covered in an anonymous section). See Appendix D for the THC Alumni Survey.
The purpose of the Program Data Collection Tool and the three individual level questionnaires is to collect the required reporting information in a standardized manner between all THC programs, and to examine characteristics of THC programs that are aligned with training in relevant and/or innovative health care delivery models. Data from the individual level surveys will be provided back to the THC program award recipients so that they can meet any additional reporting requirements to HRSA and receive program feedback from graduates. Survey tools will ultimately be made available to THC program award recipients to continue post the THC Evaluation contract period.
The questionnaires collect individual level information required by THC program award recipients to meet their annual reporting requirements. Implementation of standardized questionnaires will ensure data can be collated from all programs. If this information is not collected there will be no data to evaluate the effectiveness and impact of the THCGME program. To date, response rates have been high for the survey measures. Over 75% of residency programs have already completed the program data collection tool. Only THC residency programs that have not yet completed it will be asked (approximately 10 residency programs). The most recent alumni survey administration had a response rate of over 70%; the graduate survey response rate was 89%; and the matriculant survey was over 85%. Typical survey response rates for measures like the matriculant and graduate surveys have average response rates of around 30-35%, and external surveys like the alumni survey average 10-15%. Thus the interest and success of the THC measure administration has been quite good. As the THC program has just had its first full round of alumni surveys completed (as residency programs are three to four years in length), this extension is particularly important so the GW evaluation team can complete survey administrations with sufficient numbers to completely answer the evaluation research questions. Preliminary analyses of the data suggest that THC graduates are practicing in primary care as well as in underserved settings more than twice as frequently as national averages.
Use of Improved Information Technology and Burden Reduction
GW has developed data collection tools and questionnaires that utilize technology to administer, collect and analyze the data. The Program Data Collection Tool is implemented using fillable pdf forms. All of the responses (100%) for each individual level questionnaire (Matriculant, Graduation, and Alumni Surveys) will be collected and submitted electronically. Each of the three surveys will be completed using a secure online survey tool with built in skip logic to minimize the number of questions for respondents. Responses for both the Program Data Collection Tool and surveys will be downloadable as collated spreadsheets.
Efforts to Identify Duplication and Use of Similar Information
Some existing THC residency programs had existing surveys they were using. The survey instruments developed for the THC Evaluation specifically took into consideration the existing survey instruments – particularly the content and format of questions. These survey instruments have been developed to collect standardized information from THC trainees and graduates, as well as meet the needs of the THCs for program feedback. The survey instruments are ultimately made available to individual THC residency programs to continue collecting relevant training information and add to as needed. Most THC residency programs have adopted the THC Evaluation surveys (in lieu of any existing surveys they had been using) and use the information for their own program administration, for completing HRSA performance measures, and for tracking their alumni. Because of this situation, there is no duplication of measures and exceptionally high response rates for the THC Evaluation survey measures.
Impact on Small Businesses or Other Small Entities
The survey instruments will be implemented with individual graduate physicians. In all cases, these individual may be considered “small businesses” and therefore no additional short forms were developed. The surveys generally contain straightforward questions around demographics and practice plans or locations that should require minimal effort and time to complete. In addition, the online survey instruments include “skip logic” to allow respondents to skip questions that are not relevant based on their answers to other preceding questions.
Consequences of Collecting the Information Less Frequently
During the remaining THC Evaluation contract period (1-2 years) all residents will be expected to respond to the Matriculant Survey once. For those residents that graduate during the contract period, they will be expected to respond once to the Graduation Survey and for those that graduate at least one year prior to the end of the contract period, they will be expected to respond once to the Alumni Survey, one year after graduation. Timing of the survey is required to gather information about the characteristics of residents choosing and entering THC residency programs as well as the career outcomes of those trained in THC residency programs. A survey at graduation is the first opportunity to assess career plans; however, a follow up survey after graduation is needed to assess actual career outcomes.
Program Directors will complete a blank Program Data Collection Tool once.
There are no legal obstacles to reduce the burden.
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
In this request, all guidelines are met and this request fully complies with the regulation.
Comments in Response to the Federal Register Notice/Outside Consultation
Section 8A:
A 60-day Federal Register Notice was published in the Federal Register on October 14, 2016, vol. 81, No. 199; pp. 71108-09. There were no public comments.
Section 8B:
The resident surveys were developed based on survey instruments shared by some of the existing THCs as well as other existing similar surveys, such as the Washington, Wyoming, Alaska, Montana, Idaho (WWAMI) Family Medicine Residency Network survey, and the Association of American Medical Colleges (AAMC) matriculant and graduation surveys. The surveys were also developed to correlate with the HRSA Bureau of Health Workforce performance measures.
The GW team consulted the following individuals in developing the survey instruments:
Paul Ford, Department of Family Medicine, University of Washington ([email protected]) – Dr. Ford administers the WWAMI Family Medicine Residency Network alumni survey. He was consulted to discuss the content, implementation, and challenges faced in implementing the survey instrument among Family Medicine residency programs in the WWAMI Network. He was consulted in 2013.
Henry Sondheimer, Association of American Medical Colleges ([email protected]) – Dr. Sondheimer administers the AAMC Medical Student Matriculant and Graduation Surveys. He was consulted to discuss the content of these surveys and the potential for future collaboration in order to compare resident responses to medical student responses. He was consulted in 2013.
In both cases, survey questions were modeled after those in the WWAMI and AAMC surveys in order to allow comparison of responses – both individuals reported there were no proprietary concerns over using the same question formats.
Explanation of any Payment/Gift to Respondents
No payments or gifts will be given to the respondents.
Assurance of Confidentiality Provided to Respondents
All data will be collected through a secure online survey site or through secure password-protected e-mail. Program level data is aggregated and therefore by nature de-identified. Individual level data will be collected with personal identifiers (ex. e-mail addresses provided by the THC programs). Personal identifiers are needed in order link surveys completed over time to examine individual characteristics that are related to different career outcomes – for example, practice in rural or other underserved areas. In addition, identified individual level data will be provided back to the THC program directors in order for THC residency programs to provide accurate information for the HRSA performance measures. HRSA performance measures are required at an individual level, reported with a unique identifier created by the THC program. The survey instruments developed for the THC Evaluation will allow THC residency programs to gather individual level data in a standardized manner that can then be reported to HRSA for the required performance measure reporting. In all cases, respondents will be informed that surveys are identified and information will be provided back to their THC residency programs in an identified manner.
Justification for Sensitive Questions
The THC Matriculant Survey asks the respondent to identify their race/ethnicity. This question is important to determine the different demographic characteristics of the individuals entering into a THC residency program.
No information is gathered on social security number, sexual behavior and attitudes, alcohol or drug use, religious beliefs, and other matters that are commonly considered private.
Estimates of Annualized Hour and Cost Burden
This section summarizes the total burden hours for this information collection in addition to the cost associated with those hours.
12A. Estimated Annualized Burden Hours
Type of Respondent |
Form Name |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden Hours |
|
THC Program Directors |
Program Data Collection Tool |
10 |
1 |
8 |
80 |
|
Graduated THC Residents |
THC Alumni Survey |
200 |
1 |
0.33 |
66 |
|
THC Residents |
THC Matriculant Survey |
200 |
1 |
0.25 |
50 |
|
THC Residents |
THC Graduation Survey |
200 |
1 |
0.25 |
50 |
|
Total |
|
610 |
--- |
--- |
246 |
|
The estimated number of respondents for the THC Alumni Survey, THC Matriculant Survey, and the THC Graduation Survey was reduced from 300 in the 60-day FRN to 200 in the 30-day FRN and this supporting statement after consultation with the evaluation team. A reduction in the number of residents supported by the THCGME program as well as awardees dropping out of the program for various reasons are responsible for the change in number of respondents.
12B. Estimated Annualized Burden Costs
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
THC Programs |
80 |
$21.441 |
$1,715 |
Graduated THC Residents |
66 |
$86.952 |
$5,739 |
THC Residents |
50 |
$24.173 |
$1,208 |
THC Residents |
50 |
$26.144 |
$1,307 |
Total |
246 |
|
$9,969 |
1 Hourly rate determined using Labor Category ID CES6500000008
2 Hourly rate determined using Occupation Code 29-1062
3 Calculated based on a mean annual 1st year resident salary of $50,274 with a 40 hour work week3
4 Calculated based on a mean annual 3rd year resident salary of $54,373 with a 40 hour work week1
13. Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs
No additional recordkeeping or capital costs are expected for respondents. All data requested reflects basic program characteristics or individual demographics, practice characteristics, or program feedback. Therefore, no additional cost burden to respondents other is expected outside of the time required to complete the survey instruments.
14. Annualized Cost to Federal Government
The systems used to collect the data will be at GWU. It is estimated that the amount of staff time needed for the contract representative and review and approval of reports is .5 FTE at the GS-13 level—for a total of $46,000. Collectively the estimated annualized cost to the government in staff time is estimated to be $46,000.
15. Explanation for Program Changes or Adjustments
The current burden inventory for this information collection is 526 hours. This current request is for 246 hours. This is due to a reduction in the estimated number of respondents and as stated in the original submission of this information collection request, future years of implementation will require less time as respondents will be asked only to update the previous year’s data.
16. Plans for Tabulation, Publication, and Project Time Schedule
Data collected through the Program Data Collection Tool and individual level surveys serve a number of important purposes including strengthening program performance, responding to federal reporting requirements, and responding to congressional inquiries. Since programs are publicly-funded, data collected may be showcased in peer-reviewed articles, conferences, and/or reports published through and/or sponsored by HRSA. In the case of publication, all personally identified information will be aggregated and de-identified.
The process for cleaning, analyzing, and reporting data will consist of the following steps:
Step 1: Data cleaning. Data will be cleaned using a series of predetermined analytic rules within 30 days of receipt. Errors or discrepancies in data will be flagged and followed up with THC residency programs where appropriate.
Step 2: Analysis. Analysis of all data will be conducted under the THC Evaluation contract at the George Washington University for the duration of the contract period (approximately 2 more years). Analysis during this time period will be descriptive. Correlational analyses of the relationships between resident characteristics and career outcomes will also be conducted.
Step 3: Reporting. Data will be reported on an annual basis to HRSA in September of each year, at the time of the required annual reporting. Any additional data requests from the THC Evaluation HRSA project officers will be provided in a time period to be determined based on the status of the data collection activities and the requirements for analysis.
17. Reason(s) Display of OMB Expiration Date is Inappropriate
There is no request to seek exemption for display of the OMB expiration date.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification.
1 Rieselbach RE, Crouse BJ, Newhausen K, Nasca TJ, Frohna JG. Academic medicine: A key partner in strengthening the primary care infrastructure via teaching health centers. Acad Med. 2013; 88:1835-1843.
2 Rieselbach RE, Crouse BJ, Newhausen K, Nasca TJ, Frohna JG. Academic medicine: A key partner in strengthening the primary care infrastructure via teaching health centers. Acad Med. 2013; 88:1835-1843.
3 AAMC Survey of Resident/Fellow Stipends and Benefits, 2012. Available at: https://www.aamc.org/download/312786/data/2012stipendsurveyreportfinal.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Instructions for writing Supporting Statement A |
| Author | Jodi.Duckhorn |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |