SUPPORTING STATEMENT
Part A
AHRQ ACTION III – Measurement for Performance Improvement in Physician Practices
Version: March 31, 2016
Agency for Healthcare Research and Quality (AHRQ)
2. How, by Whom, and for What Purpose Information Will Be Used 7
3. Use of Improved Information Technology 7
4. Efforts to Avoid Duplication 8
5. Involvement of Small Businesses 8
6. Consequences if Information Collected Less Frequently 8
8. Federal Register Notice and Outside Consultations 8
9. Payments/Gifts to Respondents 9
10. Assurance of Confidentiality 9
11. Questions of a Sensitive Nature 9
12. Estimates of Annualized Burden Hours and Costs 9
13. Estimates of Annualized Respondent Capital and Maintenance Costs 11
14. Estimates of Annualized Cost to the Government 11
16. Time Schedule, Publication and Analysis Plan 12
17. Exemption for Display of Expiration Date 13
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999, is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care;
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and,
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
This two year project is an important first step to fully understanding measurement for performance improvement in medical groups. This exploratory research is expected to set the stage for informing future research and policy discussions, both of which could ultimately have a more direct impact on providers, payers, and patients. As a critical first step, however, this research breaks new ground in an important area of health care research by looking at what the current landscape is to better understand how medical groups are using measurement internally to improve performance and what that means to them, and how internal measurement relates to external measurement obligations, and identifying where the gaps are.
Project success for this exploratory work will be seen in a more complete understanding of the current landscape of performance measurement, gleaned through an environmental scan, expert input, and qualitative data collection. Ultimately, success will be measured by our ability to answer the research questions stated below that are guiding this research project.
The overall goal of AHRQ’s Measurement for Performance Improvement in Physician Practices project is to identify the current gaps in our knowledge about how practices are using data, if at all, for performance improvement. AHRQ has developed this project to address the lack of current evidence on internal performance measurement in medical groups, identifying the following project objectives and their corresponding research questions:
Specific Project Objectives
Identify gaps in the research literature regarding management for performance improvement in medical groups.
What gaps exist in the research literature regarding management for performance improvement in medical groups?
Identify specific measures/metrics, benchmarks, and comparisons used internally by medical groups to assess performance and support improvement activities, as well as how these measures are derived and reported.
How are internal performance measures derived and reported? What specific measures, benchmarks, and comparisons are used?
Describe how internal measurement activities/measures are used in medical groups to support improvement in individual, team, or organizational performance including, but not limited to, how these activities are tied to “internal” financial incentives.
How are measures used to support internal management and improvement processes?
What factors, both internal and external, drive efforts to use measurement to improve medical group performance?
What additional activities support use of internal performance measures?
Identify types of costs and other types of burdens (e.g. staff resources, IT resources, etc.), directly related to internal measurement and reporting activities. Assess the feasibility of capturing information on costs and burdens of internal and external performance measurement, and, if feasible, collect data on internal and external performance measurement’s actual costs and other associated burdens.
What are the perceived benefits of internal measurement activities? What types of costs and other burdens are directly associated with internal measurement? How feasible is it to specify actual costs of reporting?
Based on the findings, identify implications, potential impacts, and future research opportunities for payers, regulators, and medical groups regarding internal measurements for performance improvement.
What implications does evidence on internal measurement for performance improvement have for payers, policy makers, executives in delivery systems, and clinical leaders?
How have physicians responded to these measurement processes?
Efforts to improve performance among health care providers through measurement and reporting have evolved over time and have taken many forms and many names. For example, Triple Aim, Public Reporting, Performance Measurement, Quality Improvement, Pay for Performance are all common concepts today.
Increasingly, physicians and other providers work in new types of complex arrangements with ownership structures and incentives that are not well understood (Casalino, 2014)1. Most health care providers, including medical groups, are monitoring their performance using a wide array of quality measures that reflect care processes, clinical outcomes, and patient experiences. Little is known, however, about how providers internally make use of measures that are required by external bodies for payment or reporting. Nor is it known what other measures providers collect and use to improve performance. Broadly, AHRQ seeks to improve the efficiency and value of U.S. health care by understanding how physicians in practice use measurement to improve outcomes and reduce costs. This project aims to fill this knowledge gap. In doing so, it may also inform payment and reporting initiatives by providing indications of the degree to which providers view externally mandated measures as valuable for their internal quality assessment and reporting efforts.
As an initial step in understanding the landscape of measurement for performance improvement, this research will look to understand how medical groups define and measure performance improvement.
To meet the goals of this study, AHRQ has contracted with the Westat team, comprised of Westat and their partners at RAND. For this study, AHRQ will conduct field data collection through semi-structured in-depth interviews. The unit of analysis for this work is the medical group. To understand measurement for performance improvement in each medical group, AHRQ will interview up to 5 administrators and frontline clinicians per medical group included in the research. Interviews with both administrators and clinicians will be facilitated using the same protocol (Attachment C). As further discussed below, given the different levels of involvement and experience with internal performance measurement, interviews will vary in detail and thus length. But, as AHRQ works to uncover the story of each medical group involved in the study, the same guiding protocol will apply. AHRQ will audio-record and professionally transcribe each interview conducted. And, all interviews will be loaded into Dedoose for coding and analysis.
AHRQ will supplement this data collection with the collection of artifacts used for internal performance measurement efforts. Artifacts may include:
Checklists;
Scorecards;
Dashboards;
Assessment tools;
Measure specifications (including numerator and denominator criteria, rules for inclusion and exclusion, and information available on performance rate calculation and/or risk adjustment);
PQRS/HEDIS measures specifications adapted for internal use; or
Redacted screenshots of the online tools.
AHRQ will collect such artifacts via site visits when interviews are conducted in person. When conducting interviews over the phone, AHRQ will ask for such materials to be sent in pre-supplied mail-out boxes.
There are two specific rationales for performing qualitative exploratory research to understand how medical groups use measurement for internal performance improvement before proceeding to a nationally-representative quantitative study. First, there is no existing national roster of medical groups from which to draw a credible, national random sample. AHRQ anticipates that such a national roster will become available in a few years and is pursuing separate lines of research to create this roster. Second, even if such a roster already existed, the best way to gather data on this study’s research questions from a nationally-representative sample is unclear (i.e., we lack knowledge of which underlying constructs to measure, how to phrase questions for maximum understandability and salience, and which lines of inquiry are unlikely to yield much knowledge). The proposed exploratory qualitative research will clarify these important methodological points and guide the possible future development of refined data-gathering tools such as surveys and short interview guides that maximize the knowledge gained from future nationally-representative quantitative research efforts while making efficient use of medical group leaders’ time.
To support this data collection and ensure that the research done for this study has the greatest possible benefit for the field and for the future of health care research, AHRQ will also engage in the following activities:
Technical Expert Panel (TEP) feedback – A Technical Expert Panel (TEP) of eight members will be used to: (a) leverage information on key informants; (b) consult on the environmental scan; (c) identify participating medical groups; (d) provide input on the development of interview materials; (e) provide input on the analysis plan and results; and, (f) assist with disseminating findings. The diversity of the TEP will allow AHRQ to gain deeper insight and varying perspectives. A list of TEP members can be found in Attachment B.
Dissemination activities – AHRQ will produce at least one manuscript suitable for publication in a peer-review journal; and produce a PowerPoint presentation of all research findings and implications for presentation to stakeholders, such as the ACTION III partners. AHRQ will submit abstracts to scientific conferences, such as Academy Health’s Annual Research Meeting, the American Association of Public Opinion Research (AAPOR), the International Congress of Qualitative Inquiry (ICQI), AHRQ’s annual conference, the Patient-Centered Primary Care Collaborative (PCPCC) annual conference, the Society for General Internal Medicine’s annual conference, etc. AHRQ will also submit abstracts to other health care quality conferences, including but not limited to conferences run by the Institute for Healthcare Improvement (IHI), Leapfrog, and the CMS Quality Conference.
AHRQ will also disseminate findings and implications through a variety of innovative approaches. Such possible approaches include collaborative learning circles of participating medical groups, podcasts, webcasts, and videos to share findings and efficiently reach a wide audience. AHRQ will discuss these various types of optional dynamic and innovative dissemination approaches available for consideration, and will consult both the TEP and our key informants on possible dissemination strategies. Regardless of dissemination method, AHRQ will ensure professional, engaging, accessible, plain language materials tailored to the appropriate audience.
This work is being conducted by AHRQ through its contractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on health care and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
The information collected in the data collection effort will be used for one main purpose:
Identify the current gaps in internal measurement in physician practices. The results from the data collection will give AHRQ a snapshot on the current practices being undertaken for internal performance measurement and inform best next steps to move beyond this exploratory research phase.
The intended target audiences expected to benefit most from the project include the medical groups using this information to improve performance, the health care professionals who work in these medical groups working to improve their care to patients, and the patients that can benefit from improved care. One way this research could benefit these audiences is by informing payment and reporting initiatives by providing indications of the degree to which providers view externally mandated measures as valuable for their internal quality assessment and reporting efforts.
The study data collection will be conducted using WebEx to facilitate the interviews when they cannot be conducted in-person. The use of WebEx will significantly reduce the cost of the study since travel will not be necessary for all interviews. WebEx facilitates secure recording and use of WebEx facilitates AHRQ’s ability to observe data collection as appropriate. The study will also use the digital qualitative research tool, Dedoose. This software allows all data from the study to be housed in a secure, cloud-based environment, which provides the full project team access to the data and the ability to view and analyze the data efficiently and completely, and it ensures efficient and complete transmission of all data and analyses to AHRQ at the conclusion of the study.
As previously mentioned, this study is an important first step to fully understanding measurement for performance improvement in medical groups. This exploratory research is expected to set the stage for informing future research and policy discussions, both of which could ultimately have more direct impact on providers, payers, and patients. As a critical first step, however, this research breaks new ground in an important area of health care research.
The study is entirely exploratory in nature, and thus, there are no current studies that are similar in nature. Although there have been prior studies on how external measurement is used in medical groups, the use of internal measurement and the motivation behind and burdens related to use of internal measurement has not been studied, especially not in a larger, diverse national sample like the one proposed for this project.
It is possible that some small group practices participating in this study will be small businesses. However, there are not any specific small business goals or considerations.
This effort is a one-time data collection study.
The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on Page 21874 of Federal Register, April 13, 2016 for 60 days (see Attachment A).
8.b. Outside Consultations
To guide the development of all Measurement for Performance Improvement in Physician Practices products, a TEP has been assembled. As indicated above, TEP activities include review of the interview protocol and analysis plan, and feedback regarding data collection. The TEP contains 8 members from various geographic locations who are experts on varying aspects of performance measurement. Attachment B lists the TEP members and their current affiliations.
We are not proposing to provide any remuneration to participants for participation in this research effort. Given experience with the pilot test, this will not be a barrier to recruitment. This is a topic of keen interest, and medical groups want to share their experiences and thoughts on performance improvement measurement. The Groups also appreciate the exploratory nature of this research and look forward to seeing the findings. AHRQ is working to determine the best ways to share findings with participating medical groups to ensure they receive this benefit for participation, as noted above.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. Identifiers such as name, organization name, and e-mail address will be collected to facilitate data collection and schedule interviews. Once data collection is complete, personal identifiers will be removed from the data and destroyed. A copy of the consent form is included as Attachment D.
We do not believe there are questions of a particularly sensitive nature included in the survey. The unit of analysis is the medical group, so only information about Groups is being collected. No questions were identified as sensitive in nature by pilot participants.
Exhibit 1 shows the estimated annualized burden hours for the participants’ time to take part in this research. To recruit medical groups to participate, AHRQ will engage Groups in a short call to assess interest and obtain a commitment to participate. AHRQ expects to need to reach out to approximately 100 medical groups to obtain a sample of 45 Groups that are conducting some type of measurement for internal performance improvement, are interested in taking part, and are able to take part during the data collection window. In-depth, semi-structured qualitative interviews will then be conducted with up to 5 staff members at 45 medical groups using a single protocol (see Attachment C). AHRQ will target small (2-9 eligible professionals (EP)), medium (10-24 EPs), and large (25+ EPs) medical groups from across the Unites States. As this is exploratory research, we will include a mix of small, medium, and large groups but it is not expected this will be an even distribution of groups across each of the three practice size types. Initial available information indicates more large groups are currently engaging in performance improvement and fewer small groups have the resources for this at this time. For that reason, we anticipate more large and medium size groups to be included in this research. The goal is to recruit approximately 3 administrators and 2 frontline clinicians in each Group, understanding that depending on the size and organization of the medical group staff members may operate in multiple roles.
Based on the pilot study conducted for this project, AHRQ estimates that the recruitment call will average 15 minutes, and that the longest interviews will be 1.5 hours. These longest interviews will be with the highest level administrators working on internal performance measurement at the most complex medical groups. AHRQ believes these will be the largest medical groups that are part of complex systems and payment relationships. These complex organizational relationships will require more time to understand in order to understand the place, role, and operation of internal measurement for performance improvement within the Group. For equivalent administrators from medium and small groups, AHRQ estimates the longest interviews will be 1.25 hours. For all other administrators and frontline clinicians, AHRQ estimates the interviews will be 1 hour.
The total annualized burden is estimated to be 295 hours. Again, interviews with both frontline clinicians and all medical group administrators will use the same protocol (see Attachment C). The screening call will be an informal conversation where AHRQ looks to learn if the medical group self identifies as using measurement for performance improvement and provides consent to take part. AHRQ will answer any questions the medical group has about the study on this call and confirm some basic, publicly available background information about the Group that AHRQ has obtained is accurate and up-to-date. This background information will help put the information learned during the interview in better context. The types of background information AHRQ is looking at includes medical group size, organizational structure, specialty mix, and payment relationships.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Hours per response |
Total burden hours |
Frontline Clinicians (Attachment C) |
90 |
1 |
90 |
Medical Group Administrators (Attachment C) |
235 |
|
|
Medical group administrators: Administrator with authority to agree to participate in the study (Screener call) |
100 |
0.25 |
25 |
Medical group administrators: Initial, highest level administrators (Attachment C) |
45 |
1.5 |
67.5 |
Medical group administrators : All other administrators (Attachment C) |
90 |
1.25 |
112.5 |
Total |
325 |
NA |
295 |
Exhibit 2 shows the estimated annualized cost burden associated with the participants’ time to take part in this research. The total cost burden is estimated to be $27,270.45.
Exhibit 2. Estimated annualized cost burden
Interviewee Type |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Frontline clinicians |
90 |
$103.54a |
$9,318.60 |
Medical group administrators |
205 |
$87.57b |
$17,951.85 |
Total |
295 |
NA |
$27,270.45 |
a Based on the average hourly wage for one physician (29-1060; $103.54).
b Based on the average hourly wage for one Chief Executive (11-1011; $87.57).
*National Industry-Specific Occupational Employment and Wage Estimates, May 2014, from the Bureau of Labor Statistics (available at http://www.bls.gov/oes/current/naics4_621100.htm [for Offices of Physicians, NAICS 622100]).
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. The only cost to the respondent will be that associated with their time to respond to the information collection, as shown in Exhibits 1 and 2.
Exhibit 3 shows the estimated total and annualized cost for this project. Although data collection will last for less than one year, the entire project will take about 2 years. The total cost for this project in its entirety is approximately $645,272. The total cost of the actual data collection activities is approximately $186,650. And, the annualized cost is estimated at approximately $192,620 in year one and approximately $452,652 for year two to the end of contract. Note that the only activities noted in Exhibit 3 that will extend to both years of the project are Project Management and Overhead. As a result, for all other tasks, the annualized cost equals the total cost of the task as the task will be completed within the span of a single year.
Exhibit 3. Estimated total and annualized cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$168,200 |
$168,200 |
Data Collection Activities |
$186,650 |
$186,650 |
Data Processing and Analysis |
$108,940 |
$108,940 |
Publication of Results |
$63,780 |
$63,780 |
Project Management |
$48,840 |
$24,420 |
Overhead |
$82,900 |
$41,450 |
Exhibit 4. Federal government personnel cost (FY 16 and FY 17)
Role/Personnel/Salary* |
% of Time |
Cost |
Project Officer/Staff Fellow GS 13 Step 10 $119,794 |
25% |
$29,948 |
Senior Advisor/Senior Social Scientist GS 15 Step 10 $160,300 |
5% |
$8,015 |
Total |
|
$37,953 |
Note that these oversight costs are included in “Overhead” in Exhibit 3.
This is a new activity.
As soon as OMB approval is received, field interviews will begin. The estimated time schedule to conduct the interviews and analysis is shown below:
Field interview data collection, including recruitment (6 months)
Data analysis, report, and final summary (3 months)
Dissemination of important findings (3 months)
The final dissemination of the findings will be made public following the analysis.
The data collection and analysis are qualitative in nature. Analysis on the data will be conducted using the qualitative software tool, Dedoose.
No exemption is being requested.
Attachment A: Federal Register Notice
Attachment B: Technical Expert Panel
Attachment C: Interview Protocol
Attachment D: Consent Form
1 Casalino LP. Identifying key areas for delivery system research. Rockville, MD: Agency for Healthcare Research and Quality; 2014. AHRQ Publication No. 14-0024-EF.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | AHCPR |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |