SUPPORTING STATEMENT
Part A
Studying the Implementation of
a Chronic Care Toolkit and Practice Coaching
In Practices Serving Vulnerable Populations
Version: December 30, 2009
Revised: August 26, 2010
Agency of Healthcare Research and Quality (AHRQ)
TABLE OF CONTENTS
A. JUSTIFICATION
A. JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
An important part of AHRQ’s mission is to disseminate information and tools that can support improvement in quality and safety in the U.S. health care community. This proposed information collection supports that part of AHRQ's mission by further refining the practice coaching delivered in conjunction with a previously developed toolkit, Implementing Integrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics. AHRQ requests that the Office of Management and Budget approve, under the Paperwork Reduction Act of 1995, AHRQ’s intention to collect information needed to determine whether practice coaching is effective in facilitating adoption of the Chronic Care Model for improving treatment and management of chronic medical conditions by primary care physicians, especially those who care for underserved populations. This project is being conducted pursuant to AHRQ’s statutory authority to conduct and support research on health care and on systems for the delivery of such care, including activities with respect to quality measurement and improvement and with respect to clinical practice, including primary care and practice-oriented research. 42 U.S.C. 299a(2) and (4). This ICR is not an evaluation to fulfill a statutory requirement. This project will be conducted by AHRQ through a contract with the University of Minnesota.
The circumstances that make collection of this information necessary are that chronic disease accounts for a huge proportion of morbidity, mortality, and health care costs in the United States, yet the acute care orientation of the U.S. health care system has not served Americans with chronic conditions well. Recognizing this, the Robert Wood Johnson Foundation (RWJF) funded “Chronic Care Initiatives in HMOs” in 1993. Under this program the Group Health Cooperative of Puget Sound produced a model for effective management of chronic disease in managed health care called the Chronic Care Model (CCM). In 1998 RWJF also instituted “Improving Chronic Illness Care” (ICIC), a national program that operates improvement collaboratives to improve care for a chronic condition, and provides technical assistance and support to organizations interested in improving chronic illness care. That same year the Health Resources and Services Administration (HRSA) launched the Health Disparities Collaboratives (HDC) to change primary health care practices in order to improve the health care provided to underserved, uninsured, and underinsured Americans and to eliminate health disparities.i The HDC initiative adapted the Institute for Healthcare Improvement’s (IHI) Breakthrough Series, creating a regional infrastructure to sustain and support dissemination of improvements in care. The ICIC also used the IHI Breakthrough Series model. Both the ICIC and HDC developed a variety of tools to assist health care delivery sites.
The implementation of the CCM through collaboratives has been evaluated and found to improve the quality of care.ii,iii,iv,v,vi,vii Participants in IHI collaboratives have also reduced waste in the form of shorter intensive care unit stays, less waiting time, and fewer doctor’s office visits, emergency room visits, and unnecessary hospitalizations.viii
Although 1500 physician practices in the U.S. and internationally have been involved in CCM quality improvement efforts, most patients still do not receive their chronic care in accordance with CCM. One factor affecting CCM implementation has been that having teams attend collaborative meetings (three two-day meetings over a nine-month period) is burdensome, especially for under-resourced providers. An attempt to use the Internet as a virtual collaborative met with disappointing results.ix Another barrier to adoption of the CCM in settings that serve vulnerable populations is the scarcity of resources to implement and sustain the CCM.
In 2006 AHRQ contracted with the RAND Corporation, Group Health’s MacColl Institute, and the California Health Care Safety Net Institute (SNI) to develop a toolkit that informs safety net providers on how to redesign their systems of care along the lines of the Chronic Care Model while attending to their financial realities. The result was Implementing Integrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics.x The Toolkit was piloted in two California safety net clinics. Recognizing that merely distributing the Toolkit was unlikely to foster adoption of CCM, the intervention included six months of practice coaching delivered by the MacColl Institute. Practice Coaches (PC) are health care or related professionals who help primary care practices in a variety of quality improvement and research activities. PCs made two site visits to each site and participated in weekly team meetings by phone. They also interacted with the sites through email and phone contact.
The lack of documentation available on coaching led to the development of a practice coaching manual, which was funded by AHRQ through a contract with the RAND Corporation. Development of the Coaching Manual entailed conducting a literature review, interviewing practice coaching experts, and incorporating evaluation results from the coaching provided in conjunction with the Toolkit. The Coaching Manual was published in the winter of 2009. The literature review and interviews revealed that there are a number of different models of practice coaching. However, knowledge is scant about how practice coaching is best performed, under what conditions practice coaching is most successful, and the costs of coaching and being coached. Pilot testing the Toolkit with a low-intensity practice coaching strategy proved insufficient to encourage practices to use the Toolkit independently. The Toolkit was subsequently streamlined based on pilot sites' reports that the initial Toolkit was not easy to use. This project, Studying the Implementation of a Chronic Care Toolkit and Practice Coaching in Practices Serving Vulnerable Populations, will explore the implementation of the revised Toolkit along with a more intensive practice coaching strategy, providing lessons on methods to improve chronic care in clinical practices that serve vulnerable populations.
This project will include the following data collections:
Key Informant Interviews with providers, staff and practice coaches from 20 safety net practices that participate in the practice coaching intervention. These will be used to describe the process and content of practice coaching, perceived changes from the coaching intervention at the practice, provider and patient levels, factors that impeded or facilitated the coaching intervention and implementation of practice changes through the coaching process, overall satisfaction with practice coaching, and recommendations for improvement (see Attachments B, C and D).
Primary Care Practice Profile (PCPP). This questionnaire will be completed by a single individual at each site, either the medical director or chief administrator and will provide an overview of each replication site that will help place intervention activities and outcomes in context for each site. It covers demographics of patients served, patient flow, disease health outcomes, most frequent diagnoses, most frequent referrals, number of staff by discipline, staff and patient satisfaction, processes of care, and organizational processes. (See Attachment E)
Physician Practice Connections-Readiness Survey (PPC-RS) – This questionnaire asks about the presence of 53 practice systems in 5 of the 6 domains of the Chronic Care Model: clinical information systems (information systems, presence of registry or organized database, and systematic monitoring of patient population); decision support (clinician reminders and alerts for lab tests, and visits or guidelines related to individual patient care), delivery system redesign (services for managing patients with chronic illness involving multiple clinicians and care between visits), health care organization (performance tracking and feedback, process of using clinical information systems to aggregate and report on key indicators, and use of data for benchmarking performance and informing QI activities), and clinical quality improvement (presence of formal processes to assess care, develop interventions, and use data to monitor the effects). (See Attachment F)
Assessment of Chronic Illness Care (ACIC) – The ACIC is contained in the Toolkit and yields subscale scores and a total score. Subscale scores reflect CCM components and include: community linkages, self-management support, decision support, delivery system design, information systems, and organization of care. (See Attachment G)
Change Process Capability Questionnaire (CPCQ) – The CPCQ assesses 30 factors and strategies that experienced quality improvement leaders ranked as most important for successful implementation. A recent validation study found good predictive validity. Items correlating with the PPC-RS were eliminated after the initial validation study so there is little to no overlap across the two measures. In addition to changes in the content of care (CCM components), these also include organizational will for change (Priority) and capacity and skill in the conduct of the actual change processes and strategies. (See Attachment H)
Primary Care Staff Satisfaction Survey (PCSSS) – This questionnaire assesses staff satisfaction with their work environment. It consists of 8 4-point likert scale items and 2 open-ended questions, and was developed by the Institute for Healthcare Improvement. (See Attachment I)
Patient Assessment of Chronic Illness Care (PACIC) – The 20-item PACIC consists of five subscales which assess components of the CCM: patient activation, delivery system design/decision support, goal setting, problem-solving/contextual counseling, and follow-up and coordination. (See Attachment J)
Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS) – This questionnaire assesses patient experiences in three areas: getting appointments and healthcare when needed; how well doctors communicate, and courteous and helpful office staff. (See Attachment K)
Chart Audits -- Chart audits will be conducted at baseline, the end of the 10 month coaching intervention, and at 3-month follow-up to assess changes in patient care quality over the course of the intervention. HEDIS or similar care quality indicators that are specific to and appropriate for the change projects selected by the participating practices will be used. A chart abstraction form will be developed to collect these data (see Attachment L for an example). This data collection will be performed by the task order project staff, not practice personnel, and will not impose a burden on the participating sites. Therefore, OMB clearance is not required for this data collection.
This project is expected to contribute to the achievement of AHRQ’s Prevention/Care Management Portfolio goal of, “supporting the evidence base and implementation activities to improve primary care and clinical outcomes through: health care redesign, clinical-community linkages, self-management support, and integration of health IT.” Because providers that serve vulnerable populations serve a disproportionate share of minority patients, this project is also expected to contribute to the reduction of racial and ethnic disparities. According to the 2007 National Healthcare Disparities Report, Hispanics and Black/African Americans fared worse than white non-Hispanic Americans on measures of chronic care quality.
This is a mixed qualitative and quantitative study that is designed to allow us to learn lessons about the feasibility and practicality and impact of implementing a Chronic Care Toolkit and practice coaching in practices that serve vulnerable populations. The focus of this evaluation is on assessing the implementation of the Toolkit and coaching intervention in multiple practices and its potential contributions to improvements in care quality, organizational capacity for quality improvement, and patient and staff satisfaction. The qualitative methods and quantitative data collected in the study will be used to document implementation of the intervention, ensuing quality improvement activities, and theirchanges in care quality, organizational capability for change, and patient and staff satisfaction impact across 20 safety net practice sites that are participating in the study.
If the implementation in this study is successful, the information collected by this project can be used to improve the delivery and efficiency of chronic care through the use of the Chronic Care Model in numerous primary care practices that serve vulnerable populations by using the Integrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics and Coaching Manual combined with practice coaching. If the implementation in the study is not successful, the lessons learned will be used to inform the development of other strategies for managing chronic medical care for vulnerable populations. A successful implementation will be defined as: 1) consistent participation of practice members with the practice coach and participation in quality improvement activities contained in the Toolkit or suggested as a result of the meeting with the coach over the 10 month coaching period; and 2) evidence of improvements in care quality, organizational capacity for engaging in quality improvement, or improvements in staff and patient satisfaction from pre to post test determined through key-informant interviews (reports of perceived improvements across elements of the CCM and organizational capacity for change), staff and patient surveys (increases in incorporation or receipt of care consistent with elements of the CCM), and changes in HEDIS or indicators specific to chronic care activities/conditions the practices select as their area of focus for the project. An unsuccessful implementation will be defined as: 1) the consistent failure (greater than 50%) of the practice to participate in scheduled activities with the practice coach and involving activities contained in the Toolkit or suggested by the coach; and 2) failure to identify areas of perceived improvement care quality, organizational capacity, or patient and staff satisfaction. Note: This study does not employ a control group and so evidence of improvement across the identified domains cannot be directly attributed to the Toolkit and coaching intervention. However, evidence of improvements from pre to post across the identified domains may be suggestive of potential benefit from the intervention and can be used to guide design of future studies as well as future interventions.
Specific research questions that will be addressed in evaluating whether the implementation of the Toolkit and coaching intervention was successful are:
1. What aspects of the CCM and/or business strategies from the Toolkit were implemented as a result of the Toolkit plus practice coaching intervention?
2. What changes do practices make to self-management support, delivery of evidence based medicine, and efficiency of care as a result of the activities undertaken as part of the practice coaching intervention?
3. Do practices’ capacity to manage and sustain change in their organizations increase as measured by Solberg’s Change Process Capability Questionnaire which assesses organizational capacity to engage in improvement activities?
4. Does the quality of diabetes or other practice-identified chronic care conditions improve after practices participate in the practice coaching intervention as measured by practitioner and staff surveys, key informant interviews, and increased adherence to HEDIS or other appropriate care indicators? Please note: If changes are detected in quality of care or other variables being assessed in this evaluation, no attribution of causality will be drawn to either the program or processes implemented, or to the Toolkit availability due to lack of a control or comparison group.
5. Do adult patients report increased satisfaction with care and increases in activities related to the four CCM dimensions: patient management self-support, delivery system design, decision support, and information systems?
6. Does provider satisfaction with the organizational and care environment improve after the intervention?
7. Do practice characteristics such as size, location (urban/rural) and structure (Academic Health Center, Community Health Center, Federally Qualified Health Center, Rural Health Center) appear to be associated with acceptance and implementation of the practice coaching intervention and Toolkit, what aspects of practice coaching and the Toolkit are used, what aspects of the CCM are targeted, and what types and degree of change is seen in processes and quality of care post intervention?
A.3. Use of Improved Information Technology
As described in Section A.1, there has been substantial research on the implementation of the CCM through collaboratives, where it has been evaluated and found to improve the quality of care.xi,xii,xiii,xiv,xv,xvi Further, as described earlier in Section A.1, in 2006 AHRQ contracted with the RAND Corporation, Group Health’s MacColl Institute, and the California Health Care Safety Net Institute (SNI) to develop a toolkit Implementing Integrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics.xvii, which was piloted in two California safety net clinics with the assistance of practice coaches. However, the RAND/MacColl/SNI team suggested the development of a practice coaching manual because a literature review and interviews revealed that there are a number of different models of practice coaching, but knowledge is scant about how practice coaching is best performed, and under what conditions practice coaching is most successful in terms both of a practice’s ability to participate in the intervention, and its impact on key variables such as care quality, organizational capacity for sustained improvement, and staff and patient satisfaction..
No previous studies have been conducted of this Toolkit and coaching intervention in CHCs and so this ICR is not duplicative of other previous research.
In their 2005 review, Mold and Apsy identified 47 articles on practice facilitation or coaching. Of these, 25 measured the impact of interventions involving PCs on patient care outcomes and supported the potential effectiveness of PC as a strategy for planned change. Frijiling et al (2002) found that feedback reports from PCs increased rates of two (diabetic foot and eye examinations) out of 7 patient care outcomes for diabetes. In a randomized, controlled trial involving 98 physicians, Dietrich et al (1992) found that a PC-led intervention involving assessment, training, and use of decision support tools increased office system interventions related to cancer screening and preventive services. Similarly, in Margolis et al’s (2004) RCT with 26 practices, nurse facilitators increased appropriate requests for hemoglobin tests by 99% compared to a 23% decline in control sites. Baskerville, Hogg and Lemelin (2001) tested a multi-component intervention involving the use of nurse facilitators to improve preventive care performance and found statistically significant improvements on an overall index of preventive performance (11.5%), up-to-dateness (7.2%) and service inappropriateness (4.4%). Kinsinger et al (1998) conducted a study assessing the impact of a PC facilitated intervention on rates of breast cancer screening in 62 primary care practices from rural counties in North Carolina. Interestingly, while the intervention achieved improvements in care processes, these improvements did not translate to actual increases in mammograms or clinical breast exams (CBEs). This study highlights the not uncommon experience in quality improvement efforts that process improvements may fail to translate to actual improvements in care and health outcomes. PCs can provide the type of tailored hands-on support that is needed to identify factors that may be impeding improvement in these areas.
Goodwin et al (2001) tested a practice-tailored approach to increasing delivery of preventive services in an RCT involving 77 Ohio family practices (STEP-UP) and found significant changes in some areas of preventive care but not in others (immunizations). Finally, Bryce et al (1995) evaluated the impact of an audit facilitator on pattern of diagnosis and treatment of childhood asthma in 12 practices. At 2-year follow-up, there were significant increases in asthma consultations, new diagnoses of asthma and more past diagnoses reaffirmed in intervention vs. control practices. Consistent with findings that implementing the CCM can increase costs in primary care settings, costs of care at the intervention sites increased slightly. However, hospitalization costs declined. This study illustrates the very real calculus CHCs and other safety-net sites must grapple with, where the cost savings resulting from improved care in their practices accrues to other organizations but not their own.
As can be seen from the above efforts to indentify duplication, the questions that the project is intended to address have not been definitively answered by previous information collection. In fact, the previous information collection identified the need to collect information to answer theThe questions that will be addressed by this project will support development of improved methods for supporting practices in providing chronic care that can be more widely disseminated and implemented in clinical settings, including those serving vulnerable populations.
While large safety net practices deliver care to substantial numbers of low-income and uninsured patients in the U.S., a substantial portion of patients receive their health care in small and solo practice settings. In Los Angeles County for example, more than 80% of managed MediCal patients enrolled in L.A. Care, one of the largest public health plans in Los Angeles, are cared for in small practices where there are two or fewer FTE Primary Care Providers) (Seidman, 2009, personal communication). Thus it is very important to include these small practices in this study, to better understand the impact of PC in these environments also. Small safety net practices, specifically small practices with fewer than 3 Primary Care Provides (PCPs) will be included in the study to allow examination of how the process and potential outcomes of practice coaching may vary by practice size. To reduce burden on these small entities, clinic administrators and PCPs will be consulted to arrange a data collection schedule that they believe will be minimally disruptive to their work day. Based on past experience working in these smaller practices, spreading data collection over a period of 2 to 3 days so that each encounter is of shorter duration (3 20-minute sessions rather than 1 60- minute session) and coordinating these with times during the day when there are fewer patients to the clinic (which varies by practice but is often immediately after lunch and again in late afternoon) helps to reduce or eliminate disruption and burden from data collection on the staff and PCPs.
This project is a one-time information collection effort only.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
A. Federal Register Notice
As required by 5 CFR 1320.8(d), a notice was published in the Federal Register on date, February 1st, 2010 for 60 days (See Attachment M). One comment was received and is shown below, along with AHRQ's response:
Public comment:
From:
jean public [mailto:[email protected]]
Sent: Monday,
February 01, 2010 10:12 AM
To:
Lefkowitz, Doris C. (AHRQ); [email protected]; [email protected];
[email protected]; [email protected]
Subject: PUBLIC COMMENT ON FEDERAL REGISTER
THIS DEPT PROJECT IS A CESSPOOL OF OVERSPENDING. THIS ENTIRE PROJECT IS OBVIOUSLY NOT WORKING. HOW MANY ETERNITIES SHOUDL AMERICAN TAXPAYERS FUND THIS NONSENSE. SHUT DOWN THIS ENTIRE PPOJECT AND LAY OFF THE MGT OF THIS PROJECT AND WORKERS. THIS DEPT IS SO NON PRODUCTIVE THAT IT HURTS. SHUT DOWN AND SUNSET PLEASE.
JEAN PBULIC 15 ELM ST FLORHAM PARK NJ07932
AHRQ's response:
AHRQ thanks the public for taking the time to provide comments on proposed information collections. As always, AHRQ we will supervise the project closely to maximize efficiency and ensure that the project benefits those with chronic illness.
B. Outside Consultations
There were no outside consultations by the LA-NET team. AHRQ had 3 internal reviewers comment on the research design as initially proposed. In developing this project, AHRQ also consulted with the Commonwealth Fund project officer that led the Fund’s Safety Net Medical Home Initiative.
No payment or gifts will be given to respondents.
Individuals and organizations contacted will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and/or social security number, will not be collected.
No sensitive questions will be asked in this information collection.
Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this 2 year study. Key informant interviews will be conducted with practice coaches at mid-point in the intervention and again at the end of the intervention. The intervention is defined as dissemination of the Toolkit combined with support from a practice coach to support quality improvement activities and development of organizational capacity for quality improvement in the practices. Key informant interviews will also be conducted with up to 3 primary care providers and 2 other staff members from each of the 20 practices (10 per year) prior to start of the intervention, and again at 3-month follow-up after the intervention is completed. Each interview takes about 1 hour.
The Primary Care Practice Profile will be administered once and will be completed by one staff person from each practice and takes 30 minutes to complete. The Physician Practice Connections-Readiness Survey (PPC-RS) will be completed pre, post and at 3-month follow-up by three individuals from each of the 20 practices (individuals with the appropriate knowledge to complete the survey will be identified by the medical director of each site). It takes 90 minutes to complete. The Assessment of Chronic Illness Care (ACIC) will be completed by 4 staff and 4 primary care providers per practice at pre, post and 3-month follow-up and takes 30 minutes to complete. The Change Process Capability Questionnaire (CPCQ) will be completed by 4 staff and 4 primary care providers per practice at pre, post and 3-month follow-up and takes 15 minutes to complete. The Primary Care Staff Satisfaction Survey (PCSSS) will be completed by 4 staff and 4 primary care providers per practice at pre, post and 3-month follow-up and takes 15 minutes to complete. The Patient Assessment of Chronic Illness Care (PACIC) will be completed by 3,000 adult patients (1,500 annually) with chronic illness and requires 15 minutes to complete. The Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS) will be completed by 3,000 adult patients (1,500 annually) with chronic illness and requires 45 minutes to complete. Both patient surveys will be administered to adult patients with a chronic disease who receive care at the practices during a 2-day data collection period immediately before, immediately after, and at 3-month follow-up. The surveys will be administered during the post visit period in the wait room, by a bi-lingual Spanish-English research assistant. The total annualized burden hours are estimated to be 1,984 hours.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of Respondents |
Number of responses per respondent |
Hours per response |
Total Burden hours |
Key informant interviews with practice coaches |
2 |
2 |
1 |
4 |
Key informant interviews with providers (3 per practice interviewed twice) |
10 |
6 |
1 |
60 |
Key informant interviews with staff (2 per practice interviewed twice) |
10 |
4 |
1 |
40 |
Primary Care Practice Profile (PCPP) |
10 |
1 |
30/60 |
5 |
Physician Practice Connections-Readiness Survey (PPC-RS) (3 per practice x 3 times) |
10 |
9 |
1.5 |
135 |
Assessment of Chronic Illness Care (ACIC) (8 per practice x 3 times) |
10 |
24 |
30/60 |
120 |
Change Process Capability Questionnaire (CPCQ) (8 per practice x 3 times) |
10 |
24 |
15/60 |
60 |
Primary Care Staff Satisfaction Survey (PCSSS) (8 per practice x 3 times) |
10 |
24 |
15/60 |
60 |
Patient Assessment of Chronic Illness Care (PACIC) |
1,500 |
1 |
15/60 |
375 |
Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS) |
1,500 |
1 |
45/60 |
1,125 |
Total |
3,072 |
na |
na |
1,984 |
Exhibit 2 shows the estimated annualized cost burden associated with the respondent's time to participate in this study. The total annualized cost burden is estimated to be $60,714.
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of Respondents |
Total Burden hours |
Average Hourly Wage Rate* |
Total Cost Burden |
Key informant interviews with practice coaches |
2 |
4 |
$42.00 |
$168 |
Key informant interviews with providers |
10 |
60 |
$77.64 |
$4,658 |
Key informant interviews with staff |
10 |
40 |
$32.64 |
$1,306 |
Primary Care Practice Profile (PCPP) |
10 |
5 |
$77.64 |
$388
|
Physician Practice Connections-Readiness Survey (PPC-RS) |
10 |
135 |
$77.64 |
$10,481 |
Assessment of Chronic Illness Care (ACIC) |
10 |
120 |
$55.14** |
$6,617 |
Change Process Capability Questionnaire (CPCQ) |
10 |
60 |
$55.14** |
$3,308 |
Primary Care Staff Satisfaction Survey |
10 |
60 |
$55.14** |
$3,308 |
Patient Assessment of Chronic Illness Care (PACIC) |
1,500 |
375 |
$20.32 |
$7,620 |
Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS) |
1,500 |
1,125 |
$20.32 |
$22,860 |
TOTAL |
3,072 |
1,984 |
na |
$60,714 |
*Based upon the mean of the average wages, May 2008 National Occupational and Wage Estimates accessed on December 14, 2009 at: http://www.bls.gov/oes/current/oes_nat.htm#b29-0000National Compensation Survey: ** Average for 4 staff ($32.64/hr ) and 4 physician clinicians. ($77.64/hr).
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 shows the estimated total and annualized cost to conduct this research. The total cost over two years is estimated to be $600,000.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$162,744 |
$81,372 |
Data Collection Activities |
$92,994 |
$46,497 |
Data Processing and Analysis (20%) |
$92,994 |
$46,497 |
Publication of Results |
$23,248 |
$11,624 |
Project Management |
$92,994 |
$46,497 |
Overhead |
$135,026 |
$67,513 |
Total |
$600,000 |
$300,000 |
A.16. Time Schedule, Publication and Analysis Plans
The following table shows the correspondence of the research questions with the data collection instruments to be used and the items within each instrument that are relevant to the research question.
Research Question |
Instruments Used |
Relevant Items |
What aspects of the CCM and/or business strategies from the Toolkit were implemented as a result of the Toolkit plus practice coaching intervention? |
Key informant interviews at practices–semi-structured interview guide |
Key informant interview for practice coaches item 3
|
What changes do practices make to self-management support, delivery of evidence based medicine, and efficiency of care?
|
PPC-RS ACIC
|
Key informant interview staff/provider item 3 Key informant interview for coaches –item 4 PPC-RS – all items ACIC - all items |
Do practices’ capacity to manage and sustain change in their organizations increase? Does provider and staff satisfaction improve? |
CPCQ Key informant interviews |
Key informant interview staff/provider item 3 Key informant interview for coaches – item 4 CPCQ all items PCSSS |
Does the quality of diabetes care or other practice-selected index condition improve after practices participate in the practice coaching intervention? |
CPCQ CAHPS-PCA Key informant interviews |
CPCQ all items CAHPS-PCA items 1-37 Chart audit |
Do adult patients report increased satisfaction with care and increases in activities related to the four CCM dimensions? |
CAHPS-PCA PACIC |
CAHPS-PCA all items PACIC all items |
Does provider satisfaction with the organizational and care environment improve after the intervention? |
CPPP |
CPPP all items |
Do practice characteristics or prior level of CCM implementation appear to be associated with acceptance and implementation of the practice coaching intervention and Toolkit what aspects of practice coaching intervention and the Toolkit are used, what aspects the CCM are targeted, and what types and degree of change is seen in processes and quality of care post intervention? |
PCPP Key informant interviews at practices–semi-structured interview guide |
Key informant interview staff-provider - all items Key informant interview for coaches – all items PCPP all items ACIC all items PPC-RS all items PACIC all items CAHPS all items CPCQ all items Chart audit checklist |
Analysis plan
The purpose of this study is to examine the acceptability, feasibility and potential effectiveness of the CCM Practice Coaching Intervention and Toolkit by studying the implementation of practice coaching in 20 safety net practices in California. It is not an experimental study to determine the efficacy of the intervention. The intervention is defined as dissemination of the Toolkit combined with provision of on-site practice coaching support for quality improvement. Data analysis will focus on describing the implementation as well as perceived and potential benefits of the intervention at the individual practice level and comparing how implementation and potential effects benefits varied across the 20 replication sites. Quantitative data will be collected for this study including staff and provider assessment of CCM implementation (ACIC), patient assessment of the receipt of care consistent with the CCM (PACIC), staff and provider assessment of organizational capacity for quality improvement (CPCQ), staff (PCSSS) and patient satisfaction (CAHPS), care quality (select HEDIS indicators) and provider assessment of the degree to which the practice reflects elements of the patient centered medical home as defined by the NCQA.
These data will be . collected for the study will be analyzed at the individual practice level using either statistical or heuristic guidelines to describe whether they improved, stayed the same or got worse. If heuristic means are used, the criteria will be determined in advance and based upon clinically important change as determined by the practice in consultation with the research team. In some instances, statistically significant changes may not reflect changes that are clinically significant, meaning they result in real improvements in care or organizational processes. In other instances, small changes that do not achieve statistical significance may be clinicially sgnificant. Heuristic criteria developed by each practice will be used to evaluate the import and potential real value of any observed changes in care, organizational capacity or satisfaction as a means of assessing the potential value of changes occurring during the course of the intervention. Any non-statistically significant findings reported from this evaluation will include language clarifying this fact. If the criterion is to be statistical and the outcome measure is expressed as a rate, then a z-test will be used for differences in proportions to determine if the change from pre to post is significant at the standard alpha. If the outcome measure is a continuous variable, independent sample t-tests will be used.
The second step of the analysis is to look at patterns of change across all 20 practices thus treating each practice as a replication. The changes being measured will include changes in processes of care (increased implementation of components of the CCM) and organizational capacity for engaging in quality improvement a nd on. The analysis is for descriptive purposes. Attributions of causality will not be made. Improvementsin pa ach practice will be rated as either improved, stayed the same or got worse on their individually identified criterion. ed on HEDIS quality indicators or those of another well recogn source. An examples of these criteria might be HEDIS indicators for diabetes care (annual foot exam, annual eye exam, appropriatel timed visits and lab tests). Care issues that practices are expected to be interested in include areas such as management of asthma, diabetes, preventive services, cardiovascular disease, depression, hypertension, obesity among others.
Please note that higher frequency of testing does not equate directly with improvements in care, is not always associated positively with improved outcomes, and is not in and of itself an effective method for assessing improvements in care. Because of this, we will use HEDIS measures to assess quality, not frequency of testing. Frequency of testing will be considered only to the degree to which it is included as a part of the HEDIS measures.
HEDIS is a tool developed by the National Committee for Quality Assurance (NCQA) to measure performance on important dimensions of care and service. It consists of 71 measures across 8 domains of care and is used by 90 percent of health plans in America to measure quality, and frequently used by researchers to assess improvements in care. More information on HEDIS measures can be accessed online at: http://www.ncqa.org/tabid/187/default.aspx
In the diabetes example provided on page 17 of the supporting statement, HEDIS measures for quality of diabetes care include (but are not limited to): a) percent of diabetic patients at practice receiving an annual foot exam, b) percent of patients at practice receiving annual eye exam, and c) percent of patients receiving annual HbA1c test. So for practices in this project that opt to focus on improving chronic care for diabetes, we will use these measures to assess improvements or declines in quality. Practices that show an increase in percentage of patients receiving care consistent with these measures will be counted as improved and visa versa.
Our focus will be only on consistency with HEDIS measures. Care that extends beyond that recommended in HEDIS will not be considered in this project. For example, a patient who receives the annual foot exam as recommended in HEDIS and in addition to this then receives an additional 3 foot exams in the year, will be considered to have received care consistent with the HEDIS standard of an annual foot exam, but will not be considered to have received incrementally better care than another patient who received only the single annual foot exam per HEDIS standards.
The majority of the variables that willoul used to explore possible improvements in these areas are expected to be categorical. Improvements in patient outcomes are not being assessed.
Each practice will be rated as either improved, stayed the same or got worse on their individually identified care criterion. The use of 3 categories was chosen to reflect the level of precision that we thought was reasonable given the nature of the observations involved and the fact that different practices would be choosing different criteria to evaluate. We were looking for a level of precision which would give us a common metric among the various possible change criteria.
The results of these individual analyses will be assessed using a Chi Square test of Goodness of Fit where the null hypothesis would be that a third of the sample would fall into each cell if the pattern was truly random. If the universe of possible change states is defined as “stayed the same”, “got better” or “got worse”, then a truly random outcome would specify that each of the conditions would have equal probability of occurrence. The chi square test of goodness of fit takes this condition as the null against which to test the distribution of sample outcomes. In the present case, we would hope that the distribution of the sample outcomes would shift toward the “improved" category and this would result in a distribution of outcome states that is no longer .33, .33, .33. For example with 21 practices the null would hypothesize a distribution of 7 practices in each outcome state. If the sample data had a distribution of 3 practices in the ‘got worse’ category, 6 practices in the stayed the same category and 12 practices in the got better category, the null could be rejected at the .05 level f significance with a chi square of 6.0 and 2 df. Other distributions of sample data would also be possible that would also lead to a rejection of the null hypothesis of equal probability.
Thresholds will be individually defined and specified at the practice level and will include careful operational definitions so that each outcome state can be reliably determined. For example, the criterion may be that a practice has to have more than a 20% increase in # of specific types of evaluations to be considered ‘improved’ and that a reduction of 10% in the number of evaluations would be categorized as ‘got worse’, while staying within the -10% to +20% band would result in a category of ‘stayed the same’.
The results of these individual analyses will be assessed using a Chi Square test of Goodness of Fit where the null hypothesis would be that a third of the sample would fall into each cell if the pattern was truly random. With 20 practices this is approach does not have high power, but the patterns of change across independent practices at different times linked to similar interventions, accompanied by clear descriptions of each intervention should allow us to evaluate the overall effectiveness of our intervention. Also – specific patterns of change that are particularly relevant to this study are more likely to show significant results. The actual results of individual chi-square analyses are dependent on both how many can be assigned to the ‘got better’ cell and the distribution of results among the other 2 cells. If they are fairly even (e.g. as many stayed the same as got worse) then many more must be in the ‘got better’ cell for a significant result. If, however, only 1 or 2 practices ‘get worse’, some ‘stay the same’ and at least 10 ‘get better’, there would be a significant chi square. Therefore if the intervention at least insures that no one gets worse, most get better, and if they don’t get better – at least stay the same, there would be statistically significant results.
Descriptive Analyses. Means and confidence intervals (adjusted for clustering as needed) will be computed for all continuous study variables. Frequencies and percentages will be computed for categorical variables. These descriptive analyses will further be stratified by practice type (CHC, FQHC, RHC, small practice, AHC) or other dimensions of interest to AHRQ (size) and measurement occasion (pre, post, 3 months post).
Analysis of Change. For continuous measures collected at the individual level (staff or patient satisfaction, adherence to clinical care model, etc.), mixed models analysis of variance will be used to assess change over time while accounting for clustering at the practice level. Practice characteristics and any unplanned variation in intervention intensity will be added to these models as covariates to assess their relationship to change in the dependent measure. Baseline compliance ratesliance (present/not present) with HEDIS (or other relevant care indicator) will be calculated for the quality of care and patient outcomes indicators at the practice level and will be compared to follow-up rates using the Z test for a single proportion where the baseline rate will define the null condition and the follow-up rate will be the alternate condition. Compliance will most frequenof compliance with HEDIS indicators for diabetes might be assessed as follows: annual foot exam (present), annual eye exam (not present), planned visit (not present), etc. One-sided tests will be used since there is no expectation that rates will worsen after the intervention. For continuous measures collected at the individual level (staff or patient satisfaction, adherence to clinical care model, etc.), mixed models analysis of variance will be used to assess change over time while accounting for clustering at the practice level. Practice characteristics and any unplanned variation in intervention intensity will be added to these models as covariates to assess their relationship to change in the dependent measure.
Qualitative Analysis. Interview data will be entered into NVivo and analyzed for content and theme using methods recommended by Crabtree and Miller (1999). Interview data will be used to provide a detailed description of the PC process at each site, to provide a context for the quantitative data collected at each site, to allow for a qualitative comparison of how the PC intervention at each site was similar to or different from those of the other 19, and to allow us to examine qualitatively how the PC process and associated changes in care may have varied across dimensions such as practice size and type and to generate hypotheses and research questions to guide future work in this area.
Ability to attribute causality. Because this is a non-randomized study design without a control or comparison group, causality cannot be attributed to the Toolkit plus practice coaching intervention.
Ability to detect which practice characteristics and which change strategies are associated with successful implementation and impact. The project budget it not large enough to allow us to work in a large enough number of practices to determine statistically which practice characteristics and which change strategies are associated with successful implementation and impact of the Toolkit-practice coaching intervention. However, there will be more than sufficient information to develop hypotheses related to both of these questions that can be incorporated as part of the Practice Coaching research agenda that will also be developed under this Task Order.
Analysis of Qualitative Data
Qualitative data will be analyzed for content and theme using methods recommended by Crabtree and Miller and Strauss and Corbin. Expert member checks with practice staff, coaches and providers will be conducted to assess the validity and reliability of finding.
Preliminary Dissemination plan
A report describing evaluation findings will be developed that will describe: 1) The course of the intervention across the 20 sites and observed variations in its implementation across these sites (for example, variations in degree of practice participation in the intervention (hours allocated, buy-in, content, activities engaged in , et), variations in ways practices engaged with or used the toolkit and practice coach, variations in focus for the various quality improvement activities, and variations in approach by practices in undertaking the activities – for example use of a teamlette vs. a full quality improvement team); 2.) Degee of observed changes in implementation of the CCM, organizational capacity for change, and staff/patient satisfaction based on interviews and surveys and variations across sites in degree and nature of change; 3) Barriers and facilitators to implementation and use of the Toolkit and practice coach and variations in these across sites; 4) Perceived value and feasibility of use of the Toolkit and practice coaching intervention and variations in this across sites; 5) Lessons learned in implementing the intervention that may have relevance to future efforts. A thorough discussion of the limitation of the study will be included in all reports. The following methods will be used to disseminate findings and products from this project: Post findings and reports on Web sites of organizations such as the Community and Migrant Health Centers, Migrant Clinician’s Network, Health Disparities Collaboratives, AHRQ’s Health Care Innovation Exchange and Institute for Healthcare Improvement, the National PBRN Resource Center, LA Net’s, NACHC, and other relevant organizations; Publish results in journals such as the Journal for the Poor and Underserved and the American Journal of Public Health and professional journals such as Health Care for the Homeless Research Update; Present at conferences such as the National Association of Community Health Centers, the North American Primary Care Research Group and AHRQ’s Annual PBRN conference; Promote among other PBRNs beginning with the PRIME Net, OKPRN, ePCRN and NRN meta-networks; Engage organizational partners such as the Association of American Medical Colleges, IHI, and the National Association of Community Health Centers to aid in dissemination. In addition, AHRQ proposes to deliver a pre-conference workshop on practice coaching and the CCM at the AHRQ National PBRN Conference each year for 3 years to engage conference participants in discussions around the practice and science of practice coaching and to provide opportunity for knowledge, skill building and multi-PBRN projects involving practice coaching in the future.
AHRQ does not seek this exemption.
List of Attachments:
Attachment A – Healthcare Research and Quality Act of 1999
Attachment B – Key informant interview guide for practice coaches
Attachment C – Key informant interview guide for providers
Attachment D – Key informant interview guide for staff
Attachment E – Primary Care Practice Profile (PCPP) Questionnaire
Attachment F – Physician Practice Connections-Readiness Survey (PPC-RS)
Attachment G – Assessment of Chronic Illness Care (ACIC) Questionnaire
Attachment H – Change Process Capability Questionnaire (CPCQ)
Attachment I – Primary Care Staff Satisfaction Survey (PCSSS)
Attachment J – Patient Assessment of Chronic Illness Care (PACIC) Questionnaire
Attachment K – Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS) Questionnaire
Attachment L – Chart Audit Abstraction Form
Attachment M – Federal Register Notice
SUPPORTING STATEMENT
Part B
Studying the Implementation of
a Chronic Care Toolkit and Practice Coaching
In Practices Serving Vulnerable Populations
Version: January 6th, 2010
Agency of Healthcare Research and Quality (AHRQ)
TABLE OF CONTENTS
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods 1
B.2. Information Collection Procedures 6
B.3. Methods to Maximize Response Rates 10
B.4. Tests of Procedures 12
B.5. Statistical Consultants 13
B.1. Respondent Universe and Sampling Methods
The primary implementation unit for this study is the safety net practice, predominately Community Health Centers (CHCs). The population of CHCs in the U.S. is 7000 based on figures maintained by the National Association of Community Health Centers (NACHC). CHCs are health center service delivery sites defined as “community, migrant, homeless, and Public Housing Health Centers that are non-profit, community-directed health care providers serving low income and medically underserved communities” (NACHC 2009). Currently CHCs serve over 18 million people throughout the United States.
This study will use purposive sampling to select sites for this study. It is impractical to select a probability sample from all of the eligible safety-net practices in the United States, since the research budget limits the project to hiring only 4 practice coaches (PCs). The level of funding available for this data collection does not permit hiring additional PCs throughout the country or having the available PCs travel extensively. Thus, the study sites must be within reasonable traveling distance of the available PCs.
Although this qualitative study will use a purposive sample, the lessons learned from a study with 20 CHCs in California will help inform quality improvement and health disparity reduction efforts both in California and in CHCs across the country. The results are likely to have relevance to other areas in the U.S. because of the basic similarities in populations served by CHCs and resources available to CHCs throughout the country.
Ten urban/suburban CHCs and 10 rural safety net practices will participate in this study from Los Angeles (N=8), San Francisco (N=2) and Northern California (N=10). Selection was based on willingness to participate, proximity (the site is within 1 hour of the PC to allow for regular on-site visits), and practice characteristics (size -small, medium, large; location-urban, suburban, rural; structure – Community Health Centers, Federally Qualified Health Centers, Academic Health Centers, and Public Health Centers). The Table below contains the names of the 20 organizations that will participate in the study.
Participating practices
Name Structure Location
Los Angeles Area
Asian Pacific Health Care Venture, Inc. FQHC 1530 Hillhurst Avenue
Los Angeles, CA 90027
Urban (LA)
Cleaver Family Wellness Clinic CHC 4368 Santa Anita Ave.
El Monte, CA 91731
Suburban (LA)
Clinica Monsignor Oscar Romero FQHC 123 South Alvarado Street
Los Angeles, CA 90057
Urban (LA)
Community Health Alliance of Pasadena (CHAP) FQHC 1855 N. Fair Oaks Ave., Suite 200
Pasadena, CA 91103
Suburban (LA)
Family Healthcare Centers of Greater Los Angeles CHC 6501 S. Garfield Ave. Bell Gardens, CA 90201
Urban (LA)
QueensCare Family Clinics
FQHC 1300 North Vermont Avenue, Suite 1002 Los Angeles, CA 90027-0005
Urban (LA)
Saban Free
Clinic CHC 90048-3476
Urban (LA)
6405 Beverly Boulevard, Los Angeles CA
Valley Community Clinic FQHC 6801 Coldwater Canyon Ave
North Hollywood, CA 91605
Suburban (LA)
San Francisco Area
Lifelong Medical Center
FQHC PO Box 11247
Berkeley, CA.
Urban/Suburban
(SF)
San Francisco General Hospital Academic 995 Potrero Avenue, San Francisco CA 94110
Redwood Community Health Coalition
Alexander Valley Regional Medical Center FQHC 6 Tarman Drive, Cloverdale, CA 95425
Alliance Medical Center FQHC 1381 University Avenue, Healdsburg CA 95448
Coastal Health Alliance CHC 3 6th Street
PO Box 910
Point Reyes Station, CA 94956
CommuniCare Health Centers FQHC 2051 John Jones Road
Davis, CA 95616
Community Health Clinic OLE FQHC 1141 Pear Tree Lane, Suite 100
Napa, CA 94558
Marin Community Clinic FQHC P.O. Box 1868
Novato, CA 94948-1868
Petaluma Health Center
FQHC 1301 Southpoint Blvd, Suite A Petaluma, CA 94954-6858
Ritter Center
CHC 16 Ritter St, PO Box 3517, San Rafael 94912
Sonoma County Indian Health Project Tribal Health Council 4400 Auburn Blvd., 2nd Floor
Sacramento, CA 95841
Sonoma Valley Community Health Center CHC 430 West Napa St., Suite F, Sonoma, CA 95476
The respondent universe for this study is the staff and providers from the 20 participating safety net practices (N=10 per practice – 5 staff and 5 primary care providers, for a total of N=200 across the 20 participating practices); the practice coaches who will deliver the intervention (N=4); and patients with chronic disease who receive care from the 20 participating practices (N=150 per practice, or 3000 across all 20 practices).
Staff and provider interviews and surveys. Surveys. Within each participating site (practice location), all staff members and primary care providers who work at least 50 percent of full-time will be included in the study and will be asked to complete surveys at the beginning and end of the study period. The average number of part timer or greater staff and primary care providers per site is 5 staff and 5 primary care providers for an average of 10 per site. This gives an estimated 200 potential respondents across the 20 sites. Key informant interviews. Up to two staff and 3 providers from each of the 20 safety net practices will be asked to complete two key-informant interviews each. The first key informant interview will occur immediately before the start of the intervention and will ask them about the implementation of the chronic care model at their practices, factors impeding or facilitating its implementation, and their perception of the practice’s capacity to support practice improvement activities to increase implementation of the Chronic Care Model. The second interview will occur immediately after the intervention and will ask them about their experiences with and perceptions of the impact of the practice-coaching intervention on implementation of the Chronic Care Model in their practices. The key-informant interviews with staff and primary care providers will use semi-structured interview guides that are provided in this application and will require approximately 1-hour of respondent time to complete.
In addition, all staff and providers employed 50% time or greater in the participating practices will be asked to complete descriptive surveys of the degree to which their practice has implemented the Chronic Care Model, its organizational capacity for change, and work satisfaction. The surveys staff and providers will be asked to complete are the: Assessment of Chronic Illness Care (ACIC), Change Process Capability Questionnaire (CPCQ), and Primary Care Staff Satisfaction Survey.
Two staff and one primary care provider per site (total individuals responding N=60 across 20 sites) will be asked to complete the Physician Practice Connections-Readiness Survey (PPC-RS) at pre, post and 3-month follow-up to assess implementation of components of the patient-centered medical home.
Finally, one staff person per site (total individuals responding N=20 across 20 sites) will be asked to complete the practice profile at the start of the intervention.
Practice Coach Interviews. The four coaches who deliver the practice coaching intervention to the 20 practices will complete key-informant interviews at mid-point in the intervention (5 months) and at the end of the intervention (10 months). The interview will ask the coaches about their observations about the process of coaching and how it varied by practice (each will coach between 4 and 8 practices each over the course of the intervention), the content of the coaching and how it varied by practice, factors that facilitated or impeded implementation of the Chronic Care Model and how these varied by practice, and perceived impact of the coaching on implementation of the Chronic Care Model and organizational level support to sustain current and future improvements in patient care and how this varied by practice. The key-informant interviews with practice coaches will use semi-structured interview guides that are provided in this application and will require approximately 1-hour of respondent time to complete.
Patient surveys. A total of 150 adults receiving care for a chronic disease at each of the 20 practices will be asked to complete a short patient satisfaction survey. The surveys will be administered three times, pre-intervention, post-intervention, and 3-month follow-up. They will be administered in the waiting room over two consecutive days to any adult with chronic disease who receive care the practice during a two-day period immediately before the start of the intervention (N=50 per practice), immediately after the intervention (N=50 per practice) and at 3-month follow-up (N=50 per practice). The target accrual for each practice for each data collection period will be 50 for a total of 150 per practice or 3000 patients across all 20 practices surveyed by the end of the study. The survey will be available in both Spanish and English and a bi-lingual research assistant will be available to assist patient with low literacy levels to complete the survey. The patient survey instruments that will be used are the Patient Assessment of Chronic Illness Care (PACIC) and the Consumer Assessment of Healthcare Providers and Systems- Primary Care Adult (CAHPS-PCA).
Chart audits. Chart audits will be conducted to assess changes in patient care quality over the course of the intervention. The process(es) of care that will be assessed will be determined by each individual practice during the first 4 practice coaching sessions (weeks 1-4). For practices without electronic data, audits will be conducted of the medical records of up to 60 patients with the index condition from each practice who received care from the practice during the 12 months prior to the start of the intervention. A chart audit form specific to the index condition/practice process of each specific practice will be used to capture the data. For practices with electronic data, we will work with IT personnel at the site to obtain these same data through their electronic health record or registries. All data abstraction will be HIPAA compliant and utilize appropriate security to protect confidentiality of data. Electronic or hand abstracted data will then be entered into a project database for analysis.
B.2. Procedures for the Collection of Information
Overview. Data will be collected immediately before beginning the 10-month practice coaching intervention, immediately after the intervention, and at 3-months follow-up to evaluate the intervention. The interviews with the practice coaches will be conducted at mid-point (5 months) and the end of the intervention (10 months). IRB approval will be obtained prior to beginning the study. Appropriate informed consent procedures will be followed to conform with human subjects protection and HIPAA requirements. All data will be collected by individuals trained in qualitative interviewing, and survey administration. Bi-lingual staff will be used to collect patient satisfaction surveys. Everyone involved in data collection will be required to have current Human Subjects certification, training in HIPAA compliance and be covered by a Business Agreement with the site as appropriate to ensure compliance with IRB and HIPAA requirements.
Staff and Provider Interviews and Surveys. The study PI or research assistant trained in qualitative interview and certified in human subjects research will conduct the Key-Informant interviews with staff and providers at each site. These will be conducted in-person or by phone depending on what is most convenient and feasible for the respondent. The interviews will last approximately an hour and will be audiotaped and then transcribed for analysis. The interviewee’s name will be removed from the transcript and replaced with a subject code. The research team will be the only individuals with access to the master key connecting name and code. Appropriate consent will be obtained from staff before data collection begins.
A research assistant will coordinate administration of the surveys to staff and providers. The surveys will be made available in paper format and also when possible, electronically, using a secure on-line survey program. in a secure on-line format. Appropriate consent will be obtained from staff before data collection begins. Surveys will be anonymous and no personal identifiers will be required. Surveys will contain a practice code so the data can be linked to the appropriate practice site.
Practice coach interviews. The project PI or research assistant trained in qualitative interview methods and certified in research with human subjects will complete the key-interviews with the practice coaches at mid point (5 months) and at the end of the intervention (10 months). The semi-structure interview guide provided in this application will be used. The interviews will require approximately 90 minutes to complete, and will be audio-taped and transcribed for analysis. Personal identifiers of the coaches and the practice locations/staff mentioned will not be removed to allow for analysis of variations across practice sites. The only individuals that will have access to the data are members of the research team and data will be kept in locked/password protected files per human subjects requirements. Key informant interviews that will be conducted with practice coaches about process and variations in implementation and observed impact are not included in the burden estimates since they will be conducted with fewer than 9 individuals.
Patient survey data. Patient surveys will be conducted in the waiting room of the 20 participating practices. A bi-lingual research associate trained in human subjects and in survey administration with low-literacy populations will be placed in each waiting room for a two-day period. All adults receiving care at the practice that indicate they have a chronic disease will be surveyed. Data will be collected during 3 2-day data collection periods: immediately before the intervention, immediately after and at 3-month follow-up. A trained bi-lingual research associate will be placed in each practice during those times, in the waiting room or post-visit area per the directions of the practice, and invite patients with chronic disease to complete the survey at the end of their visit. The research associate will assist those with low-literacy levels to complete the survey. The survey will be available in paper format or on a computer located in a private area of the waiting room or post visit area and patients will be offered a choice of either paper or electronic format. The survey will be anonymous with no personal identifiers requested. Each survey will contain a code linking it to the practice to allow comparison across practices. Respondents will be provided with a privacy envelope to place their surveys in after completion and will be asked to place their paper surveys in a locked survey box that will be maintained at the site by the research staff member during the data collection period. This individual will be responsible for collecting and mailing the paper-based surveys to the research offices at LA Net in Long Beach California at the end of the 2-day data collection period via Fed Ex. Upon receipt by LA Net central office, paper surveys will be stored in a locked file drawer and entered into a password protected database on a desktop computer at the office.
Chart audits. After the 4th practice coaching visit when the practice and coach have identified the target for the practice coaching intervention, a research associate will work with the practice IT personnel to identify patients who have received care in the 12 months prior to the intervention with the target condition using billing records for patient in practices without EHRs and through the EHR for practices that do have them. In non-EHR enabled practices, the research associate will then work with the medical records staff to pull paper charts for 60 randomly selected patients from the list, and audit the chart for presence or absence of key quality indicators based on HEDIS measures. In EHR enabled practices, the research assistant will work with the practice IT staff to create an excel data file containing patient visit data on these same quality indicators. All personal identifiers for patients will be removed and a subject code assigned. The master key linking patient identifiers with care data will be kept in a separate locked and password protected file. Registry and chart data sources that will be used to evaluate improvements on quality of care indicators are not included in the estimates of respondent burden for this supporting statement, since they do not require information collection from respondents.
Variables and measurement
This is a study of the implementation of the practice coaching intervention in 20 safety net practices. The primary goal of the study is to increase the understanding of the practice coaching process and how its process, content and outcomes may vary across different practice environments, and to examine its potential value for supporting greater implementation of the Chronic Care Model in safety net practices. A list of the measures that will be used to assess each key area of interest, and the source of the measure, is provided in the table below.
All surveys that will be used in the study are already existing and validated survey tools. Key-informant interview protocols were developed by the study PI, Dr. Knox and members of the research team. Dr. Knox has more than 15 years of experience in qualitative research and developing qualitative interview protocols. A more detailed description of each tool is provided in Part A, Section 1.
Variable
Measure
Source
Practice variables
Practice characteristics (co-variate) Practice profile Providers/staff/admin
Components of Chronic Care Model (CCM)
Self-management support ACIC
PPC-RS Providers/staff/admin
Delivery system design ACIC
PPC-RS Providers/staff/admin
Decision support ACIC
PPC-RS Providers/staff/admin
Clinical information systems ACIC
PPC-RS Providers/staff/admin
Process and Clinical indicators
Quality of care (example condition: diabetes) HbA1c test twice in last 12 months at least 90 days apart
Eye exam in last 12 months
Foot exam in last 12 months
LDL cholesterol in last 12 months
Microalbuminuria screening in past 12 months Registry, electronic health record (HER), or random chart audit
Patient satisfaction and experience
Patient experiences of care and satisfaction CAHPS-PCA, and CAHPS-Spanish
PACIC & PACIC-Spanish Patients/staff
Organizational capacity for improving medical practice
Priority of change CPCQ Providers/staff/admin
Change process capability CPCQ Providers/staff/admin
Clinician/staff/administrator satisfaction Primary Care Staff Satisfaction Survey Providers/staff/admin
Content and process of intervention
Intensity of intervention (co-variate) PCEF Practice coach
Strategies used in intervention (co-variate) PCEF Practice coach
Process and content of intervention by site including use of Toolkit content PCEF
Key-informant interview Practice coach, providers/staff/admin
Barriers and facilitators to implementation of intervention and of CCM and observed impact PCEF
Key-informant interview Practice coach, providers/staff/admin
B.3. Methods to Maximize Response Rates
This is a multiple replication study that will examine variations in implementation, process and impact of practice coaching across 20 safety net organizations. This is not an experimental study and it is not seeking to develop patient or practice level data that can be generalized to the larger population. The goal of this study is to document site-specific impacts of practice coaching on quality of care indicators, and practice, provider and patient indicators, and implementation of the intervention, and then to evaluate how these factors vary across the 20 replication sites in order to develop “lessons learned” that can be used to inform future practice coaching interventions and development of a research agenda for AHRQ on the process and effectiveness of practice coaching as a practice improvement strategy.
Safety net provider and staff interviews and surveys. Response rates to key-informant interview requests will be maximized through personal invitations, scheduling interview at times that are convenient for the respondents, and offering the option of a telephone interview. Response rates for the survey will be maximized by administering questionnaires during scheduled clinical staff meeting times, during pre-intervention and post-intervention visits by the Practice Coaches, protecting anonymity, and offering the option of both paper and on-line versions of the survey, through repeated invitations to complete the survey, by in-person or telephonic follow-ups with providers and staff who fail to complete the surveys on first and second requests. Based on prior experiences administering satisfaction and other types of surveys to clinic staff, AHRQ has consistently obtained response rates as high as 85%. With 200 potential respondents, this will yield 170 respondents (target accrual is 160) which will provide sufficient power to detect change across the outcomes of intervention within each site, and will provide adequate sample size to identify potential variations in outcomes across sites by site characteristics.
Practice coach interviews. Because these individuals are employed as part of the project we anticipate a 100% response rate from the coaches to the interviews.
Patient survey. Response rates from patients on their satisfaction questionnaire will be maximized by placing staff in the practices who will collect data until target accrual is reached. Many of the larger potential sites for this study routinely see 50 to 100 adult patients per day. The smaller ones may see 25- 50 a day. With a 65% response rate, based on prior experiences with wait room surveys in safety net settings, AHRQ requires 1-3 days per site to meet the target accrual of 50 completed questionnaires per site. We have sufficient staff to collect these surveys and do not anticipate a problem reaching our target accrual over time. Bi-lingual research assistants will available to assist with non-English speaking patients which will also increase response rate.
Chart audit. Project staff will perform chart audits of the medical records of 60 patients with the index condition selected by each practice will be conducted at baseline, at the end of the intervention, and at 3-month follow-up. At each practice, depending on the practice size, several hundred to several thousand patients have chronic health conditions including Type 2 Diabetes, Asthma, COPD, or CVF, likely foci for the practice coaching interventions; and the data that will be used for the chart audits are routinely collected during patient care and are readily accessible as part of patients’ paper base or electronic health records maintained by the practices.
B.4. Tests of Procedures
All survey tools being used in this study have been previously validated, published and used in previous research. The key-informant interview protocol was piloted with 3 CHC staff in July 2009 and information was solicited about comprehensibility and clarity of the tool. Only minor revisions were indicated by the feedback and these changes have been incorporated into the tools. For purposes of formative and qualitative interviewing, piloting with 3 respondents is sufficient to assess the appropriateness of an interview protocol.
B.5. Statistical Consultations
Dr. Pamela M. Diamond, Ph.D., Assistant Professor of Behavioral Sciences and Biostatistics at the University of Texas at Houston School of Public Health was used as the primary consultant for the quantitative analysis plan for this study. Dr. Diamond has been PI on a number of federal grants involving health and public health studies, and has collaborated with the PI for this Task Order for more than 20 years on federally funded research studies.
References
i See www.healthdisparities.net/hdc/html/home.aspx for more information on HRSA’s Health Disparities Collaboratives.
ii Coleman K, Austin B, Brach C, et al. Forthcoming. Evidence on the Chronic Care Model in the new millennium. Health Affairs,.
iii Vargas RB, Mangione CM, Keesey J, et al. 2004. Do collaborative quality improvement programs reduce cardiovascular risk for persons with diabetes? In Annual Research Conference. San Diego: AcademyHealth.
iv Baker DW, Asch SM, Keesey JW, et al. 2005. Differences in education, knowledge, self-management activities, and health outcomes for patients with heart failure cared for under the chronic disease model: the improving chronic illness care evaluation. J Card Fail 11 (6):405-13.
v Asch SM, Baker DW, Keesey JW, et al.. 2005. Does the collaborative model improve care for chronic heart failure? Med Care 43 (7):667-75
vi Huang E, Chin M. 2005. Health disparities collaboratives: changing practices, changing lives: assessing the impact of the HRSA health disparities collaboratives and what comes next In 2005 Community Health Institute. Miami, FL: National Association of Community Health Centers. Tsai AC.
vii Morton SC, Mangione CM, et al. 2005. A meta-analysis of interventions to improve care for chronic illnesses. Am J Manag Care 11 (8):478-88.
viii Coffey RM, Matthews TL, McDermott K. Diabetes Care Quality Improvement: A Resource Guide for State Action. (Prepared by The Medstat Group, Inc. and The Council of State Governments under Contract No. 290-00-0004). Rockville, MD: Agency for Healthcare Research and Quality, Department of Health and Human Services; September 2004. AHRQ Pub. No. 04-0072.
ix Health Care Improvement Process Made Accessible to Safety-Net Providers via Breakthrough Series on Internet Foundation, Robert Wood Johnson. 2003. Health Care Improvement Process Made Accessible to Safety-Net Providers via Breakthrough Series on Internet. Available at: www.rwjf.org/portfolios/resources/grantsreport.jsp?filename=040709.htm&iaid=142.
xIntegrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics. Rockville, MD: Agency for Healthcare Research and Quality. Available at http://www.ahrq.gov/populations/businessstrategies/
xi Coleman K, Austin B, Brach C, et al. Forthcoming. Evidence on the Chronic Care Model in the new millennium. Health Affairs,.
xii Vargas RB, Mangione CM, Keesey J, et al. 2004. Do collaborative quality improvement programs reduce cardiovascular risk for persons with diabetes? In Annual Research Conference. San Diego: AcademyHealth.
xiii Baker DW, Asch SM, Keesey JW, et al. 2005. Differences in education, knowledge, self-management activities, and health outcomes for patients with heart failure cared for under the chronic disease model: the improving chronic illness care evaluation. J Card Fail 11 (6):405-13.
xiv Asch SM, Baker DW, Keesey JW, et al.. 2005. Does the collaborative model improve care for chronic heart failure? Med Care 43 (7):667-75
xv Huang E, Chin M. 2005. Health disparities collaboratives: changing practices, changing lives: assessing the impact of the HRSA health disparities collaboratives and what comes next In 2005 Community Health Institute. Miami, FL: National Association of Community Health Centers. Tsai AC.
xvi Morton SC, Mangione CM, et al. 2005. A meta-analysis of interventions to improve care for chronic illnesses. Am J Manag Care 11 (8):478-88.
xviiIntegrating Chronic Care and Business Strategies in the Safety Net: A Toolkit for Primary Care Practices and Clinics. Rockville, MD: Agency for Healthcare Research and Quality. Available at http://www.ahrq.gov/populations/businessstrategies/
| File Type | application/msword |
| File Modified | 2010-08-26 |
| File Created | 2010-08-26 |