Supporting Statement A Care Coordination - with Responses to All OMB Questions and attachments B C D E - 10.7.09

29

SUPPORTING STATEMENT

Part A

Coordinating Care across Primary Care and Specialty Care Practices

Version: October 7, 2009

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1 Circumstances that make the collection of information necessary 3

2. Purpose and use of information 9

3. Use of Improved Information Technology 15

4. Efforts to Identify Duplication 15

5. Involvement of Small Entities 15

6. Consequences if Information Collected Less Frequently 15

7. Special Circumstances 15

8. Federal Register Notice 15

9. Consultation outside the Agency 16

10. Payments/Gifts to Respondents 16

11. Assurance of Confidentiality 16

12. Questions of a Sensitive Nature 16

13. Estimates of Annualized Burden Hours and Costs 16

14. Estimates of Annualized Respondent Capital and Maintenance Costs 17

15. Estimates of Annualized Cost to the Government 17

16. Changes in Hour Burden 18

17. Time Schedule, Publication and Analysis Plans 18

18. Exemption for Display of Expiration Date 23

List of Attachments 23

Attachment A: Healthcare Research and Quality Act of 1999 …………………Separate

Attachment B: Script for Introducing the Project and Description

of the Focus Group for Recruiting Purposes ……………………………………..24

Attachment C: Revised Focus Group Guide ………………………..……………25

Attachment D: Coordinating Care Focus Group Introduction…….……………..27

Attachment E: Implementation Log………………………………..………………28

Attachment F: Federal Register Notice…………………………………………. .Separate

Section A: Justification

A.1. Circumstances of Information Collection:

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, safety, efficiency, and effectiveness of health services for all Americans, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care;

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

AHRQ, through its contractor, the Boston University School of Public Health (BUSPH), propose an evaluation of the redesign of the transitions of care between primary care and specialty care services. The purpose of the redesign is to remedy inefficiencies in the current referral processes that threaten care quality and safety, and system efficiency. This redesign is being implemented at the Boston Medical Center (BMC), and two affiliated health centers.

ANSWERS TO A1 and A2 (Question 1 in email):

The previous system included a process that was minimally standardized with referral forms generated from the health centers electronic record and then printed and faxed to the specialist. From there, the appointment was booked and the visit either occurred or did not. Reports were sent back to referring providers in a non-standardized way. There was frequently missing information because there was no standardization of the all various steps in the process (e.g., communication between providers, no shows), no agreed upon contingencies for system failures (particularly around clear lines of responsibility), delays due to missing information, unclear responsibilities for the patient, and inconsistent systems aids. Communication loops were often not closed.

A major part of this project is the implementation currently being done by the redesign team. A major role of the redesign is to address the issues of concern in the current process:

1. Standardized referrals from the patient’s Medical Home (demographics, questions, standard labs, time expectation).

2. Agreed upon protocol with regard to patients who do not keep their appointment; which clarifies responsibilities of all parties.

3. Standard consultation content (answers to questions, recommendations and documentation of clear follow up expectations for each of the 3 parties involved)

4. Standards for in what situations direct, real-time physician-to-physician communication is expected in each direction.

5. Standards for closing the communication loop at key steps.

6. Patient tool which addresses common questions, clarifies expectations, and gives a HELP number to call at anytime to assist with communication around the referral.

7. Protocol for ongoing monitoring of the standards and intermediate outcomes to ensure consistency.

Care coordination has been identified by the Institute of Medicine (IOM) as a key strategy with potential to improve the effectiveness, safety and efficiency of the health care system. At the same time, care coordination, particularly in transitions among sites of care, is often lacking. Research shows that problems in coordination of care and common failures in patients’ transitioning between and among systems typically create serious quality concerns in many settings. Individuals moving across systems of care and between care providers are vulnerable to fragmented and disjointed care (Coleman et al, 2004). Uncoordinated and fragmented transitions can lead to a wide range of costly problems and threats to patient safety including greater use of hospital and emergency services (Coleman et al, 2004), ordering and completion of redundant tests (Coleman & Berenson, 2004), prescription and medication errors and use of poly-pharmacy by multiple providers (Coleman & Berenson, 2004). The end result is often confusion about conflicting care plans and lack of follow-up care. The aim of this evaluation is to address this confusion and fragmentation by expanding knowledge of how to improve the experience and outcomes for patients in transitions of care between primary care and specialty practices.

The initial focus is on referrals between primary care and two specialties: gastroenterology (GI) and obstetrics (OB). We elected to focus on GI and OB because both are specialties that offer multiple procedures and that require multiple levels of coordination. Both have a general office type referral AND a procedure type referral. We also elected these specialties because they are in very different situations at baseline of the project: OB has recognized the problems in coordination and transitions of care and had taken some small steps to improve, while GI had not begun to address any concerns prior to this project. The only criteria specific to these referral situations will be the specific tests/procedures identified on the referral forms. We hope the redesign process is successful and can be duplicated in other specialty settings.

The redesigned referral system will be tested by implementing it in three participating primary care sites and two specialty clinics. We expect that the lessons learned from this evaluation will provide a model and tools that can later easily be tested and applied to other sites and specialties in the BMC system and provide lessons learned to other systems seeking to sustainably improve their referral processes.

The overall aims of the evaluation are to provide a rigorous assessment of the success of the redesigned referral system in meeting its improvement goals and to gain an understanding of the implementation of the redesigned system.

More specifically, the evaluation is designed to provide both formative and summative information to meet four objectives:

1) Document the implementation process;

2) Assess fidelity of redesign implementation (Are providers conforming to the new system and is it in use as planned?);

3) Assess success of outcome performance (Is the new system meeting its goals?); and

4) Identify factors that affect redesign fidelity and outcome performance.

Data to address the objectives will be drawn from three sources, as described in more detail in the following sections:

1) Secondary analysis of medical record data related to the referral processes;

2) Focus groups with providers, clinical staff, and administrative staff involved in the redesigned referral process; and

3) Project team intervention logs and meeting notes to document the processes by which the redesigned system is implemented.

ANSWERS TO A4 and A5 (Question 3 in email):

We have 11 outcome measures that will be obtained from the electronic medical records that are considered our Key Outcomes. These will indicate to us that the new system is being used as planned.

Key Outcome	Data source to determine outcome
Referral information is sent from PCP site to specialty site.	Paper records and EMR
Of those referrals, what percent were complete?	Paper records and EMR
Percent of time an appointment scheduled for patient?	EMR
Time between referral made and initial specialty care appointment made	Paper records and EMR
Percent of time PCP was made aware appointment was scheduled?	EMR
Percent of time there is a note in medical record that referral received by specialist?	EMR
Percent of time PCP is notified if patient does not attend specialty appointment	EMR
Percent of time clinical information is available on referral form by the time of specialty appointment	Paper records and EMR
Percent of time specialist sends note to PCP after appointment?	EMR
Percent of time there is note in the medical record that PCP received communication back from specialist	EMR
Percent of time the specialist makes a note of follow-up plan and responsibility	Paper records and EMR

While we can measure some aspects of the referral system from existing data in the medical records, and will utilize those data fully, some aspects of the system are not captured in existing records. Reflections and feedback directly from providers and staff using the referral processes are needed to fully understand the components of the redesigned referral system that are working well in relation to transition of care goals and to identify the remaining problems that need to be addressed. Similarly, while the redesign team can and will document the processes by which the new referral system is introduced to staff and its early implementation supported, feedback is also needed directly from the providers and staff who are key users of the redesigned referral system. Gathering data in this manner will inform further implementation efforts at BMC, and provide lessons learned to other health care systems.

The initial redesign process and subsequent implementation and evaluation will be informed by three complementary conceptual frameworks.

ANSWERS FOR QUESTION A6-A11 (Question 3 in email):

Answer to A6: We integrated three conceptual frameworks into our project, as AHRQ requested consideration of these frameworks. These include the Stepping Up to the Plate Alliance (SUTTP) framework, the Replicating Effective Programs (REP) model, and the Organizational Transformation Model (OTM). BU is not wed to any of these frameworks, but means to indicate they are being used to inform our redesign, implementation, and evaluation processes. Our evaluation and synthesis of them resulted in our design and implementation plan. They are not specifically linked to individual research questions.

However, according to these models, improvement in care transitions requires many changes, including changes in structure and culture. One example of an intervention consonant with our frameworks is that of obtaining support from the clinical units for the new procedures, which we are doing through developing a service agreement and obtaining buy-in via our redesign process. Another example of how we encourage structural and cultural adjustments is to provide ongoing feedback to providers as part of the monitoring process, which will aid in buy-in for culture change and support learning via feedback to monitor the new skill/process.

The adoption of the new process—informed in its design by the conceptual models as described above-- will then be measured according to the key outcome measures described in the response to question A5 above, as well as through the results from our focus groups.

• The redesigned referral system will be consistent with the principles for care transitions outlined by the Stepping Up to the Plate Alliance (SUTTP). The SUTTP approach takes a broad perspective on transitions, recognizing that system improvement will require changes beyond forms and procedures, including adjustments in structure and culture. The improvement challenges include thorough communication; accountability and shared responsibility to ensure providers are committed to, and accountable for, timely information sharing. By recognizing patients as active participants in the process, improved transitions also provide an important step toward more patient-centered care. (http://www.abimfoundation.org/publications/pdf_issue_brief/F06-05-2007_6.pdf)

• ANSWER TO A7: The SUTTP Alliance if the American Board of Internal Medicine has representation from medical specialties such as internal medicine and its sub-specialties, family medicine, and surgery. The alliance was formed in 2006 and has been working on care coordination across multiple settings and specialties. The SUTTP Alliance has developed a set of principles for care transitions that include accountability, communication, timely feedback, involvement of the family and care givers, and respect for the hub of coordination of care. They also developed a set of standards that include coordinating clinicians, care plans, communication infrastructure, standard communication formats, transition responsibility, timeliness, community standards, and measurement. For a health care system to adhere to these principles and standards more changes need to be made than simply instituting the use of a new patient check sheet at a clinic. These adjustments may be made in each clinic or health care system that attempts to undertake challenge to better coordinate their patients’ care. Clinical practice teams and systems need to work together more effectively to make changes in the structure and processes of their work and in the culture of the system. The goal of this project is to inform the process of making structural and cultural adjustments by examining some best practices for coordinating and smoothing transition of care. The standards and principles from the SUTTP framework will help guide the design and process of the quality improvement ( or clinical process renovation) efforts in this study.

• Answer to A8: The principles of improved communication and appropriate accountability inform the motivation and plan for this entire project. Each of the key outcome measures listed below will be derived from the electronic medical records.

Key Outcome	Principle Examined
Referral information is sent from PCP site to specialty site.	Communication
Of those referrals, what percent were complete?	Accountability
Percent of time an appointment scheduled for patient?	Accountability, shared responsibility
Time between referral made and initial specialty care appointment made	Accountability
Percent of time PCP was made aware appointment was scheduled?	Communication, share responsibility
Percent of time there is a note in medical record that referral received by specialist?	Communication
Percent of time PCP is notified if patient does not attend specialty appointment	Communication, share responsibility, accountability
Percent of time clinical information is available on referral form by the time of specialty appointment	Communication, share responsibility, accountability
Percent of time specialist sends note to PCP after appointment?	Communication, share responsibility, accountability
Percent of time there is note in the medical record that PCP received communication back from specialist	Communication, share responsibility
Percent of time the specialist makes a note of follow-up plan and responsibility	Communication, share responsibility, accountability

• The implementation of the redesigned referral system will also be informed by the CDC’s Replicating Effective Programs (REP) model. The REP framework addresses implementation across four conceptually separate, though sometimes chronologically overlapping stages: Pre-Conditions; Pre-Implementation; Implementation; and Maintenance/Evolution. The core tasks of the framework are to assess relevant organizational factors that support or impede implementation, locally adapt the intervention for maximal uptake, train staff, and provide ongoing technical assistance. The model’s approach is intended to maximize fidelity to the planned intervention while allowing opportunities for flexibility to ensure use and sustainability. (Kilbourne et al., 2007)

• ANSWER TO A9: The REP model provides a common-sense framework for the undertaking of any new process implementation. Our processes with regard to the initial redesign work with the clinical teams were consistent with this framework at every step along the way. For example, we ascertained that there were appropriate and willing clinical leaders in both of the specialty areas targeted for the intervention; we put in place plans for training of staff and plans for ongoing availability of the implementation team for ongoing technical assistance. For more information on the model, please see http://www.cdc.gov/hiv/topics/prev_prog/rep/resources/qa/process.htm#QA4.

• ANSWER TO A10: It was never stated that measuring fidelity helps to increase or maximize fidelity. In the REP model the core elements are used as a tool to help researchers adapt interventions to fit agencies’ settings, circumstances, and populations while maintaining fidelity to what the research has determined to be effective. Core elements serve as parameters for the parts of an intervention that should not be changed, benchmarks for intervention fidelity, and references for quality assurance of intervention implementation. http://www.cdc.gov/hiv/topics/prev_prog/rep/resources/qa/process.htm#QA4

• The redesign, implementation and evaluation will be framed within a higher-level model of organizational change and transformation, the Organizational Transformation Model (OTM). OTM recognizes that successful implementation and sustainability of new practices require attention to the larger organizational context in which a specific intervention lies. It focuses on five organizational features that are critical to successful transformation of patient care: 1) impetus to transform; 2) leadership commitment to quality; 3) improvement initiatives that actively engage staff in meaningful problem solving; 4) alignment to achieve consistency of organizational goals with resource allocation and actions at all levels of the organization; and 5) integration to bridge traditional intra-organizational boundaries among individual components. (Lukas et al., 2007)

• ANSWER TO A11: As stated in our response to A9, the OTM provides conceptual support for the background and system change methods undertaken in this study. It is not the direct intent of the study to measure whether the institutions impetus to transform was large or small or was significant or not in the success (or failure) of the intervention to impact changes in key outcomes. That being said, these issues are expected to be explored in the team meetings that are held at least once, if not twice a month during the project. The information will be captured in logs and meeting notes that are being used to document the incremental progress and inevitable speed bumps that affect the change process. Information related to these organizational features will also be captured in the focus groups that are held with clinicians and office staff at the primary and specialty care clinics. The focus group appendix and table one describes more specifically how some of the information will be gathered.

The evaluation design has been developed in close collaboration with the ACTION Task Order Officer and AHRQ senior members of the Task Order team. In the ACTION program model, AHRQ staff develops the Request for Task Order that defines the objectives and scope of the project, selects the winning contractor from competitive bids and provides oversight throughout the contract. Within the Boston University project team, BUSPH evaluators work closely with BMC clinicians, consistent with the ACTION emphasis on pairing researchers and clinicians to bring clinical innovations into routine practice in healthcare organizations. The evaluation protocol has been submitted for approval to the Institutional Review Board at the Boston University School of Medicine.

A.2. Purpose and Use of Information:

Table 1 summarizes for each data source the frequency of data collection and the evaluation objectives for which it will be used. In addressing each evaluation objective, the information from each data source will be used formatively by the project team to further strengthen the redesigned referral system and the processes by which it is implemented in each setting. The information will be used summatively by the project team and the leadership of BMC and the health centers to assess its success, and if successful, to plan the spread of the redesigned referral processes and implementation lessons to other primary care sites and specialties in the BMC network. The summative information will also be made available to other healthcare systems that may be able to draw lessons from the BMC and health center experiences.

Data from each source will contribute to different aspects of the broad evaluation questions as described here:

• Medical record data will be used to analyze aspects of the referral process (such as percentage of items on referral forms filled in, proportion of specialty appointments made, time between referral and initial specialty appointment), not patients’ personal health data. This data will be used to measure both the fidelity of the redesigned system in practice to plan and success in meeting redesign improvement goal (outcome) indicators.

  • ANSWER TO A12 (Question 3 from email): The outcome indicators are the 11 key outcomes described above. These items will enable us to measure problems with the old system such as lack of follow-up care and the other issues described below, which can lead to overuse of inpatient care, and other concerns related to the old system. Although we do not plan to look at hospitalizations, we are looking at items that can lead to inappropriate hospitalization. As indicated in our measures above, we will be looking at follow-up procedures to a degree. The medical record data will be extracted by project staff and will not impose a burden on the participating health care sites. Therefore, OMB clearance is not required for this part of the evaluation.

• ANSWER TO A13 (Question 11 from email): As described above in our response to comments A4 and A5, we will be looking at 11 key outcomes via the electronic medical record systems at each site and for each specialty. These data will be used to enable us to assess, more quantitatively, the effectiveness of the redesign system and whether providers are using the new tools as designed. We have developed a quantitative analysis plan for these data, which was included in Section A16, II. (Data Sources, Medical record data).

  • Standard empirical methods will be used to analyze the quantitative data collected from the medical charts and electronic data system. All data will be entered into a database using SPSS 16.0. After data are collected, we will perform data checking on a 10% sample to check for data quality and missing data. As needed, any errors will be remedied by going back to the medical charts and the electronic record. Once the database is completed, we will perform standard statistical analyses, specifically descriptive statistics including frequencies, measures of central tendency, and measures of variability. Data will be stratified by site, by provider type (i.e., primary care provider versus specialist, MD versus other clinical staff, including RNs and PAs), by referral type requested and potentially by other characteristics identified in the redesign process. We will examine change from pre-implementation to post-implementation using bi-variate analyses, such as t-tests and chi-square tests, as appropriate. If we have sufficient data, we will estimate regression models using the different performance measures as our outcome variables. Explanatory variables will include site, provider type and procedure/consultation type requested.

  • The limitations of these data are the same as that from any statistical analysis in which one is taking only a sample of data over a circumscribed time period. Because there will be some discussion at the sites about the use of the new system prior to implementation, it is possible that there may be a Hawthorne affect during the pre-implementation period. In addition, we recognize that this sample cannot be generalized to all clinics. There are also limitations of administrative data, including incomplete data.

• Focus groups with providers, clinical staff and administrative staff will be conducted in each primary care site and in each specialty practice (see Attachment B). The group sessions will pursue three topics: the extent to which the new system is being used as intended; the perceived effectiveness of the new system as implemented; and the organization and culture of the clinical setting. Themes from the focus groups will be used to assess fidelity of implementation, performance outcomes and factors affecting fidelity and outcomes.

Focus group participants will be physicians and staff working in the participating clinics. Because of the limited number of potential participants, recruitment will be done informally by clinical leaders in that practice, who will make it clear that participation is voluntary.  No letter will sent about the purpose, time and location, as meetings are scheduled at their sites during an established provider meeting or at the convenience of the participants.  The Boston University IRB has determined that the focus group participants will be knowledgeable agents and not human subjects because they will be asked about referral processes, not about personal information.  Therefore no signed informed consent is involved.  However, information about the project will be presented in the introduction to each focus group (see Attachment C).

ANSWER TO A14 (Question 4 from email):

Our study sites are community health centers and an ambulatory clinic located within a safety net hospital. These sites have both similarities and differences from other sites. Like all sites serving patients who are at high risk, there are challenges in coordination, many patients who are poor and who have problems with literacy, and overwhelmed providers. They are likely representative of all federally qualified health centers and community health centers designed to treat the under-served. They have a strong focus on the medical home model and primary care. They may also be different from other practices in that all providers are also on the faculty at a medical school. Although this may not be typical, they all have an electronic medical record, which is becoming more typical of medical practices.

However, despite any differences, as we redesign we are trying to think broadly so that our system and tools could be used anywhere. We are designing something that does not have to be electronic, although it can be used electronically. It is important to note that studies show that problems in coordination of care and common failures in patients’ transitioning between and among systems typically create serious quality concerns in many settings, despite similarities and differences in those settings.

Recruitment for the focus groups will be done by the clinical leaders at each site who are members of our redesign team. Rather, at a staff meeting or other team meeting, the clinician will tell his/her colleagues about the plan for a focus group and ask for volunteers to participate. We have taken OMB’s suggestion and added the standard introduction statement below to be used by the clinicians:

Script for Introducing the Project and Description of the Focus Group for Recruiting Purposes: The goal of the Coordinating Care project has been to assess the current referral systems and identify mechanisms for improving both the referral process and overall coordination of care. Working with a redesign team, we have developed a set of new tools and processes for referring patients between primary care at three health centers and two specialties, OB and GI. You have had the opportunity to use these new procedures for the past several months. We now want to know how the new procedures have worked for you. Therefore, to evaluate this redesign, we are conducting a series of focus groups and would like to invite you to participate. The focus group session will last 45 minutes, and will be facilitated by members of the study team from Boston University. Only information pertinent to the referral system will be gathered; no personal information will be elicited or discussed. The focus groups will be audio-taped; however, all data will be analyzed in the aggregate and kept private to the extent permitted by law.

• Implementation logs and meeting notes kept by the project team throughout the redesign implementation will document the implementation process, including factors affecting the process, challenges encountered, and strategies for dealing with the challenges. This component of the evaluation will not impose a burden on the participating health care sites; therefore OMB clearance is not required.

ANSWER TO A15 (Question 11 from email):

Our “implementation logs” will be logs used by the team as we move toward implementation. We believed that this is not “information” as they are simply an informal mechanism for documenting our process during the period between and just prior to implementation. They will be filled out only by members of the redesign team and therefore not included under the PRA. They do not track data elements, but are simply used to enable us to assess our progress, to build our tools, and address any implementation issues that arise. They also allow us to keep track of ideas as they arise in our meetings and to include them in our redesign. A sample is included at the end of this document.

Table 1 (Original): Data Sources by Evaluation Objectives

[Months are calculated from time of OMB approval]

Evaluation objectives: Data sources:	Document implementation process	Assess fidelity of system redesign implementation	Assess success of redesign outcomes	Identify factors affecting implementation and effectiveness
Medical record review: Baseline: Months 0-2 [referrals in prior 8 months] Follow-up: Months 10-12 [referrals in prior 8 months]		√	√
Provider/ staff focus groups: Months 10-11		√	√	√
Implementation logs and meeting notes process -- ongoing Months 3-12	√			√

Table 1 (Revised): Data Sources by Evaluation Objectives [Months are calculated from time of OMB approval]
Evaluation objectives: Data sources:	Document implementation process	Assess fidelity of system redesign implementation	Assess success of redesign outcomes. Revised to match updated key outcomes.	Identify factors affecting implementation and effectiveness
Medical record review: Baseline: Months 0-2 [referrals in prior 8 months] Follow-up: Months 10-12 [referrals in prior 8 months]		- Number of referrals to specialties generated - Referral information is sent from PCP site to specialty site. - Of those referrals, what percent had all boxes filled in?	- What percent of time was an appointment scheduled? - Time between referral made and initial specialty care appointment made - What percent of time was PCP made aware appointment was scheduled? - Is there a note in medical record that referral received by specialist? - If patient does not attend specialty appointment, is PCP notified? - If patient does attend specialty appointment, is clinical information available for specialist at time of appointment? - Does specialist write note to PCP after appointment? - PCP sees specialist’s note/Note in the medical record that PCP received communication back from specialist - Is note clear about next steps and who will resume care (retain care) of the patient next?
Provider/ staff focus groups: Months 10-11		- Is the redesigned system being used as intended? - Was the process by which the redesigned system was put into place successful?	- Has the redesigned system improved the referral process?	- Are there remaining problem areas or areas needing further improvement? - Were there problems with the implementation process? - What barriers and facilitators affect the referral process?
Implementation logs and meeting notes process -- ongoing Months 3-12	- How was the new system introduced? - Are the introduction and technical assistance working as planned? - Are the introduction and technical assistance as planned? - What additional assistance was provided?			-What problems are encountered? - Were there staffing and resource challenges in using the new system? - What are the lessons for improvement in the implementation process?

A.3. Use of Improved Information Technology:

To reduce respondent burden, the majority of data on the referral process and its outcomes will be abstracted from existing medical records. The bulk of the abstraction will be done electronically with paper charts used as a back up. Other sources of data collection, i.e., focus groups and documentation of implementation, are not available or appropriate via technological applications. Focus groups will be conducted at each of the three primary care sites and two specialty care sites once and will be limited to 45 minutes. There will be separate focus groups for providers, other clinical staff, and administration staff. Basic aspects of the implementation processes will be documented by project staff as part of their participation in those processes.

A.4. Efforts to Identify Duplication:

The information being collected is unique to the study organizations and their practices. To date, quality improvement in the area of ambulatory transitions from primary care has been limited to ad hoc interventions that tend to focus on transitions to one specialist, or a single specialty. Few, if any, have been evaluated or described in the scientific literature. This project will feature a broad application of the intervention and a rigorous evaluation. Results could be of interest to systems-level policymakers, delivery-level clinicians and patients, and will support the goals of the AHRQ Prevention/Care Management Portfolio.

To avoid duplication of data collection efforts internally and externally, the study’s data collection instruments are designed to gather the data needed using the most efficient methods available. AHRQ has reviewed the instruments and has confirmed that the data are not available from another source. A preliminary literature review was conducted and we are presently unaware of other duplicative research being conducted.

A.5. Involvement of Small Entities:

The proposed data collection does not involve small business entities.

A.6. Consequences if Information is Collected Less Frequently:

The clinician and staff focus groups will only be conducted once. Medical record data extraction, implementation logs and meeting notes will involve project staff only, so there is no additional burden imposed on the clinic staff associated with these data collections. There are no legal obstacles to reducing the burden.

A.7. Special Circumstances:

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d) (2). No special circumstances apply.

A.8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on March 2, 2009 for 60 days (see Attachment D). One comment was received. No changes were made based on this comment.

ANSWER TO A17 (Question 6 from email): Comment: “THIS AGENCY IS VERY VERY UNPRODUCTIVE. THEY LOVE TO SIT IN BUREAUCRATIC SPLENDOR PRODUCING ABSOLUTELY NOTHING OF VALUE FOR AMERICAN HEALTH. AMERICANS HEALTH IS GOING DOWN THE TOILET. IT IS THIRD WORLD COUNTRY STATUS AT PRESENT DESPITE THE FACT THAT WE SPEND BILLIONS ON THIS AGENCY. THEY LIKE TO TAKE SURVEYS AT THIS AGENHCY AND PRODUCE NO REAL HELP FOR ANY AMERICAN AT ANY TIME. PUT THIS AGENCY TO SLEEP PLEASE. IT ACCOMPLISHES NOTHING OF VALUE FOR AMERICA. B SACHAU 15 ELM ST FLORHAM PARK NJ 07U392”

8.b. Outside Consultations

The following are the list of persons consulted:

Michael Shwartz, PhD, Richard D. Cohen Professor in Management and Professor of Health Care and Operations Management at the Boston University School of Management, 617-353-4243.

A.9. Payments to Respondents:

No payments will be made to respondents.

A.10. Assurance of Confidentiality:

ANSWERS TO A18 and A19 (Question 5 from email):

No protected health information will be collected. We believe that the clinicians and staff participating in the focus groups fall under the definition of a knowledgeable agent rather than a human subject because they are only providing information about the organization in which they work. Focus group notes will not contain names of participates and results of the focus group will only be reported in the aggregate. We will inform participants that we will keep their responses private to the extent permitted by law.

The data files will be secured consistent with Boston University IRB regulations. All project staff have completed research and human subjects certification from both Boston University School of Medicine and the US Department of Veterans Affairs.

A.11. Questions of a Sensitive Nature:

Questions of a sensitive nature are not being asked in the proposed data collection.

A.12. Estimates of Annualized Burden Hours and Costs:

Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this two year evaluation. Focus groups will be conducted with about 21 clinical staff at each of the 3 primary care sites and 2 specialty care sites (Exhibit 1 shows 2.5 sites per year). At each of the 5 sites there will be separate focus groups for providers, other clinical staff, and administration staff. Each focus group session will last about 45 minutes. The total annualized burden is estimated to be 39 hours.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents' time to participate in this project. The total annualized cost burden is estimated to be $1,463.

Exhibit 1.  Estimated annualized burden hours

Form Name	Number of Respondents	Number of responses per respondent	Hours per response	Total Burden hours
Focus groups	2.5	21	45/60	39
Total	2.5	21	45/60	39

Exhibit 2. Estimated annualized cost burden

Form Name	Number of Respondents	Total Burden hours	Average Hourly Wage Rate*	Total Cost Burden
Focus groups	2.5	39	$37.50	$1,463
Total	2.5	39	$37.50	$1,463

* The hourly wage is based upon the weighted mean of the average wages for physicians ($58.76, n=45), clinical administrative staff ($17.64, n=30) and other clinical staff ($25.48, n=30). National Compensation Survey: Occupational Wages in the United States, “U.S. Department of Labor, Bureau of Labor Statistics.” June 2007, Summary 07-03, http://www.bls.gov/ncs/ocs/sp/ncbl0910.pdf. Accessed December 10, 2008.

A.13. Estimates of Annual Respondent Capital and Maintenance Costs:

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

A.14. Estimates of Annualized Costs:

Exhibit 3 shows the estimated total and annualized cost for this two year evaluation. The total cost is $155,110 and includes $23,267 for project development, $32,573 for data collection activities, $31,022 for data processing and analysis, $15,511 for the publication of results, $12,408 for project management and $40,329 for overhead.

Exhibit 3.  Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$23,267	$11, 633
Data Collection Activities	$32,573	$16,287
Data Processing and Analysis	$31,022	$15,511
Publication of Results	$15,511	$7,756
Project Management	$12,408	$6,204
Overhead	$40,329	$20,164
Total	$155,110	$77,555

A.15. Changes in Hour Burden:

This is a new data collection effort.

A.16. Time Schedule, Publication and Analysis Plan

1. Purpose and main research question: The overall purpose of the project is to work with primary care providers in three settings (a hospital ambulatory care clinic and two community health centers in Boston, Massachusetts) and with two high-volume specialties (Obstetrics/Gynecology (OB) and Gastroenterology (GI)) to implement and evaluate an improved system of referral transitions across sites of care.

We will formally evaluate the redesigned referral system to meet four objectives: 1) document the implementation process; 2) assess fidelity of redesign implementation (Are providers conforming to the new system and is it in use as planned?); 3) assess success of outcome performance (Is the new system meeting its goals?); and 4) identify factors that affect redesign fidelity and outcome performance.

ANSWER TO A20: We will compare the three primary care sites in order to determine whether the redesigned system and implementation process are applicable across these settings or whether modifications are needed to tailor it to different practice settings. The project does not have a control or comparison group because the criterion for success is meeting improvement goals, not net effectiveness. The design for the evaluation of the redesigned referral system is a pre-post test comparison.  From a research perspective, this design is not as strong as a control or comparison group study.  However, the focus of the project is on quality of improvement, not net effectiveness.  The criterion for success is meeting the improvement objectives. Qualitative and quantitative evaluation data will be used formatively to understand the dynamics of implementation and to make mid-course corrections to further strengthen the redesigned referral system.  At the end of the project, the data will be used summatively by the project team and the leadership of Boston Medical Center (BMC) and the participating health centers to assess its success of the redesigned referral system, and if successful, to plan the spread of the redesigned processes and implementation lessons to other primary care sites and specialties in the BMC network.  While the results of this initiative will not be formally generalizable, the summative information will also be made available to other healthcare systems that may be able to draw lessons from the BMC and health center experiences. A clear statement of the limitations of this study will be presented with any (formal or informal) sharing of results as well.

Process of reaching consensus on thresholds we will use to measure each key outcome:

Multiple steps will be involved in setting standards for assessing the success of the redesign efforts in meeting the BMC/health center improvement goals. This is not an area of research where there are well-established benchmark standards in the literature. Therefore, standards will be set using a consensus process and based on management and clinical experience and judgment. Since the referral process is not in itself a form of direct medical treatment of patients, the process that will be used will involve consensus building between clinicians and managers, as described below.

The first step in the process is to identify current indicators of standards already used in the BMC/HealthNet system. The project team has already completed this step and the indicators identified are reflected in the table below. Note that not all indicators currently have standards established at BMC/health centers. The research team will attempt to set baselines for thee indicators in step two. If a baseline cannot be set then that indicator may be removed from the research protocol.

Second, following OMB clearance, we will review medical records for the last 3 months following the protocol submitted to OMB. [Note, this will constitute the first half of the 6-month pre-implementation baseline for evaluation purposes; the second half will be reviewed immediately preceding the start of implementation.] For the purposes of setting standards, we will conduct this initial 3-month review to determine current practice, which will serve as baseline for each of the 11 indicators.

Third, with the information learned from steps one and two, the redesign group will develop draft standards using the review of current standards and current practice as the baseline to address a series of questions such as: What is current performance? From a clinical and management perspective, what are goals that can be set that are stretch goals but not impossible for the practices to reach? Where the gap between current practice and desired performance is large, what are interim goals that can build toward final success?

Fourth, the proposed draft standards will then be reviewed by health center and BMC ambulatory leadership and the clinicians in the participating health center and two specialties. The redesign team will present the proposed standards to leadership and clinicians at regularly-established meetings of each group. If there are differences of opinion, the process will require several meetings to share views and reach consensus. If full consensus cannot be reached, the leadership of BMC ambulatory and the health centers will make the final decisions, taking the clinical perspectives into account. Prior to final decision making, we will review the standards with AHRQ to obtain their approval.

Key Outcomes	Is there an established standard at BMC/Health Centers? If so, what is it?	Data source to determine outcome
Referral information is sent from PCP site to specialty site.	Yes - 100%	Paper records and EMR
Of those referrals, what percent were complete?	Yes - 80% -measured by inclusion of all key elements	Paper records and EMR
Percent of time was an appointment scheduled for patient?	Yes - 100%	EMR
Time between referral made and initial specialty care appointment made	Yes - Standard for other BMC services is 80% within 2 weeks	Paper records and EMR
Percent of time was PCP made aware appointment was scheduled?	Yes - 100%	EMR
Percent of time there is a note in medical record that referral received by specialist?	No current standard yet. We will determine this standard through the process that has been described in the introduction.	EMR
Percent of time PCP is notified if patient does not attend specialty appointment	No current standard yet. We will determine this standard through the process that has been described in the introduction.	EMR
Percent of time clinical information is available on referral form by the time of specialty appointment	Yes - 80%	Paper records and EMR
Percent of time specialist sends note to PCP after appointment?	Yes - 100%	EMR
Percent time there is note in the medical record that PCP received communication back from specialist	Yes - 100% - General expectation is that the PCP acknowledges that this comes back	EMR
Percent of time specialist note documents follow-up plan and responsibility	Yes - 100% - There is a general expectation that a follow up plan is documented but no standard for assigning responsibility	Paper records and EMR

Table 1, referenced in section A.2, reflects the strength of the evaluation design in using multiple data sources to address each evaluation objective, thus triangulating multiple perspectives. The timing of data collection for all data sources is estimated from time of OMB approval for ease of presentation even though not all are being reviewed.

II. Data sources: Evaluation data will be obtained from four sources:

• Medical record data related to referral processes. Medical record data will be abstracted by the BUSPH evaluation team from the electronic system, augmented where necessary by paper charts. The pre-implementation data will be drawn at Baseline: Months 0-2 and cover referrals made in the previous 8 months, prior to implementation of the redesigned referral system. ANSWER TO A21 (Question 7 from email): We believe this information was misunderstood. Months 0-2 meant WHEN we would perform the activity, rather than what data we would look at. The data we are talking about is electronic medical record data. We indicated that we would conduct the baseline during the time period of months 0 to 2 before implementation. We will be looking at medical record data from 8 months before to immediately before implementation, not during the first two months.

Post-implementation data will be drawn at Follow-up: Months 10-12 and will cover referrals for the previous 8 months, from the start of the implementation up to the final months of the project. ANSWER TO A22 (Question 7b in email): This is not a true effectiveness study, and therefore we did not plan to include a control group. We are going to be determining, via a pre-post comparison, whether our new process and design was successful in the clinics. This will be assessed by comparing referral component rates from the medical record review (pre-test) to rates after implementation of the project (post-test). The level of success will be measured by comparing the pre and post-test data to performance thresholds that will be set in the process described above. This project does not have sufficient resources for us to include a control group.

ANSWER TO A23: Separate samples will the abstracted for GI and OB referrals. Based on current referral levels, we estimate approximately 2185 GI referrals and 270 OB referrals for the diagnoses/procedures targeted in each 8 month review period. From this universe of referrals, we will review all OB referrals and will sample GI referrals stratified by primary care site to yield approximately 315 GI cases for review. Our sample will be selected using a random numbers table at each site and taking every 7^th referral.

• Focus groups with providers, clinical staff and administrative staff: Focus groups will be conducted in each primary care site and in each specialty practice in months 10-11. Focus groups will have 6-10 participants. We will organize separate focus groups for providers, other clinical staff and administrative staff in each practice. If a focus group for any grouping would contain fewer than three individuals (for example, if there are only 2 administrative staff in the specialty practice), we will conduct individual interviews using the same protocols. (In discussing focus groups in this document, we include any individual interviews that might be conducted for ease of presentation.) ANSWER TO A24: At each site there is an administrative manager or director, one or two referral coordinators, and three or four additional office staff for a total of 5-7 administrative staff per site.

Focus groups will be scheduled for the convenience of the participants with encouragement from senior clinic and practice leadership that this is an important activity. The focus group protocol is included in Attachment B. By necessity, the details of the questions will be tailored to the emerging discussion of each focus group as influenced by group dynamics, the professional perspectives and sites of care of the participants. The focus groups will be conducted by two-person teams from BUSPH with extensive experience in leading focus groups. Qualitative data will be collected on encrypted laptops and treated as confidential. ANSWER TO A25: We will inform participants that we will keep their responses private to the extent permitted by law. Only the BUSPH evaluation team members with the responsibility for data analysis will have computer access to the interview notes and to the subsequent coding and analysis documents, all of which will be stored on a secure network drive. Data will be reported only at a group level and no individuals will be identified.

• Implementation logs and meeting notes: These will be kept by the project team on a regular basis throughout the implementation of the redesigned system. The logs and notes will document the implementation process, including factors affecting the process, challenges encountered and strategies for dealing with the challenges.

III. Tabulations and statistical analyses

Medical record data. Standard empirical methods will be used to analyze the quantitative data collected from the medical charts and electronic data system. All data will be entered into a database using SPSS 16.0. After data are collected, we will perform data checking on a 10% sample of the data to check for data quality and missing data. As needed, any errors will be remedied by going back to the medical charts and the electronic record. Once the database is completed, we will perform standard statistical analyses, specifically descriptive statistics including frequencies, measures of central tendency, and measures of variability. Data will be stratified by site, by provider type (i.e., primary care provider versus specialist, MD versus other clinical staff, including RNs and PAs), by referral type requested and potentially by other characteristics identified in the redesign process. We will examine change from pre-implementation to post-implementation using bi-variate analyses, such as t-tests and chi-square tests, as appropriate. If we have sufficient data, we will estimate regression models using the different performance measures as our outcome variables. Explanatory variables will include site, provider type and procedure/consultation type requested.

Focus group, implementation logs and meeting notes. These data will be analyzed qualitatively. Using an explanation-building analytic strategy, we will analyze in depth the dynamics of each site and identify organizational factors across sites. ANSWER TO A26 (Question 8 from email): An explanation-building analytic strategy is simply a fancy way of saying that we will analyze our data from the ground up, based on a content analysis, such as is done in grounded theory. Consistent with grounded theory in qualitative analysis, we will use the following process to analyze the qualitative data that we collect from the focus groups:

All transcripts will be entered as text files into HyperResearch, a qualitative software program, for data analysis. Analytic procedures will follow the general procedures of grounded theory methodology. ^{i, ii} Data will be analyzed through a process of thematic-content analysis. Transcripts will be marked by code words to identify passages indicating conceptually distinct themes. Passages associated with a given codeword will be extracted, collected from all subjects, and reviewed as a set marked by a codeword.

In conducting our analysis, we will use the data-driven inductive approach described by Boyatzis (1998),ⁱⁱⁱ which allows themes to emerge from the data using inductive coding. This inductive process of coding involves finding repetitive statements or ideas within the raw data and coding them prior to interpretation, thereby searching for themes that emerged from the data as being important to addressing the goals of our investigation. The process of thematic analysis involves identifying themes through “careful reading and re-reading of the data” (Rice & Ezzy, 1999, p.258).^iv This method is a form of “pattern recognition within the data, where emerging themes become the categories for analysis” (Fereday & Muir-Cochrane, 2006, p.4).^v This method of coding the data will allow us to organize the information to identify and develop themes to interpret the data.

Consistent with Miles and Huberman’s guidelines for comparative case studies (1994), we will work to understand each case before proceeding to cross-site explanations, and then will cycle back and forth between analytic strategies aimed at understanding case dynamics and understanding the effect of key variables. ANSWER TO A27 (Question 3 from email): Our key variables are the 11 outcomes from the medical records listed in our responses to comments A4 and A5.

Summaries organized by the OTM factors along with documentation of the implementation process will be the starting point for developing our understanding of each case. The analysis framework will also remain open to capture other system experiences and dynamics that appear to have a strong impact on the project. Based on the summaries, the evaluation team will develop analytic memos that describe the key factors which appear to influence redesign implementation and outcomes within each site. For cross-case analysis, we will code and sort the focus group and implementation team notes into descriptive meta-matrices organized by: 1) the OTM factors and, if appropriate, emerging factors; 2) levels of implementation fidelity; and 3) success in meeting redesign goals. The evaluation team will re-visit the individual case analyses in order to ensure that the cross-site explanations of important components are consistent with the explanations developed within each site.

4. Time Schedule and Publication Plan

Table 2: Timetable for Data Collection, Analysis, and Publication

[Months are calculated from time of OMB approval]

Activity	Expected Month (s)
Focus groups conducted	10-11
Medical record review (Baseline)	0-2
Medical record review (Follow-up)	10-12
Data preparation	2-12
Data analysis	10-13
Draft report preparation	12-13
Final report submission	14

A.17. Expiration Date Display Exemption:

An expiration date display exemption is being not sought.

List of Attachments:

Attachment A: Healthcare Research and Quality Act of 1999 (separate, not in this document)

Attachment B: Script for Introducing the Project and Description of the Focus Group for Recruiting Purposes

Attachment C: Revised focus group guide

Attachment D: Coordinating Care Focus Group Introduction

Attachment E: Implementation Log

Attachment F: Federal Register Notice (separate, not in this document)

Attachment B: Script for Introducing the

Project and Description of the Focus Group for Recruiting Purposes

The goal of the Coordinating Care project has been to assess the current referral systems and identify mechanisms for improving both the referral process and overall coordination of care. Working with a redesign team, we have developed a set of new tools and processes for referring patients between primary care at three health centers and two specialties, OB and GI. You have now had the opportunity to use these new procedures for the past several months. We now want to know how the new procedures have worked for you. Therefore, to evaluate this redesign, we are conducting a series of focus groups and would like to invite you to participate. The focus group session will last 45 minutes, and will be facilitated by members of the study team from Boston University. Only information pertinent to the referral system will be gathered; no personal information will be elicited or discussed. The focus groups will be audio-taped; however, all data will be analyzed in the aggregate and kept private to the extent permitted by law.

ANSWER TO A28: Yes, the focus groups will be recorded and transcribed. All data and audiotapes will be kept in locked files in the office of the PI or Co-PI. When the tapes are transcribed, participants will be given false names. All data will be analyzed in the aggregate and individuals or positions will not be identified in the transcripts.

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

QUESTIONS A29-33 addressed here:

Attachment C: Revised Focus group guide

Introduction: (Interviewer please read) Thank you for agreeing to participate in this discussion group today.  We will be talking about the redesigned system for referring patients between primary care and two specialties, OB and GI, and the new tools and processes that have been put into place to implement the new system.  The goal of this project is to look at the referral systems currently in place and evaluate how those systems could be streamlined to improve the referral system and the overall coordination of care. We would like to talk with you about how the new system is working.  Please remember that your participation is entirely voluntary, you can chose to stop or leave at any time, and we expect the discussion to last between 45-60 minutes.  We will be recording the conversation so that we can code it and qualitatively analyze it later. The information you provide will be aggregated with information from other respondents and kept private to the extent permitted by law.  Should you have any questions, please feel free to contact the Project PI, Carol VanDeusen Lukas at [email protected] or by phone at (857)364-5685

As you are likely aware, over the past few months, the system for referring patients between primary care and two specialties, OB and GI, has been redesigned and new tools and processes have been put into place to implement the new system. We would like to talk with you about how you believe the new system is working.

1. How would you describe the redesigned referral system?

• What are the biggest changes you have noticed?

• What things do you like about it?

• What do you not like?

2. How well did the old processes work? (Probe for):

• What where key elements of the old processes?

• What were the major strengths and problems?

• Was redesign needed?

• Were you and your colleagues receptive to the changes?

• What concerns did you have about change to the system?

3. Do you think that the redesigned system has increase system efficacy?

• If yes, how? Probe for:

  • What is most and least efficient about it?

  • What tools are being used?

  • Are there good ways to work around problems with the referral process?

4. Has the redesigned system improved the referral process?

• If yes, in what ways?

• What parts of the new system are most successful? Probe for:

  • Tools and procedures working smoothly

  • Referral times shortened

  • Higher proportion of patients get to specialists

  • Specialists receive adequate and timely information from the primary care physician about the referral

  • Primary care physicians receive timely information back from specialists

  • Lapses and miscommunications are sharply reduced

  • Trust and joint accountability between primary care and specialty practices has increased

  • Patients know what to expect and what to do

  • Patients receive test results and other relevant feedback

5. Are there remaining problem areas or areas needing further improvement?

• If so, what are they and how could they be addressed? Probe for:

  • Procedures and tools

  • Communication patterns

  • Resource support

  • Patient involvement

6. In general, was the process by which the redesigned system was put into place successful? Probe for information about:

   • Way it was introduced

   • Training provided

   • Technical assistance provided

   • Other resources needed

7. Were there any problems with the implementation process? Probe for information about:

• the training provided

• the way it was introduced

• the technical assistance provided?

• Other concerns? How could the process be improved?

8. What barriers and facilitators affect the referral process? Probe for:

   • Priorities competing for attention

   • Clinic/practice leadership support for the redesigned system

   • History of relationships between primary and specialty care

   • Organizational culture including receptivity to change in their clinic and practices

   • Experience with strategies used to successfully introducing new clinical tools and processes

Attachment D: Coordinating Care Focus Group Introduction:

Thank you for agreeing to participate in this discussion group today.  We will be talking about the redesigned system for referring patients between primary care and two specialties, OB and GI, and new tools and processes have been put into place to implement the new system.  The goal of this project is to look at the referral systems currently in place and evaluate how those systems could be streamlined to improve the referral system and the overall coordination of care. We would like to talk with you about how the new system is working.  Please remember that your participation is entirely voluntary, you can chose to stop or leave at any time, and we expect the discussion to last between 45-60 minutes.  We will be recording the conversation so that we can code it and qualitatively analyze it later. The information you provide will be aggregated with information from other respondents and we will keep your responses private to the extent permitted by law.  Should you have any questions, please feel free to contact the Project PI, Carol VanDeusen Lukas at [email protected] or by phone at (857)364-5685

[Approved by Boston University IRB, May 2009)]

Attachment E: Implementation Log

To be completed by redesign team members responsible for facilitating implementation of redesign system in each practice.

Implementation site:

Implementation facilitator(s):

Time period covered:

Participants:

Goals for this period, activities that took place, technical assistance provided:

Problems encountered, changes from planned activities, why changes made:

Actions to resolve problems:

Lessons for improving implementation process:

Additional comments:

i (Strauss AL. Qualitative Analysis for Social Scientists. 1987, Cambridge/New York: Cambridge University Press; Glaser BG, Strauss AL. The Discovery of Grounded Theory; Strategies for Qualitative Research. Observations. 1967, Chicago: Aldine Pub. Co.).

ii (Strauss AL. Qualitative Analysis for Social Scientists. 1987, Cambridge/New York: Cambridge University Press; Glaser BG, Strauss AL. The Discovery of Grounded Theory; Strategies for Qualitative Research. Observations. 1967, Chicago: Aldine Pub. Co.).

iii (Boyatzis RE. (1998). Transforming Qualitative Information: Thematic Analysis and Code Development, Thousand Oaks, California: Sage Publications.)

iv (Rice PL, Ezzy D. (1999). Qualitative Research Methods: A Health Focus, South Melbourne, Australia: Oxford University Press.).

v Fereday J, Muir-Cochrane E. (2006).Demonstrating rigor using thematic analysis: A hybrid approach of deductive and inductive coding and theme development. International Journal of Qualitative Methods, 5(1):1-11.

/home/ec2-user/sec/disk/omb/icr/200905-0935-003/doc/11892902

DRAFT 11-4-08