Supporting Statement B - Workflow Assessment for Health IT Toolkit Evaluation 3-16-2012

SUPPORTING STATEMENT

Part B

Workflow Assessment for Health IT Toolkit Evaluation

Version: March 15, 2012

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 5

4. Tests of Procedures 6

5. Statistical Consultants 6

References 6

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The goal of this study is to evaluate the usefulness of AHRQ’s Workflow Assessment for Health IT toolkit (Workflow toolkit), which was published on AHRQ’s National Resource Center for Health IT Web site in July 2011. The target end users of the toolkit are:

1. Small- and medium-sized ambulatory care practice managers, clinicians, and staff; and

2. Clinic intermediaries such as professional organizations and associations, the Health Information Technology Regional Extension Centers (RECs), Office of the National Coordinator for Health IT (ONC) Curriculum Resource Center, and quality improvement organizations (QIOs).

There is no sampling frame available and users of the Workflow toolkit were purposefully selected from eligible practices to participate in the evaluation. Therefore study findings cannot be statistically generalized to the respondent universe. However, findings will be relevant to inform AHRQ about the Workflow toolkit’s usefulness, thus informing potential strategies for the refinement of the Workflow toolkit to better meet its users’ needs.

A purposive sample of 18 practices will participate in the project who are members of the AHRQ-sponsored Practice-based Research Networks (PBRNs) in Oregon and Wisconsin to achieve a sample of that is inclusive across the key attributes of:

1. current stage of health IT implementation;

2. practice size;

3. experience with health IT and IT implementation; and

4. degree of independence from external support and resources.

In addition, representatives from the RECs from the States of Oregon and Wisconsin will participate in the evaluation as they provide technical assistance to practices that are implementing health IT systems.

Exhibit 5 identifies the individual respondent types within each practice and REC, and outlines the total numbers of potential individual respondent interviews and respondent organizations in each category. Within practices, examples of potential Workflow toolkit users include: clinicians, office managers, front office staff, medical assistants or nurses, nurse care managers, social workers, health educators, information technology specialist, and quality improvement directors.

Examples of potential REC Workflow toolkit users include health IT specialists (IT implementation consultants and experts) or technicians, whose responsibilities include providing direct support to primary care practices in the implementation of health IT.

While there will be personnel from a variety of roles, uniform data will be collected across all roles.

Exhibit 5. Target Respondent Organizations and Individual Respondent Types

Organization Type	Number of Organizations	Individual Respondent Types	Maximum Number of Respondents per Organization
PHYSICIAN PRACTICES	18		14
Physician practices	18	Clinicians Office managers Front office staff Medical assistant or nurse Nurse care manager Social worker Health educator Information technology specialist Quality improvement director	252
REC RESPONDENTS	2		3
Regional Extension Center	2	Health IT specialists Technicians	6
PROJECT TOTAL	20		258

2. Information Collection Procedures

Data will be collected through one-on-one interviews and observations. These interviews will be performed by a trained interviewer and the observations will be conducted by a trained observer. There will be four types of data collections performed in support of this project:

1. Pre-Workflow Toolkit Interviews will consist of semi-structured one-on-one interviews with evaluation participants prior to using the toolkit. This interview will focus on practice and REC staff understanding and perceptions of workflow and of health IT implementation (see Attachments A and B for the Pre-Workflow Toolkit Interview Guides for practices and for RECs).

2. Observations will be conducted by research coordinators from the ORPRN and WREN teams, during four visits to each practice and REC. The ORPRN and WREN researchers will play a “participant-observer” role in these sessions, participating with the practice and REC staff in their workflow assessment activities while keeping field notes during and afterward about their observations regarding Workflow toolkit use and other workflow assessment activities (see Attachment C for the Workflow Toolkit Practice Activities and Perspectives Observation Log).

3. Usage Logs: Participants from the practice and REC staff will be asked to keep a record of workflow assessment activities and workflow toolkit use on a weekly basis when ORPRN or WREN research staff are not present. The record will include date and purpose of activities, use of workflow assessment toolkit, results or outcomes of the activity, and issues or concerns regarding toolkit use (see Attachment D for the Workflow Assessment Usage Log).

4. Post-Workflow Toolkit Interviews will consist of semi-structured one-on-one interviews with evaluation participants following ten weeks of workflow assessment with the toolkit. This interview will focus on changes in practice and REC staff understanding and perceptions of workflow and of health IT implementation, and on the impact of the process on themselves, the practice or REC, and its patients (for practices) or clients (for RECs) (see Attachments E and F for the Post-Workflow Toolkit Interview Guides for practices for RECs).

Each practice and REC will be engaged in data collection for ten weeks, with interviews, observation sessions, and other activities taking place on a mutually agreed upon schedule.

3. Methods to Maximize Response Rates

Our study design promotes defined periods of activity and engagement with practices and RECs such that we are unlikely to have participants stop contributing data midway through the project. This design includes project team members traveling to each practice and REC to introduce each practice and REC to the project and conduct interviews, followed by additional visits scheduled by project team members occurring at regularly occurring intervals over the evaluation period. This limited-duration visit schedule has proved successful with other studies conducted by the project team.

The study team will conduct interim data analysis prior to completion of each evaluation period to identify any missing or incomplete data, and will follow-up as needed with practices or REC staff to collect any missing data.

4. Tests of Procedures

Data collection and analysis procedures were developed by the ORPRN project team, led by Deborah Cohen PhD, an expert in qualitative methods.¹ Interview protocols and questions were developed by the ORPRN team and Dr. Cohen based on prior experience examining health IT implementation and practice innovation in small and primary care practices. Dr. Cohen will design and conduct the training of the project team members in Oregon and Wisconsin who will perform data collection activities. When training has been completed, we will identify a volunteer practice with 9 or fewer personnel to participate in a pilot test of data collection methods. This pilot test will be used to test the clarity and consistency of data collection methods and based on this pilot the data collection method instructions may be refined. During the initial data collection phase, Dr. Cohen and other senior researchers on the project team will conduct on-site supervision of the data collection process and prompt review of field notes to further ensure methods are being applied consistently in the field.

5. Statistical Consultants

This study will employ qualitative methods for data collection and analysis. Project coinvestigator Dr. Cohen is an expert on these methods. No statistical consultants were contacted.

AHRQ’s contractor, ORPRN, will be responsible for overseeing the recruitment of participants, conducting all of the data collection, and analyzing and reporting the findings. The principal investigator is Lyle J. Fagnan, M.D., and project director is Paul Gorman, M.D. Dr. Fagnan can be reached by phone at 503-494-1582 or by email at [email protected]. Dr. Gorman can be reached by phone at 503-494-4025 and by email at [email protected].

References

1. Cohen D, Crabtree B. "Using Qualitative Methods in Healthcare Research: A Comprehensive Guide for Designing, Writing, Reviewing and Reporting Qualitative Research." Robert Wood Johnson Foundation, July 2006. http://www.qualres.org/index.html

6