Form #1 Form #1 Interview Guide for Demonstration Staff

Evaluation of Phase I Demonstrations of the Pharmacy Quality Alliance

Interview Guide for Demonstration Staff

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

Thank you for your participation today. The purpose of this interview is to gain a better understanding of management and implementation personnel perspectives on the PQA measures, and quality improvement.

This is an independent evaluation funded by the Agency for Healthcare Research and Quality (AHRQ). Individual responses will be kept confidential;private; Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c); results will only be shared in aggregate at the site level.

The interview should take approximately 1 hour to complete. Depending upon your position at the [demonstration site name], the interviewers may skip some of the questions.

This interview guide has been reviewed by the Institutional Review Board at Thomas Jefferson University and approved for exemption.

Demonstration staff member name: ___________________________________

Date of interview: __/__/____

Demonstration site name: ___________________________________

Title: ___________________

Organization type [Circle appropriate letter. The letter which is recorded here will be used to direct questions to project staff based on the skip patterns described in brackets at the top of each section in this interview guide. ]:

A)Pharmacy service provider

B)Pharmacy benefit manager

C)Insurer or health plan

D)University department or center

E)Pharmacy association/society

Check here if demonstration staff member is a Demonstration Project Leader: □

Public reporting burden for this collection of information is estimated to average 1 hour and 15 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 540 Gaither Road, Room # 5036, Rockville, MD 20850.

Permission to participate and audio record the interview. [Please read the following statement to the Demonstration Staff member] Before we begin the interview questions, I would like to obtain your audio-recorded consent to participate. In addition, I would like your permission to audio record this interview in its entirety. The audio recording will enable the evaluation team to review Because we will obtaining open-ended responses to many of these questions that may be difficult to capture on paper, particularly open-ended questions where you are encouraged to elaborate in your response. , we would like your permission to record this interview. Do you agree to participate, and to allow us to record this interview?

• Yes

• No

A. DEMONSTRATION STAFF CHARACTERISTICS (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED TO ALL ORGANIZATION TYPES]

1.How long have you been employed by [demonstration site name]?

2.How long have you been at your current position?

3.Will you please describe your background [probe: experience, training]?

4.What are the duties/responsibilities in your current position?

B.DEMONSTRATION PROJECT STAFF AND QUALIFICATIONS¹ (5 minutes)

[THIS SECTION TO BE COMPLETED ONLY BY THE PRIMARY DEMONSTRATION PROJECT TEAM LEADER AT EACH SITE]

Please complete the following table by listing the individuals who participated in the PQA Phase I Demonstration project at [demonstration site name], along with their title, qualifications (degrees/skills), role(s), and time commitment.

PQA Project Team Member Name	Job Title	Qualifications (e.g., degrees, skills)	Role(s)	Time Committed to the PQA Project
				Hours per Week	Number of Weeks

C.TIME COMMITMENT FOR PROJECT TASKS² (5 minutes)

[THIS SECTION TO BE COMPLETED BY ALL MEMBERS OF THE DEMONSTRATION STAFF]

In the table below, please write list in the number of hours per week as well as the number of weeks you worked on each listed specific task. If you were not involved in the task, mark the boxanswer NOT as N/A or NOT APPLICABLE.

Task	Time you committed to this task (hours per week)	Number of weeks you worked on this task
Obtaining data use agreements
Obtaining IRB approval
Acquiring the data for pharmacy reports
Developing data files for pharmacy reports
Coding the measure specifications
Customizing the template for the pharmacy reports
Tabulating performance data for pharmacy reports
Populating the pharmacy reports with performance results
Populating the pharmacy reports with consumer survey results
Maintaining the reporting system website
Reviewing pharmacy performance measure results
Reviewing pharmacy consumer survey results
Participating in the site’s training session
Obtaining feedback from pharmacists on the reports
Drafting interim report(s) for PQA
Drafting final report for PQA
Attending PQA meetings and conference calls
Other (please specify)

D.ORGANIZATION’S BACKGROUND IN QUALITY IMPROVEMENT (5 minutes)

[THE FOLLOWING QUESTIONS WILL ONLY BE ASKED TO ORGANIZATION TYPES A, B, AND C]

Perceived importance of quality measurement within the organization

These next questions pertain to [demonstration site name] experience in quality improvement outside of the PQA Phase I Demonstration project.

Quality improvement is a team effort of identifying opportunities for improvements, measuring performance, and involving the frontline providers and staff members to find ways to improve performance. Quality improvement goes through repetitive cycles of measuring performance, testing change concepts and then re-evaluation of outcomes measures³

1.Based on the definition above, is [demonstration site name] currently involved in quality improvement initiatives?

2.If so, to what extent is [demonstration site name] currently engaged in quality improvement activities?

3.Does formal staff training in quality improvement exist within [demonstration site name]?

4.In your opinion, how important is it to [demonstration site name] to improve pharmacy quality?

5.In general, how aware do you feel [demonstration site name] is about pharmacy quality?

E.EXISTING QUALITY MEASUREMENT INITIATIVES AT THE PILOT SITE (5 minutes)

[THE FOLLOWING QUESTIONS WILL ONLY BE ASKED TO ORGANIZATION TYPES A, B, C]

These next questions pertain to [demonstration site name]’s experience in quality improvement outside of the PQA Phase I Demonstration project.

1.What quality improvement topics or issues, if any, were discussed and/or acted upon at [demonstration site name] during the past year? [probe: specific measures, accreditation providers]

2.Can you describe cases in which quality data were used as the basis for decisions and other actions outside of a formal quality improvement process?

3.If [demonstration site name] is not currently engaged in quality improvement activities, what would need to occur to begin to consider implementing these activities? [probe: market forces, pharmaceutical industry initiatives were offered to measure quality, government regulations were implemented which required measurement of quality, senior management initiatives]

4.Were you provided with adequate resources to support your role? [probe: why or why not]

5.If resources were not adequate, what was lacking (i.e., IT, office space, analytic support, management support)?

6.Overall, did you feel capable of measuring and reporting pharmacy quality with the resources provided?

F.ANALYTICAL METHODS AND DATA SOURCES (10 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED TO ALL ORGANIZATION TYPES]

Performance Reports

1.Who completed the analyses for performance measure calculations?

2.Did the specifications provided for each measure by PQA, with support from NCQA, meet the project’s analytical needs? [probe: if not then why?]

3.How does [demonstration site name] consider statistical error in reporting actual performance differences among pharmacists?

4.How did [demonstration site name] employ case-mix and severity adjustment to make fair comparisons?

5.How did [demonstration site name] include or exclude outlier cases in the measurement?

Pharmacy Consumer Survey

1.How was the sample selected to complete the Pharmacy Consumer Survey? [probe: random sample versus convenience sample]

2.How many customers were administered the Pharmacy Consumer Survey?

3.How many customers responded to the Pharmacy Consumer Survey?

4.Was an incentive offered to customers who completed the questionnaire? [if so, what was the incentive that was offered to customers who completed the questionnaire]

Data Verification

1. How did [demonstration site name] verify or audit the data generated for the performance reports?

2. Who verified the results generated for the performance reports and the pharmacy consumer survey?

G.PERSONNEL TRAINING ABOUT PHARMACY QUALITY (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED TO ALL ORGANIZATION TYPES]

1.Please describe the training program you [attended if demonstration staff member/administered if demonstration project leader]?

A)What were the topic(s) covered?

B)Who participated in the training?

2.What was the format of the training? [probe: in-person, web-based, paper, other]

3.Besides the initial session, were any additional training sessions offered?

4.Did your [demonstration site name] already ask you to evaluate the training program? [If yes, then what was the evaluation process?]

5.Overall, how would you rate the training program?

6.On a scale from 1 to 5, where 1 is “Low” and 5 is “High”, how would you rate your comfort level in relation to the topics covered in the training?

Before training 1 2 3 4 5

After training 1 2 3 4 5

7.On a scale from 1 to 5, where 1 is “Low” and 5 is “High”, how would you rate your overall knowledge of the topic(s) covered in the training?

Before training 1 2 3 4 5

After training 1 2 3 4 5

8.How satisfied were you with the training personnel’s knowledge [if in-person training]?

9.Were you satisfied with the location of the training session [if in-person training]?

10.Were you satisfied with the quality of the training?

11.What could be done to improve the quality of the training?

H.PERFORMANCE MEASURE EVALUATION (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED OF ALL ORGANIZATION TYPES]

The next series of questions pertain to your opinions about the 15 PQA Performance Measures.

Measure Importance

1.Were the measures easily interpreted?

2.Did the measures address significant health conditions?

3.Did the measures relate to activities that have high financial impact to the pharmacy? To the healthcare system?

Scientific acceptability

4.Do the measures make sense logically and clinically?

Feasibility

PART II.Do you think the measures impose an inappropriate level of burden on the pharmacy system?

PART III.Do you have doubts about the accuracy of the measures?

PART IV.Do you have doubts about the measures overall?

PART V.Do you have doubts about the relationship between the measures and the quality of customer care at the participating pharmacy?

A.DATA OR MEASURES THAT DO NOT EXIST BUT WOULD BE USEFUL (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED TO ALL ORGANIZATION TYPES]

1.Thinking about the 15 PQA measures are there any diseases that were not but should have been included in the measure set?

2.Thinking about the 15 PQA measures are there any additional elements that should be included in the measure set?

B.USABILITY OF THE PERFORMANCE REPORTS (10 minutes)

[THE FOLLOWING QUESTIONS WILL ONLY BE ASKED TO ORGANIZATION TYPES A, B, C, AND E]

PART VI.How confident are you that the performance reports are accurate measures of quality performance? [probe for explanation]

PART VII.Did you find the performance reports easy to interpret? [if not easy to interpret, then why?]

PART VIII.Were the performance reports easy to access? [probe: why or why not?]

PART IX.Did you find the performance reports to be user-friendly? [probe: why or why not?]

PART X.Did [demonstration site name] provide supporting information that served to supplement or explain the performance reports? [probe: educational materials, additional training, tailored/customized feedback]

PART XI.Did the reports allow each pharmacy to compare its performance to other pharmacies? [probe: by region or pharmacy type]

PART XII.Did the [demonstration site name] seek your feedback on the reports? If so, how? [SKIP IF PROGRAM LEADER]

PART XIII.Was the feedback used to improve the performance reports?

A.USE OF INCENTIVES AND/OR PENALTIES TO ENCOURAGE PARTICIPATION (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED TO ALL ORGANIZATION TYPES]

Incentives to promote pharmacy staff participation

PART XIV.Was any type of incentive program implemented as part of this project?

PART XV.If so, do you think the incentive program increased participation in the project?

Penalty / negative consequences for pharmacy staff non-participation

PART XVI.Did the [demonstration site name] use any form of negative consequences for staff who did not participate or fail to complete the demonstration project?

PART XVII.If so, did implementation of these consequences increase participation or the number of pharmacists who completed the program?

A.ORGANIZATIONAL CULTURE (10 minutes)

[THE FOLLOWING QUESTIONS WILL ONLY BE ASKED TO ORGANIZATION TYPES A, B, C, AND E]

Leadership Support

1.Was there a commitment by the executive leadership of [demonstration site name] in supporting the PQA Phase I Demonstration project?

2.How did executive leaders express support of the PQA Phase I Demonstration project?

Appreciation for Pharmacy Quality and Sense of Empowerment

1. Has your understanding and appreciation for pharmacy quality measurement increased as a result of this project?

PART XVIII.If not, what could have been done differently to increase your understanding and appreciation for pharmacy quality measurement?

PART XIX.Did participation lead individuals to feel empowered to improve quality in the pharmacy?

PART XX.Among what personnel, if any, did you observe an increased interest in pharmacy quality [for program leaders only]?

A.FUTURE DIRECTIONS (5 minutes)

[THE FOLLOWING QUESTIONS WILL BE ASKED OF ALL ORGANIZATION TYPES]

PART XXI.Will the [demonstration site name] continue to report on the pharmacy quality measures after completion of the PQA Phase I Demonstration project?

PART XXII.Is the [demonstration site name] planning to continue to use the Pharmacy Consumer Survey?

PART XXIII.What were the key lessons learned about pharmacy quality measurement as a result of this demonstration project? [probe: pearls and pitfalls]

1. 4. [if time permits] Do you have any final thoughts or comments that you would like to share with us about the project in the time remaining?

CLOSURE STATEMENT: This completes our interview. On behalf of AHRQ and the evaluation team at CNA and Thomas Jefferson University, we greatly appreciate your time today, and the time that you have spent with us throughout the project. Thank you for your participation.

1 This section will be distributed to the primary project leader at each PQA Phase I Demonstration Site prior to the Evaluation Team’s site visit and will serve to prepare the project leader to furnish detailed data about the personnel and time requirements of the Phase I Demonstration project. The project leader will be provided with this material at least 1 week prior to the site visit so that they are prepared to respond at the time of the interview.

2 This section will be distributed to each interview participant prior to the Evaluation Team’s site visit and will serve to prepare the interviewee to furnish detailed data about the personnel and time requirements of the Phase I Demonstration project. Interview participants will be provided with this material at least 1 week prior to the site visit so that they are prepared to respond at the time of the interview.

3 American College of Physicians. Available at: http://www.taim.org/i4a/pages/index.cfm?pageid=798. Access date: February 27, 2009.

CNA / Jefferson Medical College 12