Supporting Statement Part A -- Evaluation of Phase I Demonstration 10-23-2009

SUPPORTING STATEMENT

Part A

Evaluation of Phase I Demonstrations of the

Pharmacy Quality Alliance

Draft dated: March 30, 2009

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that make the collection of information necessary 3

2. Purpose and use of information 4

3. Use of Improved Information Technology 4

4. Efforts to Identify Duplication 4

5. Involvement of Small Entities 4

6. Consequences if Information Collected Less Frequently 4

7. Special Circumstances 5

8. Consultation outside the Agency 5

9. Payments/Gifts to Respondents 5

10. Assurance of Confidentiality 5

11. Questions of a Sensitive Nature 6

12. Estimates of Annualized Burden Hours and Costs 6

13. Estimates of Annualized Respondent Capital and Maintenance Costs 7

14. Estimates of Annualized Cost to the Government 7

15. Changes in Hour Burden 8

16. Time Schedule, Publication and Analysis Plans 8

17. Exemption for Display of Expiration Date 9

List of Attachments 9

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

AHRQ proposes to conduct an independent evaluation of five Phase I demonstrations undertaken by the Pharmacy Quality Alliance (PQA). The PQA launched the five demonstration projects to test the feasibility of implementing a pharmacy provider report card system, which will be used to provide feedback to pharmacies on their performance. Each of the five demonstration projects involves pharmacists at multiple pharmacies, in addition to its own demonstration project staff. Staff descriptions vary by demonstration model but typically include a Project Leader, analysts, IT staff, and health plan and retail pharmacy leadership. The demonstration staff for each project are responsible for creating pharmacy performance reports for each pharmacy in their project, and for providing guidance to the end user pharmacists as to how to interpret the reports.

AHRQ’s overarching goal for the evaluation is to provide an external, independent analysis of the five demonstration projects, addressing factors that affected implementation of the pharmacy performance reports. By identifying factors that facilitate and hinder implementation, the evaluation is meant to inform future efforts to establish pharmacy quality reports.

The evaluation will be conducted for AHRQ by its contractor, the CNA Corporation and Thomas Jefferson University.

2. Purpose and Use of Information

To accomplish AHRQ’s goal of analyzing the implementation of pharmacy performance reports, the evaluation team will need to study the following aspects of implementation in each demonstration project:

• Steps taken

• Time spent and costs incurred

• Reactions to the performance reports among end-user pharmacists (including both the usefulness of the performance measures and the way in which the measures were reported).

In order to obtain information to analyze these elements of implementation, the project will need to collect data from the following two sources: (1) key staff from each demonstration project and (2) pharmacists from the pharmacies participating in the demonstration projects. Information from the demonstration staff will be collected using an interview guide (see Attachment B). Information from the pharmacists will be collected using a paper survey instrument (see Attachment C).

Prior to the start of demonstration staff interviews, the project leader at each site will receive an email communication (see Attachment H) which will 1) inform them of the nature of the interview, 2) request their assistance in identification of interview participants who hold specified job functions; 3) suggest that they gather necessary information ahead of time with respect to what resources were required to complete the project, 4) provide potential date(s) for the site visit, and 5) specify that timing of the interviews will be communicated directly to participants via email following confirmation of the site visit date.

3. Use of Improved Information Technology

For the on-site interviews, the interview guide will be built into a password-protected Microsoft Access platform with user-friendly data entry interface. Data will be captured during the interviews using a laptop that includes the electronic interview guide. The laptop will be operated by a trained research assistant from AHRQ’s contractor team while the second member of the team conducts the interview. Every attempt will be made to capture verbatim responses. However, in the event that parts of the interview requires clarification, the research assistant will play back relevant portions of the audio tape and will enter the accurate verbatim responses into the interview guide on the laptop. Each interview will be followed by a short 15-30 minute break during which such clarifications will be made. Only the interviewers and key personnel involved in data capture will know the password.

It should be noted that the AHRQ contractor thoroughly explored an online survey approach. Unfortunately, some of the participating pharmacies do not allow Internet access and therefore would not be able to complete an online survey. Although these pharmacies tend to have access to an internal company intranet, the survey would have to be uniquely built on each participating organization’s corporate platform in order to offer an online survey using that capability. Since the latter approach would require greater time and incur greater costs, and since mixed methods of data collection (e.g., paper survey for some pharmacies but an online survey for others) were not desirable, the evaluation team concluded that the only suitable option was to administer a paper survey.

4. Efforts to Identify Duplication

There are no formal efforts to identify duplication because CNA and Thomas Jefferson University (Jefferson) staff, along with AHRQ program staff, through extensive contacts with organizations and individuals in both the private and public sectors, know that there are no similar data available.

5. Involvement of Small Entities

It is not anticipated that the collection of information will substantively impact small businesses or other small entities, since three of the five demonstration sites involve large corporate pharmacy chains as opposed to small business pharmacies. For the remaining two demonstration sites, the participating pharmacies are a mix of chain, small chain, and independent pharmacies. Some of these pharmacies may be small businesses, so we have kept the pharmacy survey short in order to minimize the response burden.

6. Consequences if Information Collected Less Frequently

This is a one-time collection.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on Tuesday, February 10, 2009. Volume 74, No. 26, page 6630 for 60 days (see Attachment D). One non-substantive comment was received; no changes were made based on this comment.

8.b. Outside Consultations

AHRQ, as the original contracting agency on this particular research effort, concurred that this method of data collection was well suited to obtaining the information desired. Because this is a one-time data collection effort, we see no circumstances that might preclude consultation with representatives of those from whom information is to be obtained.

9. Payments/Gifts to Respondents

No gifts or payments will be made to respondents.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). Specifically, they will be told that their responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.

.

11. Questions of a Sensitive Nature

No questions of a sensitive nature, such as sexual behavior and attitudes, or religious beliefs, will be asked as part of the data collection. Questions will be asked about their knowledge, perceptions, and experiences related to pharmacy quality measures.

At the start of each interview, a consent statement will be read and participants’ consent will be audio recorded. The pharmacist survey will not require a signed consent form but rather states that the participant’s return of the completed survey will serve as his/her implied consent. Since the pharmacists in our sample are likely to have been engaged in the Phase I demonstration project by their employer, use of an implied consent will allow us to ensure anonymity and therefore encourage greater candor from the respondents and result in a higher response rate. Participants are further informed that their participation in the interviews or survey is voluntary, and refusal to participate will involve no penalty. The informed consent will be obtained in two steps. First, an introductory letter or direct contact will be made to respondents. The letter or direct phone call will outline the purpose, nature, and sponsorship of the research project, the confidentiality of the responses and the methods used to ensure it, as well as the description of the potential benefits of the participation to the respondents. The second step in obtaining informed consent involves reading an introduction to the respondents before conducting the in-person interviews and providing a similar written introduction to the survey. All interview participants will have a chance to review and discuss the informed consent with the data collectors, as well as actively consent to research participation before interviews. Contingent upon the consent of the respondents, the in-person interviews will be recorded for later reference and analysis. Recording the interviews is preferred because the interview questions are open-ended and therefore it will be useful to be able to refer to respondents’ full comments after the interview in order to ensure that all detail was captured in the written notes taken during the interview.

Although respondent positions and length of time in their current positions (as well as respondent names in the in-person interviews) will be collected, the final reports will not include any information that could identify specific respondents.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for this one year data collection. On-site interviews will be conducted with 6 staff members from each of the 5 demonstration projects and will last about 1 hour and 15 minutes. The survey of pharmacists will be completed by about 75 staff members from each demonstration project and is estimated to take 30 minutes to complete. The total estimated annualized burden is 226 hours.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents’ time to participate in this evaluation. The cost burden is estimated to be $10,753.

Exhibit 1.  Estimated annualized burden hours

Form Name	Number of projects	Number of responses per project	Hours per response	Total Burden hours
Demonstration Staff Interviews	5	6	1.25*	38
Survey of Pharmacists	5	75**	30/60	188
Total	10	--	--	226

* Includes average estimated pre-interview preparation time of 15 minutes.

** We expect that some demonstration projects will have fewer than 75 responses, but we are indicating 75 responses here to avoid underestimating the response burden.

Exhibit 2. Estimated annualized cost burden

Form Name	Number of projects	Total Burden hours	Average Hourly Wage Rate*	Total Cost Burden
Demonstration Staff Interviews	5	38	$47.58	$1,808
Survey of Pharmacists	5	188	$47.58	$8,945
Total	10	226	--	$10,753

*Based on the national average wage for pharmacists (29-1051), National Compensation Survey: Occupational wages in the United States May 2007, U.S. Department of Labor, Bureau of Labor Statistics.

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

The estimated total cost to the Federal government is $208,874. Exhibit 3 shows a breakdown of the costs.

Exhibit 3. Estimated Annual Costs to the Federal Government

Component	Total
Developing the interview guide and survey instrument	$33,905
Preparing OMB clearance submission	$6,704
Site visits to each demonstration	$73,368
Analyzing the data from each demonstration site.	$54,835
Preparing a final report	$40,062
Total	$208,874

15. Changes in Hour Burden

This is a new collection of information.

16. Time Schedule, Publication and Analysis Plans

Exhibit 4. Key Project Activities

Task #	Activities and Deliverables*	Dates
1.a	Initial Meeting	September 26, 2008
1.b	Draft Project Design Report	Due October 10, 2008
1.b	Final Project Design Report	Due November 7, 2008
1.c	Progress Reports	Monthly
1.d	Conference Calls with AHRQ and PQA	Monthly
1.e	Conference Calls with Demonstration Sites	Monthly
1.e	Introductory Phone Calls with Demonstration Sites	October 13, 2008 – October 31, 2008
1.e	Follow-up Calls with Demonstration Sites	Ongoing (as needed)
2.1	Review Site Plans	October 13, 2008 – November 14, 2008
2.2	Define Contractor and Site Roles	October 13, 2008 – November 14, 2008
2.3	Draft Evaluation Design Report	Due November 14, 2008
3.1	Meet with Sites, PQA, and AHRQ	November 19-20, 2008
3.2	Final Evaluation Design Report	Due within 2 weeks of receiving comments from AHRQ
4.1	Interview Guide, First Draft	Due November 14, 2008
4.1	Interview Guide for OMB Submission	Due December 29, 2008
4.1	Interview Guide, Final Revisions	December 29, 2008 – June 12, 2009
4.2	Survey Instrument, First Draft	Due November 14, 2008
4.2	Survey Instrument for OMB Submission	Due February 19, 2009
4.2	Survey Instrument, Final Revisions	February 19, 2009 – June 12, 2009
5.1	Submission of OMB Clearance Materials to AHRQ	Due December 8, 2008
5.2	Revise OMB Clearance Submission	December 8, 2008 – June 12, 2009
6.2	Determine Respondents and Schedule In-Person Interviews	January 26, 2009 – June 30, 2009
6.2	Conduct In-Person Interviews	July 1, 2009 – August 14, 2009
6.3	Determine Sample for Pharmacists Survey	March 1, 2009 – June 19, 2009
6.3	Administer Pharmacists Survey	June 15, 2009 – August 14 2009
7.1	Analyze Data for Each Demonstration	March 1, 2009 – November 27, 2009
7.2	Share Analysis and Results	November 30, 2009 – December 11, 2009
8	Provide Assistance to PQA Research Coordinating Council re Their Preparation of the Evaluation Design Plan for Phase II Demonstrations	March 11, 2009 – June 28, 2009
9.1	Draft Final Report	Due January 15, 2010
9.2	Final Report	Due February 12, 2010
10	Meeting with AHRQ and PQA	March 1, 2010 – March 12, 2010

* Note that the dates for activities and deliverables that are dependent on OMB clearance may change according to when OMB clearance is actually received.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

Attachments:

Attachment A -- AHRQ's Authorizing Legislation
Attachment B – Interview Guide for Demonstration Staff
Attachment C -- Survey of Pharmacists
Attachment D -- 60 Day Federal Register Notice, Vol. 74, No. 26, p. 6630, February 10, 2009.
Attachment E – Advance Letter
Attachment F – Cover Letter
Attachment G – Reminder Letter

Attachment H – Advance email to directors of the demonstration projects

10