OMB Submission Form - Customer Satisfaction with AHRQ's HIE Evaluation Toolkit

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0935-0179)

T ITLE OF INFORMATION COLLECTION: Customer Satisfaction with HIE Evaluation Toolkit

Purpose:

The Agency for Healthcare Research and Quality (AHRQ) is a leading Federal agency in the development and dissemination of evidence and evidence-based tools on how health information technology (IT) can improve health care quality, safety, efficiency, and effectiveness.

In support of its health IT initiative, AHRQ developed the National Resource Center (NRC) for Health IT Web site. This site contains a range of information and evidence-based tools that support the health IT initiative’s work and aims, including the Health Information Exchange (HIE) Evaluation Toolkit.

With this project, AHRQ seeks to gather qualitative feedback for one of the tools available on the NRC site: the Health Information Exchange (HIE) Evaluation Toolkit. Developed in 2007, the HIE Evaluation Toolkit was designed to help guide users through the process of devising a realistic and achievable HIE evaluation plan. The HIE Evaluation Toolkit was revised and republished in 2009.

The goal of the project is to use qualitative feedback to improve the usefulness, comprehensiveness, relevance, effectiveness, and efficiency of the HIE Evaluation Toolkit for its users.

To achieve these goals, AHRQ will recruit up to fifteen (15) individuals that are materially involved in the development of an evaluation of an HIE project and who agree to use the tool in their specific HIE project to develop an evaluation plan at any time during the six-month project period. AHRQ will follow-up with participants on a monthly basis via e-mail to assess their progress in the development of the evaluation plan (see Attachment A for the Monthly E-mail Check-in Text) and if participants do not respond via e-mail AHRQ will follow-up with them via phone (see Attachment B for the Interview Guide for Monthly Check-In Phone Follow-up). At the conclusion of each participants’ involvement in the development of an evaluation plan for an HIE project, AHRQ will conduct in-depth interviews (see Attachment C for the In-depth Interview Guide).

Specifically, the in-depth interviews will assist in answering key questions including the following:

• How useful was the toolkit in helping in the formulation of the evaluation plan?

• What are the strengths and weaknesses of the toolkit?

• Which sections and components were used?

• Does the toolkit provide adequate level of detail?

• Do the topic areas of the toolkit apply to the participants’ experiences with HIEs and HIE evaluation?

• Were users able to successfully apply the toolkit to develop their evaluation plans?

• What improvements to the toolkit would assist them as they develop their evaluation plans?

Contact materials for recruiting and screening participants are included as Attachment D and Attachment E, and the final mail follow-up for non-responsive participants is also included as Attachment F.

This study is being conducted by AHRQ through its contractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to health care technologies. 42 U.S.C. 299a(a)(5).

Description of Respondents:

Respondents will be representatives of a health information exchange (HIE) organization or project, which can include employees, consultants, or staff from organizations participating in the exchange. The background and expertise of respondents are expected to vary. Many respondents are expected to have some graduate level training in research and evaluation methods and experience in evaluating programs using evaluation measures. If respondents are from more mature HIEs with academic partners they will likely be experienced evaluators of health information technology impact. Some respondents are expected to have limited experience in program evaluation and in conducting evaluations, or evaluation experience but not in the area of health IT research. Some of the respondents may be HIE staff such as managers who have little to no evaluation experience. Respondents must be materially involved in the development of an HIE evaluation plan and will have committed to using the HIE Evaluation Toolkit to develop their evaluation plan. “Materially involved” is defined as participating in design decisions and contributing directly to a significant portion of the evaluation plan.

TYPE OF COLLECTION: (Check all that apply)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ ] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [X] Other:

1. Screening Questionnaire (phone) for participation in project

2. Monthly Check-in (e-mail or phone)

3. In-depth Interviews (phone)

Certification:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Doris Lefkowitz

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected?

[X] Yes [ ] No

2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act of 1974?

3. [ ] Yes [X] No

4. If Applicable, has a System or Records Notice been published?

[ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants?

[X] Yes [ ] No

AHRQ will offer eligible persons $100 as an incentive to participate in monthly check-ins and in-depth interviews. This is currently the standard payment to health IT professionals for participation in interviews lasting up to 90 minutes. To be eligible participants will have to agree to use the toolkit within a given six-month time period, respond to monthly check-ins via e-mail (or phone), and participate in a telephone in-depth interview to discuss their experience using the toolkit. Although it may be possible to conduct this study while providing a smaller incentive amount to participants, experience suggests that doing so would increase the resources needed for recruiting participants with the desired background, thus increasing the overall project cost to the government.

Category of Respondent: Private Sector

Burden Hours:

The screening questionnaire will be completed by 30 representatives of an HIE organization or project and will take about 10 minutes to complete; 15 will screen-in and will be included in this project. Monthly Check-in Follow-up will be completed monthly for 5 months by one representative from each of the 15 participating HIEs and is expected to take 5 minutes. In-depth interviews will be conducted once with one representative from each of the 15 participating HIEs and will last approximately 90 minutes. The total burden for all data collections is estimated to be 34 hours.

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Screening Questionnaire	30	1	10/60	5
Monthly Check-in Follow-up	15	5	5/60	6
In-depth Interviews	15	1	1.5	23
Total	60	na	na	34

 

FEDERAL COST: The estimated annual cost to the Federal government is $55,000; data collection will not exceed one year.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe?

[ ] Yes [X] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

A convenience sample of participants will be recruited from health IT professional and trade associations, through contacts with participants in relevant Federal grant and other HIE initiatives, and through direct contact to personnel at health care organizations. Associations from which potential participants will be recruited include eHealth Initiative, the Healthcare Information Management Systems Society (HIMSS), the American Medical Informatics Association (AMIA), and the American Health Information Management Association (AHIMA). Relevant Federal grant or other HIE initiatives from which potential participants will be recruited include: Office of the National Coordinator for Health IT’s (ONC’s) State Health Information Exchange Program, AHRQ-funded HIE research projects under current and past grant programs, the Department of Veterans Affairs’ (VA’s) Virtual Lifetime Electronic Record (VLER) pilot, and ONC and AHRQ Health IT Research Center (HITRC) Collaboration Portal participants. Organizations from which personnel may be directly contacted and invited to participate include hospitals and large physician groups that have started private HIEs. Individuals who may be contacted are those who serve as chief information officer (CIO), chief medical information officer (CMIO), or practice managers, as appropriate.

Information about the project will be sent by e-mail (see Attachment D); organizations or associations will be asked to distribute the e-mail through their list serves and/or membership lists, participants of current HIE initiatives or individuals contacted directly will be sent the information either by their Federal program contact or project staff. The information will include a toll-free number for interested individuals to call that will be monitored by project staff who will then screen participants for suitability using a screening questionnaire (see Attachment E) and engage them in the project if eligible.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[X] Web-based or other forms of Social Media

[X] Telephone

[ ] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used?

[X] Yes [ ] No

List of Attachments

Attachment A – Monthly E-mail Check-in Text

Attachment B – Monthly Check-in Phone Follow-up

Attachment C – In-depth Interview Guide

Attachment D – Recruitment E-mail

Attachment E – Screening Questionnaire

Attachment F – Final Mail to Non-Responders to Two Monthly Follow-ups

6