Supporting Statement Part B -- Synthesis Reports for THQIT 10-12-10

Supporting Statement

Part B

Synthesis Reports for Grants and Cooperative Agreements for Transforming Healthcare Quality through Information Technology (THQIT)

Version: October 12, 2010

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Information Collection Procedures 1

3. Methods to Maximize Response Rates 2

4. Test of Survey Procedures 2

5. Statistical Consultants 3

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

For the 3 grantee surveys the universe of relevant grantees (n=118) is not sufficiently large to allow for sampling so the entire universe of grantees will be included.

For the in-depth interviews (n=30), grantees will be purposefully selected based on such characteristics as the topic they chose to focus on (selection will emphasize, but not be limited to those with rural hospital electronic health records [EHR] and/or health information exchange [HIE] projects), their having completed a final report to AHRQ of reasonable quality, their having completed the web-based survey, and their location and type of care settings involved, as the aim is to include projects from a mix of geographic regions and types of care settings.

The data from this proposed information collection will help AHRQ enhance the evidence base to support effective information technology (IT) implementation and add to knowledge about health IT by synthesizing and drawing lessons from its Transforming Healthcare Quality through Information Technology (THQIT) program.

2. Information Collection Procedures

For the grantee surveys data collection will use a self-administered web-based survey. The questionnaires for this survey were developed by Mathematica collaboratively with AHRQ, taking into account input from a technical expert panel (TEP). Three questionnaires were developed–one for each grantee type (value, planning, and implementation).

The web-based surveys will use a password-protected, secure website that will allow respondents to log in to their survey via the Internet. Respondents can complete the survey at their own pace and according to their own schedule. The survey invitation email and three reminder emails are contained in Attachment F and the instruments are included in Attachments B, C and D.

An advance email from AHRQ will be sent to all 118 grantees explaining the purpose of the survey and requesting their participation. The email underscores that AHRQ will use the information that respondents share to inform policy and technical assistance to those implementing health IT, as well as for refining their future grant-making and creating a practical tool to assist rural hospitals. The letter also notifies the grantees that they will receive an invitation email from Mathematica with a web link to the survey. All electronic communications with grantees are included in Appendix F.

For the in-depth interviews, data collection will be conducted via telephone. Once the grantees are selected, AHRQ will send an email introducing the study and asking for their participation. Mathematica will then email the grantees to request their assistance and outline the topics to be covered. In a follow-up call or email (as preferred by the principle investigator (PI)), the Mathematica scheduler will work with the grantee to identify the best one, two, or three respondents (depending on the project’s structure and complexity) to cover the protocol topics. We will be flexible according to the sense of the grantee as to the level and background of the individuals we speak with, and whether one, two, or three individuals are included.

3. Methods to Maximize Response Rates

This data collection has not been conducted previously but the estimated response rate is at least 80 percent. We anticipate this high response rate for three reasons. The subject matter and purpose of the survey is highly salient to the grantees, and the TEP that was convened indicated that if the instruments were designed to supplement and not duplicate other information collected by AHRQ (which is the case), grantees would feel the subject matter was important and would be willing to provide information about planning and implementing their health IT projects, and the resultant outcomes, if the survey was relatively short. Moreover, because of the length of the surveys, we have added plans to encourage response with an incentive described below. Third, Mathematica has a track record of achieving high response rates in part through effective initial contacts and non-response follow-ups.

Several tactics will be used to maximize response rates. First, AHRQ will send an email to the PIs of all the THQIT grantees introducing the effort, explaining its importance, and encouraging participation in the survey and the follow-up interviews, if selected. Second, grantees will be offered a $25 payment for each completed survey. Previously this has worked well to boost survey response rates.¹ A third way we have worked to maximize responses is to create self-administered web-based questionnaires, which allows respondents to complete the survey at their own pace and on their own schedule. Mathematica will also make a hard copy of the survey available if grantees prefer to complete it on paper. Up to three reminder emails will be sent to non-responding grantees (included in Attachment F). Finally, Mathematica will telephone non-respondents to remind them about the survey and to see if there are other accommodations that can be made to facilitate survey completion.

For the in-depth interviews, we will maximize participation by being very clear, straightforward, and specific from the start about what we need to know and what time commitment we are requesting from them (per the introductory email and topic lists provided in Attachment F). In our experience, this enhances response because the targeted individuals get an immediate sense that this effort is well-organized and will supplement not duplicate other information on their project. In addition, grantees will be offered $50 for participation in the in-depth interviews. We will follow up at approximately two-day intervals after the introductory email for up to three attempted contacts with an individual. An alternate individual working with the same grantee project may also be approached after that if an alternate contact is available. When a grantee refuses to participate (passively by not responding to us, or actively), we will substitute another grantee with similar characteristics (to the extent possible).

4. Test of Survey Procedures

The planning, implementation and value grant survey questionnaires were pretested with fewer than 10 respondents to discover any problems respondents might experience in providing the requested information and to make appropriate changes to the questionnaire. Pretest responses were collected via the web to emulate the self-administration that will be used for the survey. Mathematica staff followed up with pretest respondents by telephone to learn their reactions and determine how to improve language. The pretest also established the average length of time for respondents to complete the survey. The results of the pretest were used to revise the questionnaires.

The discussion guides for the in-depth interviews were not pretested, but the success of qualitative research with interview guides depends less on exact wording of questions than on the knowledge of the team developing the guides and the use of experienced staff to lead the interviews. These guides were developed by individuals with at least five years of experience involving health IT implementations (one had more than a decade of such experience). Similarly, the interviews will be led by senior team members who will actively listen to whether the grantee is responding well to the question and will clarify the intent of the discussion topic if the respondent seems confused or is not responding directly.

5. Statistical Consultants

Since this data collection will not involve a statistical design, no statisticians were consulted on the design of this project.

1 For example, two rounds of Mathematica surveys funded by NIH and targeted to executives within health plans (comparable levels to many of the individuals who will be interviewed here) have used $50 incentive payments and achieved 90 percent or higher response rates.

v