SUPPORTING STATEMENT
Part B
Evaluation of the AHRQ Healthcare Horizon Scanning System
Version May 30, 2014
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 4
1. Respondent universe and sampling methods 4
2. Information Collection Procedures 6
2.a. Unusual Operation Issues 6
2.b. Statistical Precision and Minimum Detectable Differences 7
PART B: SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION
This package requests clearance to conduct an expert survey, stakeholder survey, and key informant interviews to support an evaluation of the AHRQ Healthcare Horizon Scanning System. The AHRQ Healthcare Horizon Scanning System alerts stakeholders (e.g., consumers, clinical researchers, health plans, health systems, government agencies, developers of new interventions) in the public and private sectors to emerging technologies and new ways of providing and managing patient care. The goal of the AHRQ Healthcare Horizon Scanning System is to enable more informed strategic planning for comparative effectiveness research (CER) priorities and for use by public and private decision makers considering new technology adoption and implementation. The evaluation is being conducted by AHRQ through its contractor, ECRI Institute; it is being implemented by Mathematica Policy Research.
The objective of the evaluation is to determine how effectively the AHRQ Healthcare Horizon Scanning System has implemented the key functions of identifying, monitoring, and assessing the potential for high impact of emerging technologies. The evaluation will also identify areas of improvement for the system.
The evaluation will analyze the accuracy, completeness, and usefulness of the Potential High Impact reports issued by the AHRQ Healthcare Horizon Scanning System. These reports discuss up to 20 individual interventions in each of the 14 AHRQ Priority Conditions deemed by expert comment processes to have potential for high impact. Primary data collection is necessary to obtain the data for the measures to evaluate the reports. Mathematica will ask experts to view a report and complete an online survey to rate the accuracy and completeness of the reports, and to provide their assessment of the potential for high impact for a set of interventions identified by the AHRQ Healthcare Horizon Scanning System. Cases of inaccurate or missing information reported by a sole expert in the expert survey will be confirmed through consultation with another expert. Mathematica will ask stakeholders to complete an online survey which will link them to a report and ask them to rate the relevance, clarity, and usefulness of the report. Mathematica will conduct semi-structured interviews with the AHRQ Healthcare Horizon Scanning System staff to learn about areas and suggestions for improvement in the identification, monitoring, and impact assessment processes. The evaluation will also include reviews of the reports by the evaluator to identify whether there were cases when an intervention should have been identified by the system earlier and to describe the variability in potential impact assessments over time. The review of reports will not involve primary data collection and therefore, are not discussed further in this clearance package.
The target population for the expert survey includes experts with in-depth knowledge of existing and emerging technologies in the Priority Condition areas. The sampling frame is the ECRI pool of external experts. ECRI maintains a database of experts, which notes their area of expertise (as identified by Priority Condition). However, Mathematica will exclude the two experts who provided direct input for the intervention report being reviewed. In the Priority Conditions with more than five experts, Mathematica will randomly select five experts. In the six condition areas where there are five or fewer experts, Mathematica will select all of the experts.
Table B.1. Number of External Experts by Priority Condition
Priority Condition |
Number of Available External Experts* |
Expert Sample |
Arthritis and nontraumatic joint disease |
7 |
5 |
Cancer |
62 |
5 |
Cardiovascular disease |
28 |
5 |
Dementia (including Alzheimer’s disease) |
5 |
5 |
Depression and other mental health disorders |
16 |
5 |
Developmental delays, attention-deficit hyperactivity disorder, and autism |
7 |
5 |
Diabetes mellitus |
6 |
5 |
Functional limitation and disability |
20 |
5 |
Infectious disease including HIV/AIDS |
15 |
5 |
Obesity |
3 |
3 |
Peptic ulcer disease and dyspepsia |
4 |
4 |
Pregnancy, including preterm birth |
5 |
5 |
Pulmonary disease/asthma |
5 |
5 |
Substance abuse |
5 |
5 |
Totals |
188** |
67 |
* This number refers to the number of available external experts after excluding the two experts who commented on a report.
** The total number of experts does not represent the number of unique experts as some individuals may serve as experts in multiple areas.
For the stakeholder survey, the target population includes the key stakeholders who are interested in CER and likely to use the AHRQ Healthcare Horizon Scanning System reports. The sample frame includes seven mutually exclusive groups and each group has its own sampling frame. The stakeholder groups include: (1) patient and consumer groups, (2) provider groups, (3) health care organization (HCO) leaders/decision makers, (4) health insurance plans and payers, (5) industry members, including pharmaceutical and device manufacturing companies, (6) government policy makers, and (7) researchers. To compile the sample frame, Mathematica will first identify the organizations and then the appropriate staff member within the organization, e.g., executive director, medical director, or principal investigator. To identify associations or groups in one of the first two key stakeholder groups (patient and consumer groups, provider groups),, Mathematica will review the Encyclopedia of Associations, 48th Edition (Gale, Cengage Learning 2009) and focus on those associations or groups that represent providers able to bill for Medicare covered services, as they would be the most likely users of CER information and would more frequently make health care decisions of interest to policymakers. Mathematica will use the 2011 Federal Register (for providers) to guide selection of organizations as associations relevant to key stakeholder groups. To identify associations and organizations that represented patients and consumers (in contrast to those that were grant-making organizations or service-delivery organizations), Mathematica will review the brief descriptions of the organizations in the Encyclopedia (and, when necessary, will consult organizational websites).
For health care organization (HCO) leaders/decision makers, Mathematica will draw the sample from the list of Medicare Accountable Care Organizations. Mathematica will use the membership list of America Health Insurance Plans, a trade association for payers, to identify the sample frame for the payers group. To identify industry organizations, Mathematica will use the membership lists of Pharmaceutical Researchers of America (PhRMA), Advanced Medical Technology (AdvaMed), and Biotechnology Industry Organization (BIO). Government policy makers will be represented by the state Medicaid directors. Mathematica will use the list of ARRA CER research grantees and the PCORI comparative clinical effectiveness awardees to develop the sample frame for the researcher group.
Mathematica will select a total of 700 stakeholders to participate in the survey. Mathematica will allocate the target sample to the stakeholder groups proportionally. For example, the sample size for the consumer and patient groups is computed as the ratio of the number in the frame for this group to the number of the total sample frame.
Table B.2. Sample Frame and Survey Sample by Stakeholder Group
Stakeholder Group |
Sample Frame |
Survey Sample |
Patient and consumer groups |
340 |
107 |
Provider groups |
495 |
155 |
Health care organization |
218 |
68 |
Health insurance plans and payers |
238 |
75 |
Industry members, including pharmaceutical and device manufacturing companies |
633 |
198 |
Government policy makers |
50 |
16 |
Researchers |
259 |
81 |
Totals |
2,233 |
700 |
For the key informant interviews, the sample frame include participants at nearly every stage of the AHRQ Healthcare Horizon Scanning System. For several positions, a single person fills the role; therefore, sampling is not an issue. For positions staffed by multiple individuals, limited resources do not allow us to interview every staff member. Mathematica will ask ECRI to provide a list of each staff member and their role and length of experience with the AHRQ Healthcare Horizon Scanning System. If staff possesses marked differences in levels of experience, Mathematica will seek to interview the most experienced people. Mathematica will also ask ECRI to provide a list of experts categorized by whether they are internal or external to ECRI and their perspective (for example, clinical, health systems). Mathematica will seek to interview experts with a variety of perspectives.
Table B.3. Interview Sampling Plan
Position |
Number of Personnel Participating in AHRQ Healthcare Horizon Scanning System |
Number of Interview Respondents |
Project Manager (PM) |
1 |
1 |
Content Team Leader (CTL) |
1 |
1 |
Analyst (A) |
7 |
6 |
Leads Manager (LM) |
1 |
1 |
Searcher (Se) |
8, |
2, |
Scanner (Sc) |
11 |
2 |
Expert (E) |
216 |
6 |
Director of Information Services (DIS) |
1 |
1 |
Project Director (PD) |
1 |
1 |
Expert Review Coordinator (PRI) |
1 |
1 |
Reference Manager (RM) |
1 |
1 |
Total |
249 |
23 |
Note: Abbreviations in parentheses show how respondents are indicated in the interview protocol.
AHRQ does not expect any unusual operation issues.
Because of the limited size of the sample and responses for the expert survey, all reported scores will be descriptive simple averages and will not be associated with any measures of statistical probability, such as confidence intervals.
For the stakeholder survey, Mathematica will analyze and report the usefulness of the reports for the full sample; Mathematica will not compare differences in usefulness across the stakeholder groups. Mathematica will report the overall usefulness score and the margin of error at a 95 percent level of confidence.
No measures of statistical probability will be produced for the interviews as we will conduct a qualitative analysis of the interview data.
To maximize completion of the expert survey, Mathematica will email the experts an invitation that will include the link to the survey website and their log-in information to facilitate access to the survey. The email will include two attachments, an endorsement letter from AHRQ and ECRI that will emphasize the importance of the evaluation and their participation. Mathematica will send three reminder emails (week 4, 8, and 11). Mathematica will also offer an incentive of $75 to compensate them for the time to complete the survey. Given their relationship with ECRI, Mathematica expects a response rate of 80 percent, producing 54 completes from the 67 sampled experts.
For the stakeholder survey, Mathematica expects to obtain email addresses for 60 percent of the sample. To maximize response to the stakeholder survey, Mathematica will email the sample an invitation that will include the link to the survey website and their log-in information to facilitate access to the survey. The email will include two attachments, an endorsement letter from AHRQ and ECRI that will emphasize the importance of the evaluation and their participation. For the sample members without email addresses, Mathematica will mail an advance letter with the two endorsement letters. Mathematica will send three reminder emails/letters (week 4, 8, and 11). Mathematica will also offer an incentive of $25 to compensate them for the time to complete the survey. Mathematica expects an ineligibility rate of 30 percent and a response rate of 65 percent from the eligible sample members, producing 320 completes from the 700 sampled stakeholders.
Given that the expected response rate is lower than 80 percent, Mathematica will conduct a non-response bias analysis to assess the degree of bias, if any, that would result when those in the sample who did not respond differ in important respects from those that did. The final analysis weights will account for the probability of selection and differential nonresponse, so that the responding sample represents all stakeholders in the target population to the extent possible. Nonresponse can lead to bias and hence inaccurate estimates when those in the sample who did not respond differ in important respects from those that did. In evaluating the response patterns for the weighting activities, Mathematica will compute response rates for different stakeholder groups to look for variation among these rates and compare the characteristics of respondents and nonrespondents that are available in the sample frame, such as type of organization and location.
Mathematica expects to achieve a 100 percent response rate for the key informant interviews. After Mathematica receives the contact information for the sample members from ECRI, they will email each sample member to explain the general goals of the evaluation and specific goals of the interview, and indicate that ECRI has provided their contact information for these interviews. We will schedule the interviews at a time that is convenient for the respondent and responsive to their schedule.
For the expert survey, Mathematica will pretest the survey with two clinicians to make sure the concepts and survey questions are clear and that the survey can be completed within 20 minutes.
For the stakeholder survey, Mathematica conducted cognitive testing with two stakeholders to explore the dimensions of usability. After refining the instrument based on the cognitive-interview data, the instrument was pretested with six individuals representative of the stakeholder categories of interest. The pretest focused on the format, the ease or difficulty of answering the questions, respondents’ understanding of the questions, and the time required for completing the questionnaire.
The key informant interview protocol will not be formally pretested as it is considered a guideline for discussions, rather than a highly structured interview guide. However, after the first two sets of interviews, Mathematica will hold a debriefing to discuss how the protocols are working and make any necessary modifications. Additionally, Mathematica will train their interviewers to understand the kind of information we need to collect and the meaning of particular questions, so they can communicate that effectively to interviewees.
John Hall, Senior Fellow, Mathematica Policy Research has been consulted on the statistical aspects of the design.
| File Type | application/msword |
| File Title | Supporting Statement Part B |
| Subject | <p>(Supporting Statement Template and Instructions; instructions are in italics) SUPPORTING STATEMENT Part B |
| Author | Heather Nalls |
| Last Modified By | Schoelles, Karen |
| File Modified | 2014-05-30 |
| File Created | 2014-05-30 |