SUPPORTING STATEMENT
Part B
Agency for Healthcare Research and Quality (AHRQ)
Evaluation of AHRQ's Effective Health Care Program
February 18, 2010
(Revised)
Agency for Healthcare Research and Quality (AHRQ)
U.S. Department of Health and Human Services
B. Collections of Information Employing Statistical Methods 2
1. Respondent universe and sampling methods 2
2. Information Collection Procedures 3
3. Methods to Maximize Response Rates 5
4. Tests of Procedures 5
5. Statistical Consultants 5
One of the data collection methods used in this project will employ statistical methods: the online survey. The other two data collection methods (key informant interviews and the Appreciative Inquiry [AI] workshop) are strictly using qualitative methods and will not employ statistical techniques.
The online survey will be administered and tailored to two distinct, yet related groups: 1) EHC Program Research Centers Staff and 2) EHC Program Users and Stakeholders. The respondent universe for the online survey is all individuals involved with the EHC program, excluding AHRQ staff (see Table 1), based on a list of Research Centers staff and a contact list of program users and stakeholders maintained by the EHC program. The IMPAQ team plans to survey the universe of potential survey respondents, which are drawn from the AHRQ EHC program contacts database.
As shown in Table 1, the IMPAQ team will survey the universe, which is estimated to be approximately 400 at the time of the survey. This universe includes approximately 144 EHC Program Research Centers Staff and 256 program users and stakeholders. Based on an estimated response rate of 80%, the target number of survey respondents is 320, which will yield a sufficiently large sample to obtain reliable estimates.
Table 1: Universe for Online Survey, by Strata
EHC Program Research Centers Staff |
Approximate Number* |
|
8 |
|
88 |
|
36 |
|
12 |
Subtotal |
144 |
EHC Program Users and Stakeholders |
|
|
13 |
|
4 |
|
8 |
|
31 |
|
32 |
|
51 |
|
7 |
|
13 |
|
58 |
|
25 |
|
13 |
Subtotal |
256 |
TOTAL ESTIMATED UNIVERSE OF THE SURVEY RESPONDENTS* |
400 |
Information Collection Procedures for the Online Survey. The online survey will be created and conducted using Survey Monkey, an intelligent survey software designed to create professional online surveys quickly and easily, and at low cost. The software provides encryption capabilities for ensuring confidential responses and for the exporting of responses for future analyses. The software also allows for the identification of responders and non-responders without compromising the confidentiality of responses.
The survey will include a variety of types of questions.
Multiple choice questions
True/false questions
Rating scale-type questions
Open-ended response questions
Data will be analyzed using a combination of Microsoft Excel, Microsoft Access, SAS (Cary, NC, USA) and/or STATA software programs. Quality control procedures such as consistency and out-of-range value checks will be performed prior to analysis. If more than half of the required item responses are missing on a questionnaire, the participant who submitted it will be excluded from the analysis. However, if the respondent provides more than three quarters of the required responses, then the missing responses will be imputed using imputation techniques (such as mean response imputation or hot-deck imputation method). If the respondent provides less than three quarters, but more than half, of the required responses, missing items will not be imputed, but the responses may included as deemed appropriate. The pool of potential donors in the imputation will include respondents deemed to be similar to the respondent who provided incomplete responses. We will determine the exact imputation techniques upon the analysis of reasons for receiving the missing responses.
Protecting Confidentiality. In the survey invitation and introduction, respondents will be fully informed and assured that their answers will be kept strictly confidential to the extent permitted by law. They will also be informed that participation is entirely voluntary. The survey will omit names, addresses, telephone numbers and other key informant identifiers in the final data file. The steps that we will employ to protect respondents’ confidentiality will include:
Minimize the use of names or other identifiers
Dissociate identifiers from survey responses
Present the findings in a way that knowledgeable observers cannot identify a respondent based on background/demographic strata or tabular presentation of results
Keep the survey data secure.
To minimize the use of names or other identifiers, we will compile a list of emails, randomly selected from the list, and separately email each potential respondent the URL for the on-line survey. When the survey is complete, we will remove all email addresses from our files, so that there is no link between respondents and their email addresses.
When presenting the analysis results, if any of the categorical “cells” in presentation tables have such a small number of respondents that a knowledgeable person could infer the identity of a respondent, we will collapse categories (rows and/or columns) in the tables so that each cell in the table holds a larger number of respondents. We will also present survey results for the group as a whole. For example, we will present the results for the center/network as a whole (e.g. the DEcIDE network), but not at the level of the 13 individual DEcIDE centers. If, however, a question is asked that only individuals from one of the 13 centers could possibly answer, the categories will be collapsed to show all respondents together for all programs/centers. Similarly, results for stakeholders will be presented as coming from Federal government, healthcare providers or other sectors, but will not indicate which part of the Federal government (e.g. FDA, NIH) or sector respondents come from.
Lastly, the assurance that the survey data are secure is a final step in protecting confidentiality. IMPAQ International (IMPAQ) has implemented stringent systems, protocols, and procedures to ensure security of our offices, computers, and data. Our resources include state-of-the-art server hardware, uninterruptible power supplies, fiber optics switches, extensive physical security measures, and wide-ranging redundancy. To prevent access to the data by unauthorized users, IMPAQ utilizes data encryption, strict access control policies for sensitive data files, closing unnecessary ports into servers, and training staff in security issues. We also work closely with an independent security firm that performs regular scans of our network, audits our procedures, and continually updates our in-house security team with current security best practices. These policies and procedures are monitored by IMPAQ staff members that have completed network security certifications. The IMPAQ team’s databases and computers are password protected, with only the data administrators having “write” authority over files. Electronic data transferred via diskette or CD-ROM to clients is encrypted and password-protected before shipping via a bonded courier.
Minimizing or Mitigating Bias. Internal AHRQ staff may be more inclined to report favorably on the program because of their roles. Additionally, because there are few internal AHRQ staff who would be included in the online survey sample, it will be difficult to sufficiently mask their identity and responses in the reported results. Therefore, we have decided to remove these individuals from the list of potential online survey respondents and instead only conduct key informant interviews with this group of AHRQ staff. We also anticipate minimizing bias in the survey by ensuring respondents that their confidentiality will be protected to the extent permitted by law (as described above).
As another step to prevent response bias, we have conducted a small-scale pre-test of the online survey instrument with a small number of people (no more than nine) who are familiar with the EHC program to detect and correct any potential problems with the content, terminology, phrasing, and structure of the survey instrument, the appropriate use of skip patterns, and other survey implementation issues that may bias the responses or response rates of particular respondent groups. For example, we will consider open-ended items to ask about both what is working well and what can be improved. That way, respondents are much less likely to feel like they are just providing the negative, but are encouraged to provide both what is and is not working. Also, the IMPAQ team will use relatively short, straight forward questions, use neutrally worded questions that are not leading, and avoid “double barreled” items.
To ensure a high response rate, we will: (1) give presentations about the evaluation to EHC program researchers to raise awareness about the survey before the data collection; (2) send an official invitation letter from the Director of the Center for Outcomes and Evidence at AHRQ, Jean Slutsky, to potential survey respondents; (3) send an EHC program-sponsored cover letter to the selected sample; (4) inform participants that the survey will take approximately 20 minutes in the pre-invitation and official invitation; (5) monitor the response rates; and (6) e-mail non-respondents with two reminder emails to encourage their participation. If the response rate becomes concerning during the data collection stage, AHRQ will assess whether providing some incentives will boost the response rate.
Additionally, most respondents are involved in the EHC program and therefore have a vested interest in participating in the survey. All respondents have email addresses and presumably regular internet access. The expected response rate for the online survey is 80%.
Pretesting of the online survey has been conducted by not more than nine (9) federal staff involved with the EHC program at AHRQ to refine the data collection instrument contents and survey length, thereby minimizing the burdens on respondents and improving utility of the instrument. Minor changes have been made to the survey as a result of pretesting, in which case the final, revised version of the data collection instrument will be further pilot-tested and submitted in a memorandum to OMB prior to implementation of the survey.
AHRQ has contracted with IMPAQ International and Abt Associates to collect and analyze the survey information.
Philippe Gwet, PhD, (phone: 301-326-9001), a statistician employed by IMPAQ International, provided inputs regarding the development, design, conduct, and analysis of the online survey.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | msacchetti |
| File Modified | 0000-00-00 |
| File Created | 2021-02-03 |