Date: August 28, 2009
Subject: Responses to OMB Inquires dated August 25, 2009 Regarding Project “CAHPS Field Test of Proposed Health Information Technology Questions and Methodology.”
This document provides responses to comments received by the CAHPS team on August 25, 2009. They are located in the file titled “ICR_CAHPSHealthIT.” Each OMB comment is provided, with the page number on which it was made. Our responses are provided directly below each comment.
SUPPORTING STATEMENT: Part A
Comment A1: Are there hypotheses on how responses will differ? If they differ, how will it be determined which scale to use for the future? (Page 4)
Response: The hypothesis is that the 6-point response scale will yield more reliable responses than the 4-point response scale. If the 6-point turns out to be substantially more reliable, in the future we will encourage organizations that want to use the instrument to consider using the 6-point version.
Comment A2: Please clarify: are you testing the AMOUNT (i.e. $0 vs $5) or TIMING (pre-paid or post-paid) or both? (page 4)
Response: We are interested in whether an incentive, in general, is helpful for participation rate. The use of incentive was at the request of one field test site which often uses incentives for their member surveys. We will randomize half the sample at this site to the post-paid incentive of $5. We will compare participation rates for those with or without an incentive at this site. The information will be very useful for CAHPS as well, information that we will share, in the future, with organizations interested in implementing CAHPS.
Comment A3: It is unclear how the questions in the instrument will allow you to make assessments about the use of health IT. The questions presume that the main benefits of health IT are things like online appointment making or ordering prescription refills over the internet. These are all process issues, however, and do not really get to the heart of health IT (e.g. the prevention of medication error). Because the sample is limited to patients and providers who have already used and adopted health IT, it does not appear to allow for comparisons between users and non-users. Given these concerns, please clarify how AHRQ is defining the terms “how well health IT is being used” and which questions will enable AHRQ to address that question. (Page 4)
Response: The comment is correct that in this field test we will not be able to compare users and non-users. However this field test contributes to the development of the final instrument, which can be used to determine who does and does not use HIT, and to compare whether there are differences in their reports of their care experiences. The final instrument can also be used for more direct assessment of different functions across settings.
The field test is designed primarily to make sure that the HIT items perform well, i.e. are understandable and relevant to consumers, and have good psychometric properties. Items that do not perform well in the field test will either be dropped or modified. It is correct that most of the items are about the process of care. There is great interest among health care organizations in understanding whether patient use of IT is associated with more positive (or more negative) perceptions of their care, as assessed with the core CAHPS questions. So for example, organizations may want to know if patient use of IT is related to their rating of their doctor (item 35), or whether IT use (e.g. secure email) is related to composite scores of doctor communication.
Comment A4: Please provide more information on the plans for future use of this instrument and who would adopt it or require it to be used. (Page 4)
Response: Once the instrument has been finalized it can be used by a variety of organizations, including health service delivery and health policy organizations. We have had inquiries from both types of organizations. Health care delivery organizations that have invested in health information technologies, especially web-based personal health records, are interested in learning the extent to which these technologies are used and improve the care experience for their members, for example, by improving communication, by making transactions easier and more efficient (e.g. online refills, online appointments), and by making information more accessible (e.g. online lab results).
Response: This is an excellent suggestion. We will add an open-ended question about how well the survey captured their use and satisfaction with personal health record systems, and ask if there are any questions the respondent would change, add, or delete.
Response: When the final survey instrument is developed it can be used to compare physicians on the basis of the experiences of their patients. In addition to the usual CAHPS measures of communication, timeliness, etc. we will now be able to compare the experience of patients with doctors who use computers to show patients information (e.g. Q41) and allow patients to email with questions (e.g. Q18).
Response: See response to Comment A2. Also we are working in partnership with the field test sites who are allowing the testing of the items with their membership. The site that is interested in using the incentive uses incentives for most of its patient questionnaires, including CAHPS, as a way of increasing response rates. The reviewer is correct that in addition to seeing the impact of the incentive on participation rates, we have an important opportunity to examine whether the incentive also influences responses to individual items, that is, does it bias individuals’ answers. We will examine this in our analyses. The use of post-paid incentives is the method that the site uses, so most accurately reflects the conditions under which the final instrument would be used (at least for this large health care delivery organization). Despite evidence that pre-paid incentives increase response rates, there is still broad interest in further research on different types of incentives because in health care organizations, organizational and administrative considerations often dictate whether incentives are pre- or post paid
Response: There are limitations to the proposed field test that would affect generalizability of results. It will use a convenience sample from health care organizations, and the respondents will be selected from among people who are users of health information technology. In addition, the anticipated 50% response rate also will limit generalizability.
Response: Our plans for publication will depend on the pilot test results. However, it is very likely that we will write an article for a health services research journal describing the psychometric characteristics of the new questions. In addition, multiple facets of this test would likely be of interest to administrators, policy makers, executives and researchers. For example we anticipate that a health/medical journal with a health services or health policy focus would be interested in our findings regarding which EMR and personal health record functions are used most by patients, and which ones appear to be most associated with overall perceptions of quality of the care experience.
SUPPORTING STATEMENT: Part B
Response: The purposive selection resulted in sites that have more resources than many sites that would have been selected if we had used random sampling. They may also be organizations that have more of a quality improvement culture or focus, which has led them to implement EHR and personal health record systems before there is a large amount of evidence that they will improve efficiency or quality and to participate in the testing of a new survey.
Comment A11: Will these two sources be distinguished from one another during analysis? In what ways might respondents from each source differ from each other?
Response: Yes, we will distinguish between health care provider and health insurer. In some instances the organizations combine both the provision and insurance functions. The possibility of respondent duplication is remote, as the participating sites represent very distinct geographic areas.
Comment A12: Will you also be stratifying by the demographic characteristics of the enrollees? If so, please include screening form and describe method. (Page 11)
Response: We will not be stratifying by demographic characteristics.
Comment A13: What cognitive testing was done on the items prior to this test? Please provide a copy of the report on these results. (Page 13)
Response: Cognitive testing was conducted on draft HIT items with patients who had experience using one or more of the HIT functions that were the focus of the HIT CAHPS project. Patients were drawn from Kaiser Permanente, Palo Alto Medical Foundation, Beth Israel Deaconess Medical Center, and the Marino Health Center. Reports of the cognitive testing are attached.
Comment A14: How many and what factors are expected to emerge? (Page 13)
Response: We will conduct exploratory factor analyses to determine the number of underlying factors that best characterize the response patterns. We expect that there will be 2-3 HIT factors distinct from the existing CAHPS composites.
Comment A15: If not equivalent, how will it be determined which method to use in the future? (Page 13)
Response: The mode analysis will determine whether the modes are comparable. If they are not, this will be important information for potential users. One practical implication of a finding that modes are not comparable would be that mixed mode approaches would not be recommended, unless there is adjustment for mode.
Comment A16: As noted earlier, include other substantive survey variables as well to examine potential differences in responses (or composition of the sample) due to the use of incentives. (Page 13)
Response: We agree that it will be helpful to examine other substantive survey variables by incentive status. Thus we will examine the association of incentive status with variables such as rating of your doctor (Q35), and computer makes it easier/harder to communicate with doctor (Q44), as well as composites relating to communication and timeliness, adjusting for casemix variables.
| File Type | application/msword |
| File Title | Date: |
| Author | Anna Caponiti |
| Last Modified By | vhabedmcinnk |
| File Modified | 2009-08-26 |
| File Created | 2009-08-26 |