Attachment B-3. Draft Report on Feedback from Users of the CAHPS Health Plan Survey Database
Draft Report on Feedback from Users of
CAHPS Database Services
Deliverable xx
Submitted by:
Westat
1650 Research Blvd.
Rockville, MS 20850
for
Contract Number HHSA 290200710024C
User Network Project for CAHPS and Patient Safety
October 2008
Table of Contents
I. Introduction 1
II. CAHPS Health Plan Survey Database 1
III. CAHPS Hospital Survey Database 3
IV. CAHPS Clinician & Group Database 3
V. Next Steps 4
Appendices
A. Detailed Feedback from Medicaid/SCHIP Sponsors A-1
B. Detailed Feedback from Researchers B-1
C. ERC Meeting Summary C-1
I. Introduction
This draft report presents a summary of feedback obtained in 2008 from users of CAHPS Database products and services. This feedback has been integral to the strategic planning process we have been engaged in this year, resulting in our Final Strategic Plan approved by AHRQ in September 2008.
In our strategic plan we noted the importance of an ongoing process of responding and adapting to the dynamic environment for CAHPS Database services. Such ongoing planning requires continually listening to CAHPS survey sponsors that are current or potential users of the CAHPS Database. We believe our responsiveness to sponsors has been a key factor in our success over the past 10 years and we are committed to maintaining this approach as we pursue the vision and priorities of our strategic plan in the coming months.
This report is organized according to our Final Plan for Getting Feedback from Users of CAHPS Databases, approved by AHRQ in March 2008. A summary of feedback to date is presented for the three major components of the CAHPS Database, consisting of the CAHPS Health Plan, Hospital, and Clinician & Group (C&G) Surveys.
II. CAHPS Health Plan Survey Database
A. Users of Sponsor Reports and Annual Chartbook
Our primary source of feedback from users of CAHPS Health Plan Survey sponsor reports and the annual Chartbook was through a series of individual telephone conversations with Medicaid and SCHIP sponsors that participated in submitting data and receiving reports during the past two years. We did not seek feedback from commercial sponsors, since beginning in 2007 we discontinued production of sponsor reports for this sector.
A detailed report on our feedback findings from Medicaid/SCHIP sponsors, including 14 State Medicaid agencies and 20 individual health plans, is presented in Appendix A. Highlights of this feedback include:
Use of sponsor reports by State Medicaid agencies is mixed. For some States, the sponsor reports are their major source of comparative CAHPS data and they are relied on for contracting requirements and creating summary reports for legislative audiences. Other states rely on their vendors for such reports or produce their own reports, sometimes drawing on the CAHPS Database sponsor reports for national benchmarks.
Most health plans do not use the sponsor reports, but rely instead on their vendors or NCQA for comparative data.
Use of the annual Chartbook is also mixed, with some sponsors finding it quite useful while others were not aware of it.
State agencies and plans that use the sponsor reports and Chartbook generally like the organization and content of both. Few had strong recommendations for improving the design of either, other than suggesting some additional features such as trending or additional benchmarks (such as regional comparisons) or shortening the length.
Both State Medicaid and health plan sponsors noted that differing benchmarks from the CAHPS Database and NCQA often create confusion and result in extra time and resources devoted to understanding differences. It was frequently mentioned that the NCQA data are ultimately deferred to, since health plan accreditation is tied to the NCQA scores and not those from the CAHPS Database.
Medicaid sponsors strongly supported making a transition away from direct sponsor submissions to a data purchasing arrangement with NCQA similar to the approach we have used successfully in the last two years for the Commercial data. Sponsors see significant value in the alignment of these two databases.
This feedback has been very useful to us in shaping our future plans for the CAHPS Health Plan Survey Database. We have incorporated these findings in our revised approach for obtaining Medicaid/SCHIP data in partnership with NCQA and for reporting all CAHPS data through the new online reporting system described in our strategic plan.
B. Users of Research Files
Feedback on the use of research files for both CAHPS Health Plan and Hospital Survey data was obtained through an email request sent to all researchers (n=17) that requested and obtained CAHPS Database research files in the past 3 years (2005-2007). Responses were obtained from all 17 researchers. A detailed summary of responses is presented in Appendix B. Highlights of this feedback include:
All researchers gave high praise to the CAHPS Database staff and to the overall process for requesting and obtaining research files.
Researchers found the data files very useful, as well as the comments provided by ERC members on the design and methods of their proposed research.
Some researchers suggested that a public use version of the data files could be made available and thereby simplify the acquisition and use of the data.
We also convened a conference call meeting with members of the Executive Research Committee (ERC) to request their comments and suggestions regarding the process we use for reviewing and approving research data requests, as well as ideas for expanding the scope of research activity conducted with these files. A summary of the ERC meeting is presented in Appendix C. Highlights of the meeting include:
Agreement on the desirability of posting the data files on the CAHPS Database web site, in order to promote access and use of the data.
Consensus that some ongoing role be maintained for the ERC to provide advice and suggestions to researchers, even if formal ERC approval of research requests were no longer required.
Our strategic plan includes a recommendation for a gradual and partial transition to public use files based on this feedback. The CAHPS Database staff will process data requests and consult with ERC members only on an as-needed basis.
III. CAHPS Hospital Survey Database
We obtained feedback from users of the H-CAHPS Chartbook and related reports through ongoing contacts with hospitals and survey vendors that have participated in submitting data. In the process of communicating with these hospitals and vendors, we gathered comments and ideas related to report design, content, and distribution methods, as well as the H-CAHPS submission process. Our findings included the following:
There is a strong and continuing desire among hospitals for the type of detailed bcnchmarks we provide through the Chartbook. The comparisons by hospital characteristics and percentile distributions we provide through the Chartbook are not available through the CMS Hospital Compare web site.
Hospitals and vendors are willing to continue contributing data to the CAHPS Database, but would support a single submission process to CMS if we could arrange to obtain the data from CMS.
The existence of two different national data sets (CMS and CAHPS Database) does create some confusion among users. The consolidation of these two initiatives into a single, common database would alleviate confusion and streamline tasks required by hospitals and vendors to contribute data.
Another major source of feedback has been our ongoing dialogue with CMS about possible collaboration around the compilation and use of H-CAHPS data. Conversations between CMS and AHRQ at various times this year led to a joint meeting in early October to discuss a process for working with the CMS contractor to obtain the analyses needed for the Chartbook. We are confident that hospitals and vendors will benefit from this collaboration as a means for continuing the production of the Chartbook, but with the added benefit of using the same data set published by CMS to maintain consistency in national benchmarks.
IV. CAHPS Clinician & Group Database
Since the C&G Database is still under development and has no current users, we have engaged in a series of conversations to gather input from prospective contributors to and users of this Database. The feedback we have gathered is briefly summarized below.
A. Regional Collaboratives
We have presented our C&G Database plans, including features and benefits to participants, to many of the various regional collaboratives that are implementing or planning to implement the C&G survey. These include the 14 Aligning Forces for Quality communities and some of the Chartered Value Exchange markets. These programs are very interested in national benchmarks and are willing to collaborate with us as we develop the C&G Database.
In Minnesota, we are serving as the aggregator of survey data for the Minnesota Community Measurement (MNCM) pilot test of the visit-specific version of the C&G survey. This role has required us to develop a set of data submission specifications that will satisfy the needs of the pilot project, as well as provide a template for submissions by others users. We have received input on the feasibility of the submission specifications from the both the medical groups and their survey vendors that has been very helpful. One of the major issues we have been exploring is what information to collect, from what sources, regarding characteristics of medical practices that can be used for constructing summary-level benchmarks. We are scheduled to have final data specs completed for the pilot by early November, and have requested that medical groups submit preliminary test files prior to the final files, in order to verify that the specs and submission process will work.
Two of the regional collaboratives, the Massachusetts Health Quality Partners (MHQP) and the Pacific Business Group on Health (PBGH), have suggested that we develop a list of questions that are comparable across the early adopters using slightly different surveys. We will be developing this list through a crosswalk analysis of these surveys, in order to provide assurance that we can develop useful comparative data in return for receiving their data files.
B. Survey Vendors
We have engaged in a series of individual conference calls with leading national vendors regarding their current and future client activities related to the C&G Survey. These conversations have been very informative and have enabled us to build even stronger collaborative relationships with vendors that will serve us well at the time we release our public data submission specifications.
We have scheduled an invitational web meeting for vendors on November 19, designed to introduce our data submission specifications and gather additional input and feedback. We will have a further opportunity to meet with many of these vendors in person at the CAHPS User Group Meeting in December.
C. Health Plans and Medical Groups
We have responded to numerous
inquiries from health plans, systems, and medical practices that are
using or considering using the C&G survey that are interested in
obtaining benchmarks for comparison. We have used these inquiries as
opportunities to gather information about their survey activities and
to encourage their participation in the C&G Database. We have
received actual data files from several of these groups.
D. American Board of Medical Specialties
Members of our team have been meeting on a regular basis with ABMS staff as they plan an implementation strategy for a short-version of the C&G survey as part of the Maintenance of Certification process.
V. Next Steps
We will continue to gather feedback from users of CAHPS Database services across each of the major Database components. This feedback will be used directly in our product delivery and as input to our ongoing strategic planning process.
Appendix A
CAHPS Database
Summary of Medicaid/SCHIP Feedback
Part 1: CAHPS Database Summary of State Medicaid Agency Sponsor Feedback
April 10, 2008
Background
CAHPS Health Plan Survey Medicaid and SCHIP sponsors that had participated in submitting data and receiving reports during the past two years (2006-2007) were contacted by an e-mail and/or individual telephone call requesting a 30 minute feedback conversation during January 2008. Several reminders were sent to non-responders.
Sponsors consisted of 21 State Medicaid agencies and 20 individual health plans. 30 minute telephone calls were scheduled with the sponsors who responded to this request (State Medicaid Agencies; n=14 and individual health plans; n=9). A discussion guide with topics that were explored with sponsors during our telephone conversations was sent in advance of each call.
Dale Shaller and Janice Ricketts participated in the first few calls. The majority of calls were then conducted by Swati Nadkarni. Notes were taken during the calls and a separate summary of responses for State Medicaid agencies and individual health plan participants follows below for each question.
Feedback responses were obtained for 14 out of 21 State Medicaid Agency sponsors.
Summary of Responses (n=14)
I. About Participating Organization
1. What type of organization are you from?
10 responded as State Medicaid Agency
2 responded as State Agency other than Medicaid
1 responded as non profit organization; unaffiliated to the State Medicaid Agency. “We provide comparative data analysis to the state’s ‘Health Connect’ program”
1 responded as Association of Medicaid Health Plans.
Other comments:
"We belong to the Office of Health Care Statistics, affiliated to State Medicaid Agency. State Medicaid contracts with us to administer the CAHPS survey”
2. Do you administer the CAHPS Health Plan Survey every year?
9 responded as every year
1 responded as every year alternating between adult and child surveys.
2 responded as every other year
1 responded as “Participation depends on the funding we receive every year.”
1 responded as “We require our Medicaid Managed Care Organizations to administer the surveys every year.”
3. Do you use a vendor to submit Health Plan Survey data to the CAHPS Database?
9 responded that they use a vendor
3 responded that they contract with the External Quality Review Board/organization (EQRO) as a state requirement.
1 responded as “We act as a vendor.”
1 doesn’t need a vendor, as it’s a coalition.
4. Do you submit Medicaid data to NCQA?
5 responded yes
9 responded no
5. What other reports does your organization internally use or need (besides the NCQA report) for publishing/reporting Health Plan Survey Data? (e.g., comparative reports as part of program evaluations or reports for legislative and advocacy groups, etc.)
6 responded that they get a detailed Binder from their Vendor with plan comparisons as well as National benchmarks.
Other comments:
“The vendor provides plan specific and aggregate reports.”
“EQRO vendor develops a report based on NCQA benchmarks. Vendor gives an executive summary that includes NCQA comparisons with MCO as well as plan to plan comparisons. It also includes methodology.”
“Vendor provides a detailed comparative report by individual program and by plan. The vendor gives us the raw data which we share with our plans.”
“EQRO vendor provides a detailed plan specific report that includes executive summary and aggregate summary reports based on NCQA benchmarks.”
“Vendor does the program evaluation and gives us a high level summary report with a snapshot of quality improvement. The report also provides trending over the time. The results are not compared to NCBD national benchmarks as the sponsor reports are not available in July/August when our reports are done internally. We compare our results with historical data with high/low state wide average.
“Vendor creates a comparative report, using both NCQA and NCBD benchmarks.”
7 responded that they do their in-house reporting
Other comments:
"We do not receive any reports from our vendor. They give us clean data and we generate our own in-house reports.”
“We create a handbook with all the data.”
“We produce a comparative report using NCBD benchmarks. It identifies the issues related to access quality and timeliness of care.”
“Our analysts slice and dice data internally. Data is not shared with the state plans.”
“We create our own in-house reports based on the NCBD benchmarks.”
“We produce a summary of CAHPS results. We do look at the NCBD benchmarks.”
“We produce our own in-house reports.”
II. Data Submission Process
1. Have you attended the CAHPS Health Plan Survey Data Submission training in the
past?
7 responded no.
Other comments:
"No need to attend as the vendor has done it for years.”
" No one from the State has attended”
“Data submission process is handled by the vendor efficiently.”
6 responded yes.
"Attended in the past. It was helpful as it gave a better understanding of the whole process and also insight into the role of the vendor.”
“Analyst attended the training mainly to know if there are any changes made to the data submission system.”
“Training is useful and other staff has attended.”
“I have done the training in the past; it was fine.”
“Web-based training. These are helpful to keep the Department aware of due dates that our Managed Care Organizations need to be aware of.”
2. As a sponsor, how is the online data submission process working for you? Any issues?
10 responded that they didn’t have any issues.
Other comments:
"No issues from the vendor.”
“No issues with the submission process. The vendor has not expressed any problems.”
Vendor has no issues with the submission process.”
"Analyst did not have any problem with the site.”
"No issues.”
“No problems.”
“This process works satisfactorily. Basically, our role is to monitor to see that our MCOs are submitting according to NCBD timelines and to ensure they have their data in timely.”
3. As a sponsor, how often do you access the online submission site to check the status of your data submission?
4 responded that they don’t check the status of their data submission.
Other comments:
"I didn’t do the CAHPS submission last year. I will be able to comment on this in 2008”
“Vendor manages the data submission process.”
"Vendor provides bi-weekly report”
7 responded yes.
Other comments:
“Like the fact that the vendor requires their permission to unlock the database and it also helps to ascertain the performance of the vendor.”
“Analyst checks it regularly.”
“Data submission process is handled by the vendor. I log on to the submission site if needed.”
“I do check the data submission status a couple of times.”
“Weekly to check status of MCO’s submission status.”
“Check the status when needed.”
4 responded that that they appreciate any communication as it helps them to keep on track and manage the submission process efficiently. The notifications/reminders keep them in the loop.
Other comments:
“They are very helpful as we are able to follow-up with the vendor. This makes me a better contract administrator.”
“Appreciate all reminders. Any kind of prompts are helpful”
“Rely on notifications.”
“Getting reminders is always a good thing because it’s hard to keep track of everything especially when you use a vendor.”
4. Are there any suggestions for improving the submission process?
8 responded “No comments”
III. Sponsors Reports and Chartbook
1. How have you used the CAHPS Health Plan Survey sponsor reports?
7 responded that they didn’t use the sponsor report.
Other comments:
“We get cleaned data from the vendor and do our own analysis to produce in-house reports. The NCBD sponsor report and national benchmarks help us verify the numbers. Though we use CAHPS macro, the data is analyzed differently than NCBD. Therefore, the numbers do not match all the time. We are aware of this issue and know exactly where the results look different.”
“NCBD sponsor reports are not useful. We pass on the NCBD sponsor report to the health plans.”
“NCBD sponsor reports are useful to a certain extent. They look at the NCBD sponsor report but use the NCQA benchmarks.”
“We do not use the NCBD national benchmarks as the sponsor reports are available after our reports are gone out. We use the NCBD benchmarks later to validate our results if we have any issues.”
“MCOs are often confused by conflicting benchmarks. They all move to NCQA requirements and so does the State Medicaid agency when setting performance benchmarks. The additional data may do more harm than good.”
“Do not recall downloading the sponsor report.”
“We do not use the sponsor reports.”
7 responded yes.
Other comments:
"Like the sponsor report and find the regional comparisons very useful.”
“Very impressed with the sponsor report and found it useful. The NCBD report is superior to the report we receive from the vendor. The sponsor report is user friendly and show full representation of the results. We have shared the NCBD report with the vendor.”
“The sponsor report satisfies our contract requirement. The national and regional benchmarks are useful. Like the bar chart presentation.”
“NCBD benchmarks are used to see how well the state plans are doing as compared to other states in certain areas.”
“To document plan performance.”
“NCBD benchmarks are used by the analyst to produce in-house reports.”
“We look at the NCBD benchmarks to create a summary level report.”
2. Are there any suggestions for improving the design of sponsor reports that would enhance their usefulness for any of the above purposes? Please describe the proposed change and the purpose(s) it would serve.
8 responded “No comments, not at this time.”
6 responded with some comments.
Other comments:
"It’s good as it is.”
“Our team reviewed the last report and we liked the way the results were presented. We would like to see more comparisons between our SCHIP data and other states/programs reporting SCHIP data. Would like to be able to compare us with neighboring or other states and would appreciate any help from the CAHPS Database to connect with other SCHIP programs. We are interested in standardized SCHIP Quality measurement.”
“Plan names are reported differently in the sponsor reports. It would be helpful if plan names are translated state by state.”
“It would benefit the Department if we could obtain the 50th and 75th percentile benchmarks used to derive the charts in Section A, Table 2 of the Sponsor Report. As it is published using means to compare the plans’ performance, this method does not satisfy the Department’s needs in comparing our MCOs’ performance in various reports we produce.”
“The NCBD sponsor reports seem useful but may be too long. Maybe needs an "at a glance" summary.”
“Like the design of sponsor report.”
“Works as it is.”
3. Are there any suggestions for improving the delivery of the sponsor report?
9 responded “No comments”.
4 responded with a comment.
Other comments:
"It’s easy.”
“Prefer a reminder to download the report before the end date.”
“Works well.”
“Works fine.”
4. How useful do you find the methodology section which accompanies the sponsor report?
6 responded “No comments”.
6 responded with some comments.
Other comments:
“Pretty useful especially when I started new as a statistician on the health plan survey.”
“Useful to the analyst. We use the methodology language in our internal report.”
“Useful. It’s a handy reference if upper management has any questions about the data analysis and compilation.”
“Vendor would be more interested in the methodology.”
“Useful.”
“Just glanced the document but did not spend much time reading it.”
5. How have you used the CAHPS Health Plan Survey annual Chartbook?
7 responded that they didn’t use the chartbook.
Other comments:
“Don’t recall seeing one.”
“Do not use the information as the State doesn’t require them to compare commercial or Medicaid findings. But it’s interesting.”
“Not familiar with the chartbook.”
“Have not seen the chartbook.”
“I can’t tell you that I used it intelligently. I’m not the best authority on this. You may want to talk to other people in the agency.”
“Do not recall seeing the chartbook.”
6 responded yes.
Other comments:
“Used the graphs and the comparative data provided in the chartbook for plan comparisons.”
"Chartbook appendices are very helpful compared to our EQRO reports and has realized that we need to ask more from our vendors in terms of vendor reports.”
“Useful. Composite items are good for comparing the plan data.”
“Previously, the Department received benchmarks for Adult and Child CAHPS through NCQA. For this reason, we have not extensively used the Sponsor Report or Annual Chartbook produced by NCBD. However, for the past two years, we have not been able to obtain the Child CAHPS benchmarks from NCQA.”
“We look at the chartbook.”
“Like the design of the chartbook. The chartbook was useful but no trending was available in the 2007 chartbook.”
6. Do you have any suggestions for improving the design or content of the Chartbook?
8 responded “No comments”
3 responded yes.
Other comments:
“Used the graphs and the comparative data provided in the chartbook for plan comparisons.”
“Want to see trending.”
“Chronic conditions items and regional benchmarks should be included in the Chartbook.”
IV. Other Products and Services Provided by the CAHPS Database
1. Do you know that the CAHPS Database provides Technical Assistance to users throughout the data submission process?
11 responded yes.
responded no.
Other comments:
“Not much used.”
“Prompt responses are appreciated.”
“It’s useful if we have any issues especially last year when we had delayed data submission.”
2. Are you aware of the CAHPS User Group Meetings (UGM)?
Do you think it would be helpful to convene a session of CAHPS Database users at the User Group Meetings? If so, what topics or issues would you like to discuss?
7 responded yes but had no suggestions about the discussion topics to be convened at the UGM. .
Other comments:
"Attended in the past when I was assigned as a project manager.”
“Aware of but never participated through the State agency.”
‘Aware of. Have not attended in the past. The state would not pay for it due to tight budget and short staff.”
“I think they are good but there is less of a story. It is essential for the plans to know how the composite is driven i.e. nuts and bolts view. We want to know how to set up a good approach to deal with some of the information given in the sponsor reports like, how do you work on it or what do you have to put into it in order to achieve this. I would like to connect to people (respondents) and know how they rate their care”
“Wants to know more about it.”
“Attended in the past.”
4 responded no.
3. Are you aware that other CAHPS survey kits are available to users?
5 responded no but showed interest in knowing and getting more information.
Other comments:
“Not aware and not useful. We just do health plan survey.”
“Rely on the vendor to provide the updates. Not at this time. Not interested in implementing other CAHPS surveys.”
5 responded yes.
Other comments:
“Aware of but these are not part of our contract.”
“Yes. Managed care is our priority. Other CAHPS surveys are not applicable to us.”
“Aware of H-CAHPS and C-G CAHPS. At this point we are going to maintain the health plan survey.”
“Not using any.”
4. Are you aware of the CAHPS Database research files and the process for obtaining access to them?
7 responded no.
1 responded yes.
V. Plans for 2008
1. Are you administering any of the following surveys in 2008?
4.0/4.0H for Medicaid adult populations
4.0 for Medicaid and SCHIP child populations without Chronic Conditions
4.0 for Medicaid and SCHIP child populations with Chronic Conditions
3.0/3.0H for Medicaid and SCHIP child populations without Chronic Conditions
3.0/3.0H for Medicaid and SCHIP child populations with Chronic Conditions
11 responded that they will administer 4.0 Adult Medicaid survey.
3 responded that they will administer 3.0 Child Medicaid without Chronic Conditions survey.
4 responded that they will administer 3.0 Child Medicaid with Chronic Conditions survey.
2 responded that they will administer 4.0 Child Medicaid without Chronic Conditions survey.
2. How does the CAHPS Health plan Survey data submission schedule fit with your survey administration dates?
3 responded that they follow the NCQA HEDIS time lines.
4 responded that it works well for them.
1 responded that it doesn’t work well for them.
Other comments:
“We field the survey in the fall every year and get the results for all the plans by December. This data isn’t submitted to the CAHPS Database until June. The plan performance is presented to the Board in April and we use the CAHPS Database Chartbook results from the previous year. Therefore, we do not find the report very useful.”
3. We have discussed the possibility of expanding our data acquisition arrangement with NCQA in the future, to include Medicaid health plan data. What do you think about this idea? What advantages/disadvantages do you see in submitting Medicaid data to The CAHPS Database through NCQA?
7 responded favorably.
Other comments:
“We do not submit data to NCQA but it’s a good idea.”
“I’m positive about the move and I think it is a good idea.
“Any effort to streamline the process would be beneficial. More collaboration would help.”
“Two out of three plans under the State are NCQA accredited. It will be useful for the non-accredited plan to submit data to NCQA.”
“It’s a good idea. Sometimes it’s better not to have too much information and get confused. EQRO vendor will provide a better answer.”
“Like the idea of NCBD and NCQA partnering on the Medicaid benchmarks and sponsor reports. This will save money and reduce confusion.”
“Good idea. There will be less confusion among plans.”
3 responded as.
Other comments:
“They are not NCQA accredited. If we decide to submit data to NCQA, the state will have to pay additional fees to us.”
“We use different composites as compared to NCQA. It will add confusion than giving a unified message. When you are telling me that there might be another report coming out, I’m a little hesitant. I don’t know how to relate to the rest of the nation with different methodologies.”
“No comment.”
4.
Do you have any other comments or suggestions for improving the value
of The CAHPS Database to survey users?
9 responded “No Comments”.
2 responded with a comment.
“Would like to receive the CAHPS Connection.”
“Would like to see certified CAHPS survey in additional languages (e.g. Russian and Asian languages). Also, we have added supplemental items to our survey in 2007 and would like to know what other states are doing in terms of adding supplemental to their surveys.”
Part 2: CAHPS Database Summary of Medicaid Health Plan Sponsor Feedback
April 10, 2008
Background
CAHPS Health Plan Survey Medicaid and SCHIP sponsors that had participated in submitting data and receiving reports during the past two years (2006-2007) were contacted by an e-mail and/or individual telephone call requesting a 30 minute feedback conversation during January 2008. Several reminders were sent to non-responders.
Sponsors consisted of 21 State Medicaid agencies and 20 individual health plans. 30 minute telephone calls were scheduled with the sponsors who responded to this request (State Medicaid Agencies; n=14 and individual health plans; n=9). A discussion guide with topics that were explored with sponsors during our telephone conversations was sent in advance of each call.
Dale Shaller and Janice Ricketts participated in the first few calls. The majority of calls were then conducted by Swati Nadkarni. Notes were taken during the calls and a separate summary of responses for State Medicaid agencies and individual health plan participants follows below for each question.
Feedback responses were obtained for 9 out of 20 Medicaid Health Plan sponsors.
Summary of Responses (n=9)
I. About Participating Organization
1. What type of organization are you from?
9 responded Medicaid Health Plans
2. Do you administer the CAHPS Health Plan Survey every year?
8 administer every year
1 every other year
3. Do you use a vendor to submit Health Plan Survey data to the CAHPS Database?
9 responded that they use a vendor
4. Do you submit Medicaid data to NCQA?
8 responded yes
1 responded no
5. What other reports does your organization internally use or need (besides the NCQA report) for publishing/reporting Health Plan Survey Data? (e.g., comparative reports as part of program evaluations or reports for legislative and advocacy groups, etc.)
All 9 responded that they get a detailed binder from their Vendor with plan comparisons as well as National benchmarks.
Other comments:
"Vendor provides a custom report with many different levels of results which we look at in many different ways.”
“Vendor provides summary and detail report comparisons with their own historical and NCQA national benchmarks.”
“Vendor provides a book of business with comparisons as well as national benchmarks (NCQA) from quality compass. It consists of an executive summary, project overview with background, response rate, and profile of the survey respondents. It also contains NCQA benchmarks and trend comparisons. The report provides the satisfaction segmentation about the respondent’s ethnicity, race and education background.”
“We receive an extensive/detailed report from the vendor based on NCQA benchmarks.”
“We get a binder with high level analysis presenting the key findings like areas for major improvement / areas with strong performance etc and then in-house analysts summarize the results for the senior leadership.”
“We get a detailed report from the vendor. They cater to our needs and give us individual health plan analyses based on NCQA quality compass results.”
In house summary report: 9 responded that the in-house analyst prepares a report using the NCQA results.
Other comments:
“Association creates a report for us where all the plans under our flagship are compared to one another. It’s more like a QI report.”
II. Data Submission Process
1. Have you attended the CAHPS Health Plan Survey Data Submission training in the
past?
8 responded no.
Other comments:
"No need to attend as the vendor manages the submission process.”
"We rely on the vendor for the data submission process”
"No need to attend as the vendor manages the submission process.”
"Vendor attends and we do not think we should get familiarized with the process.”
1 responded yes.
"Found the training useful.”
When asked if there was any better way to provide the training material, the response was; "If things are status quo, it would be nice to have something downloadable with “How-To-Do steps” with highlights on new changes; highlight the major changes first if nothing changed drastically and then request the repeat users to drop off the call.”
2. As a sponsor, how is the online data submission process working for you? Any issues?
8 responded that they didn’t have any issues.
Other comments:
"No issues from the vendor: Website is fairly easy to work with.”
"No issues from the vendor.”
"Have not heard of any problems from the vendor and they do their job well.”
"Vendor has no issues.”
“No issues about NCBD submission.”
1 had problem with the submission process.
Other comments:
"Username and password sent out as a part of the registration process completion is confusing.”
3. As a sponsor, how often do you access the online submission site to check the status of your data submission?
7 responded that they don’t check the status of their data submission.
Other comments:
"Vendor checks.”
"Vendor takes care of all aspects of the submission process.”
"Rely on the vendor for data submission.”
"Trust the vendor.”
“Vendor takes care of the submission process.”
2 responded yes.
Other comments:
"Done it more towards when the vendor completes the whole submission process.”
“I Make sure that we are at top of it and like the idea of receiving notifications.”
5 responded that that they appreciate any communication as it helps them to stay on track and manage the submission process efficiently. The notifications/reminders keep them in the loop.
4. Are there any suggestions for improving the submission process?
8 responded “No comments”
III. Sponsors Reports and Chartbook
1. How have you used the CAHPS Health Plan Survey sponsor reports?
8 responded that they didn’t use the sponsor report.
Other comments:
“Can’t recognize two different versions of CAHPS satisfaction. It’s hard to detail and figure out two different sets of results.”
"Not familiar with the NCBD sponsor report and do not recall seeing any report.”
"We review the NCBD report but spend more time on our vendor report as the two methodologies are different. Rely more often on NCQA benchmarks and Vendor reports.”
"We do look at the National benchmarks from the CAHPS Medicaid sponsor report but follow the results provided by the vendor which uses the NCQA benchmarks.”
"We look at NCBD benchmarks but consistently use NCQA results. Prefer to use one set of results.”
"We look at the National benchmarks from the CAHPS sponsor report but follow the results provided by the vendor which uses the NCQA benchmarks.”
"We don’t look at it closely. Can’t remember an instance where we grabbed the data and made use of it. Use NCQA benchmarks to create reports for higher leadership.”
“No we don’t use it.”
1 responded yes.
Other comments:
"We do look at the National benchmarks from the CAHPS Medicaid sponsor reports. Like the reporting of the sample size but generate our own in-house reports.”
2. Are there any suggestions for improving the design of sponsor reports that would enhance their usefulness for any of the above purposes? Please describe the proposed change and the purpose(s) it would serve.
6 responded “No comments”.
3 responded with some comments.
Other comments:
"It would help to insert a note if the CAHPS results were different than the NCQA. Prefer uniformity in the CAHPS and NCQA rating item scales.”
"Like the sponsor report design. Medicaid report we get from the vendor looks similar to the CAHPS sponsor report with the three part bar charts.”
"Would like to see scores by percentage rather than 3 point scores and a main row for composite roll up with each question that goes with that composite. Like the idea of getting a color coded summary report.”
3. Are there any suggestions for improving the delivery of the sponsor report?
8 responded “No comments”.
1 responded with a comment.
Other comments:
"Delivery of report works fine.”
4. How useful do you find the methodology section which accompanies the sponsor report?
4 responded “No comments”.
5 responded with some comments.
Other comments:
“Can’t recognize two different versions of CAHPS satisfaction. It’s hard to detail and figure out two different sets of methodologies.”
“Not gone through the section.”
“Can’t speak directly as have not gone through it.”
“Methodology section is important especially for the new data analyst. Good reference place at the back of the sponsor report.”
“May be used by the plans which are not accredited.”
5. How have you used the CAHPS Health Plan Survey annual Chartbook?
7 responded that they didn’t use the chartbook.
Other comments:
“Looked at it. I can’t say it’s useful or not as I don’t use it.”
"Not familiar with the NCBD chartbook. Did not recall seeing it.”
"It would be useful if we had more time. It’s a time factor. Chartbook is designed well. No issues.”
"Like the chartbook but have not used it much. But it’s always good to have more information from various sources.”
"Haven’t seen it.”
2 responded yes.
Other comments:
"Like the chartbook, particularly the sample size numbers.”
"It’s interesting to see broader view across the land. The chartbook results are helpful in knowing the general trend across various areas. We review the chartbook and include the pieces that are relevant to our internal reporting.”
6. Do you have any suggestions for improving the design or content of the Chartbook?
9 responded “No comments”
IV. Other Products and Services Provided by the CAHPS Database
1. Do you know that the CAHPS Database provides Technical Assistance to users throughout the data submission process?
6 responded yes.
3 responded no.
Other comments:
"Not aware of.”
2. Are you aware of the CAHPS User Group Meetings (UGM)?
Do you think it would be helpful to convene a session of CAHPS Database users at the User Group Meetings? If so, what topics or issues would you like to discuss?
7 responded yes but had no suggestions about the discussion topics to be convened at the UGM.
Other comments:
"Aware of, but not our focus.”
"Does not fit into our budget.”
"Have read about it but these are not accreditation requirements and no plans of attending one.”
"Due to budget constraints, we attend the NCQA updates.”
"We do not have the budget.”
“Like a lot. Attended in 2006 when the instrument changed from 3.0 to 4.0.”
2 responded no.
3. Are you aware that other CAHPS survey kits are available to users?
5 responded no but showed interest in knowing and getting more information.
4 responded yes.
Other comments:
“Has just done cursory check as it’s a time factor. Our focus is on health plan survey. We have our own Medical Group and hope they would use the C-G CAHPS survey.”
“We just do health plan survey as it’s required by the State.”
4. Are you aware of the CAHPS Database research files and the process for obtaining access to them?
9 responded no.
V. Plans for 2008
1. Are you administering any of the following surveys in 2008?
4.0/4.0H for Medicaid adult populations
4.0 for Medicaid and SCHIP child populations without Chronic Conditions
4.0 for Medicaid and SCHIP child populations with Chronic Conditions
3.0/3.0H for Medicaid and SCHIP child populations without Chronic Conditions
3.0/3.0H for Medicaid and SCHIP child populations with Chronic Conditions
8 responded that they will administer 4.0 Adult Medicaid survey.
3 responded that they will administer 3.0 Child Medicaid without Chronic Conditions survey.
3 responded that they will administer 3.0 Child Medicaid with Chronic Conditions survey.
2. How does the CAHPS Health plan Survey data submission schedule fit with your survey administration dates?
7 responded that they follow the NCQA HEDIS time lines.
2 responded that it works well for them.
Other comments:
“Works well.”
“Works fine as they have an Association Meeting in October and CAHPS health plan reports are available before the meeting.”
“NCBD report comes out much later. We wrap our reports in July/August.”
3. We have discussed the possibility of expanding our data acquisition arrangement with NCQA in the future, to include Medicaid health plan data. What do you think about this idea? What advantages/disadvantages do you see in submitting Medicaid data to The CAHPS Database through NCQA?
All 9 responded favorably.
Other comments:
“It’s most confusing when we get results from the State as they use the NCBD benchmarks and compare to the NCQA results. We don’t know if they are better or worse. It would be immensely wonderful to collaborate with NCQA. It would eliminate a whole bunch of confusion. It’s hard to know where you are with two different scores, methodologies all around satisfaction. It will be great if we settle down on one methodology.”
“It’s a corporate decision. It would be nice to get the reports free.”
“Like the idea of submitting to one place.”
“NCQA and NCBD need to come to an agreement on the rating item scoring scale (8, 9, 10 Vs 9, 10). Because of these discrepancies we have been using NCQA benchmarks. We look at the NCBD results but they are not a driving force for us. It’s a waste of everybody’s time and money if we don’t get this issue straightened out. I’m pleased to hear about the possibility of expanding the Medicaid data acquisition arrangement with NCQA in the future. Consolidated report would be most useful.”
“Like the idea of getting one report.”
“It would be nice to have just one submission and receive one report from NCQA and the CAHPS Database with single analytical method. NCQA has not been providing quality compass tool for Medicaid and more collaboration would always help if data were collected by NCQA without the fees involved for accreditation and without conflicting reports. Large fees deter a lot of plans to purchase quality compass tool for different survey versions.”
“Good as long as every participant agrees. Good idea.”
“Good idea to receive one report with out conflicting results.”
“It will be nice to get one report.”
4.
Do you have any other comments or suggestions for improving the value
of The CAHPS Database to survey users?
8 responded “No Comments”.
1 responded with a comment.
"I’m happy to get good information and ideas about the other CAHPS products and to explore its use to our own advantage.”
Appendix B
CAHPS Database
Summary of Researcher Feedback
March 6, 2008
Background
An email request for feedback was sent to all researchers (n=17) that requested and obtained CAHPS Database research files in the past 3 years (2005-2007). The email was sent as a personal request by Dale Shaller, CAHPS Database Managing Director, on February 15, 2008. The request consisted of 10 questions regarding the process of requesting and using research files from the CAHPS Database. A return of responses was requested by February 29, 2008. As of March 6, 16 responses had been received from the 17 researchers. A summary of responses follows below for each question. Detailed responses are available upon request.
Summary of Responses (n=16)
1. How did you learn about the availability of CAHPS Database research files?
5 from the CAHPS or AHRQ Web site
5 from collaborators/colleagues
2 from an Internet search
2 from the research literature
2 as members of the CAHPS Consortium
2. How helpful was the CAHPS Database staff in responding to your questions?
15 responded either helpful or very helpful, e.g.: "went beyond the call of duty"; "the staff has been very responsive to my requests. Their responses have been very useful and thorough."
1 "moderate"
3. How easy was it to follow the proposal guidelines and submission process?
15 responded easy, very easy, or straight forward
1 responded not easy: "unnecessarily complex for a database of this type"
Other comments:
"The only problem is that it took quite a long time to get all of the linkages done."
"One wrench in the works is that I'm using Stata instead of SAS, so that has made a few questions difficult to answer.”
4. How useful were the comments (if any) that you received from members of the Executive Research Committee?
11 responded helpful or very helpful, e.g., "The comments were helpful especially in determining if there were other similar studies being conducted."
3 responded no comments or could not remember
2 responded somewhat or mixed
5. How useful were
the data files that you received?
13 responded useful, very useful, or extremely useful, e.g.: "immensely"; "invaluable"
2 responded they are still working with them.
1 responded not useful: "The primary questions were about racial differences in pt experiences. However, there were very few minority patients in the sample for us to conduct valid analyses. After a few preliminary runs to determine whether we could conduct our proposed analyses, we concluded we could not. "
Other comments:
"I wish the process of transforming questions into composites was a bit more transparent, but other than that, I haven't had any problems so far."
"Excellent and very relevant to real world tasks for future health care managers.”
6.
What suggestions do you have for improving the process for obtaining
CAHPS Database research files?
11 responded no suggestions, e.g., "good experience", "a complete success story", "you do a fantastic job"
5 specific suggestions:
"I think a version of this database should be made available for public use, either in a downloadable file or through an interactive tool on the AHRQ web site. This would make the information contained in the database much more accessible and allow researchers to include patients’ experience of care in discussions of quality much more readily."
"I am surprised by the big confidentiality issue given that all of the hospitals will have public data in a couple of months. In our case, the linking led to a big loss of time."
"The only thing I found rather bewildering was the website itself. It had a lot of information that was somewhat repetitious and hard to find the answers to some questions about the databases--mostly from a LOT of information for very different kinds of users."
"The process was fine. If you could provide some basic crosstabs of some of the measures, it would provide information about the feasibility of proposed analyses because researchers would have an idea as to whether there are enough observations for the items of interest."
"Send the scoring for the different domains with the data base.”
7.
Would you recommend the CAHPS Database research files to other
researchers?
15 responded yes, absolutely, definitely, or already have
1 responded maybe
8.
Do you plan to make future requests to obtain CAHPS Database research
files?
7 responded yes
5 responded maybe or possibly, it depends
4 responded no, e.g., "My primary area is in health disparities, and there were not enough observations available for us to conduct valid analyses.”
9.
Do you have any other comments or suggestions for improving this
service?
12 responded no, e.g., "great service", "thank you for your help"
4 comments:
"The only deficiency I remember in the database is the paucity of information about the sampling frame (versus those who completed a survey) and incomplete information about language of survey administration."
"Put some documentation on the Web site for computing variables from the data.”
"Documentation with data could probably be improved - I prefer frequency tables that show formatted and unformatted values. A "changes this year" feature could also be useful."
"The service has been great so I do not have any specific suggestions. From a research perspective, it would be helpful to receive data on the sampling frame used by the sponsor, such as response rates for each sponsor and if possible response rates by demographic variables.”
10. What research products have you developed using the CAHPS Database files? (please list publication and presentation titles and citations, if available)
9 responded with specific publications or products
5 responded that products are in progress
2 responded that no publications or products are planned
See list of publications and products attached, by responding researcher, if available. A complete summary of research projects and publications will be updated soon and made available on the CAHPS Database Web site and upon request.
List of Publications and Products of Responding Researchers
Renee Schwalberg, Altarum Institute:
Information from the CAHPS database was included in the 2007 and 2008 editions of HRSA’s annual Women’s Health USA publication.
Mary Segal, Temple University:
Evaluating the HealthChoices Program: Pennsylvania’s Medical Assistance Program for Persons with Disabilities. Gibbons E, Segal M, and the staff at the Institute on Disabilities. Final report to the Pennsylvania Developmental Disabilities Council on Grant #4100027247, May 2007.
Satisfaction reported by Pennsylvania’s Medical Assistance program enrollees with and without disabilities. Segal M, Caruso G, Bryen D. Presentation at the annual meeting of the Association of University Centers on Disabilities, Washington DC, November 2007.
Satisfaction with a Medical Assistance program reported by persons with and without disabilities. Segal M, Gibbons E, Caruso G. Submitted to the Review of Disability Studies, February 2008.
John Zweifler,OPA:
We have submitted a manuscript to the American Journal of Managed Care and are awaiting a decision.
Marc Elliot, RAND:
CMS REPORT TO CONGRESS: Plan to Implement a Medicare Hospital Value-Based Purchasing Program, November 21, 2007
Ron Hays, RAND:
Marshall, G. N., Morales, L. S., Elliott, M., Spritzer, K., & Hays, R. D. (2001). Confirmatory factor analysis of the Consumer Assessment of Health Plans Study (CAHPS) 1.0 core survey. Psychological Assessment, 13, 216-229.
Morales, L. S., Elliott, M. N., Weech-Maldonado, R., Spritzer, K.L., & Hays, R. D. (2001). Differences in CAHPS® adult survey ratings and reports by race and ethnicity: An analysis of the National CAHPS® Benchmarking Data 1.0. Health Services Research, 36, 595-617.
Elliott, M. N., Swartz, R., Adams, J., Spritzer, K. L., & Hays, R. D. (2001). Case-mix adjustment of the National CAHPS® Benchmarking Data 1.0: A violation of model assumptions? Health Services Research, 36, 555-573.
Weech-Maldonado, R., Morales, L. S., Spritzer, K., Elliott, M., & Hays, R. D. (2001). Racial and ethnic differences in parents’ assessments of pediatric care in Medicaid managed care. Health Services Research, 36, 575-594.
Weech-Maldonado, R., Morales, L. S., Elliott, M., Spritzer, K. L., Marshall, G., & Hays, R. D. (2003). Race/ethnicity, language and patients’ assessments of care in Medicaid managed care. Health Services Research., 38, 789-808.
Hargraves, J. L., Hays, R.D., & Cleary, P.D. (2003). Psychometric properties of the Consumer Assessment of Health Plans Study (CAHPS) 2.0 adult core survey. Health Services Research, 38, 1509-1527.
Weech-Maldonado, R., Elliott, M., Morales, L. S., Spritzer, K. L., Marshall, G., & Hays, R. D. (2004). Health plan effects on patient assessments of Medicaid managed care among racial/ethnic minorities. Journal of General Internal Medicine., 19, 136-145.
Reise, S. P., Meijer, R., R., Ainsworth, A. T., Morales, L. S., & Hays, R. D. (2006). Application of group-level item response models in the evaluation of consumer reports about health plan quality. Multivariate Behavioral Research, 41, 85-102.
Fongwa, M. N., Cunningham, W., Weech-Maldonado, R., Gutierrez, P. R., & Hays, R. D. (2006). Comparison of data quality for reports and ratings of ambulatory care by African Americans and White Medicare managed care enrollees. Journal of Aging and Health, 18, 707-721.
Reise, S. P., Morizot, J., & Hays, R. D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16 (supplement 1), 19-31.
Weech-Maldonado, R., Fongwa, M., Gutierrez, P., & Hays, R. D. (in press). Language and regional differences in evaluations of Medicare managed care by Hispanics. Health Services Research.
Howard Degenholz, University of Pittsburgh:
Data used for didactic purposes in a course on Managing the Health of Populations. Students analyze the data and complete several assignments.
Judith Lynch, RTI:
Technical Reports prepared as follows.
TRICARE Inpatient Satisfaction Survey – Overall Survey Results of Hospital Inpatients: July-September 2006: Final Report, Technical report prepared for the Office of the Assistant Secretary of Defense (Health Affairs), Health Program Analysis and Evaluation by RTI International, October 30, 2007.
TRICARE Inpatient Satisfaction Survey – Survey Results of Hospital Inpatients in Air Force Military Treatment Facilities (MTFs): July-September 2006: Final Report, Technical report prepared for the Office of the Assistant Secretary of Defense (Health Affairs), Health Program Analysis and Evaluation by RTI International, October 30, 2007.
TRICARE Inpatient Satisfaction Survey – Survey Results of Hospital Inpatients in Army Military Treatment Facilities (MTFs): July-September 2006: Final Report, Technical report prepared for the Office of the Assistant Secretary of Defense (Health Affairs), Health Program Analysis and Evaluation by RTI International, October 30, 2007.
TRICARE Inpatient Satisfaction Survey – Survey Results of Hospital Inpatients in Navy Military Treatment Facilities (MTFs): July-September 2006: Final Report, Technical report prepared for the Office of the Assistant Secretary of Defense (Health Affairs), Health Program Analysis and Evaluation by RTI International, October 30, 2007.
Eric Schone, Mathematica Policy Research:
Data are used in the TRICARE Beneficiary Reports and TRICARE Consumer Watch which are posted on the TRICARE website of the Department of Defense.
Rhonda BeLue, Pennsylvania State University:
Race/ethnic Disparities in Patient Perceived Quality of Hospital Care: The effects of care setting and health care context- -to be submitted this month (March 2008).
Robert Weech-Maldonado, University of Florida:
Weech-Maldonado, R., Elliott, M., Schiller, K.C., Oluwole, A., and Hays, R.D. Survey Response Style and Differential Use of CAHPS Rating Scales by Hispanics. Medical Care (in press).
Weech-Maldonado, R., Fongwa, M.N., Gutierrez, P., and Hays, R.D. 2008. Language and Regional Differences in Evaluations of Medicare Managed Care by Hispanics. Health Services Research (in press).
Fongwa, M.N., Cunningham, W., Weech-Maldonado, R., Gutierrez, P., and Hays, R.D. 2006. Comparison of Data Quality for Reports and Ratings of Ambulatory Care by African American and White Medicare Managed Care Enrollees. Journal of Aging & Health, 18(5):707-21.
Weech-Maldonado, R., Elliott, M., Morales, L.S., Spritzer, K., Marshall, G., and Hays,.D. 2004. Health Plan Effects on Patient Assessments of Medicaid Managed Care among Racial/Ethnic Minorities. Journal of General Internal Medicine 19(2): 136-145.
Weech-Maldonado, R., Morales, L.S., Elliott, M., Spritzer, K., Marshall, G., and Hays, R.D. 2003. Race/Ethnicity, Language and Patients’ Assessments of Care in Medicaid Managed Care. Health Services Research, 38(3): 789-808.
Weech-Maldonado, R., Morales, L., Spritzer, K., Elliott, M. and Hays, R.D. 2001. Racial and Ethnic Differences in Parents’ Assessments of Pediatric Care in Medicaid Managed Care. Health Services Research, 36(3): 575-594.
Morales, L., Elliott, M., Weech-Maldonado, R., Spritzer, K., and Hays, R.D. 2001. Differences in CAHPS Adult Survey Ratings and Reports by Race and Ethnicity: An Analysis of the National CAHPS Benchmarking Data 1.0. Health Services Research, 36(3): 595-618.
Presentations:
Weech-Maldonado, R., Elliott, M.N., Schiller, C., Hall, A., and Hays, R. “Does Hospital's Adherence to the CLAS Standards Influence Diverse Patients' Experiences with Inpatient Care?’ presented at the American Public Health Association Annual Meeting, Washington, DC, November 2007.
Weech-Maldonado, R., Elliott, M.N., Schiller, K.C., and Hays, R.D. “Differential Use of 0-10 Rating Scales by Racial/Ethnic Minorities in CAHPS” presented at the Joint Statistical Meeting, Seattle, WA, August 2006.
Weech-Maldonado, R. Panelist at Gerontological Society of America’s Workshop “Approaches for Developing/Adapting/Evaluating Health-Related Concepts and Measures in Diverse Older Populations,” Orlando, FL, November 2005.
Weech-Maldonado, R., Gutierrez, P., and Hays, R.D. “Hispanic Ethnicity, Language, and Patient Experiences with Care” presented at the Gerontological Society of America’s Annual Meeting, Washington, D.C., November 2004.
Weech-Maldonado, R., Elliott, M., Oluwole, A., and Hays, R.D. “Racial/Ethnic Differences in Assessments of Managed Care: Medicaid versus Commercial Insurance Status” presented at the AcademyHealth Annual Meeting, San Diego, June 2004.
Weech-Maldonado, R., Gutierrez, P., and Hays, R.D. “Hispanic Language and Patient Assessments of Medicare Managed Care” presented at the AcademyHealth Annual Meeting, San Diego, June 2004.
Weech-Maldonado, R., Morales, L.S., Elliott, M., Spritzer, K., Marshall, G., and Hays, R.D. “Health Plan Effects on Patient Assessments of Medicaid Managed Care among Racial/Ethnic Minorities” presented at the AcademyHealth Annual Meeting, Nashville, June 2003.
Weech-Maldonado, R., Morales, L.S., Elliott, M., Spritzer, K., Marshall, G., and Hays, R.D. “Impact of Race/Ethnicity and Language on Patients’ Assessments of Care in Medicaid Managed Care using CAHPS 2.0” presented at the American Public Health Association Annual Meeting, Philadelphia, November 2002.
Weech-Maldonado, R., Morales, L.S., Elliott, M., Spritzer, K., Marshall, G., and Hays, R.D. “Impact of Race/Ethnicity and Language on Patients’ Assessments of Care in Medicaid Managed Care using CAHPS 2.0” presented at the Academy for Health Services Research and Health Policy Annual Meeting, Washington, DC, June 2002.
Weech-Maldonado, R., Morales, L., Spritzer, K., Elliott, M. and Hays, R.D. “Latino Parents’ Assessments of Pediatric Care in Medicaid Managed Care: The CAHPS Experience” presented at the American Public Health Association Annual Meeting, Chicago, November 1999.
Appendix C
CAHPS Database
Executive Research Committee
Summary of Conference Call Meeting
March 6, 2008
I. Meeting Participants
The meeting was convened at 9:00 a.m. ET with the following participants:
ERC Members |
CAHPS Database Team |
Paul Cleary Steve Garfinkle Foster Gesten Liz Goldstein Ron Hays Ernie Moy (representing AHRQ) |
Bryan Davis John Rauch Janice Ricketts Dale Shaller
|
II. Meeting Objectives:
Dale thanked the ERC members for their time and service on the ERC. He reviewed the following objectives for the conference call:
Obtain ERC comments and suggestions regarding the process for reviewing and approving research data requests
Explore ideas for promoting the use of research files
III. Summary of Feedback from Researchers
Dale reviewed a summary of feedback obtained from researchers (see attached). Overall, the feedback on the process for obtaining and using research files was very positive. Several of the specific suggestions offered by researchers for improving the process led to a productive exchange with ERC members, summarized in the following section.
IV. Ideas and Strategies for Future Research File Services
Public use files: To promote access and use of the data, we could post the data files on the CAHPS Database Web site as public use files. To gain access, researchers would register on the site with basic information about the purpose of their study, etc. Information would be posted to refer interested individuals to the ERC for technical assistance if desired. The site could include a description of previous research (as we have compiled in the past). Initially, we may want to consider posting the legacy files, and not post new data sets until all the bugs are worked out.
ERC role: As Paul noted, the ERC has never disapproved an application. Most of the review consists of advice to researchers on similar studies or suggestions regarding methods. This function could continue as a service offered upon request, not as a required step. Ron expressed some concern about losing the ERC connection to researchers if review becomes totally voluntary.
IRB issue: All agreed that these data will always be exempt since no individual identifiers are included. Ernie noted that MEPS public use files say nothing about IRB at all. We could post a notice reminding researchers to follow their IRB process, and include language similar to the current statement in the Data Release Policy (regarding the Westat IRB determination).
Security requirements: Foster asked about the implication of public use files for data security, since the current proposal and DUA process include several provisions related to storage and protection of files. Dale responded that these provisions were largely adopted from CMS documents. Liz clarified that CMS uses a standard DUA because some of the CMS research files do include more sensitive identifying information. Dale commented that it would not be a problem to drop current provisions related to data security for the CAHPS Database files.
Response rates: Information related to response rates is currently not provided with the files because of variation in the quality of the data submitted. We could consider adding this information where available with appropriate language about the self-reported nature of this information.
Sampling frame: Data related to sampling frame are not currently requested. This information could be added to our data submission requirements, to test the ability to get useful sampling frame data.
Promotion: Ernie commented that the CAHPS data files have been requested less often than he expected. Moving to public use files would likely increase use, by removing barriers of submitting a proposal, waiting for approval, etc. Foster noted that we don't really have any information on reasons why researchers may have decided not to request data files; the high use of files from a relatively small pool of researchers suggests that the files may be underused. In addition to moving to public use files, other ideas for promoting use include: 1) webinars; 2) promotion as part of the AHRQ annual meeting "data café"; 3) promotion at the CAHPS User Group Meeting; and 4) more prominent promotion on the CAHPS Web site.
ERC burden: ERC members do not feel burdened by the current process, largely because use levels have been fairly low. The burden could increase if use increases, but in a public use file approach, that would depend on researcher decisions to obtain technical assistance. A large volume of technical assistance requests could be triaged to ERC members according to area of interest or simple rotation of duty.
CAHPS Database team burden: The current process is not burdensome to Westat staff. Even in the current process, researchers could be directed to a secure site for downloading approved files. If we move to public use, the administrative burden shifts primarily to maintenance of the files, triaging of technical assistance requests, and monitoring use of files.
V. Next Steps
CAHPS Database staff will review the suggestions from this meeting and develop a proposal for review by ERC members. The proposal will become part of the overall strategic plan to be developed for the CAHPS Database.
CAHPS Database staff will also update the summary of research projects undertaken with the research files.
| File Type | application/msword |
| File Title | Draft Plan for Getting Feedback from Users of CAHPS Database |
| Author | Dale Shaller |
| Last Modified By | Janice Ricketts |
| File Modified | 2009-11-17 |
| File Created | 2008-10-29 |