CMS-10393 11th_SOW_Supporting Statement_Part_B_ 02 15 2018

CMS 10393 Supporting Statement – Part B

Beneficiary and Family Centered Data Collection

Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The target populations for the data collection initiatives are Medicare beneficiaries and their representatives who have used the services of the Beneficiary and Family Centered Care Quality Improvement Organization (BFCC QIO). For analytic purposes, the population is divided into four sub groups as follows:

1. Filed an appeal;

2. Filed a complaint that was addressed using a quality of care Medical Record Review;

3. Filed a complaint that was addressed using Immediate Advocacy; and

4. Did not formally file an appeal or complaint.

Data collection initiatives include a range of approaches to support the parallel goals of providing CMS will valid and reliable data for QIO contract evaluation, and providing CMS and the QIOs necessary information for use in on-going quality improvement efforts including the design of future QIO scopes of work (SOWs). The data collection initiatives and their primary purposes are as follows:

1. Experience Survey (QIO contract evaluation);

2. Direct Follow-up (on-going quality improvement); and

3. General Feedback Web survey (providing an opportunity for every beneficiary to have a “voice”).

The sampling and data collection methodologies have to be efficient, minimally burdensome for respondents, frequent enough for use in on-going quality improvement efforts, and rigorous enough to support scoring and reporting about QIO performance. To achieve all of the above described goals, CMS will collect and report the data monthly with formal analysis and scoring conducted annually.

The Experience Survey (Attachment A) will be administered via telephone to beneficiaries/ representatives after the complaint/appeal case has been closed to assess their overall and specific experiences with the BFCC QIOs. Given the desire to report survey findings for each of the 5 QIO Service Areas and the relatively small annual total volume of complaints cases addressed using Medical Record Review (n~4,900) and complaints cases addressed using Immediate Advocacy (n~1,300), CMS will include a census of these cases in data collection. The annual total volume of appeals cases is much larger (n~147,000) so CMS will draw a stratified sample using simple random sampling within each of the five QIO Service Areas.

The Experience Survey relies on Service Area-stratified sampling for appeals cases to obtain statistically valid and reliable quantitative data. The sample sizes reflect the goal of using survey data for QIO contract evaluation and quality improvement efforts at the Service Area level.

The sample design for beneficiaries who have filed a complaint (Medical Record Review or Immediate Advocacy) is consistent with the design previously approved by OMB. A census of complaints cases were fielded for data collection in order to produce a data set of an adequate size for national analysis. There is no known reason to believe that the volume of complaints cases will change notably in the 11^th SOW and as such, a census of complaints cases are recommended for inclusion in data collection in order to support analysis at the Service Area level. Again, as previously approved by OMB, we will field a sample of the much larger universe of appeals cases, approximately a six percent sample in order to obtain sufficient data to support analysis at the Service Area level.

• The survey will be administered to all (census) beneficiaries/representatives who file a complaint and have it addressed through medical record review (n=4,900).

• The survey will be administered to all (census) beneficiaries/representatives who file a complaint and have it addressed through immediate advocacy (n=1,300).

• The survey will be administered to a sample of beneficiaries/representatives who file an appeal (n=8,800).

Direct Follow-up calls (Attachment B) will be made to a Service Area-stratified sample of beneficiaries/ representatives who have contacted the BFCC QIO. This sample includes individuals who contacted the BFCC QIO help desk for any reason including those that did not result in the formal initiation of an appeal or complaint case. This data collection will provide information about beneficiary experiences with the QIOs that extend beyond processing of complaint and appeal cases.

The Direct Follow-up calls also use sampling, though a much smaller sample is required to fulfill the goal of supporting quality improvement activities. This initiative is intended to ensure that CMS and the QIOs understand the experiences of all individuals who call the help desk, not just those who formally initiate an appeal or complaint case. With quality improvement as the goal, a small sample is sufficient to provide a sense of beneficiary experience and support QIOs in ensuring their approaches are beneficiary and family-centered.

• The survey will be administered to a sample of beneficiaries/representatives who contact the BFCC QIO help desk for any reason. (n=200).

General Feedback Web Survey (Attachment C) will be open to all beneficiaries/ representatives who file a complaint or appeal. A link to a General Feedback Web Survey will be included with each case closure letter sent by the QIOs.

The General Feedback Web Survey will be provided to the universe of closed appeal and complaint cases [census]. This serves as a person-centered way to offer every beneficiary/ representative an opportunity to voice their experience with the QIO whether or not they are selected to receive the Experience Survey.

• The General Feedback Web Survey link will be sent to all (census) beneficiaries/representatives who file a complaint or an appeal (n=150,200).

Table 1 presents the respondent universe, sample, anticipated response rate, and resulting number of responses by sub group and data collection initiative.

Table 1: Annual Sampling Approach

	Appeal	Complaint Medical Record Review	Complaint Immediate Advocacy	No Case Initiated
Universe*	144,000	4,900	1,300	Unknown
Experience Survey
Sample	8,800 (sample)	4,900 (census)	1,300 (census)	N/A
Response Rate %	65	65	65
Respondents	5,720	3,185	845
Direct Follow-up
Sample	200 beneficiaries who contact the QIO help desk
Response Rate %	65
Respondents	130
General Feedback Web Survey
Sample	144,000	4,900	1,300	N/A
Response Rate %	10	10	10
Respondents	14,400	490	130
*Based on May 2016 case volumes

* During the 9^th SOW, Beneficiary Experience Survey data were collected using computer assisted telephone interviewing. Through this data collection, a 75 percent response rate was achieved.

2. Procedures for Data Collection

The national sample to be drawn for the Experience Survey and Direct Follow-up will be distributed proportionally among the five QIO Service Areas to support analysis and reporting at the Service Area level. Explicit strata for oversampling any particular group of beneficiaries will not be employed. Service Area-stratified sampling will be used to select a representative sample of beneficiaries/representatives.

The data collection methodology for the Experience Survey and Direct Follow-up flows from the proposed sampling approach and the need for on-going data for quality improvement. Based on recent literature on survey methodology and response rates by mode, we recommend using a data collection that is primarily telephone with mail or web options available upon request. This methodology will achieve the goals of being efficient, effective, and minimally burdensome for beneficiary respondents. Substantial differences in the demographics of mail and early phone respondents confirm that mixed-mode data collection methods are critical in reducing nonresponse bias. Hence, this approach is being implemented to maximize response rates by taking into account that repeated contact attempts may yield higher returns by mail for some groups and higher returns by phone for others.¹

During the 9^th SOW, Beneficiary Experience Survey data were collected using computer assisted telephone interviewing. Through this data collection, a 75 percent response rate was achieved. The data collection methodology for the 11^th SOW also relies on obtaining survey response by telephone, with mail or web options as requested. With response rates falling steadily, we estimate that we will achieve a 65 percent response rate using this methodology. The data collection approach in place to support the current OMB-approved survey was yielding a 35 percent response rate with a mail-only methodology. With the addition of up to three pre-survey verification calls and up to three additional non-response follow-up reminder calls, the response rate for the current survey increased to 46 percent. We feel that moving to a telephone data collection strategy with up to five calls offering telephone data collection (rather than simple reminders) and supplemented with mail and e-mail non-response follow-up will permit us to increase response rates to 65 percent.

Data will be collected monthly during the 11^th SOW. That is to say, the annualized sample and burden hours will be allocated evenly across 12 months. The on-going data collection yields will be used to implement mid-stream improvements to data collection processes, as needed. The data collection methodology will minimize the data collection field period while maximizing the response rate. Data will be cleaned and interim data delivered to CMS for CMS and QIO quality improvement review. We will complete sampling, primary data collection, data cleaning, and delivery within a four to six-week period. Results will be provided to CMS and QIO via on-line reporting dashboard.

3. Methods to Maximize Response Rates and understand non-response bias

In support of telephone data collection initiatives, we will place up to five calls to each sampled case. Calls will be made on different days of the week, and during different times of day. To the extent possible, specific interview appointments will be set with respondents to facilitate completion of the survey in a beneficiary-centered manner. Bilingual telephone interviewers will be available to complete the survey in Spanish, if needed. Upon request and for cases we are unable to reach by telephone, we will provide hard copy surveys and a web option to beneficiaries/representatives. Hard copy surveys will be available in English, Spanish, and large print.

To gauge the degree to which the current survey is meeting CMS needs, minimizing bias and obtaining valid and reliable data, the data collection contractor prepared three analytical reports:

• Response Rate of Select Variables  - which examined response rate data to identify any subgroups less likely to respond to the survey and examined potential bias in the survey results;

• Demographic Comparison – which identified the demographic characteristics of beneficiaries/representatives who filed complaints or appeals and declined consent or refused to participate in a follow up survey about their experience with the QIOs, and,

• Geographic Comparison – which identified the geographic characteristics of Medicare beneficiaries who filed complaints or appeals and declined consent or refused to participate in a follow up survey about their experience with the QIOs.

The reports demonstrated that there was little non-response bias by age or sex but some differences existed by race and ethnicity as well as geography. The areas of greatest concern regarding survey bias were the large gaps in survey frame inclusion resulting from beneficiaries refusing consent at the time of case closure with the QIO. Initially, with an overall consent to survey fame inclusion of 29 percent, beneficiaries residing in some areas, particularly heavily populated coastal regions were included in the sample less than one third of the time, while beneficiaries in other regions, typically less populated, central areas, were included at rates over 90 percent. Similarly, beneficiaries who were Black were represented in the sample frame at a rate of 34 percent, while beneficiaries who were Asian were represented at a rate of 58 percent.

To improve the sample frame and reduce potential bias, the survey contractor worked with the QIOs to ensure the highest possible rate of agreement to survey frame inclusion. Most recently, this rate has been at 93 percent for beneficiaries across all areas.

In addition to looking at sources of bias regarding the sample frame, the contractor also considered possible survey response bias. Analysis showed that beneficiaries of all ages, sexes, and geographic areas were responding at similar rates. Some response differences were however noted by race and ethnicity. Beneficiaries of Hispanic origin were the least likely to respond (14 percent response) and beneficiaries who were White were the most likely (32 percent response). To address this concern, the survey contractor added pre-survey verification calls and non-response follow-up reminder calls conducted with bilingual staff to help better identify and support beneficiaries who may prefer to complete the survey in Spanish as well as those who need support understanding the purpose of the survey. Annual survey maintenance including non-response analysis is scheduled to be conducted again in June 2018, though the recent overall increase in the survey response rate, coupled with literature demonstrating the effectiveness of mail and telephone protocols for reducing non-response bias suggest that survey efforts may have been effective at reducing disparities response by minority racial and ethnic groups.

4. Testing

Each of the data collection instruments will undergo expert review and testing. While the methodology of extensively tested instruments including the CAHPS surveys has been used as a model in development, each instrument was reviewed and tested with members of CMS’ Beneficiary and Family Advisory Committee representing beneficiaries or family representatives. Review and testing will ensure the following:

1. Determine if the survey wording is clear and unambiguous;

2. Verify respondent’s ability to recall interactions pertaining to their complaints and appeals cases;

3. Ensure appropriate and consistent flow question wording and overall survey administration; and

4. Ensure data capture and data output are functioning flawlessly.

Review and testing was conducted in late 2017 and resulted in some recommended updates to some of the survey outreach materials (survey covering letters). All other survey content was deemed to mirror the language that beneficiaries use to describe their experience with filing a complaint or appeal and to be cognitively simple enough for beneficiaries to understand the frame of reference. As the QIO program has been substantively restructured under the 11^th SOW, expert review and testing served as an important opportunity to learn from beneficiaries and their families about how they describe their experiences under the new QIO structure.

A second round of cognitive testing has been deemed to be unnecessary unless changes are recommended or required as a result of OMB review.

5. Individuals Consulted

The following individuals were consulted in the development of the data collection instruments, sampling, and methodology:

Organization	Name	Contact Information
CMS	David Russo	617-565-1310 [email protected]
	Sally Berko	410-786-6211 [email protected]
Avar Consulting	Z. Joan Wang	301-977-6553, ext. 222 [email protected]
	Wendy Gary	337-385-2144 [email protected]
	Steven Fink	301-977-6553, ext. 219 [email protected]
	David Bercham	301-977-6553, ext. 230 [email protected]
Econometrica	Thomas Jackson	240-395-2271 [email protected]
	Munseok Seo	240-395-2283 [email protected]
Westat	W. Sherman Edwards	301-294-3993 [email protected]
	Vasudha Narayanan	301-251-2257 [email protected]
	Stephanie Fry	301-294-2872 [email protected]

1 Burkhart, Q., A Haviland, P. Kallaur, P. Kallaur, C. Edwards, J. Brown, and M. Elliott. 2015. How much do additional mailings and telephone calls contribute to response rates in a survey of Medicare Beneficiaries? Field Methods. Vol 27(4) 409-425.

CMS 10393 11^th SOW Supporting Statement Part B

4/4/2017 6