CMS-10569 Supporting Statement B CROWNWeb CY 2020 (11-15-2019)

Supporting Statement – Part B

Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The Data Validation Contractor will randomly sample 300 facilities (roughly 5% of all dialysis facilities), per contract and Quality Incentive Program (QIP) rule guidelines, for participation in the validation project. As a random sample, this should be a representative sample of all included facilities nationally. The sample pool will consist of Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb in order to Meet Section 4943108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities.

The 300 facilities will be asked to submit records that will be validated for CMS-designated Critical Performance Measures (CPMs). The patient sample size is limited to 10 patients per facility, as per contract and QIP rule guidelines. The Data Validation Contractor will sample 10 patients (or the maximum patients possible) from each selected facility for CPM reviews. Historically, facility response rates have been solid. The response rate for the 2018 validation study was 100% of the 300 facilities selected for participation.

Sample Size Estimates

Wes are taking a fundamentally different data stratification approach compared to what was used for prior years of the study. We are stratifying sampled facilities by CMS Network Number and by affiliation with major dialysis organizations (DaVita, DCI, Fresenius, US Renal Care, and all others as Independent) as shown in Tables 1 and 2, respectively.

Using the ESRD QIP rule guidelines of randomly selecting 300 facilities from the total population of eligible facilities, and randomly selecting 10 records per facility, the Validation Contractor determined the distribution of patient records by Network Number and affiliation.

Table 1: Distribution of Patients within Network Number

Network Number	Number of Patient Records per Month	% Total Patients
1	170	2.05
2	519	6.26
3	226	2.73
4	395	4.77
5	340	4.1
6	905	10.92
7	383	4.62
8	303	3.66
9	882	10.64
10	367	4.43
11	736	8.88
12	248	2.99
13	406	4.9
14	641	7.73
15	442	5.33
16	243	2.93
17	370	4.46
18	712	8.59
Total	8288	100.00

Table 2: Distribution of Patients within Affiliation

Affiliation	Number of Patient Records per Month	% Total Patients
DaVita	2839	34.25
DCI	271	3.27
Fresenius	3420	41.26
US Renal Care	411	4.96
Independent	1347	16.25
Total	8288	100.00

Some smaller facilities had less than 10 patients treated for the period; in these cases, we selected all the patients treated at the facility during the study period for validation. Table 3 depicts the methodology used when sampling for patients for CPM reviews.

Table 3: Sampling Methodology for CPM Reviews

Sampling Source	Sample to be Taken
CROWNWeb Extract	Random selection of patients available, up to 10

Sampling Time Frame

The 300 facilities to be sampled for validation will be chosen within 10 days of receiving the corresponding Facility/Patient data file from CROWNWeb. The Validation Contractor will receive a CROWNWeb extract that contains all data reported into CROWNWeb during the selected second quarter time frame (April – June 2020).

This timeframe was selected after considering several factors. To ensure that the validation can be completed during the period of performance, the Validation Contractor considered the data reporting periods allowed to facilities to submit clinical data into CROWNWeb. Facilities are given 60 days from the end of any month to enter CROWNWeb clinical data. The mandated reporting period limits the time frame we can validate expeditiously, as we will not be able to obtain an extract until after the close of the data-reporting period.

Another important consideration is that it is mandatory for us to give facilities up to 60 days to submit records. Taking into consideration these factors as well as the need to ensure that there is adequate time to perform analysis and prepare reports, we decided on the second quarter of 2020 validation time frame. A breakdown of the mandated reported deadlines that were taken into consideration is displayed in Table 4.

Table 4: Mandated Reporting Deadlines

Submission Type	Mandated Reporting Deadlines
CROWNWeb Data Submission	60 days after month close (Q2 – August 31, 2020)
Facility Record Submission Deadline	60 days after request receipt per QIP rule

Assuming the CROWNWeb data team will need at least one week to export and send the data, the Validation Contractor has estimated preliminary dates for data availability. Table 5 provides the Validation Contractor’s estimates for when the CROWNWeb data will be received for each corresponding data set.

Table 5: Estimated Timeline for Receiving CROWNWeb Data

Type of Data	Data Reporting Period	Estimated Receive Date
CPM	April, May, June 2020	Starting mid-November through End of December 2020

Due to the tight timeframe for data abstraction, effective coordination and management as well as adherence to established schedules will be crucial to the project’s success.

2. Describe the procedures for the collection of information including:

- Statistical methodology for stratification and sample selection,

- Estimation procedure,

- Degree of accuracy needed for the purpose described in the justification,

- Unusual problems requiring specialized sampling procedures, and

- Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Please see response to question 1 for statistical methodology for stratification and sample selection, including estimation procedure. As noted below in response to question 4, there are no unusual problems requiring specialized sampling procedures as our previous experience on past CMS CROWNWeb CPM validation efforts have shown near universal compliance with medical record requests. The period for data collection cycles is expected to be no more frequently than annually.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield 'reliable' data that can be generalized to the universe studied.

Facilities will be contacted in early winter via QualityNet using the Secure File Transfer option and will be asked to participate in the validation effort. The letter will provide instructions on the types of records to be submitted, methods to submit records to the Validation Contractor, and identify patients selected for validation. Facilities that do not respond to the initial request for records are contacted via phone by the Validation Contractor and receive a final request letter in early March 2020. To aid in maximizing facility response rates, our team coordinates through CMS COR to send a townhall notification around January 2019 to increase facility exposure to our validation study.

We also communicate/coordinate extensively with all facilities, using web conferences to facilitate on-time and accurate medical records submission by participating clinics. Facilities that do not respond to the request for records are subject to a 10-point reduction to their Total Performance Score (TPS). The response rate for the 2018 validation study was 100%; of the 300 facilities selected for participation, all eligible participating facilities responded and complied with our records request. For future validations, we plan to follow the same records request methodology, follow-up, and ESRD community outreach approach we’ve used in the past since it has been effective in producing desired response rates.

Data Validation

The main objective of this analysis is to perform a single comparison of the CROWNWeb system data against CPM element data obtained from the facilities’ records, leading to an evaluation of the reliability (i.e. the data are reasonably complete and accurate) and validity (i.e. the data represent what is being measured) of CROWNWeb data.

• Reliability: Reliability means data are reasonably complete and accurate, meet intended purposes, and are not subject to inappropriate alteration. Where:

• Completeness refers to the extent that relevant records are present and the fields in each record are populated appropriately, and,

• Accuracy refers to the extent recorded data reflect the actual underlying information.

A more formal definition of reliability is the extent to which results are consistent over time and an accurate representation of the population under study:

• The degree to which a measurement, given repeatedly, remains the same,

• The stability of the measurement over time, and

• The similarity of measurements within a given time period.

• Validity: Validity (as used here) refers to whether the data actually represent what one believes is being measured. Several measures are commonly used to assess validity of any measure.

To ensure the reliability of data collected by reviewers, we use two reviewers for each patient record. We systematically measure differences between reviewers for all patient records and provide ongoing training as needed to correct reviewer error tendencies. All discrepancies are reconciled by the second reviewer.

Implementing this element of the study design enables us to focus on reviewer accuracy rather than reviewer agreement. We use a system named CROWNWeb Abstraction Processing System (CAPS) that presents the second reviewer a split screen page review, displaying the first and second reviewer results. This page provides the second reviewer the capability to identify any differences and make needed updates to the second reviewer findings. Consequently, we always use second reviewer results in our analysis.

Additionally, reviewers make full use of the Consult feature of CAPS. Whenever either reviewer needs to reach out to a more experienced reviewer, the person moves the patient record to the Consult phase. There the two of them resolve the issue and then move the record back to the point where regular review processing was interrupted.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

As noted above, the sample pool will consist of Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb to meet Section 494.108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities. The previous experience on past CMS CROWNWeb validation efforts have shown near universal compliance with medical record requests. No additional tests of procedures or methods to be undertaken are expected.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Delia Houseal, Centers for Medicare and Medicaid Services, (410) 786-2724