CMS-10639_NHSN Supporting Statement B_FY2017_Revised-2.7.12.2017

Supporting Statement – Part B

Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The purpose of this validation study is to compare the data entered by facilities into CDC’s NHSN system against what is reported in medical records. The Validation Contractor will randomly sample 35 facilities (0.5% of the approximately 6800 dialysis facilities in the United States), per contract and End-Stage Renal Disease (ESRD) Quality Incentive Program (QIP) rule guidelines, for participation in this validation study. As per the QIP CY 2017 rule, “we will randomly select 35 facilities to participate in an NHSN dialysis event validation study for two quarters of data reported in CY 2017.” We decided to only validate 10 records from 35 facilities as this represents the maximum amount of data that CMS could collect under the monetary resources available to conduct the study. The sample pool will consist of current Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb in order to meet Section 494.108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities. The 35 facilities will be asked to submit records that will be validated for NHSN dialysis event elements. The patient sample size is limited to 10 patients per facility, as per contract and QIP rule guidelines. The Validation Contractor will sample 10 patients (or the maximum patients possible) from each selected facility for NHSN reviews.

Sample Size Estimates

As noted above, the purpose of the validation is to compare the data entered by facilities into CDC’s NHSN system against what is reported in medical records. Three strata of facilities were chosen to allow for the creation of a better representation of the source population based upon facility size. Using data from 2014 and 2015, we estimate the numbers of facilities per three (3) strata by the numbers of patients reported per year. Using the ESRD QIP rule guidelines of no more than ten (10) records per facility and no more than 35 facilities sampled, we estimate the number of records to be obtained from each of the three strata of facilities. The size of the facility is hypothesized to influence the quality of the data inputted into the NHSN system, as smaller facilities may have less resources for completing this administrative task. However, whether facility size impacts NHSN data quality needs to be further evaluated with this validation study. While the sample size by strata will allow for an initial exploration as to whether facility size impacts NHSN data quality, sample size estimates for patient records to be reviewed for the NHSN study are limited by the total number of records that can be selected from each facility and the size of the facilities.

The number of facilities (35) and the number of records from each facility (10) was decided based upon conversations between CMS and the CDC regarding the budgetary constraints. Given the funding available, it was determined that 350 records could be validated. To ensure that a sufficient number of records was being validated from each selected facility, we determined it was reasonable to validate 10 records from each of 35 facilities selected for participation in the validation study.

Table 1: Sample Size Estimates

Number of Patients per Quarter	% Total Facilities	% Total Patients	Number of Facilities	Number of Records to Sample per Facility	Number of Records
0 to 25	14.51%	3.25%	5	10	50
26 to 85	54.48%	40.09%	19	10	190
>85	31.01%	56.66%	11	10	110
Total	100.00%	100.00%	35		350

The methodology finalized for PY 2019, and described in the CY 2017 ESRD PPS Final Rule (81 FR 77895) and in the accompanying Supporting Statement A, states that the Validation Contractor will send the facilities selected for participation requests for medical records for all patients with “candidate events” during the evaluation period. Facilities will have 30 calendar days to respond to the request for medical records based on candidate events either electronically or on paper. We realize that some smaller facilities may have fewer than 10 patients treated for the period. In that case, if the Validation Contractor determines that additional medical records are needed to reach the 10-record threshold from a facility to validate whether the facility accurately reported the dialysis events, then the contractor will send a request for additional, randomly selected patient records from the facility. The facility will have 30 calendar days from the date of the letter to respond to the request for NHSN reviewed facilities, the facilities will submit up to 10 patients based upon an NHSN extract that will contain patients for which a dialysis event has been reported. If there are not 10 patients with dialysis events for a particular facility, the Validation Contractor will then pull a random sample from the general pool of patients at the facility. The random sample of patients will allow for an examination of possible underreporting of dialysis events in the NHSN system. Table 3 depicts the methodology to be used when sampling for patients for NHSN reviews.

Table 3: Sampling Methodology for NHSN Reviews

Sampling Source	Sample to be Taken
NHSN Extract	Maximum number of patients available, up to 10
General Pool of Patients from Patients within CROWNWeb	Remainder of patients needed to equal 10 after NHSN patients are sampled

Sampling Time Frame

The Validation Contractor intends on validating records for the first and second quarter of 2017. This time frame was selected after considering several factors. To ensure that the validation can be completed during the period of performance, the Validation Contractor considered the data reporting periods allowed to facilities to submit clinical data into CROWNWeb and NHSN. Facilities are given 60 days to enter data from the previous month for CROWNWeb clinical data and are given three months after quarter close to submit dialysis event data in NHSN (for Q2 the data submission deadline is September 30). These mandated reporting periods limit the time frame we can validate expeditiously, as we will not be able to obtain an extract until after the close of the data-reporting period. We also took into account operational concerns including the time it would take to request and receive the CROWNWeb and NHSN data extracts. Due to operational considerations and logistics, CMS and the CDC are in agreement that it may take the CDC up to a month to provide the data extract to be used for sampling. Another important consideration is the ESRD QIP rule that makes it mandatory for us to give facilities up to 60 days to submit records. Taking into consideration these factors as well as the need to ensure that our reviewers have adequate time to complete NHSN reviews and that there is adequate time to perform analysis and prepare reports, we decided on using the first two quarters of 2017 as the validation time frame. A breakdown of the mandated reported deadlines that were taken into consideration is displayed in Table 4.

Table 4: Mandated Reporting Deadlines

Submission Type	Mandated Reporting Deadlines
CROWNWeb Data Submission	60 days after month close (Q2 – August 31, 2017)
NHSN Data Submission	90 days after quarter close (Q2 – September 30, 2017)
Facility Record Submission Deadline - NHSN	60 days after request receipt per QIP rule

Assuming the CROWNWeb data team will need at least 10 days and the CDC data team will need at least one month to export and send the data, the Validation Contractor has estimated preliminary dates for availability of January, February, March, April, May, and June 2017 data. It is anticipated that the NHSN data will be received during the month of November. Due to the tight time frame for data abstraction, effective coordination and management, as well as adherence to established schedules, will be crucial to the success of the project. As noted below in response to question 3, our response rate for the NHSN Feasibility Study was high; of the nine facilities selected for participation, nine responded (100%).

2. Describe the procedures for the collection of information including:

- Statistical methodology for stratification and sample selection,

- Estimation procedure,

- Degree of accuracy needed for the purpose described in the justification,

- Unusual problems requiring specialized sampling procedures, and

- Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Please see response to question 1 for statistical methodology for stratification and sample selection, including estimation procedure and degree of accuracy needed for the purpose of this work. As noted below in response to question 4, there are no unusual problems requiring specialized sampling procedures as our previous experience on past CMS NHSN validation efforts have shown near universal compliance by dialysis facilities with medical record requests. The period for data collection cycles is expected to be no more frequently than annually.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield 'reliable' data that can be generalized to the universe studied.

As part of our previous work to collect medical records for the 2015 NHSN Feasibility Study, upon which the NHSN project is based, facilities were contacted via certified letter in January 2016 and were asked to participate in the validation effort. The letter provided instructions on the types of records to be submitted, methods to submit records to Team AST, and identified patients selected for validation. Facilities that did not respond to the initial request for records were contacted via phone by Team AST and received a final request letter in March 2016. Facilities that did not respond to the request for records were subject to a 10-point reduction to their Total Performance Score (TPS). The response rate for the feasibility study was high; of the nine facilities selected for participation, nine responded (100%). Selected facilities reported 16 positive blood culture dialysis events. Each patient medical record associated with each event was requested for the selected time frame and received for validation.

Data Validation

The main objective of this analysis is to perform a single comparison of the NHSN system against NHSN “candidate event” data obtained from the facilities’ records, leading to an evaluation of the reliability (i.e., the data are reasonably complete and accurate) and validity (i.e., the data actually represent what is being measured) of NHSN data. Candidate events include positive blood cultures, intravenous antimicrobials, or vascular site infection (e.g. pus, redness, or increased swelling).

Reliability: Reliability means data are reasonably complete and accurate, meet intended purposes, and are not subject to inappropriate alteration, where:

• Completeness refers to the extent that relevant records are present and the fields in each record are populated appropriately, and,

• Accuracy refers to the extent recorded data reflect the actual underlying information. In this instance, that the data in the NHSN system accurately reflects the data contained within the source documents, i.e. the facilities’ medical records.

Validity: (as used here) refers to whether the data actually represent what one believes is being measured. A number of measures are commonly used to assess validity of any measure.

In our interpretation of these measures, we identify the key sources of overall disagreement between the NHSN data and the patients’ medical records, which would serve as the “gold standard.” Typical sources of disagreement include missing information about events, inaccurate dates and inaccurate dialysis catheter information.

We propose to use Cohen’s Kappa () because it is an overall measure of agreement between the test and reference databases. The kappa coefficient, κ, is calculated as κ = (p_o – p_e) * (1 – p_e), where p_o is the observed agreement between two classifications and p_e is the expected agreement between two classifications based on the marginal distributions.

Cohen’s Kappa is usually presented as a proportion; we will convert key measures of agreement from proportions to percentages for easier interpretation. All measures will range from -100 % (perfect disagreement) to +100% (perfect agreement). Landis and Koch (1977) have suggested overall agreement between two classifications using Cohen’s Kappa be interpreted as shown in Table 6.

Table 6: Interpretation of Cohen's Kappa

Kappa ()	Interpretation
< 0%	Poor
0% to 20%	Slight
20% to 40%	Fair
40% to 60%	Moderate
60% to 80%	Substantial
80% to 100%	Almost Perfect

Although the Landis and Koch interpretation is an old one, it is still widely referenced and is still the dominant one used to indicate the strength of agreement (Cunningham, 2009).

Other coefficients of agreement have been suggested but as Fleiss pointed out (1981),  has a number of qualities that has made it an attractive option for the measurement of agreement:

• If there is complete agreement,  = +1,

• If observed agreement is greater than or equal to chance agreement,  ≥ 0, and

• If the observed agreement is less than or equal to chance agreement,  < 0.

However, over the years a number of difficulties in the interpretation of Cohen’s Kappa have been pointed out and several statistical fixes have been proposed. Kappa not only measures agreement, but it is affected in complex ways by the distribution of data across categories that are used (“prevalence”) and by bias that may be inherent in the measures used. These are the problems associated with Kappa (Feinstein and Cicchetti, 1990):

1. If the expected agreement (p_e) is large, the correction process can convert a relatively high value of the observed agreement (p_o) into a low value of Kappa ().

2. Unbalanced marginal totals produce higher values of  than balanced totals.

Kappa is also affected both by any bias between the two measures of gender and by the overall prevalence (the relative probability of the responses – the “Yes” and “No” responses). Byrt, Bishot, and Carlin (1993) introduced measures of prevalence (prevalence index – p_index) and bias (bias index – b_index) that can be used to compensate for the biases and suggest that these measures are reported together with . The p_index and b_index can then be used to produce a prevalence-adjusted, bias-adjusted kappa (PABAK) that takes on the values of -100% when there is no agreement and +100% when there is perfect agreement and 0 when the agreement is equal to 50%. Additionally, PABAK is linearly related to p_o.

Given the following standard table with two measures and a dichotomous response, we can therefore add the following measures of agreement shown in Table 7 below (Byrt, Bishot and Carlin, 1993; Cunningham, 2009).

Table 7: Standard Fourfold Table

Measurement A	Measurement B, Yes	Measurement B, No	Total
Yes	a	b	a + b
No	c	d	c + d
Total	a +c	b +d	N

1. The observed proportion of agreement, po = (a+d)  N

2. Expected proportion of agreement, p_e = ((a + c)(a + b)+(b + d)(c + d))  N²

3. Proportion of positive agreement, p_pos = (2a)  (N + a – d)

4. Proportion of negative agreement, p_neg = (2d)  (N – a + d)

5. Prevalence Index, p_index = (a – d)  N

6. Bias Index, b_index = (b – c)  N

7. Prevalence-adjusted, bias-adjusted Kappa, PABAK = 2p_o – 1

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

As noted above, the sample pool will consist of Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb in order to meet Section 494.108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities. Our previous experience on past CMS NHSN validation efforts have shown near universal compliance with medical record requests. No additional tests of procedures or methods to be undertaken are expected.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Jon Mark Hirshon, University of Maryland-Baltimore, 1 (410) 328-7474

Joseph Kufera, University of Maryland-Baltimore, 1 (410) 328-4161

Darren Childers, Allegheny Science & Technology, 1 (304) 612-1571

Heather Duvall, Healthcare Management Solutions, 1 (304) 816-5262

8