CMS-10639 NHSN_Supporting_Statement B_CY2018 July 30 2018

Supporting Statement – Part B

Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

RELI Group will randomly sample 35 facilities per contract and Quality Incentive Program (QIP) PY2019 rule guidelines, for participation in the validation project. As a random sample, this is expected to be an unbiased collection of facilities.

Chart 1 displays the distribution across the nation of all eligible facilities for each CMS assigned Network Number. We use this as a reference to determine whether the randomly selected facilities show a similar distribution pattern.

Chart 1: Distribution of Eligible Facilities within Network Number

Chart 2 shows the distribution across the nation for the selected facilities.

Table 1 shows the sample percentage distribution across the nation for the selected facilities.

Table 1: National Versus Sample Percentage Distribution within Network Number

Network Number	National Percentage	Sample Percentage
1	3%	3%
2	4%
3	3%	3%
4	5%
5	6%	11%
6	10%	14%
7	6%	9%
8	6%	3%
9	9%
10	4%	11%
11	7%	9%
12	5%	3%
13	5%	9%
14	9%	9%
15	5%	3%
16	3%	6%
17	4%	9%
18	6%

We conclude that the requirement to randomly select facilities has been met.

The sample pool consists of Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb to meet Section 494.108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities. The 35 facilities were asked to submit records that will be validated for NHSN dialysis event elements. The patient sample size is limited to 10 patients per facility, as per contract and QIP rule guidelines. The response rate for the 2017 validation study is 89%.

Sample Size Estimates

We segmented facilities by CMS assigned Network Number and by affiliation in the 2017 study. There are 18 unique Network Numbers that group facilities largely by geographical region as shown in Chart 1. Table 1 shows the sample size distribution according to Network Number. Some smaller facilities may have fewer than 10 patients treated for the period; in these cases, the facilities will submit records for all the patients treated at the facility during the study period for validation. If there are not 10 patients with dialysis events for a facility, the facility will pull a random sample from the general pool of patients at the facility to result in selecting a total of 10 patients. The random sample of patients will allow for an examination of possible underreporting of dialysis events in the NHSN system.

Sampling Time Frame

In consultation with CDC, RELI agreed to carry out the following steps.

Step 1: RELI will provide CDC with a line list of patient medical charts reviewed with data on variables listed in Table 2. Data should be provided in MS Office Excel format.

1a. Test File Data transmission: To conduct a test of the cross-match process, RELI will send a test file to CDC on or before COB May 15, 2018. This test file should include a minimum of 5 observations with a positive blood culture and 5 observations from patients with no positive blood culture from the completed chart reviews.

-          Data will be transmitted via QualityNet

o   RELI sender: Khalil Abdul-Rahman

o   CDC recipient: Preeti Ravindhran

-          Data will be transmitted from RELI to CDC on or before COB May 15, 2018. CDC Recipient (Preeti Ravindhran) will conduct the test cross-match, complete the excel template and provide any necessary edits to the template to RELI by June 1, 2018.

1b. Complete cross-match file transmission: The final line list of positive blood cultures and patient information for the NHSN cross match will be sent by the RELI group to CDC. This will include all the positive blood cultures in the charts reviewed for the validation of CY 2017/PY 2019.

-          Data will be transmitted via QualityNet

o   RELI sender: Khalil Abdul-Rahman

o   CDC recipient: Preeti Ravindhran

-          Data will be transmitted from RELI to CDC on or before COB June 22, 2018.

Step 2: CDC will conduct the cross match of the line list provided by RELI with positive blood cultures entered in NHSN. The list of return variables from cross-match analysis are listed in Table 3.  

Data transmission

-          Data will be transmitted via QualityNet

o   CDC sender: Preeti Ravindhran

o   RELI recipient: Khalil Abdul-Rahman

-          Data will be transmitted from CDC to RELI on or before July 13, 2018

Table 2. List of extracted variables from chart review for NHSN cross-matching

Columns (Variables)	Description	Format
CCN	Facility's CMS Certification Number (CCN) The six-digit CMS provider identifier, also referred to as provider number or billing number.	“6-character text”
Facility Name	Name of Facility being validated	Alphanumeric
Facility Street	Facility street address	Alphanumeric
Facility City	Facility city	Character
Facility State	2 Character U.S. Postal Service State Code	Character
Facility Zip	Facility zip code	Numeric
Abstraction Control Number	ACN assigned by the Validation Sample Selection system to cases selected or created for validation	Alphanumeric
Reporting Quarter	Reporting Quarter. Example “2017Q1”	Alphanumeric
MRN	Medical record number as recorded in the patient medical record	Alphanumeric
Patient Identifier	If applicable; other patient identifier recorded in the patient medical record (e.g., NHSN ptID)	Alphanumeric
Birthdate	Patient date of birth	MM-DD-YYYY Date format
First Name	Patient first name	Character
Last Name	Patient last name	Character
Sex	Male, Female or Unknown	Character M, F or U
PBC1	Did this patient have a positive blood culture 2017Q1 or 2017Q2?	Y, N
CultureDate1	[optional] Date of positive blood draw; If patient did not have a positive blood culture, report as missing ("PBC1" variable for that patient should equal "N")	MM-DD-YYYY Date format
PBCDate1	[required] Event date, as reported to NHSN per Protocol. This field should be populated with the date of positive blood draw (same value as “CultureDate”), or date of earliest Event if multiple Events occurred as part of the same patient problem (see “Multiple Dialysis Events” Page 6 of CY2017 Dialysis Event protocol); If patient did not have a positive blood culture, report as missing ("PBC1" variable for that patient should equal "N")	MM-DD-YYYY Date format
PBCReportable1	Validator’s determination from chart review whether the specific PBC was reportable	Y,N
PBC2*	Did this patient have a second positive blood culture 2017Q1 or 2017Q2?	Y, N
CultureDate2	[optional] Date of positive blood draw; If patient did not have a positive blood culture, report as missing ("PBC2" variable for that patient should equal "N")	MM-DD-YYYY Date format
PBCDate2	[required] Event date, as reported to NHSN per Protocol. This field should be populated with the date of positive blood draw (same value as “CultureDate”), or date of earliest event if multiple Events occurred as part of the same patient problem (see “Multiple Dialysis Events” Page 6 of CY2017 Dialysis Event protocol); If patient did not have a positive blood culture, report as missing ("PBC2" variable for that patient should equal "N")	MM-DD-YYYY Date format
PBCReportable2	Validator’s determination from chart review whether the second PBC was reportable	Y, N
*If multiple blood cultures are found during the medical chart review, then additional variables should be added on the template: PBC3, PBCDate3, PBCReportable3. PBC1 data should be populated based on the patient’s first PBC of the validation period, PBC2 populated with the patient’s second PBC, and so forth.

Table 3. Variables resulting from cross-match analysis. CDC will return the original data file, with the following fields added:

Columns (Variables)	Description	Format
PBCResult1**	CDC determination of whether validator’s positive blood culture matches a positive blood culture reported to NHSN, and was reported with the exact same Event date; 0 = no PBC match in NHSN, 1= PBC match in NHSN.	0, 1
PBCResultb1**	CDC determination of whether validator’s positive blood culture matches a positive blood culture in NHSN, but was reported with a different Event date; 0 = no PBC match in NHSN, 1= PBC match in NHSN.	0, 1
** Additional variables will be added for each PBC included in the RELI line list (e.g., PBCResult2, PBCResultb2)

2. Describe the procedures for the collection of information including:

- Statistical methodology for stratification and sample selection,

- Estimation procedure,

- Degree of accuracy needed for the purpose described in the justification,

- Unusual problems requiring specialized sampling procedures, and

- Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Please see response to question 1 for sample selection. As noted below in response to question 4, there are no unusual problems requiring specialized sampling procedures as our previous experience on past CMS NHSN validation efforts have shown near universal compliance by the hospitals with medical record requests. The period for data collection cycles is expected to be no more frequently than annually.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield 'reliable' data that can be generalized to the universe studied.

As part 2017 NHSN Study, the facilities were contacted via certified letter in March 2018 and were asked to participate in the validation effort. The letter provided instructions on the types of records to be submitted, methods to submit records to RELI, and identified patients selected for validation. To aid in maximizing facility response rates, the facilities that did not respond to the initial request for records were contacted via phone by RELI, and letters resent to specific contact to ensure response.

Facilities that did not respond to the request for records were subject to a 10-point reduction to their Total Performance Score (TPS). The response rate for the NHSN study in 2017 was high; of the thirty-five facilities selected for participation, thirty-one responded (89%). For future validations, we plan to follow the same records request methodology, follow-up, and ESRD community outreach approach we’ve used in the past since it has been effective in producing desired response rates.

Data Validation

The main objective of this analysis is to perform a single comparison of the NHSN system against NHSN “candidate event” data obtained from the facilities’ records, leading to an evaluation of the reliability (i.e., the data are reasonably complete and accurate) and validity (i.e., the data actually represent what is being measured) of NHSN data. Candidate events include positive blood cultures, intravenous antimicrobials, or vascular site infection (e.g. pus, redness, or increased swelling).

To ensure the reliability of the data collected by reviewers, we use two reviewers for each patient record. We systematically measure differences between reviewers for all patient records and provide ongoing training as needed to correct reviewer error tendencies. All discrepancies are reconciled by a third reviewer.

Implementing this element of the study design enabled us to focus on reviewer accuracy rather than reviewer agreement. We used a system named CROWNWeb Abstraction Processing System (CAPS) that presented to the third reviewer a split screen page review that displayed the first and second reviewer results. This page provided the third reviewer with the capability to identify any differences and make needed updates. Consequently, we always used third reviewer approved results in our analysis.

Additionally, reviewers made full use of the Consult feature of CAPS. Whenever either reviewer needed to reach out to a more experienced reviewer, the person could move the patient record to the Consult phase. There the two of them could resolve the issue and then move the record back to the point where regular review processing was interrupted.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

As noted above, the sample pool will consist of Medicare-certified dialysis facilities that are required to submit administrative and clinical data into CROWNWeb to meet Section 494.108(h) of the 2008 updated Conditions for Coverage for ESRD Dialysis Facilities. Although previous experience on past CMS NHSN validation efforts have shown near universal compliance with medical record requests, new procedures or methods may need to be undertaken to ensure that an expanded number of sampled facilities can be managed appropriately.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Khalil Abdul-Rahman, RELI Group, (410) 533-2384

Siva Bala, RELI Group, (440) 382-7415

9