CMS-10238.OASIS Supp State- Part B_07_16_07

Supporting Statement For Paperwork Reduction Act Submission

“Home Health Quality Measures and Data Analysis”

Part B: Collections of Information Employing Statistical Methods

1 . Description of the potential respondent universe and sam­pling/other respondent selection methods to be used.

The data will be collected from home health agencies in three states: Colorado, Massachusetts, and Ohio. According to Home Health Compare, there are 139 agencies in Colorado, 119 in Massachusetts, and 420 in Ohio. Based on the average number of patients served per home health agency^1, we estimate that the total patient population in these three states is 231,074.

As described elsewhere, the primary objective of this project is to assess the burden of proposed changes to the OASIS assessment and to conduct reliability testing and process item verification at up to 11 home health agencies – a maximum of 5 agencies in Ohio and up to 3 each in Colorado and Massachusetts. We anticipate that 95-100 percent of the agencies we identify to participate in the study will consent to participate. This expected response rate is based on our past experience on similar types of projects and derives from established relationships with agencies in the 3 states and our concerted approach to recruiting volunteer agencies, explaining the nature of data collection thoroughly at the outset of work to each potential study site. Thus, agencies will be selected based on their motivation and willingness to participate. Volunteer agencies will be compensated $1000 for their participation in the study

We anticipate that 80 percent of the patients to whom agencies initially offer study participation will consent and take part in the study. Again, this expected response rate is based on our past experience on similar types of data collection efforts and relies on the agency’s knowledge of patient status and ability to explain the study to patients. If a patient declines study participation, the agency will identify and request consent from another appropriate respondent until the sample has been met.

2. Procedures for the collection of information

a. Statistical Methodology for Sample Selection

The home health agencies included in burden, reliability, and process item verification tests will be selected from Medicare-certified home health agencies in Colorado, Massachusetts, and Ohio. We will invite interested agencies to participate in the study and will attempt to obtain participation from agencies that are diverse in their geographic location, urban/rural status, ownership structure and patient population.

Patient respondents for the burden testing will be selected randomly from each agency to reflect the home health population. A total of 180 assessments will be performed, 60 Start of Care, 60 Recertification, 20 Transfer and 40 Discharge. Respondents for the Start of Care assessments will be selected from all patients who are scheduled to have one of the assessments performed for the agency’s COP mandated OASIS collection. When possible, Recertification, Transfer and Discharge assessments will be performed on patients who participated in the Start of Care assessment.

We plan to conduct 320 paired assessments on 160 patients for the inter-rater reliability testing Respondents will be selected purposively to ensure that the clinical conditions and time points that trigger the items are included. Diagnostic categories to be included are heart failure, diabetes and pressure ulcers. Process item validity testing will be conducted on OASIS data collected for the assessments completed as part of the burden and reliability testing. Patients will be selected to include clinical conditions and time points that will maximize the number of process items to be validated.

b. Estimation Procedure:

For the burden testing, the estimation procedures to be used will consist of standard approaches to estimating measures of central tendency, primarily the mean number of minutes required to complete the assessment and the standard deviation of the mean. This will be done for each of the assessment types, Start of Care, Recertification, Transfer and Discharge.

We will test the reliability of the new OASIS items using a formal reliability sample that consists of 160 patients from sample agencies. Because some of the new items are collected only at discharge/transfer and others are collected only for patients who have certain clinical conditions, we will design the sample to include patients and assessment points meeting those criteria. To measure reliability, we will use standard statistics such as percent agreement and the kappa statistic (or weighted Kappa for items with more than two response categories), measuring the inter-rater reliability of data collected from two agency HHA clinicians or trained research nurses. A weighted Kappa is appropriate for OASIS items with multiple ordered response categories (e.g., ADL items); an unweighted Kappa is appropriate for the process items and OASIS items that only measure presence or absence of an attribute.

For the process measure testing, we will measure the extent to which information from the OASIS assessment is consistent with information in the patient’s medical records. We will use a measure of percent agreement and the kappa statistic to measure the extent of agreement between information from the two sources. We anticipate that 80 assessments will be included in the process measure testing. We will also produce a descriptive analysis of the frequency of responses selected for each process measure and whether information related to patient outcomes for the process items could be located in the full patient OASIS.

c. Degree of Accuracy Needed:

Burden estimates: For the burden estimates, our goal is to estimate the mean time required to complete assessments within a confidence interval of approximately +/- 5 minutes. The proposed sample size for each type of assessment and the resulting confidence interval are shown in Table 1 below. We propose a sample of 180 assessments, consisting of 60 Start of Care, 60 Recertification, 40 Discharge, and 20 Transfer assessments. Note that limited data on the distribution of current OASIS completion times is available. The standard deviation estimates in the table are based on a midpoint of results from other studies and expert judgment from clinicians and researchers familiar with the OASIS instrument and proposed item revisions.

Table 1 OASIS Burden Estimates: Required Sample Size to Estimate OASIS Burden With A +/- 10 Minute Confidence Interval Mean and Standard Deviation from OASIS Cost and Benefits Survey
		OASIS Burden		Estimated Confidence Interval
Type of Assessment	Sample Size	Mean	Std. Deviation	Std. Error of Estimate	95% Lower C.I.	95% Upper C.I.
Start of Care	60	150	20	2.58	144.86	155.14
Transfer	20	30	10	2.24	25.4	34.6
Recertification	60	100	20	2.58	94.9	105.1
Discharge	40	75	15	2.37	70.2	79.8
Note: Mean and standard deviation figures for start of care, recertification, and discharge assessments are based on the OASIS Cost and Benefits Survey and represent facility-level estimates of reported OASIS burden. Time estimates for transfer assessments are based on the NAHC Study of Time Required for OASIS Activities, with the standard deviation estimated

Reliability Testing: The reliability testing will include 160 assessments, but some items may not be available for all patients in the study, due to skip patterns or certain items not being included on certain types of assessments. The precision of our estimates depends on the sample size available for testing each item and the statistical distribution of the item (number of categories, distribution across categories). For items that are available for all 160 patients in the reliability study, we anticipate being able to estimate Kappa to within +/- 0.11 for an item with a Kappa of 0.51 and to within 0.09 for an item with a Kappa of 0.71 (Table 2). (Note that this is a weighted Kappa for a hypothetical item with four response categories with an approximately uniform distribution.) For an item that is available for only half the sample (n=80), the estimate is considerably less precise, with a confidence interval of +/- 0.16 for an item with a Kappa of 0.51 and +/- 0.12 for an item with a Kappa of 0.71.

Table 2 OASIS Reliability Testing: Hypothetical OASIS Item With Four Response Categories
Sample Size	Number of Response Categories	Estimated Weighted Kappa	Asymptotic Standard Error	95% Lower C.I.	95% Upper C.I.
80	4	0.51	0.083	0.343	0.670
80	4	0.71	0.063	0.588	0.834
160	4	0.51	0.059	0.391	0.622
160	4	0.71	0.044	0.624	0.800
Note: Confidence intervals are for a weighted kappa item with 4 response categories.

Process Item Verification: We anticipate that 80 assessments will be included in the process item verification study. Given this sample size, the estimated margin of error for an item with an agreement rate of 50 percent is +/- 10.93 (with a 95 percent confidence interval); the margin of error decreases to +/- 9.5 percent for an item with an agreement rate of 75 percent and +/- 6.6 percent for an item with an agreement rate of 90 percent. This is sufficient statistical precision on which to base conclusions about the reliability of the process items.

d. Unusual Problems Requiring Specialized Sampling Procedures:

No specialized sampling procedures are required for this project.

e. Use of Periodic Data Collection Cycles:

This is a one-time study with no periodic data collection cycles.

3. Methods to maximize response rates and to deal with issues of non-response.

The expected initial response rate for the data collection activities is 80 percent. The contractors have collected data of this nature in health care settings on past research projects, and their approach has been refined over a period of approximately ten years. Basically, it entails explaining to selected respondents the nature of the data collection process and its associated burdens. If a patient declines to participate after a thorough discussion of what is entailed in terms of data collection, another patient is then selected until the necessary sample size is met.

In addition to the above approaches to maximizing response rates and minimizing burden, training sessions will be held for agency staff involved in the data collection. These sessions, to be conducted by senior nurse researchers from Case Western Reserve University (CWRU), Abt Associates and University of Colorado Health Sciences Center (UCHSC) will be targeted toward training clinicians to fully understand the revised OASIS instrument and guidance on assessment methods for collecting the data.

Regular telephone consultation will occur to provide technical assistance to data collectors at the various agencies. These conversations will be initiated by project staff members if agency staff has not contacted them. Research nurses will also visit individual agencies to provide additional support. Contractor staff take very seriously the need to establish a positive rapport with all those influenced by data collection procedures, attempting to familiarize them with the project, its purpose, and to make them feel a part of the project team. The 95-100 percent patient response rate that the contractors have experienced using such techniques in the past is therefore projected to be a realistic estimate of the response rate for this project.

Non-response analysis

Non-response is a potential issue with the burden survey, if we find that there is a relationship between non-response and OASIS completion time (e.g., if sicker patients, for whom OASIS assessment may take more time, are less likely to consent to be in the study). We will first analyze whether there is a relationship between OASIS completion time and patient conditions. If we find no relationship, then non-response is irrelevant for the burden estimates. If there is, we will then analyze whether there was non-response bias by comparing the proportion of patients with certain types of conditions in the burden survey compared to all home health patients (i.e., for items that are available on both the current and new assessment). If necessary, we may need to weight certain assessments (those for under-represented groups) more heavily in the burden calculations.

There is a non-random (purposeful) sample for the reliability and process item analyses, so non-response is not a concern, as we will continue to select patients with the conditions of interest until the desired sample size is reached.

4. Tests of procedures and/or methods to be undertaken

Assessing burden of revised OASIS instrument

For the burden estimate, the revised OASIS instrument will be conducted by home health clinicians at selected volunteer HHAs on a total of 180 home health patients. This data collection will be done within the time period normally scheduled for the agency’s COP mandated OASIS collection. Clinicians from volunteer agencies will complete a paper version of the revised OASIS, and will record in-home time spent conducting the assessment as well as separately recording additional time spent coding or collecting additional information outside the home.

Reliability testing of new/revised OASIS items and process items

Inter-rater reliability testing will be based on a formal reliability sample consisting of 160 patients in selected volunteer agencies. Some items will be collected only at discharge/transfer and some only of patients with certain clinical conditions, so the sample will be designed to include patients and assessment points meeting those criteria. The inter-rater reliability test will require two agency HHA clinicians or trained research nurses to collect the same data on each patient either at the time of start of care, follow-up or discharge. Each person will independently conduct the “mini-assessment”, consisting of all new and revised OASIS items relevant to that patient at that time point. The second clinician will be completed a mini-assessment on the same patient within one day of the first. Pearson correlation coefficients and Cohen's kappa statistics will be calculated and examined for each item in order to assess item reliability and to provide information for further refinement of item wording and structure.

Process item verification and descriptive analysis

To verify the process items in the revised OASIS, a research clinician will review patient charts for 80 of the assessments completed as part of the burden and reliability testing, examining the chart for evidence that may support or contradict the process data reported. The process item data collected for each selected assessment will be encoded and the full OASIS assessment data will be obtained for the corresponding patient care episodes. We will then examine the relationship between process items and relevant information in the OASIS, such as level of pain experienced during the home health stay, development of new pressure ulcers, wound healing status and adverse events leading to emergent care or hospitalization. We will produce a descriptive analysis of our findings, including the frequency of responses selected for each process item and whether information that could be used to calculate patient outcomes for the process items could be located in the OASIS.

5. Individuals responsible for statistical design, data collection, and/or data analysis

Data will be collected and analyzed as part of Contract Number HHSM 500-2005-00018I T.O.#2, “Home Health Quality Measures and Data Analysis”. The following table lists the name and contact information for individuals responsible for the design, collection and analysis of the data.

Name, affiliation	Area of responsibility	Contact information
Doug Brown, CMS, OCSQ	CMS Project Officer for the contract under which this study is being conducted	[email protected] 410-786-0028
Henry Goldberg, Abt Associates	Project Director - overall project design and implementation	[email protected] 617-349-2482
Dr. David Hittle, UCHSC	OASIS instrument refinement	[email protected] 303- 724-2430
Dr. Alan White, Abt Associates	Statistical aspects of design and analysis	[email protected] 919- 294-7719
Dr. Elizabeth Madigan, Case Western Reserve University	Collection of data for burden assessment, inter-rater reliability and verification/validity of process items	[email protected] 216-368-8532

1 According to data from CMS, there were 2,835,600 home health patients served in 2004. According to Home Health Compare, there are 8,320 home health agencies in the United States. Thus, the average number of home health patients served per agency is 340.

Supporting Statement For Paperwork Reduction Act Submission – Part B Page 6

“Home Health Quality Measures and Data Analysis” 07/12/2007