CMS-10178.Supporting Statement Part B

Supporting Statement – Part B

Collections of Information Employing Statistical Methods

1. To measure improper payments for PERM, 17 States from a total of 50 States plus the District of Columbia (i.e., ‘51’ States) were selected each year to create a three year rotation cycle. The States were rank-ordered by their past Federal fee-for-service (FFS) expenditures and grouped into the four strata of 17 States each for three PERM cycles (3x17=51). This distribution of States is shown in Table 1.

Table 1: State Strata Defined

Stratum	FY 2006/2009	FY 2007/2010	FY 2008/2011
1A	3	3	3
1B	3	3	2
2	6	5	6
3	5	6	6
Total	17	17	17

Claims are grouped into 10 strata by payment amount before sampling. The FFS annual sample size for each State in the FY 2008 and FY 2009 is 1,000 claims for FFS from each of the Medicaid and the CHIP programs, plus 40 Medicare Premium Payment claims only from the Medicaid FFS universe. The managed care annual sample size for each State is 520 claims from each of the Medicaid and the CHIP programs. Claims are grouped into the 10 strata with equivalent total payments in each stratum. The sampling in then conducted as shown in Table 2. The result is a much higher likelihood of sampling high-dollar payments than low-dollar payments for each State program and payment method.

Table 2: Randomly Claims Sampled per Fiscal Quarter from Each Stratum in FY 2008 and FY 2009

Program Universe	S01	S02	S03	S04	S05	S06	S07	S08	S09	S10	S11
Medicaid FFS	25	25	25	25	25	25	25	25	25	25	10
CHIP FFS	25	25	25	25	25	25	25	25	25	25
Medicaid Managed Care	13	13	13	13	13	13	13	13	13	13
CHIP Managed Care	13	13	13	13	13	13	13	13	13	13

The sampling follows a 3-stage, stratified, prioritized probability design. The stages are for beneficiary claims that are nested within payment type (i.e., FFS or managed care), which is nested within program (i.e., Medicaid or CHIP), which is nested within States. States and claims are both stratified to accommodate the wide ranging variance across States in their total Medicaid and CHIP payments. The sampling is prioritized to yield greater probabilities of sampling the higher dollar claims.

All sampled FFS claims receive a data processing review; sampled FFS claims that are not denied or Medicare crossover claims receive medical review. Sampled managed care claims receive only a data processing review. The final FY 2008 and FY 2009 PERM payment error rate reports will contain national Medicaid and CHIP rates that include FFS, managed care, and eligibility components, as prescribed by Public Law 107-300.

The anticipated response rate for all facets of PERM should approach 100% due to the regulatory requirement under Final Rule CMS 6062-F 42 CFR 431.970. Previous periods of performance of PERM have shown that most States comply at the 100% level for their programs.

2. The overarching procedures for the collection of information, including sample selection and stratification, are described in #1 above. ¹ FFS samples were determined to need to contain approximately 1,000 claims per State program, using an assumed error rate of up to a maximum of approximately 10% and ten strata used for sampling. The maximum error rate parameter was based on the FY 2006 PERM results, the early FY 2007 results, and the tendency for error rates to decrease over time due to the responsible agencies learning the operationalized program. Under these conditions, the federal precision requirements should be met, which include no more than an anticipated +/- 3% margin of error at a 95% confidence level for payment error rates at the State program level, and no more than an anticipated +/- 2.5% margin of error at a 90% confidence level for payment error rates at the national program level.

In more statistical depth, sample size determinations follow the approach outlined in Cochran (1977). The process starts with a variance estimation. The total variance for a staged sampling model from Cochran is as follows (of note, Cochran shows a two-stage model, whereas the variance for a three-stage sample nests Cochran’s equation within one more stage to account for payment method, FFS or managed care): ^{2, 3}

(1)

where:

N=total States in universe

n=total States sampled

f₁=proportion of States sampled

Y_i=projected dollars in error for a State

M_i=total units in State i

m_i=total units sampled in State i

f_2i=proportion of units sampled from State i

S_2i²=sample variance of errors within State i

The goal is to determine the expected value of the variance of the projected dollars in error.

Letting

(2) σ_YB² =

denote the variance in projected error between States, using the identity Y_i=R_iP_i, and assuming that all R_i are equal, then σ_YB²=R²* σ_PB², where σ_PB² is the variance in payments between States.

(3) Next, for each State-level estimate, the variance of the error rate is given by:

(4)

where P_i is the total payments for the State, and σ_EW² is the variance of the projected dollars in error within a State.

The goal is to determine the expected variation in projected dollars in error within a State, which depends on the State selected (the first stage).

Therefore:

(5)

which is attained only when assuming independence between the error rate and payments, and this appears to be a reasonable assumption.

Additionally, note that σ_Ri² is currently designed such that the precision met is .03, with 95 percent confidence.⁴ For generality, the desired State level precision denoted as d, then note that σ_Ri²=d²/z² where z is the standard normal of a designated level of confidence.

Combining this, equation (1) can be rewritten as:

(6)

All that remains is to find the variance of the error rate, which for the method used by the Office of Inspector General (OIG) (a “difference” estimator) would simply be:

(7)

Note: The sample size calculations in these derivations were for simple random sampling schemes. For stratified random sampling schemes, the same procedure is used, partitioned into the respective strata, and each individual result combined in a standard fashion.

The overall formula for the sample size needed to construct a rate with a specified precision is given by:

where

z_α/2 = the standard normal value with α/2 in one tail of the distribution.

d = desired precision

k = coefficient of variation, assumed constant across strata

π = probability a sampled unit is in error, assumed constant across strata

The formula above provides the overall sample size requirement for the study, and this sample size would be divided into strata. These particular calculations assume that the sample will be allocated based on the proportion of payments in each stratum. Note: The value of π is technically the probability that a sampled unit (e.g., line item) is in error, and is not the dollar weighted error rate. In practice; however, the statistical contractor would use the dollar valued error rate in lieu of this value.

Sampling with strata is done as follows:

1. Sorted the data first by paid amount

2. Calculated the total payments for universe

3. Defined strata: sorted claims in descending order, such that each stratum represents 10 percent of expenditures

4. Determined the skip factor for each stratum (denoted by k_i).

Let

denoted the universe number of claims for the i^th stratum in a State

5. Determined a random start value for each stratum (denoted by start_i), such that (i denotes the i^th strata)

6. Sampled every ^th item within the i^th stratum

The estimation procedure thus accounts for the nesting of claims within payment methods within program types within States. The error rate calculations utilize the Intra-class Correlation Coefficient to properly adjust for similarities within the nested structures in the data. In doing so, the PERM SC has chosen to use a Separate, Separate, Combined Estimator (SSC). This method represents a mixture of two methods: the combined ratio estimator and the separate ratio estimator. It is not documented in standard sampling textbooks, but the estimator and its standard error are straightforward to formulate and have been used for PERM in FY 2006 and FY 2007. The discussion is divided into the three steps for the estimator.

Stage 1: SRE for Combining Stratified Results

First, the PERM sample design has four State strata, determined by their expenditure amounts. The estimates and standard errors can be assumed to be produced for each State stratum.

The SR estimator is given by:

(10)

where

share of expenditures for State stratum i (sum across all strata equals 1)

error rate for stratum i, as determined by a ratio estimator, to be described later

i denotes the State stratum (i=1 to 4)

The variance for the SSC is given by:

(11)

Note the variance of the stratum specific error rate will be derived in later steps.

Stage 2: SR estimator for State Stratums

Within each State stratum, individual rates are estimated. The application of the separate ratio estimator occurs again when creating these State stratum rates. The State stratum rate will be the weighted combination of the State specific rates, with the weights being the relative shares of expenditures. Therefore,

(12)

where

share of expenditures for State j in State stratum i (sum of all strata equals 1)

error rate for State j in stratum i, as determined by a ratio estimator, to be described later

i denotes the State stratum (i=1 to 4)

j denotes the State (i=1 to n_i)

The variance follows the properties of a three stage sample design, where the selection of States is the first stage, the selection of program type is the second stage, and the selection of the sampling units (claims) within payment method is the third stage. The variance of this portion of the estimator is given by:

(13)

Let

(14)

such that

(15)

Then continuing from (13),

(16)

The estimated variance is given by

(17)

Stage 3: Combined estimator for State Stratums

Where can vary based on the estimator employed for estimating rates at the State level. For the combined ratio estimator, the State level error rates are estimated by:

(18)

where:

m_k are the number of claims sampled from strata k

M_k are the number of claims or line items in the universe from strata k

e_kl represents the error on the lth claim in the kth stratum

p_kl represents the payment on the lth claim in the kth stratum

Then estimated variance is given by:

(19)

The needed accuracy is provided by the IPIA and should be no more than an anticipated +/- 3% margin of error at a 95% confidence level for payment error rates at the State program level, and no more than an anticipated +/- 2.5% margin of error at a 90% confidence level for payment error rates at the national program level.

On occasion, the distribution of claims within a universe can lead to the dollar value of a single or a small number of claims comprising over 10% of the State’s universe of claims. Under that condition, the 10-strata structure is maintained and the claims that would have been sampled from the stratum, if it had contained enough claims, are evenly spread across the remaining strata.

On other occasions, States are late with some data for varied reasons. The PERM SC has implemented a roll-over policy to sample the claims along with those from later submissions, rather than to create a supplemental sample. The reason is that the statistical properties of the rolled-over sampling units are more closely aligned with the theoretically optimal properties than they are under the condition of a supplemental sample.

In order to meet the requirements of IPIA, all selected States must fully participate.

3. Most States have been quite responsive, so non-response is a minimal issue for PERM. The accuracy and the reliability for PERM are specified by federal regulations and supported by appropriate sample sizes. For these reasons, the information collected should be appropriate for its intended purposes. Reliable data are expected because the PERM SC compares the States’ data with their CMS 64 and CMS 21 submissions for Medicaid and CHIP, respectively. Further, States are subject to an OMB audit on their PERM submissions.

4. Not applicable.

5. Contact information

Livanta LLC

9175 Guilford Road, Suite 102

Columbia, MD 21046

Main Number: 240-568-9434

Program Director: Linda Daily, PMP x231

1 A stratified sample is simply a series of simple random samples, combined together.

2 Note: In practice, PERM will utilize a slightly different estimator that utilizes the error rate for each State, as opposed to the projected dollars in error for each State. However, the above formula could be used as well, and the resulting sample size calculation of States would yield similar results with a much simpler presentation herein.

3 The number on the equations is the original found in Cochran (1977).

4 In the FY 2007 error rate, the precision at the State level will be less than this because of reductions in sample sizes.

7