CMS-10166 REVISED-Supporting Statement Part B-May 2013 clean copy

Supporting Statement Part B
1. To measure improper payments for PERM, 17 States from a total of 50 States plus the
District of Columbia (i.e., ‘51’ States) were selected each year to create a three year
rotation cycle. The States were rank-ordered by their past Federal fee-for-service (FFS)
expenditures and grouped into the four strata of 17 States each for three PERM cycles
(3x17=51). This distribution of States is shown in Table 1.
Table 1: State Strata Defined
Stratum
1A
1B
2
3
Total

Cycle 1
3
3
6
5
17

Cycle 2
3
3
5
6
17

Cycle 3
3
2
6
6
17

Claims are grouped into strata by service type before sampling. The FFS annual sample
size for each State is state specific but the base sample size is 500.
The sample design for PERM is typically referred to as a two-phase sampling approach,
where the first stage refers to the sampling of states and the second stage refers to the
sampling of line items within a state.
All sampled FFS claims receive a data processing review; sampled FFS claims that are
not denied or Medicare crossover claims receive medical review. The final PERM
payment error rate reports will contain national Medicaid and CHIP rates that include
FFS, managed care, and eligibility components, as prescribed by Public Law 107-300.
The anticipated response rate for all facets of PERM should approach 100% due to the
regulatory requirement under Final Rule CMS 6062-F 42 CFR 431.970. Previous periods
of performance of PERM have shown that most States comply at the 100% level for their
programs.
2. The PERM program must estimate a national Medicaid error rate that covers the fifty
states and District of Columbia. According to IPIA requirements, the estimated national
error rate must be bound by a 90% confidence interval of 2.5 percentage points in either
direction of the estimate. To achieve this goal, the PERM program will sample 17 states,
without replacement, each year. Sampling a different 17 states each year ensures that all
states are sampled only once in a span of 3 years.
The variance of the estimator for this design is quite complex, and it is difficult to
numerically solve for the sample size. The equations and assumptions used are
described below.

1

From Cochran (1977), the variance for a total from a two stage sampling is1:

N2
 Yi  Y   N
(1) Var ( Yˆu ) 
( 1  f1 )
n
N 1
n
2

M i ( 1  f 2i )S 22i

mi

where:
N=total states in universe
n=total states sampled
f1=proportion of states sampled
Yi=projected dollars in error for a state
Mi=total units in state i
mi=total units sampled in state i
f2i=proportion of units sampled from state i
S2i2=sample variance of errors within state i
The goal is to determine the expected value of the variance of the projected dollars in
error.
Letting

 Yi  Y 
=

2

(2) σYB2

N 1

denote the variance in projected error between states, using the identity Yi=RiPi , and
assuming that all Ri are equal, then σYB2=R2* σPB2, where σPB2 is the variance in payments
between states.
(3) Next, for each state-level estimate, the variance of the error rate is given by:

ˆ ) 2 
(4) Var ( R
i
R
i

M i ( 1  f 2i )S 22i
mi
Pi2



2
 EW

Pi2

where Pi is the total payments for the state, and σEW2 is the variance of the projected
dollars in error within a state.

1

Note that in practice, PERM will utilize a slightly different estimator that utilizes the error rate
for each state, as opposed to the projected dollars in error for each state. However, the above
formula could be used as well, and the resulting sample size calculation of states would yield
similar results.

2

The goal is to determine the expected variation in projected dollars in error within a
state, which depends on the state selected (the first stage). Hence, we want:
2
(5) E(  iw
)  E(  R2 * Pi2 )   R2 E( Pi2 )   R2 (  P2   P2 )
i
i
i

which is attained only when assuming independence between the error rate and
payments, which appears to be a reasonable assumption.
Additionally, note that σRi2 is currently designed such that the precision met is .03, with
95% confidence. For generality, we will denote the desired state level precision as d, then
note that σRi2=d2/z2 where z is the standard normal of a designated level of confidence.
Combining this, equation (1) can be rewritten as:
(6) Var ( Yˆu ) 

N2
N  d 2  2
( 1  f 1 )R 2 2p  n
(  P   P2 )
n
n  z 2 

All that remains is to find the variance of the error rate, which for the Office of Inspector
General (OIG) “difference” estimator would simply be:

ˆ )
(7) V ( R
u

V ( Yˆu )
P2

Note that the sample size calculations in these derivations were for simple random
sampling schemes. For stratified random sampling schemes, the same procedure is used,
partitioned into the respective strata, and each individual result combined in a standard
fashion.2

Sampling with strata is done as follows:
1. Sorted the data first by paid amount
2. Calculated the total payments for universe
3. Defined strata: sorted claims in descending order, such that each stratum
represents 10 percent of expenditures
4. Determined the skip factor for each stratum (denoted by ki).
Let

Ni denoted the universe number of claims for the ith stratum in a State

ki 

Ni
ni

5. Determined a random start value for each stratum (denoted by starti), such that
1  starti  k i (i denotes the ith strata)

2

A stratified sample is simply a series of simple random samples, combined together.

3

6. Sampled every k i th item within the ith stratum
The estimation procedure thus accounts for the nesting of claims within payment methods
within program types within States. The error rate calculations utilize the Intra-class
Correlation Coefficient to properly adjust for similarities within the nested structures in
the data. In doing so, the PERM SC has chosen to use a Separate, Separate, Combined
Estimator (SSC). This method represents a mixture of two methods: the combined ratio
estimator and the separate ratio estimator. It is not documented in standard sampling
textbooks, but the estimator and its standard error are straightforward to formulate and
have been used for PERM in FY 2006 and FY 2007. The discussion is divided into the
three steps for the estimator.
Stage 1: SRE for Combining Stratified Results

First, the PERM sample design has four State strata, determined by their expenditure
amounts. The estimates and standard errors can be assumed to be produced for each State
stratum.
The SR estimator is given by:
a

(10) Rˆ SSC   S i Rˆ i
i 1

where

Si 

t up

i

t up

share of expenditures for State stratum i (sum across all strata equals 1)

Rˆ i  error rate for stratum i, as determined by a ratio estimator, to be described
later
i denotes the State stratum (i=1 to 4)
The variance for the SSC is given by:
(11) Var ( Rˆ SSC ) 

a

 S Var ( Rˆ )
i 1

2
i

i

Note the variance of the stratum specific error rate will be derived in later steps.
Stage 2: SR estimator for State Stratums

Within each State stratum, individual rates are estimated. The application of the separate
ratio estimator occurs again when creating these State stratum rates. The State stratum
rate will be the weighted combination of the State specific rates, with the weights being
the relative shares of expenditures. Therefore,

4

(12) Rˆ i 

ni

S
j 1

ij

Rˆ ij

where

S ij 

t upij
t upi

share of expenditures for State j in State stratum i (sum of all strata

equals 1)
Rˆ ij  error rate for State j in stratum i, as determined by a ratio estimator, to be
described later
i denotes the State stratum (i=1 to 4)
j denotes the State (i=1 to ni)
The variance follows the properties of a three stage sample design, where the selection of
States is the first stage, the selection of program type is the second stage, and the
selection of the sampling units (claims) within payment method is the third stage. The
variance of this portion of the estimator is given by:

 



  

(13)  R2ˆ  Var Rˆ i  Var E Rˆ i  i  E Var Rˆ i  i
i



Let
(14) Rˆ i 

ni

ni

Ni

j 1

j 1

j 1

 S ij Rˆ ij   S ij Rˆ ij   S ij/ Rˆ ij

such that

S i   i
0   i

(15) S ij/  

Then continuing from (13),

 

 ni

 Ni / 2 2
ˆ


(16) Var Ri  Var   S ij Rij   E   S ij  Rˆ
eij
 j 1

 j 1

 n S2 R 
ij

ij

n
N






ni

S 
j 1

2
ij

2
Rˆ eij

The estimated variance is given by

5

ni


S ij Rˆ ij

ni
 ˆ
j 1

 S ij Rij 
ni
j 1


n  
(17) Vˆar Rˆ i  ni 1  i 
ni  1
 Ni 

 

2







  n
N

ni

S
j 1

ˆ R2ˆ

2
ij

eij

Stage 3: Combined estimator for State Stratums

Where ˆ R2ˆ can vary based on the estimator employed for estimating rates at the State
eij

level. For the combined ratio estimator, the State level error rates are estimated by:
(18) Rˆ  f (tˆe , tˆp ) 

tˆe

tˆp

a

mk

k 1
a

l 1
mk

Wk  ekl
W  p
k

k 1

l 1

kl

where:
a

Mk
k 1 mk

tˆe  
a

Mk
k 1 mk

tˆp  

mk

a

mk

l 1

k 1

l 1

mk

a

mk

l 1

k 1

l 1

 ekl  Wk  ekl
 pkl  Wk  pkl

mk are the number of claims sampled from strata k
Mk are the number of claims or line items in the universe from strata k
ekl represents the error on the lth claim in the kth stratum
pkl represents the payment on the lth claim in the kth stratum
Then estimated variance is given by:
 nk
  (ekl  Rˆ p kl  (ek  Rˆ p k )) 2
a
a
1
1
(19) Vˆar ( Rˆ )  2 Wk2 nk Vˆar (ekl  Rˆ p kl )  2 Wk2 nk  l 1

nk  1
tˆp k 1
tˆp k 1










The needed accuracy is provided by the IPIA and should be no more than an anticipated
+/- 3% margin of error at a 95% confidence level for payment error rates at the State
program level, and no more than an anticipated +/- 2.5% margin of error at a 90%
confidence level for payment error rates at the national program level.

6

In order to meet the requirements of IPIA, all selected States must fully participate.
3. Most States have been quite responsive, so non-response is a minimal issue for PERM.
The accuracy and the reliability for PERM are specified by federal regulations and
supported by appropriate sample sizes. For these reasons, the information collected
should be appropriate for its intended purposes. Reliable data are expected because the
PERM SC compares the States’ data with their CMS 64 and CMS 21 submissions for
Medicaid and CHIP, respectively. Further, States are subject to an OIG audit on their
PERM submissions.
4. Not applicable.
5. Contact information
The Lewin Group
3130 Fairview Park Drive
Falls Church, VA

7