Reliability Model for Test Preparation Study

Reliability Model for Test
Preparation Study

where Λ is a 20 × 20 diagonal matrix of subtest standard deviations and J is a 20 × 10 matrix of stacked
10 × 10 identity matrices I:
∙
¸
Here we describe the planned approach for assessI
J=
.
ing the impact of test preparation on test score preciI
sion. This approach estimates the test prep precision
impact from the covariance matrix of observed sub- In this model, the diagonal elements of Φ are contest scores. This covariance matrix among 20 sub- strained to 1.
tests is constructed from two administrations of the
Maximum Likelihood (ML) estimates of the free
ASVAB to a group of military applicants: 10 subtests parameters specified in (1) can be obtained using
of the first battery administered before test prepara- the COSAN model implemented by PROC CALIS in
tion and 10 subtests of an alternate form adminis- SAS. The null hypothesis (that test preparation has
tered after test preparation. Test score data will be no effect on measurement precision) can be tested
obtained from examinees who choose to take the test using a likelihood ratio statistic
twice and happen (as a matter of their own choosh
i
ing) to engage in test preparation inbetween the two
LR = −2 log L(ˆθr ) − log L(ˆθu ) ,
administrations.
Let Φ represent the 10 × 10 correlation matrix where ˆθ are the ML estimators of free parameters
r
of subtest (GS, AR, ..., AO) scores that occur be- for the restrictive, nested model and ˆθ are the esu
fore (and therefore are unaffected by) test prepara- timators for the model without restrictions. Here,
tion. Then the elements of Φ (denoted by φij , for the restrictions are imposed by setting R = I (i.e.,
i = 1, ..., 10; j = 1, ..., 10) represent the correlations setting R to an identity matrix) which constrains
among the subtests under pre test-preparation con- ρ∗ = · · · ρ∗ = 1, consistent with the outcome exGS
AO
ditions. Further, we construct a 20 × 20 diagonal pected
if test prep has no effect on measurement ermatrix
rors. The LR statistic has a limiting χ2 distribution
⎡
⎤
with df = 10.
1
⎢
⎥
..
⎢
⎥
.
0
⎢
⎥
⎢
⎥
1
⎢
⎥ ,
R=⎢
∗
⎥
ρGS
⎢
⎥
⎢
⎥
..
⎣
⎦
.
0
∗
ρAO
where ρ∗GS , ..., ρ∗AO denote the reliability indices associated with the 10 subtests of the alternate form
administered after test preparation. Then using the
classical test theory formula for modeling the effects
of measurement error1 (i.e., the correction for attenuation formula), the observed covariance matrix Σ
among all 20 subtests can be expressed as
Σ = ΛR1/2 JΦJ 0 R1/2 Λ ,

(1)

1 In

the present case, we equate the more general concept
of measurement error to the special case of random errors
introducted by test preparation.

1