BLS Sign-off On Statistical Methodology

.
u.s. DEPARTMENTOF LABOR
Bureau of Labor Statistics Concurrence of Surveys Employing Statistical Methods
TO: Daniel Gillman[gillman. danie/@b/s.gov]

DATE: 4252011
FAX:

FROM:

Bonnie Naradzay, ETA Compliance Officer for PRA

For Review by John Eltinge, Ph.D.
ACTION: Review the attached survey, indicate approval or disapproval and return to the
sponsoring agency or program. Comments shall not be made on the proposed report but in the
comments space below.
TITLE OF SURVEY OR EVALUATION
WIA God Standard Evaluation (Survey)
CLEARANCE

D Approved

D Disapproved

(See comments below)

rgJ Approved with caveats (See comments below)
COMMENTS
1. BLS does not have sufficient information to verify the accuracy of some statements in this
package. In addition, BLS has no responsibility for any aspects of the proposed work, beyond
review of some mathematical material provided in the package.
2. Per page 1 of Part B, the package for the second portion of the proposed data collection has
not been received by BLS to date and thus is not approved. Our understanding is that at some
point in the future the ETA plans to handle this second portion of the data collection in a separate
OMB PRA statistical package. Our understanding is that this prospective second OMB package
would be subject to standard reviews by BLS and the OMB Statistical Policy Office. Any
potential approval of the second package would depend on the content of that second package.
3. Per the first paragraph of Section 1 of Part A, no valid statistical inferences can be drawn from
these data to any population outside of those served by the program during the period covered by
the evaluation.
4. The proposed design is not robust against biases potentially arising from PSU-Ievel
nonresponse.
5. The use of "replacement" sites for nonresponding PSUs is not approved, and a final decision
on use of replacement sites is left to the discretion of the OMB Statistical Policy Office. Primary
problems with the proposed use of "replacement" sites include the following.
5.a. This method has been known for over fifty years to involve serious risks of bias due to
nonresponse. Some parts of the package (e.g., page 10 of part B) appear to confuse the important

distinction between a known random selection mechanism established a priori, and unknown
nonresponse mechanisms.
5.b. Due to the moderate number ofPSUs involved, under general mathematical conditions the
proposed use of "replacement" sites will be inefficient and will incur additional risks of bias,
relative to use of appropriately applied propensity and regression methods.
Comment s ace ex ands to accommodate all necessa comments
Disclaimer: BLS is approving only the statistical methodology as presented in the written documentation, and not
necessarily passing judgment on the questionnaire itself nor on the)le,cessity to conduct the collection of
information.
ATE
SIGNATURE OF BLS REVIEWER 7.../ ,.e~~ {J
W:II

'401

Mt/ll"4<:kcv~

p..(.f, ',~

..f«-acA.ot E:-1-4i"'J~ t',re--",..,'i

File Typeapplication/pdfFile Modified2011-06-23File Created2011-06-23