Attachment 2 - QCEW-CES RAS Final Report

August 21, 2009

Differences in Seasonality between the CES and QCEW Programs:
Results from the 2008 Response Analysis Survey

Sally Anderson, LAUS
Margaret Applebaum, CES
Michele Eickman, CES
Greg Erkens, SMS
Kristin Fairman, QCEW
Jeffrey Groen, ERPDS
Steve Kroll, CES
Chris Manning, CES
Polly Phipps, OSMR

The team thanks other important contributors to this project: Victoria Battista (CES), Brian
Dahlin (CES), Jennifer Harris (CES), Pat Jones (L-3 Communications), Justin McIllece (SMS),
Paresh Patel (L-3 Communications), Orest Stuparek (BAE), and the data-collection staff in
Niceville, Florida.

2

Contents
Executive Summary .......................................................................................................................................5
1. Introduction...............................................................................................................................................11
2. Background ...............................................................................................................................................11
3. Description of Seasonality RAS ...............................................................................................................17
4. Seasonality RAS Results and Analysis.....................................................................................................28
5. Summary ..................................................................................................................................................57
6. Recommendations....................................................................................................................................60
7. Figures and Tables ...................................................................................................................................65
Appendix A: RAS Questionnaire..................................................................................................................91
Appendix B: Regression Trees .....................................................................................................................119

3

4

Differences in Seasonality between the CES and QCEW Programs:
Results from the 2008 Response Analysis Survey

EXECUTIVE SUMMARY
Background
•

The QCEW and CES programs are linked in two fundamental ways. First, the QCEW
serves as the sample frame for the CES. Second, the CES annually benchmarks its
estimates of employment to universe employment counts derived mainly from the
QCEW.

•

At the aggregate level, QCEW and CES have different seasonal patterns, and this
difference is consistent over time from year to year.
o The seasonal differences between the two series are present in most industries.
o The differences between the two series seem to be getting smaller over time.

•

Both programs collect monthly employment data from an establishment using the same
reference period, so the monthly employment figures reported by an establishment to
each program should generally be identical. However, there are some differences
between the programs that could contribute to differences in monthly employment
figures.

•

These differences include: treatment of workers not covered by UI, timing of data
collection, methods used to compile and report the figures, the handling of non-response,
and procedures used by payroll-processing firms and software.

Response Analysis Survey (RAS)
•

The RAS focused on reasons behind the different seasonal patterns in the CES and
QCEW as well as reasons for establishment-level differences in the reported CES and
QCEW employment figures.

•

The RAS sample consisted of 3,002 actively reporting establishments of various sizes
and industries, most of which were identified as exhibiting different reporting patterns
between the CES and QCEW from January 2006 to March 2007.

5

•

The establishments in the RAS sample with reporting discrepancies fell into at least one
pre-defined error group. The error groups were developed to target specific types of
reporting differences that have been identified through past research and analysis. About
3% of the sample was allocated to establishments with no reporting differences.

•

The RAS questionnaire was divided into two sections with identical questions: one
section focused on the monthly CES report, the other on the Quarterly Contribution
Report (QCR), which is the source of QCEW data.

•

The questionnaire gathered information on the following topics: payroll, data sources,
reporting procedures, record keeping, reference period, whether checks were counted to
compute employment, the types of employees included or excluded in employment
counts, and the respondent’s opinion of the reason for employment differences between
CES and QCEW.

•

The RAS was conducted by phone from January to June 2008. The calls were made from
a call center in Niceville, Florida, using a CATI instrument specifically designed for this
survey.

•

Out of the sample of 3,002 establishments, 2,117 responded – for an overall response rate
of 71%. Of establishments that completed one or both parts of the questionnaire, 56%
completed only the CES section, 44% completed both sections, and less than 1%
completed only the QCEW section.

•

Item non-response was a problem for the QCR section. For 34% of the responding
establishments, an outside organization (such as a payroll provider firm -- PPF, or an
accountant) was responsible for preparing the QCR figures, and the CES respondent
wasn’t familiar enough with the report to complete the QCR portion of the questionnaire
or wasn’t willing to allow the interviewers to contact these outside organizations. This
resulted in a larger number of completed questions for the CES section and higher item
non-response for the QCR questions, which made comparisons across the two programs
difficult for some cases.

6

Findings
Reference Period
•

Establishments used the correct reference period (pay period that includes the 12th of
month) for the CES counts far more often than for the QCR.

•

Establishments using an incorrect reference period for the QCR have greater seasonal
differences between the CES and QCR employment counts (greater build up in the
QCEW).

•

Establishments using an incorrect reference period for the QCR are more likely to display
constant employment within or across quarters or a stair-step pattern for the QCEW.

•

Establishments using an incorrect reference period for the CES are more likely to display
constant employment across months for the CES.

Payroll
•

Establishments with more complex payroll situations have greater seasonal differences
between CES and QCR employment counts.

•

Having multiple payrolls is associated with differences in seasonal patterns between CES
and QCR. Among establishments with single payrolls, those that pay on a monthly basis
have smaller seasonal differences than those with weekly payrolls.

Report Preparation
•

Establishments that obtain their CES or QCR employment from a source other than
payroll tend to report different employment values to the CES and QCR.

•

When the CES and QCR are prepared by the same person, the reports are less likely to
have employment differences.

•

Preparation of the CES or QCR employment report by an outside organization is
associated with differences in employment.

•

Changes to software, reports, or other data sources used to compile the employment
figures are a reason for employment differences for a small percentage of respondents.

Check Counting
• A small percentage of respondents reported counting the checks issued rather than the
number of persons receiving pay, but check counting did not appear to be a major
contributor to employment differences.

7

Employee Type Inclusion/Exclusion
• Overall, respondents are reporting most employee types correctly. There are a couple of
employee types where there is a higher incidence of incorrect reporting: erroneously
including out-of-state employees and excluding corporate officials and executives.
• The incorrect reporting of employees is primarily associated with CES reporters.
Respondent-Provided Reasons for Differences
•

Including or excluding employee types and use of a reference period other than the pay
period that includes the 12th of the month are the most common reasons for the
discrepancy cited by respondents.

•

The incorrect inclusion or exclusion of employee types is largely related to the CES,
along with worksite differences, data source or timing differences, and the counting of
checks.

•

Reference-period problems are largely associated with the QCR; responses indicate it is
more likely for the QCR to be a monthly count of employees. In addition, automated
reporting issues with payroll software or a payroll processing company are one of the
reasons for differences primarily associated with the QCR.

•

Establishments that reported they were not using the correct reference period were more
likely to exhibit higher over-the-year growth in the QCEW.

•

Approximately 25% of all reasons for employment differences included a “don’t know”
answer; some of these respondents were not the respondent during the 2006-07 time
frame.

December-to-January Seasonal Differences: Tree Regression Analysis
• There are a number of groups that account for a large proportion of the December-toJanuary seasonal difference, measured by the influence that each group has on CES and
QCEW difference (see Appendix B for technical details).
• Six mutually exclusive groups were identified that account for large portions of the
seasonal error.
o Three groups in which the QCR is not filled out by the CES respondent account
for nearly 50% of the seasonal error. In particular, groups in which a PPF fills out
the QCR account for 31% of the seasonal error.

8

o One group includes establishments using multiple payrolls and/or pay periods,
accounting for 16% of the seasonal difference. Nearly all of the establishments in
this group say that they are reporting correctly to the CES.
o Another group includes a small subset of establishments with the same person
reporting for the CES and the QCR, and these respondents typically use the same
source for the CES and QCR. These establishments account for 42% of the
seasonal difference. Most of these respondents said that they use the correct
reference period for both programs. These results are counterintuitive, as one
would expect this group to be reporting correctly; we have not been able to
identify any specific problems with this group using the RAS data.
o A final group includes establishments using the incorrect reference period for the
QCEW, accounting for 10% of the seasonal difference.
Recommendations
•

Some reporting errors are likely to be corrected through respondent education and reeducation. This could take place within the normal framework of respondent contact in
the two programs. In particular, it may be helpful for the CES program to reference the
quarterly report during the initiation process. This would likely have the largest impact
in establishments where the same respondent completes both reports (41% of the RAS
respondents).

•

Additionally, we recommend developing procedures to identify, on a more routine basis,
establishments that are reporting differently to the two programs and contacting them
immediately after collection of the QCR to reconcile any differences.

•

Reporting errors can also be addressed by redesigning the forms used to collect CES and
QCR data, especially with regard to the reference period, multiple payrolls, and particular
types of employees. BLS should also recommend that state QCR forms be redesigned in
a manner that makes them more standard across states.

•

We recommend cognitive research to explore and better understand respondent
difficulties with the BLS definition of the reference period (pay period including the 12th
of the month).

9

•

We recommend conducting research on respondents who report their CES and QCEW
data via EDI. These respondents were excluded from the scope of the seasonality RAS
but nevertheless comprise 44% of the CES sample and therefore merit future
investigation regarding their role in differences between CES and QCEW.

•

Likewise, we recommend that BLS undertake a review of current EDI edit and screening
parameters and explore the possibilities of incorporating more formal comparisons of
CES and QCR data at the establishment level.

•

We recommend further investigation into the role of payroll processing firms (PPFs),
including follow-up with the PPFs that were interviewed in 2007 about their procedures
for reporting CES and QCR data.

•

We also recommend an analysis of the establishment characteristics of the two PPF
groups accounting for a large proportion of the December-to-January seasonal difference,
as identified by the tree regression analysis.

•

Research on PPFs should also be done using QCEW data for California, which maintains
a set of agent codes for payroll provider firms on their QCEW files.

•

We recommend that BLS require all states maintain an agent code (with consistent
coding) on their QCEW files. If this isn’t feasible because of resources, BLS should
consider expanding the practice to several more large states.

•

Further analysis of data from the seasonality RAS could prove fruitful in understanding
employment and seasonal differences. In particular, analysis using multivariate methods
should be supported, although the missing data for particular RAS variables poses
problems for such analyses.

•

The method of calculating the influence that each establishment has on CES and QCEW
differences provides possibilities for future CES-QCEW data quality research. Research
could be conducted to pinpoint respondents for data reconciliation who repeatedly have a
large absolute influence in a particular month. Also, establishments could be stratified by
influence in future RAS sample designs to place emphasis on establishments with large
seasonal differences.

10