Pretest memo

APPENDIX B
PRETEST MEMO

MEMORANDUM

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone (609) 799-3535
Fax (609) 799-0005
www.mathematica-mpr.com

TO:

Stefanie Schmidt

FROM:

Julita Milliner-Waddell

SUBJECT:

Pretest Summary—National Longitudinal Study of
Unemployment Insurance Recipients (NLS-UI)

DATE: 10/1/2013
NLS-UI - 25

A. Introduction

Mathematica conducted a comprehensive pretest of the first interview questionnaire for the
National Longitudinal Study of Unemployment Insurance Recipients (NLS-UI) in preparation
for data collection. The goals of the pretest were to confirm respondents’ ability to provide the
requested information, test question clarity and flow, check entrance conditions and skip
instructions for each question, and estimate respondent burden.
To optimize the value of the nine pretests allowed by the Office of Management and Budget
(OMB), we conducted pretest interviews in two rounds. In the first round (August 21 through
August 27), four pretest interviews were conducted. All pretests were reviewed for timing,
clarity, and accuracy and were recorded to facilitate review and revision. A member of the
project team also monitored pretest interviews in real time to assist with administration.
The first four pretest interviews revealed that the questionnaire items are generally clear and
would yield valid responses. The general flow and sequencing of questions worked well.
However, administration time in the first pretest round was approximately 10 minutes longer
than the budgeted length of 25 minutes, requiring cuts to questionnaire items to balance coverage
and timing goals. We made questionnaire improvements, including question deletions, before
proceeding with additional pretest interviews.
Four additional interviews were conducted in the second round of pretesting (September 16
through September 20) using the revised version of the questionnaire that incorporated changes
and lessons learned in the first round. The second round of pretesting confirmed respondents’
ability to provide responses and the clarity of items. Administration time for interviews
conducted during the second round of pretesting aligned better with our target than first-round
administration times. We discuss interview timing in detail next.
B. Pretest Sample

To enhance the utility of the pretest, we sought to test the survey questionnaire with recent
UI claimants who would be similar to respondents to the NLS-UI first interview. To identify and
contact recent UI claimants without the benefit of OMB clearance, an internal email sent to staff
at all Mathematica office locations asked for referrals of friends and associates who recently filed
An Affirmative Action/Equal Opportunity Employer

MEMO TO: Stefanie Schmidt
FROM:
Julita Milliner-Waddell
DATE:
10/1/2013
PAGE:
2
for and began receiving UI benefits within the past two months. Trained telephone interviewers
recruited 12 pretest candidates. Eight of the 12 candidates completed interviews.
Table 1 provides the UI filing, employment, and household characteristics of the eight
pretest respondents. As shown, six of the pretest respondents had filed for UI compensation
benefits more than eight weeks before being interviewed—a situation we expect to differ
somewhat from the actual survey sample. (We expect to conduct a first interview with sample
members two to three months after the initial claim is filed.) Six of the pretest respondents had
held one job since filing for UI and were working at the time of pretest. Two respondents were
unemployed and had not held any jobs since filing for UI benefits. Seven respondents reported
household sizes of two people or more. Pretest respondents were from six states—Maine (1),
Maryland (1), Massachusetts (2), Michigan (1) New Jersey (2), and Illinois (1).
Table 1. Characteristics of Pretest Respondents and Survey Length
Pretest #

Filing
State

UI Claim
Month/Year

Pretest
Month/Year

Employment
Status

Household
Size

Survey Length
(minutes)

Currently
working
Unemployed

1

36

3

26

Currently
working
Currently
working

3

37

2

42

First Round Pretest Interviews
1

NJ

July 2013

August 2013

2

MD

July 2013

August 2013

3

ME

March 2013

August 2013

4

MI

June 2013

August 2013

Average Length of First Round Pretest Interviews

35.25

Second Round Pretest Interviews
5

MA

April 2013

6

IL

June 2013

7

MA

July 2013

8

NJ

June 2013

September
2013
September
2013
September
2013
September
2013

Average Length Of Second Round Pretest Interviews

Currently
working
Currently
working
Currently
working
Unemployed

3

33

2

32

2

32

2

27
31.00

C. Pretest Administration

Pretests for the NLS-UI were restricted to the first interview questionnaire. (Questionnaires
for the second and third rounds of data collection will closely mirror the first interview
questionnaire, with many questions skipped because the information has already been gathered

MEMO TO: Stefanie Schmidt
FROM:
Julita Milliner-Waddell
DATE:
10/1/2013
PAGE:
3
and a few new questions added.) We trained three interviewers to administer the pretest
questionnaire in a two-hour session that included a summary of the project, a detailed review of
the questionnaire, and role-play interviews. Role-plays used scenarios that were developed to
ensure that different paths through the questionnaire were reviewed.
To use resources most efficiently, we will perform computer programming of the
questionnaire for web and computer-assisted telephone interviewing (CATI) using the
questionnaire version that benefitted from pretesting. Therefore, interviewers used hard-copy
versions of the questionnaire to administer the pretest. To facilitate the administration of the
survey on hard copy, we developed a cheat sheet on which to record critical and repetitive item
fills such as the UI claim date, marital status, job search and current employment status for use at
the interviewers’ discretion. Nevertheless, navigating skip instructions using paper and pencil
requires more time than both the web and CATI versions.
Table 1 shows the length of each interview, which is influenced in part by the employment
history and household composition of respondents. In the first pretest round, administration time
ranged from 26 minutes for a respondent who had not worked since becoming unemployed to 42
minutes for a respondent who had multiple job offers and held one job since becoming
unemployed. The average administration time for the first four interviews was 35 minutes.
Table 2 lists the topics covered in the first interview, by questionnaire section. As shown, the
sections on the pre-UI claim job and financial well-being took the longest to administer.
Table 2. Average Length of Interview, by Questionnaire Section and Pretest Round
Average Length (minutes)
Section

Topic

Round 1

Round 2

Average

A

Case Management

1.25

1.50

1.38

B

Demographic and Household Characteristics

1.75

2.00

1.88

C

Pre-UI Employment and Unemployment

6.00

5.00

5.50

D

Job Search and Offers

4.75

3.50

4.13

E

Reemployment Expectations

0.50

0.50

0.50

F

Reemployment

3.75

3.00

3.38

G

Financial Well-Being

9.75

9.25

9.50

H

Customer Satisfaction

2.75

1.75

2.25

I

Closing and Contact Information

4.75

4.50

4.63

35.25

31.00

33.13

Total

As noted earlier, questionnaire items were deleted between the two rounds of pretesting to
lessen administration time. The average administration time for the final four pretest interviews
after deleting these items was 31 minutes, with a range of 27 minutes for a respondent who was

MEMO TO: Stefanie Schmidt
FROM:
Julita Milliner-Waddell
DATE:
10/1/2013
PAGE:
4
unemployed at the time of the interview to 33 minutes for a currently employed respondent with
a three-member household. We anticipate that the interview length will decrease by 5 to 8
minutes with the efficiency of web and CATI administration. (For the WIA Gold Standard
Evaluation, hard-copy pretesting averaged 40 minutes; actual interviews conducted using CATI
average 32 minutes.) This suggests an average administration time of 23 to 26 minutes,
achieving our target length of 25 minutes. Table 3 shows questionnaire items deleted between
the two rounds of pretesting and the rationale for the deletion.
Table 3. Questionnaire Items Deleted Between Pretest Rounds
Topic

Question

Pre-UI
Employment and
Unemployment

Previous UI
experience

Job Search and
Offers

Job offers requiring
change in
occupation

Job Search and
Offers

Change in
occupation of new
jobs held
Expectations about
the need to change
occupation
Characteristics of
jobs held between
pre-UI and current
job
Specific types of
investment
accounts at job
separation
Amounts received
from income
support programs

Reemployment
Expectations
Reemployment

Financial WellBeing
Financial WellBeing

Financial WellBeing
Customer
Satisfaction

Application for
income support
benefits
Satisfaction with
the promptness
with which
questions were
answered

Rationale
This item provided insight on repeat filers who might be more
familiar with the UI filing system. We might be able to address
this issue by looking at whether the respondent expected to be
recalled.
This item provides insight about occupational mismatches
between the skills of unemployed workers and the occupations
of job openings; it was deleted due to data quality concerns—
an occupational change might be difficult for respondents to
identify.
Occupational coding of other survey items will provide this
information.
This item was deleted due to data quality concerns—an
occupational change might be difficult for respondents to
identify.
Specific details on type of occupation, industry, and changes in
occupation and benefits will be asked only about jobs held at
the point of the interview. Start and stop dates, hours worked,
and pay rates will still be asked for all jobs.
Separate categories were combined into a single investment
account category, using these specific types of accounts as
examples.
Collecting specific dollar amounts for each type of income
support adds a significant amount of time. We retained
questions on receipt of each type of income support. Whether
income from the various programs increased due to job loss
and lower household earnings should be addressed to some
degree in the questions about total household income.
The third interview will ask about applications for benefits since
job separation if the respondent reported in all prior interviews
that he or she had not received a specific benefit type.
This question is very similar to another question about the
efficiency with which staff members did their jobs.

MEMO TO: Stefanie Schmidt
FROM:
Julita Milliner-Waddell
DATE:
10/1/2013
PAGE:
5
D. Summary

The two rounds of pretesting conducted for the NLS-UI were valuable in finalizing our
questionnaire development of all three study questionnaires. The pretest helped to identify areas
of the questionnaire that had to be shortened, assisted staff in identifying skip errors and
clarifying which questions would be asked of specific subgroups, and highlighted topics that
might require extra emphasis during interviewer training.
Pretest respondents did not report any major issues comprehending the questions. At the end
of each interview, respondents answered a series of questions to gauge their overall reaction to
the survey experience and to specific questions. Debriefing questions solicited opinions on
survey length and asked about concepts discussed in the survey, such as the definition of
household, what they considered when answering the question about employment outlook, the
ease or difficulty of answering questions on financial well-being, and the likelihood they would
participate in this study if they received a letter from the U.S. Department of Labor. All
respondents reported that the survey length was reasonable and noted that they would be very or
somewhat likely to participate in a similar study.

cc: Karen Needels, Alicia Leonard, Wendy Costa