March 22, 2017
NOTE TO THE REVIEWER OF: |
OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
|
FROM: |
Jennifer Edgar Research Psychologist Office of Survey Methods Research
|
SUBJECT: |
Submission of Materials for the Pretesting of CPS UI Non-Filer Supplement |
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.
The total estimated respondent burden hours for this study is 78 hours.
If there are any questions regarding this project, please contact Jennifer Edgar at
202-691-7528.
Introduction and Purpose
The Department of Labor has contracted the Bureau of Labor Statistics (BLS) to update the supplement for Unemployment Insurance (UI) Non-filers. This supplement focuses on unemployed respondents who may have chosen not to apply for UI benefits, with the goal of understanding their experiences and motivations.
The purpose of this testing is to ensure that the questions in the supplement work as intended, that respondents can understand and answer them both for themselves and for other members in their household.
2. Research Design
To accomplish this goal, traditional cognitive interviews and an online survey will be conducted.
Individual Cognitive Interviews. Cognitive interviews provide an in-depth understanding of the participant’s thought processes and reactions to the questions. These interviews will be done one-on-one in the Office of Survey Methods Research (OSMR) laboratory at the BLS. Interviews will be conducted by staff who are experienced in conducting these types of interviews. Participants will be asked select survey questions from the Current Population Survey (CPS), to provide context, and then asked the proposed UI Non-filer supplement questions. After the survey questions, participants will be asked debriefing questions to better understand their responses and reactions to the questions, both when reporting for themselves and when reporting for other household members. The survey questions and full cognitive interview testing protocol is included in Appendix A.
Online survey. As unemployed participants who have not applied for UI benefits is likely to be a difficult group to recruit, we will also conduct an online survey (Appendix B) to collect information from a larger, and more targeted sample not possible in the laboratory. The online survey will use participants recruited from Amazon.com’s Mechanical Turk.
The findings from both collections will be evaluated qualitatively, and used to arrive at conclusions about the effectiveness of the proposed wording. As this is pretesting, we expect that modifications to the question wordings or response options may be made during the course of the study based on initial results. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the interview questions. Modifications are likely to range from slight changes to question wording to the order of the questions.
3. Participants
Cognitive Interviews. Up to twenty participants will be recruited for cognitive interviews using OSMR’s existing participant database, as well as targeted advertisements in newspapers and online/local bulletin boards (see Appendix C). Participants will be screened to result in a sample diverse in terms of race, age, gender, education, and employment situations. We expect to have to screen up to 40 participants to find 20 eligible for the study. The Recruitment Script is contained in Appendix D.
Online Survey. Up to 150 Amazon Mechanical Turk participants will be recruited. This sample size was determined by similar online probing studies. This sample size also takes into account break-offs, incomplete data, and participants who do not follow the task instructions. Recruiting of participants will be handled by Amazon Mechanical Turk, using screening requirements specified by BLS (currently unemployed, lost job through no fault of their own, would like to work, have held a job in the past 12 months). Once participants are recruited into the study, they will be given a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers.
In addition, we will pretest the survey instrument via TryMyUI.com with 5 participants. TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. These pilot participants will be asked to think aloud while completing a subset of the survey questions. This pretesting will ensure that participants understand how to complete the roster for their household, that the task is clear, the questions are worded clearly, and the instrument works as intended. The pilot tests will be conducted iteratively so that any modifications can be tested with pilot participants before launching the full study.
4. Burden Hours
The maximum total burden hours for this study will be 78 hours.
|
# of Participants Screened |
Minutes per participant for Screening |
Total Screening Burden |
Maximum number of Participants |
Minutes per participant for data collection |
Total Collection Burden Minutes |
Total Burden Minutes (Screening + Collection) |
Cognitive Interviews |
40 |
10 |
400 |
20 |
60 |
1200 |
1600 |
Mechanical Turk |
150 |
0 |
0 |
150 |
20 |
3000 |
3000 |
TryMyUI |
5 |
0 |
0 |
5 |
20 |
100 |
100 |
|
|
|
|
|
Total |
4,700 Minutes |
|
|
|
|
|
|
Total |
78 hours |
5. Payment to Participants
Up to $1,237 will be spent on payments to participants.
Cognitive Testing. The 20 participants will each be paid $40 for their participation in testing, and there is a $3.60 administrative fee for each payment, for a total of $872.
Online Survey. Mechanical Turk participants will receive $1.50 for participating in this survey, a typical rate for similar tasks. A total of 150 participants will be screened. The costs for the main survey total $315.00 (1.50 x 150, plus a $90 commission fee to the company). The pretesting pilot participants recruited from TryMyUI will receive the standard TryMyUI fee of $10 each for their participation. The payment for these additional 5 pilot participants will total $50.
6. Data Confidentiality
Cognitive Interviews Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used for internal purposes to improve the design of a national employment survey. Participants participating in cognitive interviews will be given a consent form to read and sign (see Appendix E). Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.
Online Survey. Participants will be informed of the OMB number and the voluntary nature of the study, with the below text appearing on the first page of the survey.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 20 minutes to complete. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. Your participation is voluntary, and you have the right to stop at any time.
Appendix A |
Interview Protocol |
Appendix B |
Online Survey |
Appendix C |
Recruitment Ads |
Appendix D |
Screening Script |
Appendix E |
Consent Form |
Appendix A: Individual Cognitive Interviewing Protocol
Cognitive Testing of the CPS UI Non-filers Supplement
Cognitive Interview Protocol Guide
Hi! Thank you for coming in today.
I am …... [This is my colleague ( ) who will be taking notes for us today]
Have you participated in any of our studies before?
Consent Form/Permission to audiotape
Explanation:
We are going to be working with some questions from the Current Population Survey, an ongoing survey that provides a continuous flow of information about the characteristics of the American workforce.
What we are going to do today is go through a few questions from the survey and get your reactions to them.
The purpose of today’s session is to help us find out more about how people respond to these questions. Basically, we’re trying to find out what you think a question is asking and how hard it is to answer. We are not here to evaluate you, we are looking to improve the questions, so there are no wrong answers. All the information you give us will be kept completely confidential, and will be used to improve the survey questions.
Any questions before we begin?
Permission to record?
Labor Force Questions
Question Number |
Questions |
Response Options |
Who Receives Question |
|
Roster1 |
Please tell me the names and ages of everyone who lives in your household |
Open ended |
Everyone |
|
EDUCA |
What is the highest level of school (name/you) (has/have) completed or the highest degree (name/you) (has/have) received? |
(31) Less than 1st grade (32) 1st, 2nd, 3rd or 4th grade (For example: BA, AB, BS) (33) 5th or 6th grade (34) 7th or 8th grade (35) 9th grade (36) 10th grade (37) 11th grade (38) 12th grade NO DIPLOMA (39) HIGH SCHOOL GRADUATE- high school DIPLOMA or the equivalent (For example: GED) (40) Some college but no degree (41) Associate degree in college - Occupational/vocational program (42) Associate degree in college -- Academic program (43) Bachelor's degree (44) Master's degree (For example: MA, MS, MEng, MEd, MSW, MBA) (45) Professional School Degree (For example: MD,DDS,DVM,LLB,JD) (46) Doctorate degree (For example: PhD, EdD) |
Everyone |
|
HSPNON
|
(Are / Is) (name/you) of Hispanic, Latino, or Spanish origin? |
(1) Yes (2) No |
Everyone |
|
RACE
|
I am going to read you a
list of race categories. You may choose one or more races. For
this survey, Hispanic origin is not a race. Are you: White; Black or African American; American Indian or Alaska Native; Asian; OR Native Hawaiian or Other Pacific Islander? |
|
Everyone |
|
LABOR |
Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___
|
Everyone |
||
PUWK |
LAST WEEK, did you/name do ANY work for pay? |
|
Everyone |
|
PUABS |
LAST WEEK, did you/name have a job either full or part time? Include any job from which you were temporarily absent. |
|
PUWK = 2, D, R |
Anyone who didn’t work last week |
LAYOFF |
LAST WEEK, were you/name on layoff from a job? |
|
PUABS=2, D,R |
Anyone who didn’t have a job last week |
RETURN |
Has your/their employer given you/name a date to return to work? |
|
LAYOFF=1 |
Anyone on layoff |
LOOKING |
Have you been doing anything to find work during the last 4 weeks? |
|
LAYOFF =2, D, R |
Anyone who didn’t have a job last week
And not on layoff |
LKMTH |
What are all the things you have done to find work during the last 4 weeks? |
Record what is said all go to LASTWORK |
LOOKING = 1 |
Anyone who said did something to find work last week |
DWWNT |
Do you/does Name currently want a job, either full or part time |
|
LOOKING=2, D, R |
Anyone who said they did not do something to find work last week |
LASTWORK |
When did you/name last work at a job or business? |
|
DWWNT = 1, 3, 4 or 5 |
Anyone who said they want a job, they’re retired, disabled or unable to work |
UI-Non Filer Questions
(UI Nonfilers asked of LASTWORK=1 (anyone who did not have a job last week, wants a job, and worked in the last 12 months)
Your cooperation is important because the data collected will be used by the U.S. Department of Labor to learn more about individuals experience with unemployment benefits. Analysis from this survey data will help improve the U.S. unemployment insurance system.
This is the lead in introduction to the supplement.
>PRESUP<
This month we are asking some additional questions about unemployment insurance benefits. People who have previously worked can apply to receive these benefits, which replace part of their income while they look for a job.
PRESS ENTER TO PROCEED
_______________________________________________________________________
Universe for Q1: HHMEM = 1, AFNOW = 2, PRTAGE = 16+, HRMIS=1,2,3,5,6, OR 7, AND PUMLR=5,6,or 7
>Q1<Are you retired FROM A JOB OR BUSINESS?
Yes – to Q3
No – to Q2
===> __
[Don’t know or refused GO TO Q2]
__________________________________________________________________________________
Universe for Q2: Q1=2,D or R
>Q2<What best describes your situation at this time? For example, are you disabled, ill, in school, taking care of house or family, or something else?
Disabled
Ill
In school
Taking care of house or family
In retirement
Something else/other
===> __
[Don’t know or refused GO TO Q3]
[all response options GO TO Q3]
__________________________________________________________________________________
Universe for Q3:[(HRMIS=1,2,3,5,6 or 7) AND (PUDWLK=-1,2,D or R) AND PUMLR=5,6, or 7
>Q3< Have you worked at a job or business at any time during the past 12 months?
Yes – to Q4
No – end survey
===> __
[Don’t know or refused END SURVEY]
_______________________________________________________________________
Universe for Q4: Q3=1 OR PUDWWK=1 OR PUJHWKO=1 OR PUNLFJH=1 (within the last 12 months) OR (PUMLR=4 AND PULKLWO=1, 2,) OR (PUMLR=4 AND PULKLL1O=1 AND PULKLWO=-2, -3, -1) OR (PUMLR=3)
>Q4<[Fill (Have you/Has name)] applied for unemployment benefits since [Fill (your/her/his)] last job?
Yes – to Q5
No – to Q10
===> __
[Don’t know or refused GO TO Q10]
________________________________________________________________________
>Q5<
[Fill (Have you/Has name)] received any unemployment benefits since [Fill (your/her/his)] last job?
Yes – to Q6
No – to Q9
===> __
[Don’t know or refused GO TO Q7]
________________________________________________________________________
>Q6< In what month and year did [fill (you/they)] start receiving unemployment benefits?
===> __
[Don’t know or refused]
=== > Go to Q7
________________________________________________________________________
>Q7<
Did [Fill (you/name)] receive unemployment benefits last week?
Yes - to Q13
No - to Q8
===> __
[Don’t know or refused GO TO Q13]
>Q8<
Why didn’t [Fill (you/name)] receive any unemployment benefits last week?
Applied and Waiting
Waiting for approval of application
Application approved, but waiting period not over
Denied Claim / Did Not Qualify
Used up (exhausted) all benefits
Left job voluntarily, quit, or dismissed for conduct or cause
Earnings or hours too low to qualify
Strike/work stoppage
Was self-employed or an independent contractor
Disqualified (fraud, failed to report on time, refused work referral/work offer, incarceration)
Withheld for child support or overpayment
Different pay periods (every two weeks, monthly)
Other (specify) ___________ - to Q8a
===> __ [1 – 10 GO TO Q13]
[11 GO TO Q8a]
[Don’t know or refused GO TO Q13]
______________________________________________________________________________
>Q8a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q13]
>Q9<
Why [Fill (haven’t you/hasn’t name)] received any unemployment benefits since [Fill (your/her/his)] last job?
Applied and Waiting
Waiting for approval of application
Application approved, but waiting period not over
Denied Claim / Did Not Qualify
Used up (exhausted) all benefits
Left job voluntarily, quit, or dismissed for conduct or cause
Earnings or hours too low to qualify
Strike/work stoppage
Was self-employed or an independent contractor
Disqualified (fraud, failed to report on time, refused work referral/work offer, incarceration)
Withheld for child support or overpayment
Different pay periods (every two weeks, monthly)
Other (specify) ___________ - to Q9a
===> __ [1 – 10 GO TO Q13]
[11 GO TO Q9a]
[Don’t know or refused GO TO Q13]
>Q9a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q13]
>Q10<
There are a variety of reasons why people might not apply for unemployment benefits. What are the reasons [Fill (you have/name has)] not applied for unemployment benefits since [Fill (your/her/his)] last job? MARK ALL THAT APPLY AND PROBE: Was there another reason?
(Note to instrument Author: Please try to fit this in one screen.)
Plan to file soon– to Q13
Voluntarily left last job, retired or was self-employed
Knowledge Reasons
Did not think eligible -- to Q11
Did not know benefits existed
Did not know where or how to apply
Used or exhausted benefits
Last employer didn’t give any information
Job-Related Reasons
Expect to start a new job
Expect to be recalled to work soon
Problems With Application
Started application, did not finish
Application was too confusing, technical or difficult to understand
Tried to get assistance with application but couldn’t get help
Tried to file by phone or internet but application system wasn’t functioning
Attitudinal / Personal Reasons
Too much like charity or welfare
Worried might impact future jobs
Does not need money / Benefits not expected to be large enough or last long enough
Expected application to be rejected
Too much work/hassle to apply or meet requirements
Other Problems
No transportation
No telephone, no computer or internet
Application or assistance is not available in their language
Other (specify) __________ - TO Q10a
N) No more
[IF tally >1 GO TO Q12]
[IF tally =1 GO TO Q13]
[Don’t know or refused GO TO Q13]
>Q10a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q5 to probe for more responses]
[Store this response as number 19 and tally it in the number of responses to Q10]
>Q11< - only if you answered 3 in Q10
Why didn’t [Fill (you/name)] believe [Fill (you were/she was/he was)] eligible for unemployment benefits?
Didn’t earn/work enough/worked part time
Didn’t have a recent job
Had voluntarily left/quit last job/retired
Was fired from last job
Was self-employed or an independent contractor
Told not eligible by former employer or office
Received severance pay
Other-Specify _______________
===> __ [1 – 7 GO TO Q10 to probe for more responses]
[8 GO TO Q11a]
[Don’t know or refused GO TO Q10 to probe for more responses]
>Q11a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q10 to probe for more responses]
[Store this response as number 25 and tally it in the number of responses to Q10]
>Q12< [if Q10 has more than one response]
Of the reasons you just mentioned, what is the main reason [Fill (you/name)] did not apply for UI benefits?
[Response options should have the reasons entered in Q10 (1 through 21) retaining the same numbers previously used.]
Example: <5> Misinformed by boss or office
<22> Had voluntarily left/quit last job
===> __
[This response must match an entry given in Q10 or Q11]
[GO TO Q13]
[Don’t know or Refused GO TO Q13]
>Q13<
[Fill (Were you/Was name)] a union member or covered by a union contract on [Fill (your/his/her)] last job?
<1> Yes
<2> No
<3> DK / R
===> All go to A1
Now, we also have a few questions about your experience looking for a new job over the last 2 months.
>A1< How many job applications, if any, have you submitted in the last 2 months?
0 - go to A4
1 to 10 – go to A2
11 to 20 – go to A2
21 or more – go to A2
DK / R – go to A2
>A2< To accept any of the jobs you applied for, would you have to move or relocate?
Yes
No
DK/R
All go to A3
>A3< How many of those applications submitted in the last 2 months led to an interview?
0
1 or 2
3 to 7
8 or more
DK / R
All go to A4
>A4<.Have you had any problems finding jobs to apply for?
Yes -- Go to A5
No -- go to B1
DK/R -- go B1
>A5< What is the main problem that you have had in finding jobs to apply for? [Interviewer does not read options out loud] MARK ALL THAT APPLY AND PROBE: Was there another problem?
Too few jobs for what I am trained to do / for my education level
The wages are too low
The benefits associated with the job (pension, health insurance) are not good
The hours of work are too few
The hours of work are not convenient
Jobs do not have flexible enough work schedule
The jobs do not have good promotion potential
Most of the suitable jobs are too far away
My health keeps me from being able to do many jobs
Not searching hard enough
Other, specify
===> __ [Range is 1 through 11 and N for no more. Accept multiple entries. Allow backup to remove an entry. Zero is not an acceptable response]
[If 11 is entered immediately store it for A5 and GO TO A5a, it will return from A5a and it should probe for more responses to A5]
[Tally number of responses entered including stored entries of option 11]
==== > go to B1
>A5a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO A5 to probe for more responses]
[Store this response as number 11 and tally it in the number of responses to A5]
>B1< How many job offers, if any, have you received since you last worked?
0 Go to B5
1 Go to Criteria for B2
2 or more Go to Criteria for B2
CRITERIA FOR RECEIVING B2):
All respondents in MIS=4 or MIS=8
If MIS= 1-3, 5-7: {just outgoing rotations, 2) people who never applied for UI, 3) people who applied but never got benefits, and 4) people who are no longer receiving benefits }
Nonfiler Q4= No
or
Nonfiler Q5= No and [Q9 =/= Waiting for approval of application or Q9 =/= Application approved, but waiting period not over]
or
Nonfiler Q7= No
If criteria matches Go to B2
If criteria does not match Go to B3
>B2< Thinking about the [fill if B1>1 best] job offer that you received, why did you turn that job offer down?
I have not turned down a job offer – go to B5
The wage was too low
The benefits associated with the job (pension, health insurance) were not good
The hours of work were too few
The hours of work were not convenient
It did not offer a flexible enough schedule
The job was not in my usual occupation/use my skills well
The job did not have good promotion potential
The job would have required me to move, relocate to another part of the country
The commute to the job would have been too long
Personal reasons for turning down the job (e.g. ill health, could not arrange child care, not physically able to do the job
Other, specify
[Response options 2-12 go to B3]
>B2a<
Other Specify: ________________________________________________
[60 characters maximum]
=== > Go to B3
>B3< How long ago did you receive your [fill if B1>1 best] offer?
Within the last week
Longer than a week ago but less than a month
1 to 3 months ago
3 to 6 months ago
6 or more months ago
DK/R
Go to B4
>B4< If you were offered that job again today, would you accept it or would you still turn it down?
Yes
No
I have not turned down the job offer
DK/R
Go to B5
>B5< If you were offered a job similar to your last job, would you be willing to accept less pay than you received before?
Yes
No
DK/R
Debriefing
Thanks for your answers so far. Now I want to go talk about some of the questions individually, so we can get your feedback on the questions and understand what you were thinking when you answered them.
To start, I want to ask you generally -
What did you think of the interview questions?
What was your reaction to the questions?
Did any of those questions stand out to you for any reason?
[proxy] Do you think that you were generally able to answer the questions correctly for the other people in your household?
Which ones were most difficult?
Intro
At the beginning of the second section, I said “This month we are asking some additional questions about unemployment insurance benefits. People who have previously worked can apply to receive these benefits, which replace part of their income while they look for a job.”
Have you heard of unemployment insurance benefits before?
What do you think they are?
Do you have any personal experience with them?
Have you heard of the term UI benefits?
Are they the same or different?
Did our explanation “replace part of your income while you look for a job” make sense?
Did it match with your understanding of unemployment insurance benefits?
Do you know any of the requirements associated with receiving unemployment insurance benefits?
Q5. One of the questions in this section asked “[Fill (Have you/Has name)] received any unemployment benefits since [Fill (your/her/his)] last job?”
What did that question mean to you?
Was it clear what job the question was referring to?
When answering for yourself, what job did you think about?
[proxy] how much do you know about [fill name]’s employment and unemployment situation?
What about their last job, do you know much about why and when it ended?
When did you/they last have that job?
Q6. [if received] The next question asked “In what month and year did [fill (you/they)] start receiving unemployment benefits? “
How easy or difficult was that to answer? Are you sure of your answer?
How did you go about remembering when your benefits started?
[proxy] How did you know when their benefits started? (e.g., share bank accounts, they told me)
Q8. [if received] Another question asked “Why didn’t [Fill (you/name)] receive any unemployment benefits last week?”
[if gave more than one reason] You said the main reason was [fill verbatim], how did you go about deciding which reason was the main one?
Was it easy or difficult to select?
I matched that to our answer choice of [filler response option]. Do feel that’s an accurate representation of your answer?
Do you think there may be other reasons you didn’t receive UI benefits last week?
[proxy] Are you clear on the reasons they didn’t receive UI benefits last week?
Q9 [if received] We also asked about why you “haven’t received any unemployment benefits since [Fill (your/her/his)] last job?”
[if gave more than one reason] You said the main reason was [fill verbatim], how did you go about deciding which reason was the main one?
Was it easy or difficult to select?
I matched that to our answer choice of [filler response option]. Do feel that’s an accurate representation of your answer?
Do you think there may be other reasons you didn’t receive UI benefits since your last job?
[proxy] Are you clear on the reasons they didn’t receive UI benefits since your last job?
Q10. [if received] We are especially interested in reasons why people may not apply for unemployment benefits, as we would like to make it as easy as possible for people eligible for these benefits. We asked you for the reasons that [you/fill name] had not applied for unemployment benefits since your last job.
[if gave more than one reason] You said the main reason was [fill verbatim], how did you go about deciding which reason was the main one?
Was it easy or difficult to select?
I matched that to our answer choice of [filler response option]. Do feel that’s an accurate representation of your answer?
[if relevant] Can you tell me about your experience trying to apply for benefits?
How well do you understand the requirements of UI benefits?
Did the requirements dissuade you from applying in any way?
[proxy] How well do you feel you know the reasons [fill name] didn’t apply for benefits?
[proxy] Did [fill name] tell you about their experience trying to apply for benefits?
Q11 [if received]. For people who didn’t apply because they thought they weren’t eligible to receive the unemployment benefits, we asked for the reasons they thought they weren’t eligible.
You said the reason was [fill verbatim], I matched that to our answer choice of [filler response option]. Do feel that’s an accurate representation of your answer?
[probe for specifics as necessary to understand their understanding of eligibility]
[proxy] do you know if [fill name] thinks they’re eligible or not?
Q12 [if received]. I asked you which of the reasons for not applying was the main reason.
How did you decide which of your reasons for not applying was the main one?
Did one really stand out to you more than the others?
[proxy] Do you feel like you have a good sense of their reasons and which are the most important?
Job Searching
After the questions about unemployment insurance benefits, there were some questions about job applications, interviews and offers.
What did you think of these questions?
What was your reaction to the questions?
Did they seem to fit in with the rest of the questions I asked?
Did any of those questions stand out to you for any reason?
[proxy] Do you think that you were generally able to answer the questions correctly for the other people in your household?
Which ones were most difficult?
A1. I asked “How many job applications, if any, have you submitted in the last 2 months?”
How did you go about answering that question?
Probe enough to get estimation strategy
[proxy] How much do you know about the job applications other people in your household submit?
Do you know how many jobs they apply for?
Do they give you regular updates?
A2. I also asked “To accept any of the jobs you applied for, would you have to move or relocate?”
What does this question mean to you?
A3. When I asked you how many of the applications submitted in the last 2 months led to an interview, you were able to answer.
How did you go about answering that question?
Probe enough to get estimation strategy
How much do you know about the interviews other people in your house go on?
A5 (if asked). I asked you to tell me about the main problem you had in finding jobs to apply for.
You said the reason was [fill verbatim], I matched that to our answer choice of [filler response option]. Do feel that’s an accurate representation of your answer?
[proxy] how well do you know the problems other people in your house have finding jobs to apply for?
Job Offers
I then asked some questions about job offers you have received.
What was your reaction to those questions?
Did any of the questions stand out to you in any way?
B2, B3 and B4 [if asked] One of the questions asked you about the best job offer you’ve received.
Was it easy to identify which offer was the ‘best’?
How did you make that decision?
[proxy] Do you think you know about all the job offers other people in your household received?
[proxy, if more than one offer] Do you think they consider their best job offer the same offer that you do?
B5. The last question asked “If you were offered a job similar to your last job, would you be willing to accept less pay than you received before?”
In your own words, what is this question asking?
Conclusion
Now that we’ve talked about the questions we’re considering, what is your reaction?
Do any of the specific questions stand out to you for any reason?
If we asked someone else in your household to answer these questions on your behalf, do you think they’d be able to?
Would they have any concerns?
Any other thoughts as we wrap up?
Appendix B. Online Survey
Sample: MTurk participants who: 1. Are currently unemployed, 2. Lost job through no fault of their own, 3 would like to work, and 4. have held a job in the past 12 months. Eligibility will be based on self report, those who meet the requirements will be sent to the Qualtrics survey.
Introduction
Thank you for participating in our two-part study, which should take less than 20 minutes. In the first part, you’ll answer a series of questions from a survey for the Bureau of Labor Statistics. The questions are from a national government survey designed to collect information about employment and unemployment.
In the second part, you’ll be asked to give your feedback on those questions. This feedback will be used to evaluate the questions. The information you provide will be kept confidential, and used for research purposes only.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. This survey will take approximately 20 minutes to complete. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. Your participation is voluntary, and you have the right to stop at any time.
Labor Force Questions
Question Number |
Questions |
Response Options |
Who Receives Question |
|
Roster1 |
Please tell me the names and ages of everyone who lives in your household |
Open ended |
Everyone |
|
EDUCA |
What is the highest level of school (name/you) (has/have) completed or the highest degree (name/you) (has/have) received? |
(31) Less than 1st grade (32) 1st, 2nd, 3rd or 4th grade (For example: BA, AB, BS) (33) 5th or 6th grade (34) 7th or 8th grade (35) 9th grade (36) 10th grade (37) 11th grade (38) 12th grade NO DIPLOMA (39) HIGH SCHOOL GRADUATE- high school DIPLOMA or the equivalent (For example: GED) (40) Some college but no degree (41) Associate degree in college - Occupational/vocational program (42) Associate degree in college -- Academic program (43) Bachelor's degree (44) Master's degree (For example: MA, MS, MEng, MEd, MSW, MBA) (45) Professional School Degree (For example: MD,DDS,DVM,LLB,JD) (46) Doctorate degree (For example: PhD, EdD) |
Everyone |
|
HSPNON
|
(Are / Is) (name/you) of Hispanic, Latino, or Spanish origin? |
(1) Yes (2) No |
Everyone |
|
RACE
|
Which of the following
best reflects your race? Are you: White; Black or African American; American Indian or Alaska Native; Asian; OR Native Hawaiian or Other Pacific Islander? |
|
Everyone |
|
LABOR |
Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday and ending on Saturday.
|
Everyone |
||
PUWK |
LAST WEEK, did you/name do ANY work for pay? |
|
Everyone |
|
PUABS |
LAST WEEK, did you/name have a job either full or part time? Include any job from which you were temporarily absent. |
|
PUWK = 2, D, R |
Anyone who didn’t work last week |
LAYOFF |
LAST WEEK, were you/name on layoff from a job? |
|
PUABS=2, D,R |
Anyone who didn’t have a job last week |
RETURN |
Has your/their employer given you/name a date to return to work? |
|
LAYOFF=1 |
Anyone on layoff |
LOOKING |
Have you been doing anything to find work during the last 4 weeks? |
|
LAYOFF =2, D, R |
Anyone who didn’t have a job last week
And not on layoff |
LKMTH |
What are all the things you have done to find work during the last 4 weeks? |
Record what is said all go to LASTWORK |
LOOKING = 1 |
Anyone who said did something to find work last week |
DWWNT |
Do you/does Name currently want a job, either full or part time |
|
LOOKING=2, D, R |
Anyone who said they did not do something to find work last week |
LASTWORK |
When did you/name last work at a job or business? |
|
DWWNT = 1, 3, 4 or 5 |
Anyone who said they want a job, they’re retired, disabled or unable to work |
UI-Non Filer Questions; will follow the same flow as the cognitive interviews
(UI Nonfilers asked of LASTWORK=1 (anyone who did not have a job last week, wants a job, and worked in the last 12 months)
Your cooperation is important because the data collected will be used by the U.S. Department of Labor to learn more about individuals experience with unemployment benefits. Analysis from this survey data will help improve the U.S. unemployment insurance system.
This is the lead in introduction to the supplement.
>PRESUP<
This month we are asking some additional questions about unemployment insurance benefits. People who have previously worked can apply to receive these benefits, which replace part of their income while they look for a job.
PRESS ENTER TO PROCEED
_______________________________________________________________________
Universe for Q1: HHMEM = 1, AFNOW = 2, PRTAGE = 16+, HRMIS=1,2,3,5,6, OR 7, AND PUMLR=5,6,or 7
>Q1<Are you retired FROM A JOB OR BUSINESS?
Yes – to Q3
No – to Q2
===> __
[Don’t know or refused GO TO Q2]
________________________________________________________________________
Universe for Q2: Q1=2,D or R
>Q2<What best describes your situation at this time? For example, are you disabled, ill, in school, taking care of house or family, or something else?
Disabled
Ill
In school
Taking care of house or family
In retirement
Something else/other
===> __
[Don’t know or refused GO TO Q3]
[all response options GO TO Q3]
__________________________________________________________________________________
Universe for Q3:[(HRMIS=1,2,3,5,6 or 7) AND (PUDWLK=-1,2,D or R) AND PUMLR=5,6, or 7
>Q3< Have you worked at a job or business at any time during the past 12 months?
Yes – to Q4
No – end survey
===> __
[Don’t know or refused END SURVEY]
_______________________________________________________________________
Universe for Q4: Q3=1 OR PUDWWK=1 OR PUJHWKO=1 OR PUNLFJH=1 (within the last 12 months) OR (PUMLR=4 AND PULKLWO=1, 2,) OR (PUMLR=4 AND PULKLL1O=1 AND PULKLWO=-2, -3, -1) OR (PUMLR=3)
>Q4<[Fill (Have you/Has name)] applied for unemployment benefits since [Fill (your/her/his)] last job?
Yes – to Q5
No – to Q10
===> __
[Don’t know or refused GO TO Q10]
________________________________________________________________________
>Q5<
[Fill (Have you/Has name)] received any unemployment benefits since [Fill (your/her/his)] last job?
Yes – to Q6
No – to Q9
===> __
[Don’t know or refused GO TO Q7]
________________________________________________________________________
>Q6< In what month and year did [fill (you/they)] start receiving unemployment benefits?
===> __
[Don’t know or refused]
=== > Go to Q7
________________________________________________________________________
>Q7<
Did [Fill (you/name)] receive unemployment benefits last week?
Yes - to Q13
No - to Q8
===> __
[Don’t know or refused GO TO Q13]
>Q8<
Why didn’t [Fill (you/name)] receive any unemployment benefits last week?
Applied and Waiting
Waiting for approval of application
Application approved, but waiting period not over
Denied Claim / Did Not Qualify
Used up (exhausted) all benefits
Left job voluntarily, quit, or dismissed for conduct or cause
Earnings or hours too low to qualify
Strike/work stoppage
Was self-employed or an independent contractor
Disqualified (fraud, failed to report on time, refused work referral/work offer, incarceration)
Withheld for child support or overpayment
Different pay periods (every two weeks, monthly)
Other (specify) ___________ - to Q8a
===> __ [1 – 10 GO TO Q13]
[11 GO TO Q8a]
[Don’t know or refused GO TO Q13]
______________________________________________________________________________
>Q8a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q13]
>Q9<
Why [Fill (haven’t you/hasn’t name)] received any unemployment benefits since [Fill (your/her/his)] last job?
Applied and Waiting
Waiting for approval of application
Application approved, but waiting period not over
Denied Claim / Did Not Qualify
Used up (exhausted) all benefits
Left job voluntarily, quit, or dismissed for conduct or cause
Earnings or hours too low to qualify
Strike/work stoppage
Was self-employed or an independent contractor
Disqualified (fraud, failed to report on time, refused work referral/work offer, incarceration)
Withheld for child support or overpayment
Different pay periods (every two weeks, monthly)
Other (specify) ___________ - to Q9a
===> __ [1 – 10 GO TO Q13]
[11 GO TO Q9a]
[Don’t know or refused GO TO Q13]
>Q9a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q13]
>Q10<
There are a variety of reasons why people might not apply for unemployment benefits. What are the reasons [Fill (you have/name has)] not applied for unemployment benefits since [Fill (your/her/his)] last job? MARK ALL THAT APPLY AND PROBE: Was there another reason?
(Note to instrument Author: Please try to fit this in one screen.)
Plan to file soon– to Q13
Voluntarily left last job, retired or was self-employed
Knowledge Reasons
Did not think eligible -- to Q11
Did not know benefits existed
Did not know where or how to apply
Used or exhausted benefits
Last employer didn’t give any information
Job-Related Reasons
Expect to start a new job
Expect to be recalled to work soon
Problems With Application
Started application, did not finish
Application was too confusing, technical or difficult to understand
Tried to get assistance with application but couldn’t get help
Tried to file by phone or internet but application system wasn’t functioning
Attitudinal / Personal Reasons
Too much like charity or welfare
Worried might impact future jobs
Does not need money / Benefits not expected to be large enough or last long enough
Expected application to be rejected
Too much work/hassle to apply or meet requirements
Other Problems
No transportation
No telephone, no computer or internet
Application or assistance is not available in their language
Other (specify) __________ - TO Q10a
N) No more
[IF tally >1 GO TO Q12]
[IF tally =1 GO TO Q13]
[Don’t know or refused GO TO Q13]
>Q10a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q5 to probe for more responses]
[Store this response as number 19 and tally it in the number of responses to Q10]
>Q11< - only if you answered 3 in Q10
Why didn’t [Fill (you/name)] believe [Fill (you were/she was/he was)] eligible for unemployment benefits?
Didn’t earn/work enough/worked part time
Didn’t have a recent job
Had voluntarily left/quit last job/retired
Was fired from last job
Was self-employed or an independent contractor
Told not eligible by former employer or office
Received severance pay
Other-Specify _______________
===> __ [1 – 7 GO TO Q10 to probe for more responses]
[8 GO TO Q11a]
[Don’t know or refused GO TO Q10 to probe for more responses]
>Q11a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO Q10 to probe for more responses]
[Store this response as number 25 and tally it in the number of responses to Q10]
>Q12< [if Q10 has more than one response]
Of the reasons you just mentioned, what is the main reason [Fill (you/name)] did not apply for UI benefits?
[Response options should have the reasons entered in Q10 (1 through 21) retaining the same numbers previously used.]
Example: <5> Misinformed by boss or office
<22> Had voluntarily left/quit last job
===> __
[This response must match an entry given in Q10 or Q11]
[GO TO Q13]
[Don’t know or Refused GO TO Q13]
>Q13<
[Fill (Were you/Was name)] a union member or covered by a union contract on [Fill (your/his/her)] last job?
<1> Yes
<2> No
<3> DK / R
===> All go to A1
Now, we also have a few questions about your experience looking for a new job over the last 2 months.
>A1< How many job applications, if any, have you submitted in the last 2 months?
0 - go to A4
1 to 10 – go to A2
11 to 20 – go to A2
21 or more – go to A2
DK / R – go to A2
>A2< To accept any of the jobs you applied for, would you have to move or relocate?
Yes
No
DK/R
All go to A3
>A3< How many of those applications submitted in the last 2 months led to an interview?
0
1 or 2
3 to 7
8 or more
DK / R
All go to A4
>A4<.Have you had any problems finding jobs to apply for?
Yes -- Go to A5
No -- go to B1
DK/R -- go B1
>A5< What is the main problem that you have had in finding jobs to apply for? [Interviewer does not read options out loud] MARK ALL THAT APPLY AND PROBE: Was there another problem?
Too few jobs for what I am trained to do / for my education level
The wages are too low
The benefits associated with the job (pension, health insurance) are not good
The hours of work are too few
The hours of work are not convenient
Jobs do not have flexible enough work schedule
The jobs do not have good promotion potential
Most of the suitable jobs are too far away
My health keeps me from being able to do many jobs
Not searching hard enough
Other, specify
===> __ [Range is 1 through 11 and N for no more. Accept multiple entries. Allow backup to remove an entry. Zero is not an acceptable response]
[If 11 is entered immediately store it for A5 and GO TO A5a, it will return from A5a and it should probe for more responses to A5]
[Tally number of responses entered including stored entries of option 11]
==== > go to B1
>A5a<
Other Specify: ________________________________________________
[60 characters maximum]
[GO TO A5 to probe for more responses]
[Store this response as number 11 and tally it in the number of responses to A5]
>B1< How many job offers, if any, have you received since you last worked?
0 Go to B5
1 Go to Criteria for B2
2 or more Go to Criteria for B2
CRITERIA FOR RECEIVING B2):
All respondents in MIS=4 or MIS=8
If MIS= 1-3, 5-7: {just outgoing rotations, 2) people who never applied for UI, 3) people who applied but never got benefits, and 4) people who are no longer receiving benefits }
Nonfiler Q4= No
or
Nonfiler Q5= No and [Q9 =/= Waiting for approval of application or Q9 =/= Application approved, but waiting period not over]
or
Nonfiler Q7= No
If criteria matches Go to B2
If criteria does not match Go to B3
>B2< Thinking about the [fill if B1>1 best] job offer that you received, why did you turn that job offer down?
I have not turned down a job offer – go to B5
The wage was too low
The benefits associated with the job (pension, health insurance) were not good
The hours of work were too few
The hours of work were not convenient
It did not offer a flexible enough schedule
The job was not in my usual occupation/use my skills well
The job did not have good promotion potential
The job would have required me to move, relocate to another part of the country
The commute to the job would have been too long
Personal reasons for turning down the job (e.g. ill health, could not arrange child care, not physically able to do the job
Other, specify
[Response options 2-12 go to B3]
>B2a<
Other Specify: ________________________________________________
[60 characters maximum]
=== > Go to B3
>B3< How long ago did you receive your [fill if B1>1 best] offer?
Within the last week
Longer than a week ago but less than a month
1 to 3 months ago
3 to 6 months ago
6 or more months ago
DK/R
Go to B4
>B4< If you were offered that job again today, would you accept it or would you still turn it down?
Yes
No
I have not turned down the job offer
DK/R
Go to B5
>B5< If you were offered a job similar to your last job, would you be willing to accept less pay than you received before?
Yes
No
DK/R
Debriefing
Thank you, now please give us your feedback on the survey questions you just answered.
What topics did the survey questions ask about? (5 open ended lines)
Did any of the individual questions stand out to you for any reason?
Yes – open ended to specify which one, followed by “what was it about this question that stood out to you?”
No
Would you have any concerns answering these questions if asked by a Bureau of Labor Statistics interviewer?
Yes – what concerns would you have? Go to question 4
No – skip to question 7
Using the grid below, mark which specific questions do you have concerns about and whether your concerns related to answering for yourself or answering for other people in your household.
|
Concerns when: |
|
Question Topic |
Answering for myself |
Answering for other people in my household |
Education |
|
|
Race |
|
|
Income |
|
|
Current employment status |
|
|
Date of last job |
|
|
Whether you/they want a job |
|
|
Whether applied for unemployment insurance benefits |
|
|
Whether received unemployment insurance benefits |
|
|
Number of job applications |
|
|
Number of job interviews |
|
|
Number of job offers |
|
|
Were there any other questions that you were concerned about?
Yes, specify
No
What categories best reflects your main concern about answering these questions? (mark all)
I didn’t understand the question
The information is too personal
I didn’t know the answer
I wouldn’t want to share that information with the government
Other, specify
Prior to this survey, had you heard the term Unemployment Insurance Benefits?
Yes – go to 4
No – go to 10
How certain are you that you know what Unemployment Insurance Benefits are?
Very certain
Somewhat certain
Somewhat uncertain
Very uncertain
Which of the following are requirements for receiving Unemployment Insurance Benefits?
Have had a job in the last 12 months
Chose to leave last job (e.g., quit or retired)
Lost job through no fault of their own
Have applied for benefits
Be actively looking for a job
Applied to at least 5 jobs
Received an interview for a new job
Not turned down any job offers
Other, specify
Thank you for your help with this important study!
Appendix C: Recruitment Advertisements
Craigslist Advertisement
Volunteers needed to test survey - $40 for 1 hour
The Bureau of Labor Statistics is looking for people who:
1. Are unemployed, retired or otherwise out of a job AND
2. Have held a job in the past 12 months.
Each participant will receive $40.00 compensation for participating in this study.
If you meet the requirements please contact Stella at 202-691-7524 or [email protected].
Appendix D: Screening Script
CPS UI Non-Filer Screener
Name |
|
|
|
First Name |
Last Name |
|
|
|
Address |
|
|
|
|
|
|
|
|
|
|
|
Primary Phone |
|
Cell Home Work |
|
|
|
Secondary Phone |
|
Cell Home Work |
|
|
|
Email Address |
|
|
|
|
|
Step 1. Screen to see if they can answer for themselves
|
|
Working full time |
not eligible for self report, go to page 4, screen for HH reporting |
Working part time |
not eligible for self report, go to page 4, screen for HH reporting |
Retired |
Continue screener |
In School |
Continue screener |
Unemployed and looking for a job |
Continue screener |
Unemployed and not looking for a job |
Continue screener |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
|
|
Yes |
Continue screener |
No |
not eligible for study, thank and end screener |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
|
|
Yes |
Continue screener |
No |
Go to Step 3, page 6 |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
|
|
Yes |
Continue screener |
No |
Collect demographics and determine eligibility |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
Step 2. See if they can answer for other household members
|
|
0 |
Not eligible for study |
1 |
Continue Screener |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
|
|
No |
not eligible for study |
Yes, retired |
Continue Screener |
Yes, unemployed |
Continue Screener |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
|
|
No |
not eligible for study |
Yes, retired person held a job in the last 12 months |
Collect demographics and determine eligibility |
Yes, unemployed person held a job in the last 12 months |
Collect demographics and determine eligibility |
Other |
get information about their situation, and tell them you’ll call back if they’re eligible |
Step 3. Collect Demographic Data, if they meet the race and education requirements, schedule participant
Based on your answers, you may be eligible for our study! I’d like to get some background information to be sure
|
|
White |
Use recruiting goals to determine eligibility |
Black or African American |
|
Asian, Hispanic or Other |
|
|
Less than high school |
Use recruiting goals to determine eligibility |
Some College |
|
College Degree or Higher |
Appendix E: Consent Form
OMB Control No: 1220-0141
Expiration Date: April 30, 2018
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you 60 minutes to participate in this research.
Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires April 30, 2018.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | October 14, 2003 |
Author | Yu, Erica - BLS |
File Modified | 0000-00-00 |
File Created | 2021-01-22 |