Note to Reviewer - OES PRD Feasibility Test

28 October 2014

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Office of Survey Methods Research
SUBJECT:	Submission of Materials for the OES Previously Reported Data Cognitive Interviews

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 30 hours.

If there are any questions regarding this project, please contact Jennifer Edgar at 202-691-7528.

Introduction

The Occupational Employment Statistics (OES) survey program has been conducting research on creating sampling and estimation methodologies for converting OES to a time series. Currently OES data is only requested from a single establishment once every three years. With the switch to time series that will change. OES will contact the same establishments more often. Most large establishments will submit electronic files which can be consistently and accurately coded by State analysts. However, many small and medium firms will submit their data using the survey form or over the telephone. Telephone solicitation of non-responders is especially important since 25-30% of responses come in after the last mail-out.

One of the biggest challenges of time series data collection is distinguishing real change from respondent error or satisficing. Presenting previously reported data (PRD) to respondents has the potential to improve data quality; if a respondent has trouble with recalling particular job titles or how they calculated wages, having the PRD can help them remember their process and recreate it reliably. On the other hand, PRD can increase errors if a respondent satisfices and simply reports the same values. There are two types of dependent interviewing techniques: reactive and proactive. During reactive interviewing, when the data is submitted and is different, the interviewer points out that it is different and asks for any corrections. In proactive interviewing the previously submitted data is shown prior to data submission. The interviewer gives them the previously submitted data and asks for updates. 

The goal of this research project is to explore two methods of using previously reported data (PRD) during the telephone solicitations in the second wave of collection. This two-phase research project will start with cognitive interviews to explore how respondents use the PRD when arriving at their responses for the second wave of data collection. A split-ballot field test, to be submitted under a separate OMB package, will explore the impact of using PRD on interview procedures, interview duration, interviewer burden and responses rates.

1. Methodology

1. Cognitive Interviews

To gain an understanding of how respondents use PRD in their response process, OSMR staff will conduct up to 20 cognitive interviews of previous OES respondents. Ideally, these interviews will be conducted in-person, in the Washington DC area, but we may conduct some on the telephone based on recruiting results. Participants will be selected from prior OES respondents who have less than 19 employees and are in the local area.

The cognitive interviews will focus on the following research questions:

1. What do participants do when presented with PRD proactively? Think of their own answers first and then compare, or think of the PRD and then reasons to change it?

   1. Does this impact their desire/need to look up information?

2. What do participants do when presented with PRD retroactively? Reevaluate their current answer? Reevaluate their previous answer? Ignore it?

3. Is there a minimum difference that participants won’t bother to correct or address?

The interview will begin by following the protocol used during production OES telephone follow-up calls. Participants will be asked to report job titles, employment count and wage rates for all employees at the establishment. Half of the participants will be given the job titles, employment count and wage rates they reported to OES in the last data collection and asked if there are any updates. The other half will not be given PRD until after they have provided their data. Then, the interviewer will share their previously reported values and ask the participant to reconcile any differences between the current report and the PRD.

For both groups, a debriefing will occur after the data has been collected. A protocol outline (Attachment I) will be followed, but the debriefing will be unscripted, a dialog aimed at understanding the participants response process and focusing on the three research questions.

2. Participants

The sample will consist of 60 businesses with less than 19 employees, drawn from past OES respondents. Certainty units (COCs) and government units will be excluded from the selection process.

3. Burden Hours

Up to 20 cognitive interviews will be conducted, each lasting up to one hour, for a total of 20 burden hours. We estimate that a total of 60 establishments will be contacted during the recruitment process, with each recruiting call lasting about 10 minutes each, for a total of 10 hours.

4. Data Confidentiality

Cognitive interview participants will be given a consent form prior to their interview (Attachment II).

5. Attachments

1. Attachment I: Cognitive Interview Protocol Outline

2. Attachment II: Consent Form

Attachment I: Cognitive Interview Protocol Outline

1. Introduction

   1. Introduce self and explain goals of the study

   2. Get consent form signed

2. Group 1 Proactive PRD

   1. Tell them their previously reported job titles and ask if they have had any changes in the last year

   2. Tell them their previously reported employment counts and ask if they have had any changes in the last year

   3. Tell them their previously reported wage rates and ask if they have had any changes in the last year

3. Group 2 Retroactive PRD

   1. Ask for the job titles of all employees at that establishment

   2. Ask for the number of employees in each position

   3. Ask for the hourly rate or salary for each employee

4. Debriefing

   1. How did they arrived at their answer?

      1. Did they use records?

      2. How certain are they that the information is accurate?

   2. Do they remember answering OES last year?

      1. Did they use records?

   3. Group 1 Proactive: how did they use the PRD?

      1. Think of an answer first and compare?

      2. Think of the PRD first and think of how it would have changed?

      3. Think of individual people/situations?

      4. Do you think having the PRD made it easier or more difficult for you to answer the questions?

   4. Group 2 Retroactive: How did they use the PRD?

      1. Think of reasons it would have changed?

      2. Re-think the current number?

      3. Think of individual people/situations?

   5. What changes in employment have there been in the past 12 months?

      1. Probe for possible differences: promotions, new hires, people leaving?

   6. How burdensome was it for you to answer these questions?

   7. In the future, another employee may answer the survey. Do you have any concerns about sharing job titles, employment count and wage rates with someone else in the company?

5. Closing

   1. Any final comments or suggestions?

Attachment II: Consent Form

CONSENT FORM

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 45 minutes to participate in this research (ranging from 30 minutes to 60 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: February 28, 2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.