Note to Reviewer - Testing of OES Small Establishment Form

OMB Control Number: 1220-0141

Expiration Date: 03/31/2021

5/9/2019

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Robin Kaplan Office of Survey Methods Research
SUBJECT:	Submission of Materials for Testing of Occupational Employment Statistics (OES) Small Establishment Form

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 183 hours.

If there are any questions regarding this project, please contact Robin Kaplan at 202-691-7383.

1. Introduction

The Occupational Employment Statistics (OES) program at BLS collects data from establishments on their employment and wage estimates. A typical OES form includes a spreadsheet with one column for the job title/description of duties and the associated wage for each employee. OES is moving toward asking respondents to report wage rate data for their employees to increase data precision. Currently, respondents are asked to report wage ranges. The Office of Survey Methods Research (OSMR) previously worked on research to determine whether or not OES respondents are able to report point estimate data for employee wages. The results from the previous study (phone interviews with a subset of OES respondents from small establishments) showed that most OES respondents in small establishments have access to point estimates, and are willing and able to report wage rates. Feedback from this previous research and the state analysts that collect OES data found that there may be a more efficient way for OES respondents to report wage information by inclusion of a third column on the form where respondents can indicate the number of employees in a particular occupation with the exact wage rate. For example, if an employer has five HR Assistants who each earn the same exact salary, the respondent could fill in the number “5” on the third column of the form rather than write out each HR specialist on an individual line. This approach has the potential to save respondents time and reduce burden.

Because smaller establishments with 20 or fewer employees are most likely to use a paper form versus uploading a spreadsheet, the change from reporting wage intervals to precise wage rates would most likely affect these smaller establishments. However, a large scale test was not conducted with the addition of the third column to enter the number of employees in a particular job with the same wage, to determine whether OES respondents would provide quality data when the task changed from reporting wage intervals to precise wage rates.

2. Methodology

To better understand the impact of requesting point estimates instead of wage interval data from small establishments on data quality, OES and OSMR will conduct a research study outside of regular OES production.

We will conduct a national sample of 1500 small establishments (those with 20 or fewer employees) stratified across a range of industries, to complete one of two integrated forms (one with the number of employees column and one without; see Attachments A and B for the forms that will be used). Respondents will be randomly assigned to receive one of the two forms. The cover letter will instruct respondents to send their data via mail or fax only so that we may assess the impact of the third column (only available in the paper form) on data quality. Respondents will have 3 months to respond. We will send one follow-up reminder (re-sending the original form) to respondents who have not completed the form within one month after the first mailing was sent.

Once we receive the data, OSMR will look for anomalies across both version of the form, including:

• Discrepancies between the total employment and the number of employees listed on each form

• The number of unique occupations listed on each form

• Salary ranges provided instead of exact wage rates (e.g., if a respondent indicates a range or average salary was provided)

• Any other unexpected patterns in the data

We will also track the following data quality indicators throughout the duration of the field period:

• Number of incoming calls and emails with questions, and information requested (we will provide an OSMR-monitored phone number and email address to document the types of questions received)

• Total number of employees reported at the establishment (#4) on the form versus the number of occupations reported

• Response rates between the two forms will be compared as a secondary research question

Once units with anomalies are identified, OSMR will conduct a follow-back debriefing study (see Attachment C for the protocol). Follow-back, debriefing research studies are common in establishment survey research methods to assess changes and get respondent feedback. To facilitate recall of the form, during the interview, the researchers will send respondents a copy of the OES forms via email for their reference. We estimate up to 40 respondents will be cooperative and willing to participate in the debriefing. We anticipate certain types of data quality issues may emerge and have provided pre-scripted probes for those situations. However, the data may reveal an unanticipated or different type of complication completing the form. As such, each protocol will be tailored for each unit that targets specific probes to gain a better understanding of any confusion that a respondent experienced while completing the form while being flexible to incorporate emergent probes, for example:

• Clarify any anomalies in the provided data (e.g., respondent provided wage range instead of ranges, respondent indicated an average range was provided)

• Any standard follow up they would do in production (e.g., respondent provided names instead of job titles; respondent didn’t provide enough information to classify an occupation)

Sample Size

We conducted a power test to determine the necessary sample size to detect differences between the two forms. Assuming cases assigned to the integrated form without the “number of employees” column will need follow-up 5% of the time, a sample of 1,500 units (750 to each form) will allow us to detect a difference in the percent of units requiring follow-up for units assigned to the integrated form with columns of three percent. If we assume only 66% of units respond, we would still be able to detect a four percent difference at the 0.05 significance level. We estimate up to 50 respondents will be contacted for the debriefing, resulting in 40 interviews with each debriefing interview lasting approximately 20 minutes.

3. Participants and Burden

A sample of 1500 former OES respondents will be sent one of the two forms via mail. The sample will be nationwide, consisting of those from size classes 1, 2, and 3 with at least 2 employees at the establishment, across a representative mix of industries and states.

Table 1. Estimated Burden Hours

	# of Participants Screened	Minutes per participant for Screening	Total Screening Burden	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden (hrs)	Total Burden (Screening + Collection)
Mail survey	0	0	0	990*	10	165 hours	165 hours
Debriefing survey	50	5	4.2	40	20	13.3	17.5 hours
Total Burden							183 hours

*Note: this assumes a maximum of 66% of respondents will complete the form.

IV. Data Confidentiality

1. The following confidentiality statement will be included on the OES collection form:

The Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent, except in the case of State and local governments. The BLS publishes statistical tabulations from this report that may reveal the information reported by State and local governments. Upon request, however, the BLS will hold the information provided by State and local governments on this report in confidence. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

2. Follow-back debreifing interviews with OES respondents: The following statement will be read verbally to participants before completing the survey over the phone, and asked for their verbal consent to participate in the interview before starting:

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

List of Attachments

Attachment A: Form with column

Attachment B: Form without column

Attachment C: Followback debriefing phone interview protocol

3