March 31, 2015
NOTE TO THE REVIEWER OF:
|
OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
|
FROM: |
Jennifer Edgar and Jim Esposito Office of Survey Methods Research
|
SUBJECT: |
Submission of Materials for the OES Previously Reported Data Feasibility Test
|
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.
The total estimated respondent burden for this study is 208 hours.
If there are any questions regarding this project, please contact Jennifer Edgar at 202-691-7528.
Introduction
The Occupational Employment Statistics (OES) survey program has been conducting research on creating sampling and estimation methodologies for converting OES to a time series. Currently OES data is only requested from a single establishment once every three years. With the switch to time series that will change. OES will contact the same establishments more often. Most large establishments will submit electronic files which can be consistently and accurately coded by State analysts. However, many small and medium firms will submit their data using the survey form or over the telephone. Telephone solicitation of non-responders is especially important since 25-30% of responses come in after the last mail-out.
One of the biggest challenges of time series data collection is distinguishing real change from respondent error or satisficing. Presenting previously reported data (PRD) to respondents has the potential to improve data quality; if a respondent has trouble determining which employees to include or how they calculated wages, having the PRD may help them remember their process and recreate it reliably. On the other hand, PRD can increase errors if a respondent satisfices and simply reports the same values.
There are two ways PRD can be used in an interview: reactive and proactive. During reactive interviewing, when the data is submitted and is different, the interviewer points out that it is different and asks for any corrections. In proactive interviewing the previously submitted data is shown prior to data submission. The interviewer gives them the previously submitted data and asks for updates.
The goal of this research project is to explore two methods of using previously reported data (PRD) during the telephone solicitations in the second wave of collection. This two-phase research project started with cognitive interviews which explored how respondents used the PRD when arriving at their responses for the second wave of data collection. This test, a split-ballot field test, will explore two PRD methods (proactive and reactive) to explore the impact of using PRD on interview procedures, interview duration, interview burden and responses rates.
Methodology
For the field test, OES staff in the State of Maine will send out an advance letter (Attachment I) to 800 previous OES respondents. They will then attempt to conduct telephone data collection with up to 300 previous OES respondents. With the exception of the PRD described below, staff will use the procedure they use for production OES (Attachment II). Participants will be randomly assigned into one of three groups:
Control group: no PRD is given, participants are asked to report job title, employment count and wage ranges as in the production OES.
Proactive group: PRD is given to the participant upfront and they are asked to provide updates.
Reactive group: participants are asked to report job title, employment count and wage ranges as in the production OES. After the data is collected, PRD is given to the participant and they are asked to reconcile any differences.
The PRD provided to the participant will be their responses to the prior OES collection. After the participant has provided a final response, debriefing questions (Attachment III) will be asked to address the research questions. The interview guide for these debriefing questions is based on cognitive interview results and input from OES field staff who conduct production telephone interviews. The debriefing is estimated to take less than 10 minutes, an estimate based on cognitive interviews using a similar protocol.
The following measures will be used to evaluate the research questions:
The length of time spent answering the OES survey questions
The number of differences in employment counts and PRD employment counts
The magnitude of the difference
The number of differences in reported wage ranges and PRD wage ranges
The magnitude of the differences
The number of differences in reported job titles and PRD job titles
Participant feedback, collected during debriefing
Interviewer feedback, collected during bi-weekly calls with OES team
Participants
The field test sample will select 800 past OES respondents with less than 19 employees, the size class that is most likely to complete the OES over the telephone. These cases will be randomly assigned to the three groups. The goal is to conduct 300 telephone follow-up interviews, 100 from each treatment group. We plan to oversample businesses to account for an expected low response rate.
For both samples, certainty units (COCs) and government units will be excluded from the selection process.
Burden Hours
We estimate that we will need to contact up to 800 respondents to conduct the intended 300 interviews. Reviewing the advance letter is expected to take less than 5 minutes and any discussion about the study or respondent concerns is expected to take about 5 minutes. Therefore we estimate a total of 133 hours for recruitment.
We estimate that the telephone data collection will take an average of 15 minutes, including the collection of OES data and the debriefing to complete. With the goal of conducting interviews with 300 businesses, we estimate a total of 75 burden hours.
Data Confidentiality
OES Standard assurances of data confidentiality will be provided in the advance letter (Attachment I).
The Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
“All information and identity provided to OES are kept in strict confidence in accordance with Bureau of Labor Statistics Data Integrity Guidelines,1 and in accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347).”
Attachments
Attachment I: Advance Letter
Attachment II: Interview Guide
Attachment III: Debriefing Guide
Attachment I: Advance Letter
[May xx, 2015]
XYZ Company
123 Main Street
Anytown, VA 12345-1212
Dear [salutation] [last name]:
I am writing to request your assistance. As the nation’s premier source of occupational information, the Bureau of Labor Statistics (BLS) is studying ways to improve our data collection procedures and determine the optimal manner for collecting employment and wage data. We would greatly appreciate your help in this effort.
The Office of Occupational Employment Statistics (OES) at BLS will be calling a limited number of businesses, including yours, which have previously submitted data to OES. We will ask for your current data on the following:
Number of employees
Job titles
Wages
We also plan to ask a few additional questions that we hope will improve our data collection process and enhance data accuracy. Your voluntary participation in this important research effort will help BLS stay abreast of changes in the labor force and better project our nation’s future occupational needs. Your assistance will help us meet the needs of decision makers and data users. Any information you provide in support of this study is strictly confidential.
The Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.
We estimate it will take you 15 minutes to complete this survey. Your participation in this research project is voluntary, and you have the right to stop at any time.
The OMB control number for this survey is 1220-0141. Without a currently valid OMB number BLS could not conduct this survey.
We will be calling you in the near future to collect your most recent data and go over any changes.
Should you have any questions regarding this research effort, please feel free to contact Carrie Jones at (202) 691-6474 or [email protected] for additional information.
In closing, we wish to thank you for your consideration of this request and for your participation.
Sincerely,
Laurie Salmon
Division Chief
Division of Occupational Employment Statistics
Office of Employment and Unemployment Statistics
What is the Occupational Employment Statistics (OES) Survey?
The OES Survey collects data from a sample of establishments and calculates employment estimates by occupation, industry, and geographic area. The survey covers all non-farm industries. Data are collected by the State Workforce Agencies in cooperation with the Bureau of Labor Statistics, US Department of Labor.
How are the data protected?
All information and identity provided to OES are kept in strict confidence in accordance with Bureau of Labor Statistics Data Integrity Guidelines,2 and in accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347).
How will the data gathered from research participants be used to enhance the OES program?
The data collected from research participants will help OES determine if it is possible to collect information consistently from the same establishments.
What will I be asked?
A representative of the Bureau of Labor Statistics will be calling you to collect the above information about your business. You won’t be asked to report for individual employees, rather for each group of employees in an occupation. For example:
Job Title |
Number of Employees |
Annual/Hourly Wage Range |
Accountant (Junior) |
3 |
$14.75 – $18.75 |
Accountant (Senior) |
2 |
$18.75 - $23.99 |
Secretary |
1 |
$9.25 - $11.74 |
Owner |
1 |
$62,920 - $80,079 |
Who do I include in my employee count?
Include: |
Exclude: |
|
|
|
|
|
|
|
|
|
|
|
|
|
What job titles should I report?
Give a descriptive title that captures the various job duties associated with a given occupation (e.g., president; controller; sales manager; programmer; human resources specialist).
What do I include for wages?
Include: |
Exclude: |
|
|
|
|
|
|
|
|
|
|
Attachment II: Interview Guide
PROACTIVE INTERVIEW
Prepare for survey (who completed it previously, review any notes).
Ask for previous respondent, go through standard introduction of self and survey
Last time it was reported that your establishment had X employees.
May calls: How many employees does your establishment have at present [Note to Interviewer: As in production collection, it may be useful to define who counts as an employee.]
June – August calls: How many employees did your company have in May of 2015?
Last time, it was reported that your establishment had (first job title), (second job title), (third job title).
May calls: Do you still have Xs, Ys and Zs? How many X’s does your establishment employ currently? Y’s? Z’s? Are there any other employees we haven’t covered? What do they do?
June – August calls: Did you still employ Xs, Ys and Zs in May 2015? How many X’s? Y’s? Z’s? Are there any other employees we haven’t covered? What do they do?
Finally, last time, it was reported the # Xs made between (specify wage range).
May calls: What is the wage range for that occupation? (repeat for all occupations x wage ranges)
June – August calls: What was the wage range for that occupation in May 2015? (repeat for all occupations x wage ranges)
REACTIVE INTERVIEW
Standard introduction of self and survey
May calls: How many employees does your establishment have at present [Note to Interviewer: As in production collection, it may be useful to define who counts as an employee.]
June – August calls: How many employees did your company have in May of 2015?
If different than PRD: Last time, it was reported that your establishment had X employees. Has there been a change in your employment since then?
If yes: What was the change?
If no: Which figure is accurate for May 2015, X or Y?
If same number, go on to next question
May calls: {Insert: NAICS tailored occupation questions: For example, if it is a restaurant, Interviewer will ask: how many waitresses work there presently? How many bartenders? Any dishwashers? Cooks? Supervisors? GMs? Others not mentioned?}
June – August calls: {Insert: NAICS tailored occupation questions: For example, if it is a restaurant, Interviewer will ask: how many waitresses work(ed) there in May of 2015? How many bartenders? Any dishwashers? Cooks? Supervisors? GMs? Others not mentioned?}
If number of employees per job titles has changed: Last time, it was reported that your establishment had X xxxxs employed there (e.g. 3 plumbers) has that changed?
If number of occupations has changed: Last time, you did/did not have an Y employed there, has that changed?
If same job titles listed, go on to next question
May calls: What is the wage range for that occupation? (repeat for all occupations x wage ranges)
June – August calls: What was the wage range for that occupation in May 2015? (repeat for all occupations x wage ranges)
If there are differences: The wage range(s) for X is different from last time, did that wage rate change?
If same wages listed, go on to debriefing
CONTROL INTERVIWEW
Standard administration of self and OES
May calls: How many employees does your establishment have at present [Note to Interviewer: As in production collection, it may be useful to define who counts as an employee.]
May calls: {Insert: NAICS tailored occupation questions: For example, if it is a restaurant, Interviewer will ask: how many waitresses work there presently? How many bartenders? Any dishwashers? Cooks? Supervisors? GMs? Others not mentioned?}
May calls: What is the wage range for that occupation? (repeat for all occupations x wage ranges)
June – August calls: How many employees did your company have in May of 2015?
June – August calls: {Insert: NAICS tailored occupation questions: For example, if it is a restaurant, Interviewer will ask: how many waitresses work(ed) there in May of 2015? How many bartenders? Any dishwashers? Cooks? Supervisors? GMs? Others not mentioned?}
June – August calls: What was the wage range for that occupation in May 2015? (repeat for all occupations x wage ranges)
Attachment III: Debriefing Guide
PROACTIVE DEBRIEFING
Thank you. In order to simplify the response process for this survey, I have a few questions about your experience and your answers. Understanding how the interview worked for you and your company will help us improve it in the future.
Your company provided data to us on {insert response date}. Do you recall having answered these questions last time?
If no: Do you know who would have provided these data last time?
(if necessary, if obvious that the R used records, probe as below) Did you use any records to answer the questions today?
If yes, what records did you consult?
What information did you use from the records?
I first asked about number of employees at your company. How easy or difficult was that to answer?
I gave you the total number of employees that was reported last time. Was that helpful when reporting this number last time?
Have you had any changes in employment since the last time your company reported?
Has anyone been hired or left the company? {probe to confirm or correct current answer}
Have there been any changes in company size or organization? {probe to confirm or correct current answer}
I also asked about job titles. Again, I gave you what was reported last time. Did having this information make it easier or more difficult to report for this year?
Have there been any changes to the list of job titles or to the number of employees associated with specific job titles since the last time your company reported? For example as a result of promotions, company reorganizations? {probe to confirm or correct current answer}
Finally I asked you about wage ranges, giving you the wage ranges reported last time. Did the wage ranges reported last time make it easier or more difficult to report for this year?
Have there been any changes in wage ranges since the last time your company reported? For example as a result of promotions or new hires? {probe to confirm or correct current answer}
That’s all the questions I have, thanks for your help in this important study! Do you have any comments or feedback on how we could make the OES data-collection process easier for respondents?
Before you go, I have one final request. To allow us to evaluate data accuracy, we ask respondents to send us data from their company. With that data, we’re able to gain a lot of insight into the quality of our data.
Would you be able to provide us with data showing the number of employees, their job titles and wage ranges for May 2015? This might be as simple as a printout from your HR or payroll system.
As is the case with all of our surveys, all of the data provided will be kept secure and confidential.
If yes: Thank you! What is the easiest way for you to get us that data? You can email it, fax it, or mail it in.
If no: I understand. Thanks for your help with this study and have a great day!
REACTIVE DEBRIEFING
Thank you. In order to simplify the response process for this survey, I have a few questions about your experience and your answers. Understanding how the interview worked for you and your company will help us improve it in the future.
Your company provided data to us on {insert response date}. Do you recall having answered these questions last time?
If no: Do you know who would have provided these data last time?
(if necessary, if obvious that the R used records, probe as below) Did you use any records to answer the questions today?
If yes, what records did you consult?
What information did you use from the records?
I first asked about number of employees at your company. How easy or difficult was that to answer?
If difference was identified, reactive PRD given: The number of employees you reported for this year was different from what was reported last time. What did you think when I pointed that out?
Was it helpful for you to know what was reported last time?
If reason for difference wasn’t identified in the interview, probe to identify it now
I also asked about job titles. How easy or difficult was that to answer?
If difference The job titles you reported for this year was different from what was reported last time. What did you think when I pointed that out?
Was it helpful for you to know what was reported last time?
If reason for difference wasn’t identified in the interview, probe to identify it now
Finally I asked you about wage ranges, giving you the wage ranges reported last time. How easy or difficult was that to answer?
If difference The wage ranges you reported for this year was different from what was reported last time. What did you think when I pointed that out?
Was it helpful for you to know what was reported last time?
If reason for difference wasn’t identified in the interview, probe to identify it now
That’s all the questions I have, thanks for your help in this important study! Do you have any comments or feedback on how we could make the OES data-collection process easier for respondents?
Before you go, I have one final request. To allow us to evaluate data accuracy, we ask respondents to send us data from their company. With that data, we’re able to gain a lot of insight into the quality of our data.
Would you be able to provide us with data showing the number of employees, their job titles and wage ranges for May 2015? This might be as simple as a printout from your HR or payroll system.
As is the case with all of our surveys, all of the data provided will be kept secure and confidential.
If yes: Thank you! What is the easiest way for you to get us that data? You can email it, fax it, or mail it in.
If no: I understand. Thanks for your help with this study and have a great day!
CONTROL DEBRIEFING
Thank you. In order to simplify the response process for this survey, I have a few questions about your experience and your answers. Understanding how the interview worked for you and your company will help us improve it in the future.
Your company provided data to us on {insert response date}. Do you recall having answered these questions last time?
If no: Do you know who would have provided these data last time?
(if necessary, if obvious that the R used records, probe as below) Did you use any records to answer the questions today?
If yes, what records did you consult?
What information did you use from the records?
I first asked about number of employees at your company. How easy or difficult was that to answer?
Last time, it was reported that there were XX employees at your company.
Probe for reasons for differences, or for confirmation of the consistency, aiming to identify “truth”
Would it have been useful for you to know how many employees were reported last time when answering for this year?
I also asked about job titles. How easy or difficult was that to answer?
Last time, it was reported that there were [job title1]s, [job title 2], [job title 3] at your company.
Probe for reasons for differences, or for confirmation of the consistency, aiming to identify “truth”
Would it have been useful for you to hear the job titles reported last time when answering for this year?
Finally I asked you about wage ranges, giving you the wage ranges reported last time. How easy or difficult was that to answer?
Last time, it was reported that there were [wage range1]s, [wage range2], [wage range3] at your company.
Probe for reasons for differences, or for confirmation of the consistency, aiming to identify “truth”
Would it have been useful for you to hear the wage ranges reported last time when answering for this year?
That’s all the questions I have, thanks for your help in this important study! Do you have any comments or feedback on how we could make the OES data-collection process easier for respondents?
Before you go, I have one final request. To allow us to evaluate data accuracy, we ask respondents to send us data from their company. With that data, we’re able to gain a lot of insight into the quality of our data.
Would you be able to provide us with data showing the number of employees, their job titles and wage ranges for May 2015? This might be as simple as a printout from your HR or payroll system.
As is the case with all of our surveys, all of the data provided will be kept secure and confidential.
If yes: Thank you! What is the easiest way for you to get us that data? You can email it, fax it, or mail it in.
If no: I understand. Thanks for your help with this study and have a great day!
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Edgar, Jennifer - BLS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |