Note to Reviewer - Testing for OES IDCF

OMB Control Number: 1220-0141

Expiration Date: 03/31/2021

10/31/18

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Robin Kaplan Office of Survey Methods Research
SUBJECT:	Submission of Materials for Testing of Occupational Employment Statistics (OES) Internet Data Collection Facilitity (IDCF)

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 35 hours.

If there are any questions regarding this project, please contact Robin Kaplan at 202-691-7383.

1. Introduction

The Occupational Employment Statistics (OES) program at BLS collects data from establishments on their employment and wage estimates. OES is interested in conducting research to determine how to implement two changes to the Internet Data Collection Facility (IDCF) website where respondents report their data. The first change involves development of a function for respondents to update their contact information. The way IDCF is currently designed, respondents do not have a simple way to keep their contact information up-to-date without leading to extraneous screens asking them to report data. OES would like to develop a set of screens for respondents to simply update their contact information without being prompted to submit data. The second component of this research is to conduct testing to determine whether or not OES respondents are able to report point estimate data for employee wages. Currently, respondents are asked to report wage ranges. OES would like to determine whether it is feasible for respondents to report point estimate data to help increase data precision. However, it is unknown as to whether respondents have access to point estimates, and are willing and able to report wage rates at that level of precision.

For both components of this research, OSMR will design mock-ups of these screens to (a) get feedback from OES respondents, and (b) conduct usability testing on these screens to ensure they are easy to navigate and that respondents are able to complete the tasks of updating their contact information and providing point estimates effectively. The testing will be iterative, making improvements as we go along and test new versions.

To assess these changes, a combination of methods will be used to ensure these changes are optimally designed. We plan to use a combination of cognitive testing and usability testing. We will first conduct interviews with former OES respondents to get their feedback on providing point estimates in the OES. Cognitive interviews are a guided one-on-one discussion with respondents to understand sources of measurement error associated with a survey task. The interviews will focus on getting OES respondents to consider the process of providing point estimates and for potential sources of measurement error. The findings from these interviews will then be used to design instructions and mockups of screens for a usability test, where we will observe participants go through the process of using the mock-ups to update their contact information and provide point estimates in IDCF. The results will be used to identify any problems with the mock-ups, what is difficult to use or understand, and how we can fix those problems.

2. Methodology, Participants, and Burden

This study will use a multi-prong approach (a combination of phone interviews, debriefings, and online surveys) to collect the necessary information in the most time and cost-efficient way possible.

1. Phone calls and debriefings with OES respondents

We will contact former OES respondents who are not currently active OES respondents across a mix of size classes and industries. We will not include multi-unit establisments, or those sampled in the 2018 panel. Follow-back, debriefing research studies are common in establishment survey research methods to assess changes and get respondent feedback. A standard cognitive interview approach will be used over the phone. The interview is expected to take between 10 to 30 minutes, depending on establishment factors (e.g., those who simply upload a spreadsheet will have less information to provide). We will complete up to 20 interviews, and expect we will need to screen about 50 respondents to reach the target of 20 interviews. We will send those OES respondents who have access to their email during the phone call an example of a data collection instrument that collects point estimates vs. ranges. The complete protocol and email is included in Attachment A.

2. In-person meetings with OES respondents

We will conduct in-person follow-up meetings with OES respondents who recently participated in the OES, as it will be fresh in their minds. Follow-back, debriefing research studies are common in establishment survey research methods to assess changes and get respondent feedback. In-person meetings also allow the advantage for us to observe respondents interact with a mock-up in real-time of the IDCF screens and get crucial feedback to help design the screens. We plan to contact up to 40 respondents, expecting to complete 10 in-person interviews with OES respondents at their place of business. The interviews will take no longer than 30 minutes to complete. The protocol for these interviews can be found in Attachment B.

3. OSMR cognitive lab testing

We will conduct traditional usability testing in the OSMR usability lab with participants recruited from the OSMR participant database. Participants will walk through the process of completing the OES report, with mock-ups and screens that mimic the real process of actual OES respondents using IDCF as closely as possible. This will allow us to observe navigation problems with the interface, and any additional usability issues that arise. We plan to test with 10 participants, expecting we will need to screen 20, and each session will last approximately 45 to 60 minutes. See the full protocol in Attachment C and consent form in Attachment D.

The burden hours for all prongs of this research are shown in Table 1 below.

Table 1. Estimated Burden Hours

	# of Participants Screened	Minutes per participant for Screening	Total Screening Burden	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden (hrs)	Total Burden (Screening + Collection)
Phone interviews with OES respondents	50	5	4	20	30	10	14 hours
In-person interviews with OES respondents	40	5	4	10	30	5	9 hours
OSMR Cognitive Lab Testing	20	5	2	10	60	10	12 hours
Total Burden							35 hours

III. Payment to Participants

1. Phone interviews with OES respondents: Respondents who complete the survey over the phone will not receive compensation; the survey is voluntary.

2. In-person interviews with OES respondents: Respondents who participate in an in-person interview at their place of business will not receive compensation; the survey is voluntary.

3. OSMR Cognitive Lab Testing: Participants who come to the OSMR cog lab for usability testing be compensated $40, the standard amount for a one-hour lab session. The total cost will be $400 paid to the participants.

IV. Data Confidentiality

1. Phone interviews with OES respondents: The following statement will be read verbally to participants before completing the survey over the phone, and asked for their verbal consent to participate in the interview before starting:

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

2. In-person interviews with OES respondents: Respondents who participate in the in-person interviews will be given a consent form (see Attachment D) with the CIPSEA confidentiality statement included at the beginning of the interviews.

3. OSMR Cognitive Lab Testing: Participants who participate in the OSMR cognitive lab testing will be given a consent form (see Attachment D) with the CIPSEA confidentiality statement included at the beginning of the study.

List of Attachments

Attachment A: Phone interview protocol with former OES respondents

Attachment B: Phone Screener and Protocol for in-person interviews with OES respondents

Attachment C: Protocol for OSMR lab testing

Attachment D: Consent Form for OSMR testing

5