Note to Reviewer - Flex Enrollment Letter Test (2022)

MEMORANDUM FOR: Reviewer of 1220-0011

FROM: Nicholas Johnson

Bureau of Labor Statistics

Office of Employment and Unemployment Statistics

Division of Current Employment Statistics

Current Employment Statistics (CES)

SUBJECT: Special Enrollment Letter Test

Early in 2022, the Bureau of Labor Statistics (BLS) Current Employment Statistics (CES) program would like to conduct a test of newly selected sample respondents by providing the option of self-enrolling into the CES program by web. The test involves mailing a special enrollment letter to these respondents with all the instructions necessary to enroll and submit their microdata online. In Supporting Statement Part B, Item #4, the CES program noted its ongoing efforts to test contact strategies to identify improvements in materials sent to respondents to increase response and reduce burden.  This test is part of those refinement efforts.

Normally, the CES program sends new respondents a letter and then follow up with a phone call to enroll them and collect their data. The idea behind this test is a proof of concept to see if we can direct respondents to report online who would have otherwise ignored our phone calls. It’s becoming increasingly more difficult to reach respondents by phone, thus making our traditional enrollment method less successful. If we can reach a segment of our respondents with this new approach, we will not only improve response rates, but also reduce respondent burden by allowing respondents to choose their preferred method for enrollment.

The Special Enrollment Letter (Attachment A) was developed in consultation with the BLS Office of Survey Methods Research and the test is being performed with consultation from that office. We intend to send this letter to a sample size of 400 units in early 2022. Details of the test are outlined in Attachment B. If the test is successful, the CES program will seek OMB approval to revise the existing collection to include the use of the special enrollment letter.

Attachment A – Special Enrollment Letter

Attachment B – Test Plan

Attachment B: Test Plan

CES will select a random sample of 800 new CES sample units and randomly split into two groups of 400. The test group will receive the attached letter that provides an option to self-enroll online or phone. The control group will be enrolled using the standard CES enrollment methodology (i.e., by phone). The primary research objective of this experiment is to identify if the test procedure impacts enrollment response rates. A secondary goal is to identify if there is substantial interest among respondents to self-enroll online (i.e., did they opt to attempt enrollment online?). The results of both of these research objectives will determine if CES will expand availability of self-enrollment to all newly sampled respondents and if it is worthwhile to commit additional development resources to expand self-enrollment functionality on our current data collection website.

Sample size determination:

Power level: 0.90. This is the probability of correctly rejecting the null hypothesis if it is false

Alpha: 0.05. The probability of falsely rejecting the null hypothesis even though it is true 

Null hypothesis: There is no difference in enrollment response rates between the control group (phone only) and the test group (web + phone)

Current standard enrollment response rates: 45%

Based on prior research, estimated response propensity for self-enrollment by web: 30%

Estimated response by phone for respondents who choose to not self-enroll by web: At least 15%

Methods:

Assume a simple random sample.

Use pwr.t.test() function in R under pwr library to perform a two-sided test.

Results:

Response Rate difference we wish to detect	Sample size for the test group
3%	366*

*Will be rounded to 400 to account for sample units that may be determined to be ineligible.