CES Enrollment Test Plan (2020)

CES Enrollment Package Test

Draft Outline 4.18.19

Updated 2.25.20

Goal: To test the impact of alternative packages on the CES enrollment rate. The CES program believes that the current package does not adequately establish the legitimacy, value, or simplicity of the survey request.

Study Design: The study involves a series of three experiments where the treatments of subsequent experiments are dependent on the outcome of earlier experiments. New CES sample units enrolled from the Kansas City Data Collection Center (DCC) will be randomly assigned to a treatment condition and mailed their assigned enrollment package. This study will have three phases, with the outcome of each phase being used to inform the versions tested in the subsequent phase.

The following are the conditions under consideration¹:

1. Advance letter test. This experiment will test the effectiveness of mailing an advance letter to sampled units prior to enrollment. The experiment will be fielded from July to September 2019 with evaluation of performance in October 2019.

2. Folder test. Units will be assigned to receive the current folder, an updated folder, or no folder. Depending on the outcome of experiment 1, sampled units will or will not be mailed an advance letter. The experiment will be fielded from November 2019 to January 2020 with evaluation of performance in February 2020.

3. Form vs letter test. The enrollment package will either contain a survey form or a letter (similar to cover of survey form). The type of folder and whether or not an advance letter is sent will depend on outcomes from experiments 1 and 2. The experiment will be fielded for three months. The three treatment conditions are listed in Table 1.

Table1

Treatment 1	Treatment 2
Address refinement	Address refinement
Form mailed	Letter Mailed
Phone contact	Phone contact
	Form Emailed

After the enrollment package mailing, normal CES enrollment procedures will be followed. The exception to this is in the third test. For units who did not receive a form with their mailing and are scheduled to receive an email after the phone contact, the DCC staff may direct respondents to the PDF form online to refer to during the call.

For each case, some additional information, listed in the Outcome measures, should be tracked to allow for a full analysis of the impact of the enrollment mailings.

Pilot Test (only for form vs letter test): To evaluate the feasibility of the third test, a pilot test will be conducted. A small number of Interviewers will test the condition where units receive a letter and no form (treatments 2). These interviewers will use the new procedures for a small number of cases and collect the requested paradata. Their feedback and results will be used to determine whether or not it is feasible (in terms of timing and logistics) to mail and/or email the form after making initial contact.

Sample: The Kansas City DCC will be used, with all enrollment cases from that DCC assigned to one of the treatment conditions in a way that evenly distributes key establishment characteristics (e.g., size and industry). Monthly sample size is approximately 1,600 cases.

Outcome Measures: The Office of Field Operations & DCC staff will determine which of the below outcome measures can be systematically tracked for the test, and the mechanism for doing so:

• Address refinement outcome: no address refinement, verified with business, verified with online or other sources

• Date enrollment package mailed out

• Mailing Address: name of respondent or no-name

• Date of enrollment

• Enrollment final status code: enrolled, not enrolled

• Number of calls to enroll

• Status of mailing during enrollment call: didn’t receive, received but didn’t have during call, received and did have during call

Cost

• There may be some cost associated with printing and mailing the alternative enrollment packages.

• Additionally, for the groups who do not receive the CES form, a separate mailing will have to be done to send them the form post-enrollment (though there may be cost savings if no folder is sent with the letter).

• Additionally, depending on what paradata is currently captured, DCC staff may need to capture some information manually.

Timeline

• Tests will be conducted under the current CES OMB clearance. Tests 1 and 3 will be approved through separate non-substantive change requests.

• Enrollment rate analysis will be conducted one month after every experiment, followed by a repeat analysis three months after to look at ‘final’ enrollment status. We also recommend analyzing response rates for 6 months to identify any impact on ongoing response rates.

• After analysis of the final test has concluded, a final report, including recommendations, will be produced.

• We propose the following timeline:

  • April 2019: Analyze current enrollment patterns - Complete

  • May 2019: Finalize test plan, submit OMB clearance request for experiment 1 (Advance letter test), create packages for experiment 3 (Form vs letter pilot testing). Work with OPUBBS to develop new folder for experiment two - Complete

  • June 2019: Train all interviewers for experiment 1 - Complete

  • July to September: 2019 Field Advance letter test - Complete

  • October 2019: Analysis of Advance letter test. Train interviewers for experiment 2 (Folder test) - Preliminary results reviewed, waiting on full dataset for analysis

  • November 2019 to January 2020: Field Folder test – Currently in field

  • February 2020: Analysis of Folder test. Train interviewers for experiment 3.

  • TBD: Pilot Form vs Letter test

  • TBD (3 months): Field Form vs letter test

  • TBD: Analysis of Form vs letter test

  • TBD: Submit report and recommendation

Other Recommendations

• DCC staff should be trained on the importance of the CES so they can “sell the survey” as needed during the enrollment process. We recommend creating a video or PowerPoint to train them on the key points. This could be implemented in a separate DCC during the test to evaluate its impact (comparing enrollment rates before and after within that DCC).

1 Depending on the results of the analysis of current enrollment rates, we may add or change a condition. E.g., if there are specific types of establishments who systematically do not enroll, we may propose a package tailored to them.

3