Attachment A - CES Enrollment Test Plan (2019)

Note to Reviewer – Attachment A

CES Enrollment Package Test Plan

2019

Goal: To test the impact of alternative packages on the CES enrollment rate. The CES program believes that the current package does not adequately establish the legitimacy, value, or simplicity of the survey request.

Study Design: The study involves a series of three experiments where the treatments of subsequent experiments are dependent on the outcome of earlier experiments. New CES sample units enrolled by the Kansas City Data Collection Center (DCC) will be randomly assigned to a treatment condition and mailed their assigned enrollment package. This study will have three phases, with the outcome of each phase being used to inform the versions tested in the subsequent phase.

The following are the conditions under consideration¹:

1. Advance letter test. This experiment will test the effectiveness of mailing an advance letter to sampled units prior to enrollment. The experiment will be fielded from July to September with evaluation of performance in October.

2. Folder test. Units will be assigned to receive the current folder, an updated folder, or no folder. Depending on the outcome of experiment 1, sampled units will or will not be mailed an advance letter. The experiment will be fielded from November to January with evaluation of performance in February.

3. Form vs letter test. The enrollment package will either contain a survey form or a letter (similar to cover of survey form). The type of folder and whether or not an advance letter is sent will depend on outcomes from experiments 1 and 2. The experiment will be fielded from March to May with evaluation in June.

After the enrollment package mailing, normal CES enrollment procedures will be followed. The exception to this is in the third test; units who did not receive a form with their mailing, the DCC staff should direct respondents to the PDF form online to refer to during the call.

For each case, some additional information, listed in the Outcome Measures below, should be tracked to allow for a full analysis of the impact of the enrollment mailings.

Pilot Test (only for form vs letter test): to evaluate the feasibility of the third test, a pilot test will be conducted. Two Interviewer 2’s (experienced staff) will test the condition where units receive a letter and no form. These interviewers will follow the new procedures for a month and collect the requested paradata. Their feedback and results will be used to determine whether or not to conduct the third test.

Sample: The Kansas City DCC will be used, with all enrollment cases from that DCC assigned to one of the treatment conditions in a way that evenly distributes key establishment characteristics (e.g., size and industry). Monthly sample size is approximately 1,600 cases.

Outcome Measures. The Office of Field Operations & DCC staff will determine which of the below outcome measures can be systematically tracked for the test, and the mechanism for doing so:

• Address refinement outcome: no address refinement, verified with business, verified with online or other sources

• Date enrollment package mailed out

• Mailing Address: Name of respondent or no-name

• Date of enrollment

• Enrollment final status code: enrolled, not enrolled

• Number of calls to enroll

• Status of mailing during enrollment call: didn’t receive, received but didn’t have during call, received and did have during call

Cost

• There may be some cost associated with printing and mailing the alternative enrollment packages. Additionally, for the groups who do not receive the CES form, a separate mailing will have to be done to send them the form post-enrollment (though there may be cost savings if no folder is sent with the letter).

• Additionally, depending on what paradata is currently captured, DCC staff may need to capture some information manually.

Timeline

• Test will be conducted under the current CES OMB clearance. The phases of the testing will be approved through separate OMB requests.

• Enrollment rate analysis conducted one month after every experiment, followed by a repeat analysis three months after to look at ‘final’ enrollment status. We also recommend analyzing response rates for 6 months to identify any impact on ongoing response rates.

• After analysis of the final test has concluded, a final report, including recommendations, will be produced.

• We propose the following timeline:

  • April: Analyze current enrollment patterns – Complete.

  • May: Finalize test plan, submit OMB clearance request for the Advance letter, create packages for Pilot testing. Work with OPUBBS to develop new folder for experiment two.

  • June: Train all interviewers for experiment 1.

  • July to September: Field Advance letter test.

  • October: Analysis of Advance letter test. Train interviewers for experiment 2. Submit OMB clearance request for the Folder test.

  • November to January: Field Folder test.

  • January: Pilot Form v letter test.

  • February: Analysis of Folder test. Train interviewers for experiment 3. Submit OMB clearance request for the Form v letter test.

  • March to May: Field Form vs letter test.

  • June: Analysis of Form vs letter test.

  • September: Submit report and recommendation.

Other Recommendations

• DCC staff should be trained on the importance of the CES so they can “sell the survey” as needed during the enrollment process. We recommend creating a video or PowerPoint to train them on the key points. This could be implemented in a separate DCC during the test to evaluate its impact (comparing enrollment rates before and after within that DCC).

1 Depending on the results of the analysis of current enrollment rates, we may add or change a condition. E.g., if there are specific types of establishments who systematically do not enroll, we may propose a package tailored to them.

2