ACS Methods Panel Testing NSC Change Request

OMB Non-substantive Change Request

Department: Commerce

Agency: U.S. Census Bureau

Title: American Community Survey Methods Panel Tests

OMB Control Number: 0607-0936

Expiration Date: 12/31/2015

2015 Replacement Mail Questionnaire Package Test and 2015 Mail Contact Strategy Modification Test

Motivation

The public are not sufficiently aware of the American Community Survey (ACS) or its value to them. In addition, many people believe the questions are intrusive or burdensome and so are opposed to completing it. This public resistance to completing the survey lowers response rates and causes less reliable estimates.

We partnered with Reingold, Inc. to conduct messaging and mail package assessment research that we expect will help us address frequent questions and concerns we hear about the ACS surrounding privacy, intrusiveness, value of the data, and burden of completion. With Reingold we conducted research to perform assessments of the ACS mail package to: increase response rates for participants (and boost early response rates); to link the value of ACS and Census data to key areas of interest for various stakeholder groups; and to increase public awareness, support, and use, of ACS data. Specifically the contractor conducted research projects and creative development for the alternative designs and messages in the mail package in preparation for field testing conducted by the Census Bureau (Hagedorn et al., 2014).

Additionally, the Census Bureau solicited feedback from Dr. Don Dillman, respected expert on mail survey methods, on the current ACS mail materials and an early version of the proposed revised designs from Reingold. Dr. Dillman provided many recommendations for revisions to the ACS mail methodology and the materials. Many of those recommendations were incorporated into the later designs by Reingold, while some were not.

Treatments

Although Reingold conducted qualitative testing to assess some of the design changes, only field testing can measure the impact these changes have on a respondent’s likelihood to complete the survey. In order to measure the impact of various design and methodological changes in the ACS mail packages, we propose a series of field tests to minimize confounding effects in the measurement of various changes. Specifically, we plan to conduct two tests this fiscal year, with an additional test planned later based on the results of the first two tests (a separate change request will be submitted for this third test at a later time). The first test is the Replacement Mail Questionnaire Package Test; the second test is the Mail Contact Strategy Modification Test.

1. Replacement Mail Questionnaire Package Test

This test will consider two experimental changes in a partially factorial design. The first change measures the impact of removing either the ACS-34 RM Instruction Card, the ACS-30 Instruction Guide, or both from the replacement mail questionnaire package, in response to Dillman’s suggestion that fewer pieces in this package would help encourage response. The second change involves modifying the approach to how we present the choice option in the replacement mail questionnaire package, when we remove the instruction card. The treatments are described below:

Control Treatment – The mail materials in this treatment are identical to the mail materials used in the production ACS. The replacement mail package includes the ACS-34 RM instruction card and ACS-30 instruction guide. There are no revisions to the messages in the package.

Remove Card and Guide, Emphasis on Choice – This replacement mail package does not include the ACS-34 RM instruction card nor the ACS-30 instruction guide. There are no revisions to the messages in the package.

Remove Card and Guide, Reduced Emphasis on Choice – This replacement mail package does not include the ACS-34 RM instruction card nor the ACS-30 instruction guide. The letter (ACS-14(LX)RPT) is revised to reduce the emphasis on choice and to provide clear instructions on how to respond.

Remove Guide Only, Emphasis on Choice – This replacement mail package does not include the ACS-30 instruction guide. The ACS-34 RM instruction card is still included. There are no revisions to the messages in the package.

Remove Card Only, Reduced Emphasis on Choice – This replacement mail package does not include the ACS-34 RM instruction card. The ACS-30 instruction guide is still included. The letter (ACS-14(LX)RPT) is revised to reduce the emphasis on choice and to provide clear instructions on how to respond.

2. Mail Contact Strategy Modification Test

This test will consider two experimental changes in a fully factorial design. The first change involves eliminating the pre-notice letter, but converting the reminder postcard into a letter containing specific log in instructions, and sending the initial package on a slightly earlier schedule. Note that in September 2014, we conducted a field test using a portion of the production sample to measure the impact of dropping the pre-notice letter, but we did not create a replacement letter in the way Dillman had proposed, so additional testing is necessary to measure the impact of that recommendation. In order to separate the confounding effects of sending the initial package earlier and converting the reminder postcard into a letter, this change will also be tested without converting the current first reminder postcard into a letter.

The second change involves modifying the approach to the additional reminder postcard that is currently sent only to households not included in the Computer Assisted Telephone Interview (CATI) operation. During this test, we would examine sending the card to all nonresponding households instead of only those households not included in the CATI operation. The treatments are described below:

Control Treatment – The mail materials in this treatment are identical to the mail materials used in the production ACS. Materials are mailed on the production ACS schedule. The following materials are included:

• Prenotice letter (including multi-lingual brochure)

  • Initial package

  • Reminder postcard

  • Replacement package

  • Reminder postcard

  • Additional postcard for households not in CATI

Treatment 1 – This treatment removes the pre-notice letter. Additionally the first reminder is sent as a letter instead of a postcard. All other materials are the same as production. The materials are sent on a modified schedule. The following materials are included (changes from the control are noted in bold):

• Initial package¹ (including multi-lingual brochure)

• Reminder letter² (ACS-20(LX)CST)

• Replacement package

• Reminder postcard

  • Additional postcard for households not in CATI

Treatment 2 – This treatment removes the pre-notice letter. The materials are sent on a modified schedule. The following materials are included (changes from the control are noted in bold):

• Initial package (including multi-lingual brochure)

• Reminder Postcard

• Replacement package

• Reminder postcard

  • Additional postcard for households not in CATI

Treatment 3 – The mail materials in this treatment are identical to the production ACS except that the additional postcard reminder currently only sent to households not in CATI is now sent to all nonresponding households. The materials are sent on the production schedule. The following materials are included (changes from the control are noted in bold):

• Prenotice letter (including multi-lingual brochure)

• Initial package

• Reminder postcard

• Replacement package

• Reminder postcard

• Additional postcard³ sent to all nonresponding households (ACS-23(X) CST)

Treatment 4 – This treatment removes the pre-notice letter. Additionally the first reminder is sent as a letter instead of a postcard and the additional postcard reminder is sent to all nonresponding households (instead of households not in CATI). The materials are sent on a modified schedule. The following materials are included (changes from the control are noted in bold):

• Initial package (including multi-lingual brochure)

• Reminder letter (ACS-20(LX)CST)

• Replacement package

• Reminder postcard

• Additional postcard sent to all nonresponding households (ACS-23(X) CST)

Treatment 5 – This treatment removes the pre-notice letter. Additionally the additional postcard reminder is sent to all nonresponding households (instead of households not in CATI). The materials are sent on a modified schedule. The following materials are included (changes from the control are noted in bold):

• Initial package (including multi-lingual brochure)

• Reminder Postcard

• Replacement package

• Reminder postcard

  • Additional postcard sent to all nonresponding households (ACS-23(X) CST)

Sample

To field this test, we plan to use ACS production (clearance number: 0607-0810, expires 6/30/2016). Thus, there is no increase in burden from this test since the treatment will result in approximately the same burden estimate per interview (40 minutes). We have divided the monthly production sample into 24 nationally representative groups of approximately 12,000 addresses each.

For the Replacement Mail Questionnaire Package Test, planned for the March 2015 panel, we will use one randomly assigned group for each of the four experimental treatment groups. The remaining cases in the March 2015 panel will comprise the control and receive all standard ACS mailings. For the Mail Contact Strategy Modification Test, planned for the April 2015 panel, we will use one randomly assigned group for each of the five experimental treatment groups. The remaining cases in the April 2015 panel will comprise the control and receive all standard ACS mailings. As we are using production cases for the tests, both tests will run through the complete 3-month data collection.

Our primary evaluation measures for the Replacement Mail Questionnaire Package test are the unit response rate and item response rates. Our primary evaluation measure for the Mail Contact Strategy Modification Test is unit response rates. Comparing the unit response rates between two treatments allows us to detect a 2.0 percentage point difference with 80% power and α=0.1. Comparisons between two treatments of 12,000 addresses each will allow us to measure approximately a 1.4 percentage point difference in item nonresponse at the household level with 80% power and α=0.1.

With approximately 12,000 addresses per treatment and four experimental treatments, the total sample size for the Replacement Mail Questionnaire Test is approximately 48,000 addresses. Similarly, with approximately 12,000 addresses per treatment and five experimental treatments, the total sample size for the Mail Contact Strategy Modification Test is approximately 60,000 addresses.

References

Hagedorn, S., Panek, M., and Green, R (2014). American Community Survey Mail Package Research: Online Visual Testing. Available at:

https://www.census.gov/acs/www/Downloads/library/2014/2014_Hagedorn_04.pdf

1 The letter included in this package (ACS-13(LX)PT) is revised to remove references to receiving a pre-notice letter.

2 This letter includes clear instructions to log in, an explicit reference to the userid, and a mandatory message. This letter will be sent seven days after the initial package whereas the current schedule for mailing sends the reminder postcard three days after the initial package (seven days after the pre-notice).

3 Language in the current card referring to possible personal visits is revised to refer to future contacts.