Nonsubstantive Change Request_2017 Mail Design Test

OMB Nonsubstantive Change Request

Department: Commerce

Agency: U.S. Census Bureau

Title: American Community Survey Methods Panel Tests, 2017 Mail Design Test

OMB Control Number: 0607-0936

Expiration Date: 08/31/2018

Request: The Census Bureau plans to conduct additional research under the American Community Survey Methods Panel Tests clearance to test three experimental mail design treatments for the American Community Survey (ACS). The research is designed to reduce respondent burden and address respondent concerns about the perceived intrusiveness of the ACS and the perceived overbearing nature of the mandatory messages found throughout the ACS materials. The Census Bureau proposes to conduct a field test of these three experimental treatments as part of the production ACS August 2017 panel. This will require a change in the mail materials for a subset of the sampled addresses. No additional burden is expected.

Background: In 2014, the Census Bureau collaborated with Reingold Inc., a communications and marketing firm, to conduct a comprehensive set of research aimed at enhancing the materials the Census Bureau sends to sampled addresses. The goal of this research was to increase public awareness of the ACS, exhibit the value of ACS data, and improve the design of the mail materials in hopes of increasing the self-response rate. This research included several iterative rounds of qualitative and quantitative testing. At the conclusion of the research, three of the recommendations that Reingold proposed for the ACS were:

• Use visual design principles to draw attention to key messages and help respondents navigate the ACS materials better.

• Demonstrate the benefits of the ACS participation to local communities.

• Streamline mail packages and individual materials (Reingold, 2014).

In response to these recommendations and recommendations from experts in the field of survey methodology, the Census Bureau designed three experimental mail design treatments to tackle two competing objectives: reduce respondent burden and perception of intrusiveness of the ACS and increase self-response.

Timeline: The test is being planned as part of the ACS August 2017 panel, adhering to the same data collection protocols as production ACS.

Method: Currently, the ACS has five mailing pieces: 1) an initial mailing package, 2) a reminder letter, 3) a paper questionnaire package (if a response has not been received ), 4) a reminder postcard (if a response has not been received), and 5) a final reminder (if an Internet or mail return is not received and the address is not eligible for telephone followup). See Table 1 below for a comparison of the four treatments (one control treatment and three experimental treatments).

Table 1. Description of Mail Materials for all Treatments in the 2017 Mail Design Test

Mailing Classification	Current Production (Control)	Softened Revised Design*	Partial Redesign	Full Redesign
Initial Mailing Outgoing Envelope Frequently Asked Questions Brochure Letter Instruction Card Multilingual Information “Why We Ask”	Your Response is Required by Law 11.5 x 6 (envelope size) YES No Callout Box YES Brochure Included NO	Your Response is Important to Your Community Open Immediately 11.5 x 6 NO Callout Box Softened Wording YES Brochure Included NO	Your Response is Important to Your Community Open Immediately 11.5 x 6 NO Callout Box Softened Wording YES** Included in the letter YES	Your Response is Important to Your Community 9.5 x 4.375 NO Callout Box Wording Changes NO Included in the letter YES
Reminder Letter
Outgoing Envelope	No message	No message	No message	No message
Letter				Wording changes
Questionnaire Package
Outgoing Envelope	Your Response is Required by Law	Your Response is Important to Your Community Open Immediately	Your Response is Important to Your Community Open Immediately	Your Response is Important to Your Community
Questionnaire	Current	Current	Design Changes***	Design Changes***
Frequently Asked Questions Brochure Letter	YES Current	NO Softened Wording	NO Softened Wording	NO Wording Changes
Instruction Card	YES	YES	NO	NO
Return Envelope	YES	YES	YES	YES
Reminder Postcard	Postcard	Postcard Softened	Postcard Softened	Postcard Wording changes
Final Reminder	Postcard	Postcard Softened	Letter**** Softened ‘Final Notice Respond Now’ on envelope	Letter Wording changes ‘Final Notice Respond Now’ on envelope

*The Softened Revised Design for this test will have a few minor changes to the materials from the Summer Mandatory Messaging Test.

**Ideally, the Census Bureau would have removed the instruction card from the Partial Redesign to be consistent with the Full Redesign. Unfortunately, the Partial Redesign letter did not have sufficient space to print the address and the Internet instructions. The Census Bureau will note this limitation in comparing the Partial and Full Redesigns for the initial mailing.

***The front page of the questionnaire will have changes to make it more informative (to replace the need for the instruction card). Thus, it is a methodology change -- not a wording change.

****The wording placement is slightly different in the Partial Redesign Treatment letter compared to the Softened Revised Design Treatment postcard.

The Softened Revised Design treatment was evaluated during the “2015 Summer Mandatory Test” (Oliver et al., 2016). This treatment aimed to improve the way the Census Bureau communicates the importance and benefits of the ACS while updating the look and feel of the ACS materials. However, because it removed or softened the mandatory messaging in the ACS materials, it reduced self-response. For this test, the Census Bureau will examine this treatment again, with minor modifications, and two new experimental treatments–the Partial Redesign and the Full Redesign. The two new treatments are based in part on the Softened Revised Design treatment, changes to ACS methodology (the actual materials provided), and wording (the messages expressed on those materials) suggested by Don Dillman, an expert in survey methodology. The Partial Redesign and the Full Redesign both make changes to ACS methodology. However, the Full Redesign also makes changes to wording in the materials.

Analysis Metrics: The Census Bureau will compare self-response rates and final response rates of each of the experimental treatments to the Control treatment. The experimental treatments will also be compared to each other. The Census Bureau will also assess the impact of each of the experimental treatments compared to the Control treatment on data collection costs and survey estimates. Please see the ACS Research and Evaluation Analysis Plan (Attachment A) for the research questions and methodology.

Additionally, the change in the cover of the questionnaire required the Census Bureau to move a date field from the cover page to the inside page of the questionnaire. The Census Bureau will assess the impact on the data quality by looking at item missing data rates for the date.

Sample: The monthly ACS production sample of approximately 295,000 addresses is divided into 24 groups, where each group contains approximately 12,000 addresses. Each group is a representative subsample of the entire monthly sample, and each monthly sample is representative of the entire yearly sample and the country. The Census Bureau will use two randomly selected groups for each treatment. Hence, each treatment will have a sample size of approximately 24,000 addresses. In total, approximately 96,000 addresses will be used for the four treatments (including the current production treatment group). The current production treatment will have the same mail materials as the rest of production but will be sorted and mailed with the other treatment materials. The sample size will be able to detect differences of approximately 1.25 percentage points between the self-response return rates of the control and experimental treatments. The power of the test, which is 80 percent and where α=0.1, assumes a 50 percent response rate.

Use of Incentives: None.

Burden: There is no change in burden to the public associated with this test. The test is being conducted using production ACS sample. No additional contacts are being made.

Attachments:

Attachment A – ACS Research and Evaluation Analysis Plan

Attachment B – ACS 2017 Mail Design Test Treatment Condition Materials Guide

Attachment C – ACS Methods Panel Burden Hour Tracker

Attachment D – 30 Day Federal Register Notice

References:

Reingold, Penn Schoen Berland, Decision Partners, (2014). American Community Survey Messaging and Mail Package Assessment Research: Cumulative Findings. Washington DC: U.S. Census Bureau. Retrieved on February 22, 2017 from

https://www.census.gov/library/working-papers/2014/acs/2014_Walker_02.html

Oliver, B., Risley, M., & Roberts, A. (2016). 2015 Summer Mandatory Messaging Test. Washington DC, U.S. Census Bureau. Retrieved on February 21, 2017 from

https://www.census.gov/content/dam/Census/library/working-papers/2016/acs/2016_Oliver_01.pdf

Contact Information:

Please contact the following individuals for further information about this test:

Data Collection:

Agnes Kee

American Community Survey Office

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1516

[email protected]

Statistical Aspects:

Anthony Tersine

Decennial Statistical Studies Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1994

[email protected]

4