Nonsubstantive Change Request_2017 Mail Design Test

NCS_2017MailDesignTest_042717.docx

American Community Survey Methods Panel Tests

Nonsubstantive Change Request_2017 Mail Design Test

OMB: 0607-0936

Document [docx]
Download: docx | pdf

OMB Nonsubstantive Change Request

Department: Commerce

Agency: U.S. Census Bureau

Title: American Community Survey Methods Panel Tests, 2017 Mail Design Test

OMB Control Number: 0607-0936

Expiration Date: 08/31/2018


Request: The Census Bureau plans to conduct additional research under the American Community Survey Methods Panel Tests clearance to test three experimental mail design treatments for the American Community Survey (ACS). The research is designed to reduce respondent burden and address respondent concerns about the perceived intrusiveness of the ACS and the perceived overbearing nature of the mandatory messages found throughout the ACS materials. The Census Bureau proposes to conduct a field test of these three experimental treatments as part of the production ACS August 2017 panel. This will require a change in the mail materials for a subset of the sampled addresses. No additional burden is expected.


Background: In 2014, the Census Bureau collaborated with Reingold Inc., a communications and marketing firm, to conduct a comprehensive set of research aimed at enhancing the materials the Census Bureau sends to sampled addresses. The goal of this research was to increase public awareness of the ACS, exhibit the value of ACS data, and improve the design of the mail materials in hopes of increasing the self-response rate. This research included several iterative rounds of qualitative and quantitative testing. At the conclusion of the research, three of the recommendations that Reingold proposed for the ACS were:


  • Use visual design principles to draw attention to key messages and help respondents navigate the ACS materials better.

  • Demonstrate the benefits of the ACS participation to local communities.

  • Streamline mail packages and individual materials (Reingold, 2014).


In response to these recommendations and recommendations from experts in the field of survey methodology, the Census Bureau designed three experimental mail design treatments to tackle two competing objectives: reduce respondent burden and perception of intrusiveness of the ACS and increase self-response.


Timeline: The test is being planned as part of the ACS August 2017 panel, adhering to the same data collection protocols as production ACS.


Method: Currently, the ACS has five mailing pieces: 1) an initial mailing package, 2) a reminder letter, 3) a paper questionnaire package (if a response has not been received ), 4) a reminder postcard (if a response has not been received), and 5) a final reminder (if an Internet or mail return is not received and the address is not eligible for telephone followup). See Table 1 below for a comparison of the four treatments (one control treatment and three experimental treatments).




Table 1. Description of Mail Materials for all Treatments in the 2017 Mail Design Test

Mailing

Classification

Current

Production (Control)

Softened Revised

Design*

Partial Redesign

Full Redesign

Initial Mailing


Outgoing Envelope








Frequently Asked Questions Brochure



Letter




Instruction Card


Multilingual Information


Why We Ask”



Your Response is Required by Law





11.5 x 6 (envelope size)


YES



No Callout Box




YES


Brochure Included


NO



Your Response is Important to Your Community


Open Immediately


11.5 x 6



NO



Callout Box


Softened Wording


YES


Brochure Included


NO



Your Response is Important to Your Community


Open Immediately


11.5 x 6



NO



Callout Box


Softened Wording


YES**


Included in the letter


YES



Your Response is Important to Your Community




9.5 x 4.375



NO



Callout Box


Wording Changes


NO


Included in the letter


YES

Reminder Letter






Outgoing Envelope

No message

No message

No message

No message


Letter








Wording changes

Questionnaire Package





Outgoing Envelope

Your Response is Required by Law

Your Response is Important to Your Community


Open Immediately


Your Response is Important to Your Community


Open Immediately

Your Response is Important to Your Community

Questionnaire

Current

Current

Design Changes***

Design Changes***


Frequently Asked Questions Brochure


Letter

YES



Current


NO



Softened Wording

NO



Softened Wording

NO



Wording Changes


Instruction Card

YES

YES

NO

NO


Return Envelope

YES

YES

YES

YES

Reminder Postcard

Postcard


Postcard


Softened

Postcard


Softened

Postcard


Wording changes

Final Reminder

Postcard



Postcard


Softened

Letter****


Softened


Final Notice Respond Now’ on envelope

Letter


Wording changes


Final Notice Respond Now’ on envelope

*The Softened Revised Design for this test will have a few minor changes to the materials from the Summer Mandatory Messaging Test.

**Ideally, the Census Bureau would have removed the instruction card from the Partial Redesign to be consistent with the Full Redesign. Unfortunately, the Partial Redesign letter did not have sufficient space to print the address and the Internet instructions. The Census Bureau will note this limitation in comparing the Partial and Full Redesigns for the initial mailing.

***The front page of the questionnaire will have changes to make it more informative (to replace the need for the instruction card). Thus, it is a methodology change -- not a wording change.

****The wording placement is slightly different in the Partial Redesign Treatment letter compared to the Softened Revised Design Treatment postcard.


The Softened Revised Design treatment was evaluated during the “2015 Summer Mandatory Test” (Oliver et al., 2016). This treatment aimed to improve the way the Census Bureau communicates the importance and benefits of the ACS while updating the look and feel of the ACS materials. However, because it removed or softened the mandatory messaging in the ACS materials, it reduced self-response. For this test, the Census Bureau will examine this treatment again, with minor modifications, and two new experimental treatments–the Partial Redesign and the Full Redesign. The two new treatments are based in part on the Softened Revised Design treatment, changes to ACS methodology (the actual materials provided), and wording (the messages expressed on those materials) suggested by Don Dillman, an expert in survey methodology. The Partial Redesign and the Full Redesign both make changes to ACS methodology. However, the Full Redesign also makes changes to wording in the materials.

Analysis Metrics: The Census Bureau will compare self-response rates and final response rates of each of the experimental treatments to the Control treatment. The experimental treatments will also be compared to each other. The Census Bureau will also assess the impact of each of the experimental treatments compared to the Control treatment on data collection costs and survey estimates. Please see the ACS Research and Evaluation Analysis Plan (Attachment A) for the research questions and methodology.

Additionally, the change in the cover of the questionnaire required the Census Bureau to move a date field from the cover page to the inside page of the questionnaire. The Census Bureau will assess the impact on the data quality by looking at item missing data rates for the date.

Sample: The monthly ACS production sample of approximately 295,000 addresses is divided into 24 groups, where each group contains approximately 12,000 addresses. Each group is a representative subsample of the entire monthly sample, and each monthly sample is representative of the entire yearly sample and the country. The Census Bureau will use two randomly selected groups for each treatment. Hence, each treatment will have a sample size of approximately 24,000 addresses. In total, approximately 96,000 addresses will be used for the four treatments (including the current production treatment group). The current production treatment will have the same mail materials as the rest of production but will be sorted and mailed with the other treatment materials. The sample size will be able to detect differences of approximately 1.25 percentage points between the self-response return rates of the control and experimental treatments. The power of the test, which is 80 percent and where α=0.1, assumes a 50 percent response rate.

Use of Incentives: None.


Burden: There is no change in burden to the public associated with this test. The test is being conducted using production ACS sample. No additional contacts are being made.


Attachments:

Attachment A – ACS Research and Evaluation Analysis Plan

Attachment B – ACS 2017 Mail Design Test Treatment Condition Materials Guide

Attachment C – ACS Methods Panel Burden Hour Tracker

Attachment D – 30 Day Federal Register Notice


References:

Reingold, Penn Schoen Berland, Decision Partners, (2014). American Community Survey Messaging and Mail Package Assessment Research: Cumulative Findings. Washington DC: U.S. Census Bureau. Retrieved on February 22, 2017 from

https://www.census.gov/library/working-papers/2014/acs/2014_Walker_02.html


Oliver, B., Risley, M., & Roberts, A. (2016). 2015 Summer Mandatory Messaging Test. Washington DC, U.S. Census Bureau. Retrieved on February 21, 2017 from

https://www.census.gov/content/dam/Census/library/working-papers/2016/acs/2016_Oliver_01.pdf


Contact Information:

Please contact the following individuals for further information about this test:


Data Collection:

Agnes Kee

American Community Survey Office

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1516

[email protected]


Statistical Aspects:

Anthony Tersine

Decennial Statistical Studies Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1994

[email protected]

4


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleACS Methods Panel -2017 Mail Design Test
AuthorErica L Olmsted Hawala
File Modified0000-00-00
File Created2021-01-21

© 2024 OMB.report | Privacy Policy