Nonsubstantive Change Request- Pressure Seal Test

OMB Nonsubstantive Change Request

Department: Commerce

Agency: U.S. Census Bureau

Title: American Community Survey Methods Panel Tests

OMB Control Number: 0607-0936

Expiration Date: 08/31/2018

Request: The Census Bureau plans to conduct additional research under the Methods Panel clearance for the testing of pressure sealed mailers. We propose to conduct a field test of these envelopes as part of the production American Community Survey (ACS) May 2017 panel. No additional burden is expected as a result of this test; rather there will be a change in mail materials for a subset of sampled addresses.

Background: In 2014, the Census Bureau collaborated with Reingold, Inc., a communications and marketing firm, to conduct a comprehensive set of research aimed at enhancing the materials we send to sampled addresses. The goal of this research was to increase public awareness of the ACS, exhibit the value of ACS data, and improve the design of the mail materials in hopes of increasing the self-response rate. This research included several iterative rounds of qualitative and quantitative testing. At the conclusion of the research, one of the recommendations Reingold made was for ACS to include the usage of pressure seal envelopes in the survey mailout materials (Reingold, 2014).

In response to the recommendation, the Census Bureau has designed the Pressure Seal Mailing Materials Test. Because pressure seal mailers have connotations with other important government-issued mail (Reingold, 2014), we hypothesize that this may mean they are more likely to be opened or looked at than other mail methods, thus potentially leading to a higher response. Furthermore, the confidential nature of pressure seal mailers gives the Census Bureau the opportunity to add an Internet user personal identification number (UserID), which cannot be provided on the existing postcards because of privacy and Title 13 restrictions. An increase in self-response would decrease overall ACS production costs, improve reliability, and potentially improve data quality.

Timeline: The test is being planned as part of the ACS May panel, adhering to the same data collection protocols as production ACS. Self-response mailings will begin in late April and continue through May. Telephone interviewing will be conducted for nonrespondents for whom we have a phone number in June, and in-person interviewing will be conducted for a subsample of nonrespondents in July.

Method: Currently, the ACS has five mailing pieces: an initial mailing package, a reminder letter, a paper questionnaire package, a reminder postcard, and an additional reminder postcard. This test will involve the reminder letter, reminder postcard, and an additional reminder postcard. A pressure sealed mailer will be used in place of one or more of these mailings, based on the experimental design shown in the table below. The Census Bureau did not consider using a pressure sealed mailer for the initial package or the paper questionnaire package. Those mailings include other materials that cannot be included in this type of envelope.

Experimental Design for the Pressure Seal Mailing Materials Test

	1st Mailing	2nd Mailing	3rd Mailing*	4th Mailing*	5th Mailing**
Current Production	Initial Package	Reminder Letter	Paper Questionnaire Package	Reminder Postcard	Additional Reminder Postcard
Treatment 1 (Control)	Current	Current	Current	Current	Current
Treatment 2	Current	Pressure seal mailing (trifold) (Attachment A)	Current	Current	Current
Treatment 3	Current	Pressure seal mailing (trifold) (Attachment A)	Current	Current	Pressure seal mailing (trifold) (Attachment C)
Treatment 4	Current	Pressure seal mailing (trifold) (Attachment A)	Current	Pressure seal mailing (bifold) (Attachment B)	Pressure seal mailing (trifold) (Attachment C)

*Only if Internet return not received

** Only if Internet or mail return not received and is not eligible for telephone followup

By eliminating the need for envelopes for the reminder letter, the pressure seal envelope/mailer (see Attachment A) will present a potential cost savings for the ACS program. Treatment 2 is designed to test this cost-saving change. A change to pressure sealed mailers results in an increase in cost for the two postcard mailings, so they were not changed for Treatment 2.

Treatments 3 and 4 will test the use of the pressure sealed mailer (see Attachments B and C) in place of the postcard mailings. Currently, all materials except the postcards include an Internet UserID to encourage Internet response. The use of the pressure sealed mailer allows us to test the inclusion of the UserID in these mailings, which we hypothesize will increase response (and offset the increased cost of the mailing).

Treatment 3 focuses on changing the additional reminder postcard mailing to a pressure sealed mailing but leaving the reminder postcard as is. The reminder postcard is sent out shortly after the paper questionnaire and is used primarily to remind respondents to send back their paper questionnaire (while also reminding them that they can go online). The Census Bureau postulates that it is less important to highlight the UserID in this mailing compared to the fifth mailing, which is sent several weeks later. We hypothesize that respondents would be less likely to still have their paper questionnaire, and encouraging them to go online would be more effective at increasing the response rate.

Despite this hypothesis, the Census Bureau felt it would be useful to test the full potential of pressure sealed mailers in Treatment 4. We are aware that messaging and visual elements become less effective as they are repeated in subsequent mailings. An additional element of the test is the bifold and trifold designs of the pressure sealed mailings. The current postcards are different sizes and printed on different colored card stock to help distinguish them from other mailings a respondent may receive, as well as from each other. Because printing on colored paper proved to be expensive for the pressure sealed mailings, we decided to mimic the design change by changing the way the mailing is folded (see Treatment 4 in the table above).

Treatment 1 will serve as the experimental control and will have all of the same mail materials as current production but will be sorted and mailed at the same time as the experimental treatment materials.

Sample: The monthly ACS production sample of approximately 295,000 addresses is divided into 24 nationally representative groups of approximately 12,000 addresses each for testing. These 24 groups are referred to as methods panel groups. For this test, planned for the May 2017 ACS panel, the Census Bureau will use two randomly assigned methods panel groups for Treatments 1 and 4 and four randomly assigned methods panel groups for Treatments 2 and 3. The total sample size for the experimental test is approximately 144,000 addresses. Finally, the remaining 12 panels will receive current production materials and will be sorted as usual.

We expect to be able to detect differences of approximately one to two percentage points for self-response, depending on the treatments being compared and mailing universe (with 80 percent power and α=0.1; this calculation assumes a 50 percent self-response rate).

Analysis Metrics: We will evaluate the effectiveness of the treatments by looking at the self-response rates at various points in the mailing schedule as well as the final response rates. We will compare the experimental treatments to the control, as well as to each other.

For the reminder letter, because it costs less to send a pressure sealed mailer than a regular envelope, the test will be considered successful as long as the use of the pressure sealed mailer does not decrease response rates.

For the postcards, the use of the pressure sealed mailer costs more. The test will be considered successful if there is an increase in the response rates sufficient to offset the cost.

The ACS currently uses a mailing strategy involving five mailouts. The annual production cost is approximately $16.3M, which includes printing (letters, questionnaires, envelopes, and brochures), docuprinting (letters and postcards), assembly, sorting, postage, labor, etc. for all the mailings.  We estimate each pressure sealed mailer to cost $0.80, which includes printing, sorting, postage, labor, etc.  We expect the cost of the pressure seal mailers to offset overall ACS production costs to both reminder mailings and nonresponse followup operations.  The use of pressure sealer mailers is expected to increase the self-response rate and would improve reliability and data quality. 

Use of Incentives: None.

Burden: There is no change in burden to the public associated with this test. The test is being conducted using production ACS sample. No additional contacts are being made; the way the mail is packaged and the content of the mailing (the inclusion of the UserID) are being modified.

Attachments:

Attachment A - ACS-20(LX)PST, Reminder Letter

Attachment B - ACS-29(LX)PST, Reminder Postcard/Letter

Attachment C - ACS-23(LX)PST, Additional Reminder Postcard/Letter

References:

Reingold, Penn Schoen Berland, Decision Partners, (2014). American Community Survey Messaging and Mail Package Assessment Research: Cumulative Findings. Washington DC: U.S. Census Bureau. Retrieved from

https://www.census.gov/library/working-papers/2014/acs/2014_Walker_02.html

Contact Information:

Please contact the following individuals for further information about this test:

Data Collection:

Agnes Kee

American Community Survey Office

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1516

[email protected]

Statistical Aspects:

Anthony Tersine

Decennial Statistical Studies Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1994

[email protected]

4