Study Plan for NSC 2017 Pressure Seal Test

AMERICAN COMMUNITY SURVEY (ACS)
PROGRAM MANAGEMENT OFFICE	January, 2017
ACS RESEARCH & EVALUATION ANALYSIS PLAN
Pressure Seal Mailing Materials Test
RS17-4-0203
Dorothy Barth and Elizabeth Poehler Authors
Agnes Kee Project Manager
Jennifer Ortman, ACSO Anthony Tersine, DSSD Division Authority

Research & Evaluation Analysis Plan (REAP) Template Revision Log

The Author should update this log with their revision history

Version No.	Date	Revision Description	Author
0.1	1/6/17	Initial Draft	D. Barth
0.2	1/18/17	Second Draft	D. Barth

Table of Contents

1. Introduction 1

2. Literature Review 2

3. Research Questions and Methodology 3

4. Potential Actions 7

5. Major Schedule Tasks 8

6. Project Oversight 8

7. References 8

8. Research & Evaluation Analysis Plan Approvals 10

Attachment A: Pressure Seal Mailing Materials 11

1. Introduction

The American Community Survey (ACS) data provide a wealth of information for both public and private sectors. Government officials rely on the data to inform decisions on matters of public interest and businesses in the private sector use the data to determine business risks and opportunities. However, the statistics provided by the ACS program are only as good as the information we collect from sampled housing units. Unfortunately, most people are unaware of the ACS (Hageldorn, Green, and Rosenblatt, 2014). As such, when asked to participate, they are often reluctant, citing the intrusiveness of the questions, wariness about the security of their information, or the time commitment necessary to participate (Zelenak and Davis, 2013). As a result, we expend more costs to get information from reluctant respondents, often resorting to personal visits to try to collect information from the most reluctant.

To address respondents’ concerns and improve the survey, in 2014 we collaborated with Reingold, Inc., a communications and marketing firm, to conduct a comprehensive set of research aimed at enhancing the materials we send to sampled addresses. The goal of this research was to increase public awareness of the ACS, exhibit the value of ACS data, and improve the design of the mail materials in hopes of increasing the self-administered response rate. This research included several iterative rounds of qualitative and quantitative testing. At the conclusion of the research, one of the recommendations Reingold made was for the ACS to include the usage of pressure seal envelopes in the survey mailout materials (Reingold, 2014). In response to the recommendation we have designed the Pressure Seal Mailing Materials Test.

This test will evaluate the effects on self-response and cost of using pressure seal mailers to replace several pieces of existing ACS mailing materials. Currently ACS production materials include five separate mailings:

1. Initial Mailing Package: Introduction Letter, Frequently Asked Questions (FAQ) Brochure, Multi-Lingual Informational Brochure, and Internet Instruction Card. This mailing urges housing units to respond via the Internet by giving them a UserID on the Internet instruction card.

2. Reminder Letter: A reminder letter sent to all addresses that were sent the Initial Mailing Package, reiterating the request to respond. (Includes UserID).

3. Paper Questionnaire Package: Sent to addresses that have not responded via the Internet. This package includes an Introduction Letter, FAQ Brochure, Paper Questionnaire (includes UserID), Return Envelope, and Internet Instruction Card.

4. Reminder Postcard: A reminder postcard sent to all addresses that were also sent the Paper Questionnaire Package, reiterating the request to respond.

5. Additional Postcard: An additional reminder postcard sent to addresses that have not yet responded and are ineligible for follow-up via computer-assisted telephone interview (CATI), as a last reminder to respond.

This test will involve the reminder letter, the reminder postcard, and the additional postcard mailings.¹ By eliminating the need for an envelope, replacing the reminder letter with a pressure seal mailer will present a cost savings on mailing materials for the ACS program. Pressure seal mailers are typically recognized as being official and important mailings since they are traditionally used for mailing functions such as personal identification numbers, report cards, bill statements or confidential results (Reingold, 2014). Furthermore, the confidential nature of pressure seal mailers gives us the opportunity to add an Internet UserID which cannot be provided on the existing postcards due to privacy and Title 13 restrictions. Currently, all materials except the postcards include an Internet UserID to encourage Internet response. Although pressure seal mailers are more costly than the current ACS production postcards, we hope that the addition of a UserID to the postcard mailings and the more official presentation of the pressure seal mailers will bring an increase in the ACS self-response rate. An increase in self-response would decrease overall ACS production costs and respondent burden as personal interviews are often perceived to be the more intrusive, burdensome modes of data collection (Poe, 2011). In addition, an increase in self-response would improve reliability and could potentially improve data quality.

Keywords: data quality, data collection methods, cost savings, response rates.

2. Literature Review

One of the recommendations that came out of the research done by Reingold, Inc. suggested that the ACS program should perform a mail messaging test with pressure seal mailings. In mail package focus groups and interviews done by Reingold, one of the highest-scoring pieces they tested was a sealed, perforated mailing. They noted that advantages of the perforated mailing included its connotations with other important government-issued mail; a sealed format conveying confidentiality and enabling more explicit instruction about inputting the user ID at the response URL; and a bifold format providing added space.² Based on the effectiveness of the piece in testing, Reingold proposed to send it to all respondents as an initial reminder postcard preceding distribution of the paper questionnaire mailing. (Reingold, 2014).

Statistics Canada used pressure seal mailers for the 2016 Canadian Census. After researching the most cost effective way to mail out letters with variable imaged addresses and secure access codes, this is the design they chose. In a meeting at the Census Bureau in October of 2016, they described the benefits of the pressure seal mailers that informed their decision.

• The mailer eliminated the requirement for a separate envelope and the need for a separate insertion into the envelope, which reduced cost.

• The mailer reduced paper waste and was thus more environmentally friendly.

• They considered the mailer to be a more “official” presentation to the public than a traditional envelope.

• Their printer had the capacity to produce 1.6 million pressure seal mailers a day, making it both a more time efficient and cost efficient option. (Graziadei, 2016).

A small scale mailout research experiment using a pressure seal mailing was conducted as a test for the 2020 Census. That test differed slightly from the proposed ACS test. The Census pressure seal mailing replaced only a postcard mailing and it did not include a login identification for the participant to respond on the Internet. The pressure seal mailer is a more expensive option than a traditional postcard, thus it is only beneficial if it increases self-response. Preliminary findings from the Census pressure seal mailer test indicated that no significant difference was found in response by replacing a postcard with a pressure seal mailer. (Eggleston, forthcoming.)

3. Research Questions and Methodology

1. What is the impact on self-response of changing the second mailing from a reminder letter with a regular envelope to a reminder letter with a tri-fold pressure seal mailer?

2. What is the impact on self-response of changing the fifth mailing to a tri-fold pressure seal mailer that also includes a UserID to encourage online response when the reminder letter (second mailing) is also a tri-fold pressure seal mailer?

3. What is the impact on self-response of changing the fourth mailing to a bi-fold pressure seal mailer with a UserID to encourage online response when the reminder letter (second mailing) and additional postcard (fifth mailing) are tri-fold pressure seal mailers?

4. What would be the relative cost impact of implementing each experimental treatment into a full ACS production year?

Table 1. Experimental Design for the Pressure Seal Mailing Materials Test

	1st Mailing	2nd Mailing	3rd Mailing*	4th Mailing*	5th Mailing**
Current Production	Initial Package	Reminder Letter	Paper Questionnaire Package	Reminder Postcard	Additional Postcard
Treatment 1 (Control)	Current Production Materials
Treatment 2	Current	Pressure seal mailing (trifold)	Current	Current	Current
Treatment 3	Current	Pressure seal mailing (trifold)	Current	Current	Pressure seal mailing (trifold)
Treatment 4	Current	Pressure seal mailing (trifold)	Current	Pressure seal mailing (bifold)	Pressure seal mailing (trifold)

*Only if Internet return not received

**Only if Internet or mail return not received and is not eligible for telephone followup

By eliminating the need for envelopes for the reminder letter (second mailing), the pressure seal mailer will present a cost savings on mailing materials for the ACS program. Treatment 2 is designed to test this cost-saving change. A change to pressure seal mailers would result in an increase in cost for the two postcard mailings, so they were not changed for Treatment 2.

Treatments 3 and 4 will test the use of pressure seal mailers in place of the postcard mailings. Currently, all materials except the postcards include an Internet UserID to encourage Internet response. The use of a pressure seal mailer allows us to test the inclusion of the UserID in these mailings, which we hypothesize will increase response (and offset the increased cost of the mailing).

Treatment 3 focuses on changing the additional reminder postcard (fifth mailing) to a pressure seal mailer but leaving the reminder postcard (fourth mailing) as is. The reminder postcard is sent out shortly after the paper questionnaire and is used primarily to remind respondents to send back their paper questionnaire while also reminding them that they can go online. It seemed less important to us to highlight the UserID in this mailing compared to the fifth mailing, which is sent several weeks later. We hypothesize that, by the time of the fifth mailing, respondents would be less likely to still have their paper questionnaire. Also, we expect that encouraging them to go online would be more effective at increasing the response rate enough to pay for the additional cost of the pressure seal envelope compared to the postcard.

Despite this hypothesis, we felt it would also be useful to test the full potential of the pressure seal mailers. Thus, Treatment 4 will use pressure seal mailers for three of the mailings. Because we are concerned that messaging and visual elements become less effective as they are repeated in subsequent mailings, we included an additional element of design variation to the test mailing materials. The current production postcards are different sizes and are printed on different colored card stock to help distinguish them from other mailings a respondent may receive, as well as from each other. Because printing on colored paper proved to be expensive for the pressure seal mailers, we decided to mimic the design change by changing the way the mailer is folded. Two of the pressure seal mailers will have a tri-fold design and one of the mailers will have a bi-fold design.

The monthly ACS production sample of approximately 295,000 addresses is divided into 24 nationally representative groups (referred to as methods panel groups) of approximately 12,000 addresses each for testing. This test will use the May 2017 ACS production sample. Treatments 1 and 4 will use two randomly assigned methods panel groups each (approximately 24,000 mailing addresses per treatment) and Treatment 2 and 3 will use four randomly assigned methods panel groups each (approximately 48,000 mailing addresses per treatment). The experimental difference between Treatments 2 and 3 is the additional postcard which is sent to non-respondents that are not in the CATI universe. Since so few addresses are sent this mailing piece we needed a larger sample size for these treatments in order to detect any significant differences between the two treatments. Treatment 1 will serve as the experimental control and will have all of the same mail materials as current production but will be sorted and mailed at the same time as the experimental treatment materials. The total sample size for the experimental test is approximately 144,000 addresses. Finally, the remaining 12 panels will receive current production materials and will be sorted as usual.

We expect to be able to detect differences of approximately one to two percentage points for self-response depending on the treatments being compared and mailing universe (with 80 percent power and α=0.1; this calculation assumes a 50 percent self-response rate).

All analyses, except for cost analysis for the fourth research question, will be weighted using the ACS sampling weight (the inverse of the probability of selection). We will use a significance level of α=0.1 when determining significant differences between treatments. For any analysis that involves multiple comparisons, we will adjust for the Type I familywise error rate using the Hochberg method (Hochberg, 1988).

Analysis Metrics:

To evaluate the effectiveness of the experimental treatments, we will calculate self-response return rates as defined below:

Self-Response Return Rate:

Self-Response Return Rate	=	Number of mailable/deliverable sample addresses that provided a non-blank3 return by mail, TQA, or a complete or sufficient partial response by Internet	*100
		Total number of mailable/deliverable sample addresses4

Reminder Postcard Mailing Self-Response Return Rates:

Reminder Postcard Mailing Self-Response Return Rate	=	Number of mailable/deliverable sample addresses sent the Reminder Postcard mailing that provided a non-blank3 return by mail, TQA, or a complete or sufficient partial response by Internet	*100
		Total number of mailable/deliverable sample addresses4 sent the Reminder Postcard mailing

Additional Postcard Mailing Self-Response Return Rates:

Additional Postcard Mailing Self-Response Return Rate	=	Number of mailable/deliverable sample addresses sent the Additional Postcard mailing that provided a non-blank5 return by mail, TQA, or a complete or sufficient partial response by Internet	*100
		Total number of mailable/deliverable sample addresses6 sent the Additional Postcard mailing

We will calculate self-response return rates combined and separately, by mode of data collection. The self-response return rates will be calculated at various points in the data collection cycle; the points will depend upon the comparison being made and are specified below. The Additional Postcard Mailing self-response return rate will only be calculated at the end of CATI. We will calculate self-response return rates before each experimental change occurs to verify that the rates are the same between treatments being compared.⁷

1. What is the impact on self-response of changing the second mailing from a reminder letter with a regular envelope to a reminder letter with a tri-fold pressure seal mailer?

To isolate the impact of replacing only the reminder letter with a pressure seal mailer, we will calculate and compare self-response return rates for Treatment 1 (Control) and Treatments 2 and 3 combined.⁸ Since an increase in self-response will decrease the cost of the following phase of the data collection cycle, we will compare self-response return rates just before the paper questionnaire mailing (third mailing) and total self-response return rates (Internet, TQA, and mail) just before the start of CATI.

We already know that the pressure seal mailer costs less than the current production letter, so we are mostly concerned if the experimental treatment shows a decrease in self-response. Therefore, we will compare Treatment 1 (Control) and Treatments 2 and 3 combined using a one-tailed hypothesis test with the null hypothesis H₀: T1 ≤ (T2 & T3) and the alternative hypothesis

H_A: T1 > (T2 & T3) and a significance level of α = 0.1.

Unless we see a statistically significant decrease in the self-response rate, we will consider changing the reminder letter to a pressure seal mailer a better option than maintaining the current production envelope and letter insert.

2. What is the impact on self-response of changing the fifth mailing to a tri-fold pressure seal mailer that also includes a UserID to encourage online response when the reminder letter is also a tri-fold pressure seal mailer?

To determine the impact of the additional postcard (fifth mailing) we will calculate and compare the additional postcard mailing self-response return rates (combined and by mode) just before the start of CAPI. Any self-response received the day after the additional postcard is mailed will be included in the numerator for these calculations. We will compare Treatment 2 and Treatment 3. We will use a two-tailed test with the null hypothesis H₀: T2 = T3 and the alternative hypothesis H_A: T2 ≠ T3 and a significance level of α = 0.1. This comparison will measure the effect of changing the additional postcard to a pressure seal mailer while also changing the reminder letter to a pressure seal mailer.

3. What is the impact on self-response of changing the fourth mailing to a bi-fold pressure seal mailer with a UserID to encourage online response when the reminder letter (second mailing) and additional postcard (fifth mailing) are tri-fold pressure seal mailers?

To determine the impact of replacing the reminder postcard with a bi-fold pressure seal mailer (fourth mailing), we will calculate and compare self-response return rates (combined and by mode) before the start of CATI and before the start of CAPI for the universe of those mailed the the reminder postcard. Any self-response received the day after the reminder postcard is mailed will be included in the numerator for these calculations. We will use a two-tailed test with the null hypothesis H₀: T3 = T4 and the alternative hypothesis H_A: T3 ≠ T4 and a significance level of α = 0.1.

4. What would be the relative cost impact of implementing each experimental treatment into a full ACS production year?

A decrease in self-response has a negative impact on operational costs, while an increase in self-response has a positive impact on operational cost savings. We will explore the impact on data collection costs and provide a relative cost impact for each experimental treatment compared to the control (Treatment 1). The relative cost impact will account for the difference in costs for printing the pressure seal mailers (as compared to current production materials) as well as differences in CATI and CAPI workloads that result from either increases or decreases in self-response. Since this analysis is looking at survey costs, it will be done unweighted.

4. Potential Actions

Based on the results of this research, the ACS program will consider changes to the design of some of the ACS production mailing materials to include the technology of pressure seal materials. The specific combination of materials to be changed will be decided upon by examining the cost analysis and self-response metrics of each experimental treatment.

5. Major Schedule Tasks

Tasks (minimum required)	Planned Start (mm/dd/yy)	Planned Completion (mm/dd/yy)	To Be Tracked in MAS (Y/N)?
Author drafts REAP, obtains CR feedback, updates and distributes Final REAP	12/01/16	1/23/17
PM/Author conducts response and cost analysis and drafts report	7/14/17	10/6/17
Author obtains CR feedback and updates report	10/10/17	11/6/17
Author develops presentation and conducts briefing to R&E WG	11/7/17	11/21/17
Author updates final report and posts to Internet	11/22/17	12/13/17
Author develops and obtains approval of the R&E Project Record (REPR)	12/14/17	12/28/17
Author presents to ACS Research Group (if desired)	TBD	TBD

6. Project Oversight

This is a Tier 4 project.

7. References

Eggleston, Casey (forthcoming). “August 2016 Experiment with Alternative Reminder Materials and Messaging About How Data Will Be Used.”

Graziadei, Connie (2016). “Overview of the 2016 Canadian Census” [PowerPoint slides]. Informal meeting with division chiefs at the US Census Bureau in October, 2016.

Hageldorn, S., Green, R., and Rosenblatt, A. (2014) “ACS Messaging Research: Benchmark Survey.” Washington, DC: US Census Bureau. Retrieved on January 17, 2017 from

https://www.census.gov/library/working-papers/2014/acs/2014_Hagedorn_01.html

Hochberg, Y. (1988). A Sharper Bonferroni Procedure for Multiple Tests of Significance. Biometrika, 75 (4), 800-802. Retrieved on January 17, 2017 from http://www.jstor.org/stable/2336325?seq=1#page_scan_tab_contents

Poe, T. 2011. “The Pros and Cons of Making the American Community Survey Voluntary.” Prepared Testimony of Ted Poe (TX-02) Before the House Oversight and Government Reform Committee’s Subcommittee on Health Care, District of Columbia, Census, and National Archives. March 6, 2011. Retreived on January 17, 2017 from

https://oversight.house.gov/wp-content/uploads/2012/03/3-6-12-Census-Poe.pdf

Reingold, Penn Schoen Berland, Decision Partners, (2014). American Community Survey Messaging and Mail Package Assessment Research: Cumulative Findings. Washington DC: U.S. Census Bureau. Retrieved on January 17, 2017 from

https://www.census.gov/library/working-papers/2014/acs/2014_Walker_02.html

Zelenak, M.F., and Davis, M. (2013) “Impact of Multiple Contacts by Computer-Assisted Telephone Interview and Computer-Assisted Personal Interview on Final Interview Outcome in the American Community Survey,” Washington, DC: US Census Bureau. Retrieved on January 17, 2017 from https://www.census.gov/library/working-papers/2013/acs/2013_Zelenak_01.html

8. Research & Evaluation Analysis Plan Approvals

Author: Dorothy Barth, DSSD		Approval Date
Author: Elizabeth Poehler		Approval Date
Critical Reviewer: David Raglin, ACSO		Approval Date
Critical Reviewer: Anthony Tersine, DSSD		Approval Date
Critical Reviewer: Nicole Scaniello, SEHSD		Approval Date
Project Management
Project Manager: Agnes Kee, ACSO		Approval Date
Division Authority (ACSO): Jennifer Ortman		Approval Date
Division Authority (DSSD): Anthony Tersine		Approval Date

Attachment A: Pressure Seal Mailing Materials

ACS-20(LX)PST – Experimental Replacement for the Reminder Letter (2^nd Mailing)

ACS-29(LX)PST – Experimental Replacement for the Reminder Postcard (4^th Mailing)

ACS-23(LX)PST – Experimental Replacement for the Additional Postcard (5^th Mailing)

1 Attachment A contains examples of the letters that will be used in the pressure seal mailers for this test.

2 Reingold suggested that these mailings could also include foreign-language text with the additional space.

3 A blank form is a form in which there are no data defined persons and the telephone number listed on the form by respondents is blank.

4 We will remove addresses deemed to be Undeliverable as Addressed by the Postal Service if no response is received.

5 A blank form is a form in which there are no data defined persons and the telephone number listed on the form by respondents is blank.

6 We will remove addresses deemed to be Undeliverable as Addressed by the Postal Service if no response is received.

7 If the rates differ significantly we will make an adjustment to the rates calculated after the experimental treatment is applied to determine the affect of the experiment on return rates.

8 Because our analysis will look at changes in return rates prior to the fifth mailing (which is the only difference between these treatments) we are combining the two treatments for the comparison with the control treatment to provide additional statistical power to the test.