NSC--Request for Additional burden hours to conduct Further Internet Response Research

OMB Non-substantive Change Request

Department: Commerce

Agency: U.S. Census Bureau

Title: American Community Survey Methods Panel Tests

OMB Control Number: 0607-0936

Expiration Date: 12/31/2012

ACS Internet Notification Strategies Follow-up Test

Motivation:

Preliminary results from the 2011 ACS Internet Mode Test indicate that the Internet notification strategies were successful in not only driving response to the Internet, but in keeping overall response very close to ACS production. Offering the choice between Internet and mail produced preliminary response rates that tracked very closely to offering mail only. This is a positive finding in light of findings from the first ACS Internet test in 2000 where response decreased by over five percentage points (on average) when respondents were offered a choice between mail and Internet (Griffin et al., 2001).

Surprisingly, even the preliminary response rates for the push Internet strategy (where we removed the paper questionnaire in the first mailing, and moved up the replacement questionnaire mailing by one week) were close to the rates when we offered mail only. This is the first test where we have seen a push strategy stay competitive with other strategies. Moreover, almost two-thirds of response in the push strategy came from Internet. Given the substantial cost savings associated with moving to this method, we propose a follow-up test in November 2011 to verify and validate current findings as well as test some changes to the mailing pieces that we hope will improve response.

Stratification:

As in the 2011 ACS Internet Mode Test, we propose testing the Internet notification strategies among two segments of the population, called Targeted and Not Targeted. The Targeted group includes areas where Internet penetration and preference is highest, and the Not Targeted group contains the balance. While the preliminary results from the 2011 ACS Internet mode test suggest similar findings between the two groups, the slight differences that exist may warrant the need for different notification strategies by group when we get the final results. Thus, as in the 2011 Internet test, the results from this follow-up test will indicate which notification strategy achieved the highest response rates (as well as highest cost savings) in Targeted and Not Targeted groups. We will then compare the strategy that received the highest response against ACS production response rates, by strata. Our intent is to use the best strategy for each group if and when we introduce the Internet in ACS production.

Treatments:

We would like to retest two of our current strategies for notifying sampled units about the Internet response, the Prominent Offer (Choice) and Push Internet of Modified Mailing schedule (#1 and #3 below). Retesting these strategies will not only allow us to verify the findings from our current test, but these strategies will serve as a baseline so we can determine if any of the proposed changes enhance response. Moreover, ACS underwent some operational changes (such as a new procedure to remove cases where the first questionnaire was returned as undeliverable by the postal service) and we need to see whether there is an impact on our current findings.

We would also like to test potential enhancements to the two treatments we are retesting. In the Prominent Offer (choice) design, we would like to add icons of a computer and pencil where we offer the choice of Internet versus paper on the questionnaire, respectively. We refer to this strategy as Prominent Choice with Icons (#2 below). We feel this enhancement may potentially drive more response to the Internet, given the evidence from cognitive testing that respondents tend to pay attention to the questionnaire and disregard the other mail materials. We will compare this new strategy to the Prominent Offer (Choice) strategy to assess the impact of the icons on overall response and response by Internet.

Moreover, we would like to test sending the replacement questionnaire for the Prominent Choice panel one week earlier, similar to the Push Internet on Modified Mailing panel. We call this strategy Prominent Choice with Icons on Modified Mailing Schedule (#3 below). By including this panel, we can tease out the effect of accelerating the replacement questionnaire mailing under the Choice condition by comparing to the Prominent Choice with Icons strategy.

Finally, we also would like to test an enhancement to the Push Internet on Modified Mailing Schedule treatment. Our main objective with the enhancement is to increase overall response. Given that the replacement questionnaire is mailed a week earlier than the ACS production schedule in this treatment, we would like to send another reminder postcard after the replacement questionnaire to encourage response to that mailing. This would be a new postcard (second reminder postcard). We call this strategy Push Internet on Modified Mailing Schedule with Second Reminder Postcard (#5 below). This postcard will be different from the first reminder postcard in color and text to ensure respondents do not easily dismiss it. We will compare this strategy to the Push Internet on Modified Mailing Schedule to assess the effects of the additional postcard.

1. Prominent Offer (Choice) - Households are given a choice of completing the ACS on paper or the Internet.  The Internet option is prominently displayed on both the letter and questionnaire of the initial questionnaire mailing, as well as on the reminder postcard and replacement questionnaire mailing. This strategy also includes the Internet instruction card (introduced in the current test) that provides the choice of response modes and instructions for responding online.

2. Prominent Choice with Icons – Same as Prominent Offer (Choice) above, but with icons (computer and pencil) displayed next to the Internet and paper options respectively on the front of the questionnaire to draw attention to the choice of modes.

3. Prominent Choice with Icons on Modified Mailing Schedule – Same as Prominent Choice with Icons above, but the replacement questionnaire will be mailed about two weeks after the first questionnaire, one week sooner than the replacement questionnaire is mailed in ACS production.

4. Push Internet on Modified Mailing Schedule – During the initial questionnaire mailing, sample addresses would normally receive a paper questionnaire. Under this design, these sample units only receive a letter and instructions on how to complete the ACS on the Internet.  They will not receive a paper questionnaire until the replacement questionnaire mailing. We will mail the replacement questionnaire about two weeks later, one week sooner than the replacement questionnaire is sent in ACS production.

5. Push Internet on Modified Mailing Schedule with Second Reminder postcard – Same as Push Internet of Modified Mailing Schedule, except we will send an additional reminder postcard after the replacement questionnaire mailing.

Sample Size and Burden Hour Estimates:

To field this follow-up test, we are requesting additional burden hours for the ACS Methods Panel Clearance (0607-0936).

We will also use ACS production for the corresponding sample month (November 2011) as a control panel since there is no Internet option. Using a sample size of 10,000 addresses for each panel, we expect to measure a 1.6 percentage point difference between ACS production and a treatment panel with 80% power and α=.1 in targeted areas. For not targeted areas, we expect to measure a 1.5 percentage point difference between ACS production and a treatment panel with the same power and α level. Comparisons between two treatment panels of 10,000 addresses each will allow us to measure a difference of roughly 2.1 percentage points with 80% power and α=.1 in both strata. With 10,000 addresses per panel, five experimental panels in each of two strata, the total sample size for the test is 100,000 addresses (See table 1 below).

In terms of burden hour estimates, we assume the same amount of time for response as ACS production (38 minutes per address). Therefore, the expected burden for this test is 38 minutes per address, times 100,000 addresses in the test sample, which results in 63,334 burden hours overall.

Table 1. Panel design with Sample Size and Burden Hour Estimates

Panel	Targeted addresses	Not Targeted addresses
Prominent Offer (Choice)	10,000	10,000
Prominent Choice with Icons	10,000	10,000
Prominent Choice with Icons on Modified Mailing Schedule	10,000	10,000
Push Internet on Modified Mailing Schedule	10,000	10,000
Push Internet on Modified Mailing Schedule with Second Reminder postcard	10,000	10,000
TOTAL Addresses	100,000
TOTAL burden hours	63,334

References:

Griffin, D., Fischer, D., and Morgan, M. (2001), “Testing an Internet Response Option for the American Community Survey,” Paper Presented at the Annual Conference of the American Association for Public Opinion Research, May 17-20.