Example of Study Materials to be Submitted to OMB

Attachment B:

Example Study Materials

Title of the Study: Proof-of-Concept Test on Email Subject Line and Address Fields

Purpose of the study

The Census Bureau plans to conduct additional research under the generic clearance for internet pretesting research (OMB number 0607-XXXX). We plan to conduct a pilot test using an online questionnaire to gather information about email subject lines and screen designs to collect residential address information. This pilot test is part of an iterative testing strategy that we plan to use to supplement the 2020 Census research program. The goal is to determine the best email subject lines to maximize self-response for the 2020 Census. Email subject lines that fail to generate significant click-through rates in the nonprobability panel are not likely to perform better in a probability panel. Eliminating poorly performing email subject lines before larger-scale testing is the purpose of the nonprobability testing we propose here. Similarly, we look to examine which set of address questions yields the most geocodable results.

Research Design and Methodology

This will be a split-ballot test. We will send survey invitation emails to a randomly drawn sample of people who opted-in to participate in Census Bureau research studies through the Census Bureau’s email subscription website, run by GovDelivery. There was no incentive to sign up and there is no incentive (other than a copy of the research report) to participate. Participation is voluntary. Currently, the panel has over 7000 emails. Because of the opt-in nature, this panel is considered a nonprobability panel. This test is a pilot test as we have not yet contacted any of the panel to participate in a research study. We will measure the open rates and click through rates to the emails with different subject lines.

There are four panels in this test. Each panel consists of a sample of 250 email addresses for a total sample of 1,000 email addresses. There are of two different versions of the online survey. The only difference in the surveys is the screen that collects residential address information. There are two different initial emails, both containing a link to the survey. The only difference in the emails is the subject line of the initial email. The design is completely crossed. The sample will receive a maximum of three notification emails:

• one of the initial emails on Monday, December 2,

• a reminder email on Thursday December 5 (if they had not yet click on the link to the survey), and

• a final reminder email on Thursday, December 12.

The survey will be closed on Friday, December 13^th at midnight.

The survey that the emails link to will collect data on address collection screen designs. The Census Bureau hypothesizes that many people will come to the online 2020 Census without their housing unit ID and thus they will have to enter their address as part of the data collection. We must make sure that the address screen collects accurate address data in order for us to match to our master address file and geocode. Without good addresses, we will not be able to count people at the correct location for the census.

After the address screens, we ask opinion questions as a part of the “debriefing.” The objective of these questions is to gather qualitative data in order to guide future iterations of this test, to gain a sense of how respondents want to be contacted about the census, to answer the census, and what these highly motivated individuals think the census collects. These data will be shared with 2020 Census staff and the communications area of the Census Bureau. The answers to the demographic questions and questions on new technologies will allow us to compare our results against future probability panels results and to look at characteristics of respondents.

The pilot test will be conducted from December 2 through December 13, 2013. Staff from the Center for Survey Measurement’s Human Factors and Usability Research Group will select the sample and send the emails through GovDelivery. The survey will be hosted on our secure servers within the Application Services Division of the Census Bureau that hosts all other secure online production surveys. The username needed to enter the survey will be the email address where the email was sent (this is the same email used to sign up to participate in Census Bureau research studies). If the respondent starts the survey but does not complete it, that person will not be allowed to re-enter the site later. The emails and questions were usability tested earlier under a separate generic clearance letter sent on June 12, 2013.

Burden

We estimate that users will spend 5 minutes on average completing the survey and approximately 5 minutes reading emails. Thus, the total estimated respondent burden for this study is 167 hours.

There is no incentive, other than offering a copy of the completed report to participants.

Contact person: contact information (e-mail and phone)

The contact person for questions regarding data collection and statistical aspects of the design of this research is XXXXXXXX.

List and description of all attachments or enclosures.

Below are the emails, data collection screens and debriefing questions that will be administered each with a brief description.

1. Initial emails for the Pilot Test (Half the sample will receive the email in Figure 1 and the other half will receive the email in Figure 2)

From: U.S. Census Bureau

Subject line: Confidential 2013 Census Study

The U.S. Census Bureau is conducting research to develop new methods to make the next census easier, more convenient and less costly for taxpayers.

Please complete the survey online at http://respond.census.gov/study

The survey will take 5 minutes or less to complete.

We are conducting this survey under the authority of Title 13 of the United States Code, Section 193. Federal law protects your privacy and keeps your answers confidential (Title 13, U.S. Code, Section 9 and 214). Participation in this test is voluntary.

Figure 1: Initial email for half the sample, “Confidential 2013 Census Study” subject line

From: U.S. Census Bureau

Subject line: Respond to the 2013 Census Study

The U.S. Census Bureau is conducting research to develop new methods to make the next census easier, more convenient and less costly for taxpayers.

Please complete the survey online at http://respond.census.gov/study

The survey will take 5 minutes or less to complete.

We are conducting this survey under the authority of Title 13 of the United States Code, Section 193. Federal law protects your privacy and keeps your answers confidential (Title 13, U.S. Code, Section 9 and 214). Participation in this test is voluntary.

Figure 2: Initial email for half the sample, “Respond to the 2013 Census Study” subject line

2. Reminder emails for the Pilot Test

From: U.S. Census Bureau

Subject line: Reminder about the 2013 Census Study

A few days ago, you should have received an email about completing an important U.S. Census Bureau study. The survey will take 5 minutes or less to complete.

If you have not yet responded, now is the time to complete the survey online at: http://respond.census.gov/study

The U.S. Census Bureau is conducting this research to develop new methods to make the next census easier, more convenient and less costly for taxpayers. We are conducting this survey under the authority of Title 13 of the United States Code, Section 193. Federal law protects your privacy and keeps your answers confidential (Title 13, U.S. Code, Section 9 and 214). Participation in this test is voluntary.

Figure 3: First reminder email –Sent to the sample which did not click the link to the survey

From: U.S. Census Bureau

Subject line: Final reminder about the 2013 Census Study

About a week ago, you should have received a couple of emails about completing an important U.S. Census Bureau study. The survey will take 5 minutes or less to complete.

If you have not yet responded, now is the time to complete the survey online at: http://respond.census.gov/study

The U.S. Census Bureau is conducting this research to develop new methods to make the next census easier, more convenient and less costly for taxpayers. We are conducting this survey under the authority of Title 13 of the United States Code, Section 193. Federal law protects your privacy and keeps your answers confidential (Title 13, U.S. Code, Section 9 and 214). Participation in this test is voluntary.

Figure 4: Final reminder for the pilot study -- Sent to the sample which did not click the link to the survey

3. Landing page

Figure 5: Landing page after clicking the link in the email (for all panels)

4. Screen shots of production address screens (horizontal design) (Half the sample will receive this design.)

Figure 6: Main screen with the default selection of Street Address

Figure 7: The screens if P.O. Box or Rural Route are selected

Figure 8: Write in screen if the respondent selects P.O. Box or Rural Route (The examples on the screen are a result of the usability testing conducted and are not the same examples in the 2014 Census production instrument.)

5. Screen shots of cognitive testing address screens (vertical design) (The other half of the sample will receive this design.)

Figure 9: Design similar to the design of the 2020 Census Coverage Cognitive Testing project

6. Opinion debriefing questions (All sample) (Response choices in Question 2 will be randomized.)

1. We want to get usability feedback on the address screens you just completed.

How would you rate the quality of your experience completing the address screens?

Excellent

Good

Fair

Poor

Very poor

2. We want to get feedback on possible email message subject lines to use in future Census Bureau studies.

Which email message subject line(s) would you be likely to click on if you saw it in your inbox? You may select more than one.

Important Census Study

Mandatory Census Study

Improve the U.S. Census Study

Help us make the U.S. Census better, answer our survey

Answer the U.S. Census– Help your community

10 minutes to improve the U.S. Census and help your community

Your Civic Duty – Answer the U.S. Census!

A message from the Director of the U.S. Census Bureau

A message from John Thompson, Director of the U.S. Census Bureau

3. Do you have any additional comments on either the address screens or the possible email subject lines?

[250 max characters] with character count down

7. Additional debriefing questions (All sample) (This will be on a new page and it will be all one page.) (Response choices in Questions 1 and 2 will be randomized.)

1. Every 10 years, the Census Bureau counts everyone living in the United States.  We do this by collecting information from each address. For the next census in 2020, how would you prefer to be contacted by the Census Bureau?

❑ Mail a form to my home

❑ Mail a letter to my home containing the link to the online form

❑ Email a link to the online form

❑ Call me on my home phone (landline)

❑ Call me on my cell phone

❑ Text message me a link to the online form

❑ Send an interviewer to my home

2. For the 2020 Census, would you rather the Census Bureau:

❑ contact you to get your information

❑complete your form for you using information you have already provided other

government agencies

3. What information do you think the census typically collects every 10 years?

 Select all that apply

• Names of all adults living at your address

• Names of all children living at your address

• Social security number

• Age

• Date of birth

• Race

• Ancestry

• Income

• Relationship

• Citizenship

4. Do you have…

   1. A cell phone?

❑Yes

❑No

2. A handheld device made primarily for e-book reading, such as a Nook or Kindle e-reader?

❑Yes

❑No

3. A tablet computer such as an iPad, Samsung Galaxy Tab, Google Nexus or Kindle Fire? ❑Yes

❑No

4. An iPod or other MP3 player?

❑Yes

❑No

5. A game console like an Xbox or PlayStation?

❑Yes

❑No

5. (if yes to 4a) Some cell phones are called ‘smartphones’ because of certain features they have. Is your cell phone a smartphone or not, or are you not sure?

❑Yes

❑No

6. (if yes to 5a) Which of the following best describes the type of cell phone you have? Is it an iPhone, a Blackberry, an Android phone, a Windows phone, or something else?

❑iPhone

❑Blackberry

❑Android

❑Windows

❑Something else

7. Are you male or female?

❑ Male

❑ Female

8. What is your age?

❑ Less than 18 years old

❑ 18-24

❑ 25-34

❑ 35-44

❑ 45-54

❑ 55-64

❑ 65 or older

9. Are you of Hispanic, Latino or Spanish origin?

❑ Yes

❑ No

10. What is your race? You may select one or more.

❑ White

❑Black or African American

❑American Indian or Alaska Native

❑Asian

❑Native Hawaiian or Other Pacific Islander

11. What is your highest level of education?

❑ Less than High School Diploma or GED

❑ High School Diploma or GED

❑ Some College

❑ College Graduate or Professional Degree

12. Which of the following best describes your current field of work?

❑ Student/Educator

❑ Government

❑ Technical/Professional

❑ Service (e.g., retail, food service, protective service, and so on)

❑ Retired or not currently working for pay

❑ Other  Please specify [ 50 characters ]

13. Would you like to receive a link to the research report for this study when it is complete?

❑ Yes

❑ No

Thank you for your participation in this survey. Your responses are very helpful in our research!

14