Memo

Generic Information Collection Request (ICR):
2020 Census Integrated Communications Campaign (ICC) –
Creative Testing – Campaign Testing

Request: The U.S. Census Bureau plans to conduct additional research under the Generic Clearance for Internet Nonprobability Panel Pretesting (OMB Control Number 0607-0978). As part of the 2020 Census Integrated Communications Contract (ICC), the Census Bureau plans to conduct online qualitative sessions and focus groups. Together the online qualitative sessions (Part 1 below) and in-person focus groups (Part 2 below) comprise “Campaign Testing.” Campaign Testing results will provide direction for refinement of creative executions for the 2020 Census ICC campaign.

Purpose: Campaign Testing assesses initial creative executions developed based on the selected platform from the Fall 2018 focus groups and tailors it to specific audiences. Campaign Testing will provide diagnostic feedback, upon which the creative executions will be adjusted and refined into final advertisements used in the 2020 Census communications campaign. It will also highlight strengths of the creative executions and identify problem areas or issues that detract from desired effects to motivate census participation.

Stimuli Tested in Campaign Testing

Team Y&R will develop creative stimuli for testing called “creative originals.” Originals will be based on the selected platform, and we will conduct Campaign Testing to evaluate these draft ads among the Diverse Mass audience and sociodemographic subgroups audiences. Each original will represent a topic or theme developed to reach a specific audience.

To create the originals, Team Y&R will craft audience-specific creative briefs including a message hierarchy based on the approved platform creative brief informed by previously fielded Census Bureau research, including the Census Barriers, Attitudes, and Motivators (CBAMS) research, the 2018 Quick Idea Platform Testing research (QIPT), and the approved mindset analysis, third-party research, and inputs from multicultural experts.

Team Y&R agencies, leveraging media data, will determine the most relevant stimuli formats (television, print, radio, digital, etc.) and will develop originals for testing (i.e., the stimuli) among each audience identified in this study plan.

Creative Terminology

Term	Definition
Campaign	The creative expression of the QIPT-tested platform that will be executed in different ways to feature specific messaging or to be expressed to reach specific racial/ethnic or other affinity-based audiences, including those meeting hard-to-count criteria. There is a tagline associated with the campaign.
Original	A representative creative piece for topically linked ads, intended for specific audiences within a campaign. Each original represents thematic-message groups or specific insight-driven creative expressions in the tactical format most likely to be used with the audience. For example, if a print-only media buy is planned, then the original will be created in an industry standard print/static image testing stimuli format. Any originals expected to be executed as video and/or TV spots will be tested in an industry standard video testing format.

Part 1: Online Qualitative Testing

Team Y&R will conduct online qualitative sessions to effectively and efficiently gain insights from the Diverse Mass and some additional sub-sets of English-speaking audiences. Integrating online testing into the creative development process helps inform both the selection of the most effective creative development. The samples will be recruited to reflect geographic and demographic diversity from populations that would be difficult to convene at specific locations and times. Online qualitative sessions will only include participants who can communicate comfortably online in written English.

Online qualitative sessions will allow participants to respond at their convenience during the weeklong discussion period. This will help facilitate testing of a large volume of materials required to support the Diverse Mass campaign. The discussion will be asynchronous, allowing community participants more time to think and provide more thoughtful, in-depth responses than they might provide in simultaneous online focus groups (Kruger & Casey, 2009).¹ Team Y&R has found the added time is particularly useful for virtual moderators to consider compelling follow-up questions. In addition, the extended length generally allows the online qualitative tool to extract more insights from participants than other research techniques can elicit.

Populations of Interest: For online qualitative sessions, Team Y&R will recruit a sample from the Diverse Mass audience (to roughly reflect the demographic distribution of the U.S. population) as well as five additional samples from each of the following audiences: military, households with young children, LGBTQ, Black/African Americans, and English-proficient Hispanics. All participants will be at least 18 years old.

Timeline: Tentative fielding dates are April 22 – April 26, 2019.

Language: English only

Method: Recruiting of participants for online qualitative sessions will be conducted via nonprobability techniques, as appropriate for qualitative research. Potential participants will receive an email from the survey panel indicating they have a screening survey available to complete. Team Y&R will review responses from potential participants to assess their eligibility and engagement with open-ended opinion screening questions. A subsequent email will invite top prospects to join and create a profile on the online testing board website. Participants must complete both steps to be eligible for the study.

To maximize feasibility, Team Y&R will recruit participants from a variety of commercially available online research panels. These panels recruit U.S.-based audiences and specialize in some multicultural sub-groups that are of interest to our study. Participation in online qualitative sessions is opt-in and voluntary; participants will be individuals who have opted into a research panel and are familiar with the process. Online panels generate interest in studies like this by offering people virtual points to complete screeners to become eligible for larger studies. These points cannot be exchanged for cash, but have approximate cash equivalence of $0.50 - $1.00.

Team Y&R’s online testing approach will not include Personally Identifiable Information (PII). Team Y&R will inform online qualitative session participants that answers they provide will remain anonymous. The contractor will not pass individual survey responses to the Census Bureau or any government systems. Data stored on Team Y&R’s research subcontractor’s survey tool during analysis will not allow contractors or other viewers to identify individual participants or trace back participants’ answers to individuals.

A variety of demographics will be integrated into randomization to ensure a balanced sample of participants; this will ensure no demographic subgroup disproportionately sees the same original first. The randomization variables will include race/ethnicity (white vs. non-white), education (less than college vs. some college and above), gender, and age (18-34 years of age vs. 35-54 years of age vs. 55 years of age and older). The Diverse Mass sample will include a wide range of general population English-speaking audiences as they naturally occur, such as Black/African Americans and people of Hispanic or Asian descent.

Participants will join the online qualitative sessions and complete three days of activities over a five-day field period. Participants will have access to a secure, customized, state-of-the-art online discussion board, where Team Y&R will post activities for completion. Participants can choose when to log on to the online community and complete the activities, but they must complete three days of activities to qualify as having completed the sessions and to earn the incentive. Activities include:

• Static Advertisement Mark-up

• Video (Animatic) Advertisement Review

• Radio Advertisement Review

• Discussion Questions

Throughout the process, Team Y&R will closely monitor responses and an online moderator will probe on insights requiring elaboration or follow-up. The moderator will also remove any PII shared inadvertently by participants. Participants will complete activities independently and will not be able to see others’ responses or discuss responses with one another in order to limit a potential breach of PII.

Team Y&R recommends testing each original stimulus with at least 16 participants of a given audience in the online sessions, as this is comparable to two focus groups’ worth of participants and is likely to achieve sufficient data saturation for testing purposes.

Incentives: Once selected to participate in the online qualitative sessions, participants are required to participate for approximately one hour on three different days over a five-day period. The use of incentives is standard practice. Organizations that set standards for conducting ethical marketing research, such as the Council of American Survey Research Organizations (CASRO), recommend the use of incentives for communications industry creative research (CASRO Code of Standards and Ethics for Survey Research). Participants who complete the screening questionnaire via online opt-in panel will receive a points-based incentive for their time. Online panel providers award points at various values, and awarded points are dependent upon length of interview, target audience, and topic. As mentioned previously, these points cannot be exchanged for cash, but have approximate cash equivalence of $0.50 - $1.00 Participants who complete all activities assigned in the three one-hour sessions (totaling three hours per person) will receive a $75 incentive, similar to participation in a live focus-group, to offset expenses such as child-care and gaining online access.

Length of Interview: The online qualitative sessions proposed in this study plan require approximately three hours of participant engagement over a five-day period. The incidence rate for general population sample completion is 60%, while the incidence rate for the oversample completion is 30%. The incidence rates account for people who are in online panels. Due to the intensiveness of the online qualitative panel, we are recruiting 100 general population participants and 32 participants per oversample group to begin the study, with the goal that at least 50 general population participants and 16 participants per oversample group completing all three hours of activities. Thus, the estimated maximum total respondent burden for this study is 838.33 hours, inclusive of screening (see Burden Hour Calculation below).

Enclosures: The following is enclosed:

• Appendix A: Online Qualitative Testing Screener Questionnaire

• Appendix B: Online Qualitative Testing Guide and Activities

Part 2: Focus Groups

Populations of Interest: We will conduct focus groups with audiences that are at-risk for low response to the 2020 Census and unreachable online due to low response factors. Additionally, these audiences are not reachable in online panels.

We will conduct 120 focus groups with the audiences described in Tables 2 and 3:

Table 2. Campaign Focus Group Audiences with English Speakers

Description	Focus Groups
American Indian and Alaska Native (AIAN)	24
Black/African American (BAA)	12
Middle Eastern or North African (MENA)	4
Native Hawaiian or Pacific Islander (NHPI)	8
People Living in Rural Locations	6
People Living in Large Households with Young Children	4
TOTAL	58
Note: Focus groups will include participants with a mix of ages, genders, and education levels.

Table 3. Campaign Focus Group Audiences with Non-English Speakers

Description	Focus Groups
Arabic	2
Brazilian (Portuguese speaking)	2
Chinese, Cantonese	4
Chinese, Mandarin	4
Filipino (Tagalog speaking)	4
Haitian Creole	2
Japanese	2
Korean	4
Polish	2
Russian	2
Spanish, Puerto Rico (island residents, not those living within the continental U.S.)	8
Spanish, U.S. Mainland (including a mix of country of origin)	20
Sub Saharan African Descent (French speaking)	2
Vietnamese	4
TOTAL	62
Note: Focus groups will include participants with a mix of ages, genders, and education levels.

Focus groups will be conducted in 36 locations across 25 states. The proposed locations presented in Tables 4 and 5 below were selected, in collaboration with Census Bureau SMEs and Team Y&R’s multicultural partners, to provide sufficient concentration of the focus groups’ target audiences. They also represent significant geographic diversity across the country. Other factors considered include the mix of country-of-origin in particular locations and minimizing travel costs for the government.

Table 4. Proposed Campaign Focus Group Locations with English Speakers

Audience	Number of Groups by Location
American Indian (AI)	(4) (2) (4) (4) (2) (2) (2)	Green Bay, WI Oklahoma City, OK Riverside, CA Bismarck, ND Seattle, WA Asheville, NC Miami, FL
Alaska Native (AN)	(4) Kotzebue, AK
Black/African American (BAA)	(3) Atlanta, GA (3) Oakland, CA (3) Jackson, MS (3) Dayton, OH
Middle Eastern or North African (MENA)	(2) Peoria, IL (2) Orange County, CA
Native Hawaiian or Pacific Islander (NHPI)	(4) Honolulu, HI (2) San Diego, CA (2) Salt Lake City, UT
People with Young Children Living in Household	(2) El Paso, TX (2) Philadelphia, PA
People Living in Rural Locations	(2) Beckley, WV (2) Jackson, MS (2) Bozeman, MT

Table 5. Proposed Campaign Focus Group Locations with Non-English Speakers

Audience	Number of Groups by Location
Arabic	(2) Paterson, NJ
Brazilian (Portuguese)	(2) Boston, MA
Chinese, Cantonese	(2) San Francisco, CA (2) New York, NY
Chinese, Mandarin	(2) Houston, TX (2) New York, NY
Filipino (Tagalog)	(2) Honolulu, HI (2) Las Vegas, NV
Haitian Creole	(2) Miami, FL
Japanese	(2) Los Angeles, CA
Korean	(2) Los Angeles, CA (2) Greater Washington, DC
Polish	(2) Chicago, IL
Russian	(2) Brooklyn, NY
Spanish, Puerto Rico (island residents, not those living within the continental U.S.)	(6) San Juan, PR (2) Utuado, PR
Spanish, U.S. Mainland (include a mix of country of origin)	(4) Yakima, WA (4) Chicago, IL (4) Miami, FL (4) Boston, MA (4) McAllen, TX
Sub Saharan African Descent (French speaking)	(2) Arlington, VA
Vietnamese	(2) Orange County, CA (2) Houston, TX

Timeline: Tentative fielding dates are March 25 – May 10, 2019.

Method: Focus groups provide a way to include insights from low response audiences, who by nature are hard-to-reach and unavailable in online panels. To be eligible for participant, potential participants are screened using a screening questionnaire (see Appendix C for Audience Screener Questionnaires). While each audience has specific screening criteria, all participants across all audiences must meet three overarching criteria to be eligible. First, participants will be screened to be at risk for nonresponse (low response criteria – see above description and citation in Online Testing section), and eligibility will be contingent upon risk for nonresponse. Second, participants must meet soft quotas for characteristics like gender, age, and education to ensure that there is an appropriate mix of people within each focus group (see Appendix D for Quota Tables). Finally, participants must have not participated in any other focus group session in the past three months. Eligible participants must also be willing to contribute to the research and must be available to attend the focus group in person at a specific location on a particular date and time. Appendix C includes Audience Criteria and Quota Tables for each audience, while Appendix D includes each audience-specific screener questionnaire.

Within each of the specific audiences for focus groups, we will seat participants who have elevated risk of nonresponse, or who are at risk for low response - demographic, family composition, or housing tenure factors that are associated with lower response rates - and to hear a broad range of perspectives. Participants must receive a minimum of two “priority audience” points to be eligible for participation in any of the audience focus groups. That is, only potential participants with a priority audience score of two or more will be scheduled to ensure we are reaching the target audiences of this phase of research.

In addition to establishing eligibility rules based on priority audience factors, we have established soft quotas for focus groups to recruit diverse subpopulations. Soft quotas ensure that the study sample represents individuals from a variety of demographic backgrounds. Census Bureau SMEs and the ICC contractor multicultural agencies provided input for the soft quota proportions for this research study (see Appendix C for Quota Tables). The team will evaluate the confirmed recruiting list and determine throughout the data collection process if any subgroup is over- or under-represented based on the study’s soft quotas. We will adjust recruitment strategies accordingly, and every effort will be made to ensure a strong mix of participants in each group.

Team Y&R will recruit between 14 and 16 participants, to seat six to eight participants per group. Groups may proceed with only six participants if show rates are lower than anticipated (even in spite of over-recruitment efforts). Campaign Testing focus groups will rely on an on-the-ground, in-the-community based recruitment strategy, in which the multicultural agencies tap into their network of community and organizational leaders to disseminate information and recruit potential participants.

Each focus group will be approximately 90 minutes in duration. Upon arrival at the facility, participants will receive a focus group information sheet and informed consent form (enclosed as Appendix F). A member of the research team will be available to answer any questions prior to the start of the discussion. Discussion will be semi structured and aligned with a discussion guide. With guidance from contractor researchers, multicultural specialist agencies, and Census Bureau SMEs, we have developed a discussion guide for the focus groups that covers key topics and identifies areas to probe for further information. Professionally trained moderators with demonstrated expertise in focus group facilitation and with specific audience groups will facilitate discussions. Moderators will facilitate non-English group in the appropriate language, and an interpreter will translate discussion into English for on-site observers located in a separate room.

The topics for discussion are as follows:

• Section A: Introduction and Icebreaker (10 min.)

• Section B: Warm Up (5 min.)

• Section C: Individual Advertisement Testing (65 min.)

• Section D: Conclusion (10 min.)

The complete discussion guide and screenshots for group activities are enclosed (see Appendices E and F respectively).

Incentives: We are requesting that OMB authorize the Census Bureau to approve an incentive amount of $75, as is typical for the Census Bureau for focus groups.

Length of Interview: Focus groups will last approximately 90 minutes. For each group, we will recruit between 14 and 16 participants, to seat eight participants per group. Thus, the estimated maximum total respondent burden for this study is 2893.33 hours, inclusive of screening and informed consent activities (see Burden Hour Calculation below).

Enclosures: The following are enclosed:

• Appendix C: Focus Group Screening Questionnaires by Audience

• Appendix D: Focus Group Audience Criteria with Quota Tables

• Appendix E: Focus Group Discussion Guide

• Appendix F: Focus Group Activities

• Appendix G: Focus Group Consent Form

Burden Hour Computation

Online Qualitative Testing

BURDEN HOUR COMPUTATION Number of respondents (X) estimated response or participation time in minutes (/60) = annual burden hours:

Type/Category of Respondent	No. of Respondents	Participation Time (minutes)	Burden (hours)
Screened Potential Participants
Screening – General Population	166.67	5	13.89
Screening – Oversamples	533.33	5	44.44
Completed Participants
Maximum Completes – General Population	100	180	300
Maximum Completes – Oversamples	160	180	480
Total Burden (Screened Participants and Completed Participants)			838.33

Assumption Notes:

• For every ten potential respondents invited, six will be interested and able to complete the screener questionnaire as a general population participant (recruit 100 total; goal is 50 completes, accounting for attrition)

• For every ten potential respondents invited, three will be interested and able to complete the screener questionnaire as an oversample participant (recruit 160 total – 32 recruited per five oversample groups; goal is 80 total completes - 16 completes per five oversample groups, accounting for attrition)

Focus Groups:

BURDEN HOUR COMPUTATION Number of respondents (X) estimated response or participation time in minutes (/60) = annual burden hours:

Type/Category of Respondent	No. of Respondents	Participation Time (minutes)	Burden (hours)
Screened Potential Participants
Questionnaire	17,440	5	1,453.33
Focus Group Participants
Maximum Focus Group Participants	960	90	1,440
Total Burden (Screened Participants and Focus Group Participants)			2893.33

Assumption Notes:

• Recruit 16 per NHPI (8 groups) and AIAN (24 groups) groups and 14 for all other groups (88 groups) to seat eight (recruit 1,744 total)

• For every ten potential participants contacted; one will be interested, eligible, and available to participate in a focus group

Point of Contact: The contact person for questions regarding data collection design of this research is listed below:

Monica Vines

Researcher

Center for New Media and Promotion (CNMP)

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-8813

[email protected]

1 Kruger, Richard & Mary Anne Casey. (2009). “Chapter 11: Telephone and Internet Focus Group Interviewing.” Focus Groups: A Practical Guide for Applied Research.

13