Online Qualitative Testing Protocol

Appendix B – Online Qualitative Testing Guide and Activities

NOTE: The images presented in this guide are for illustrative purposes only; they are intended to exemplify what the participants will see and are not associated with this project or the U.S Census Bureau. Team Y&R will create a customized portal with project-specific images.

The Online Qualitative Dashboard and Process

During five days of the online session, participants will have access to our online activity platform where Team Y&R posts tasks for them to complete at times convenient for them throughout this period. Participants can choose when to login to the online community and complete the activities, but they must complete three days of activities to qualify as a “complete” and earn the incentive. These activities will include interactive mark-up activities on static originals, as well as discussion prompts regarding the test stimuli originals (across static, video, and audio formats) that participants are exposed to each day. Participants will receive an email notification when new activities or comments are added to their study.

Throughout the process, Team Y&R will closely monitor the responses and a Team Y&R moderator may probe for clarity. All activities will be individual-based, i.e., participant cannot see others’ responses to limit a potential breach of personally identifiable information (PII). The moderator will also remove any PII inadvertently shared by participants.

Figure 1. Sample Activity Dashboard for an Online Qualitative Session

EXAMPLE

Audiences

For the online qualitative sessions, Team Y&R will recruit a sample of Diverse Mass and five additional samples for each of these select audiences: Black/African Americans; English-proficient Hispanics; people who identify as LGBTQ; individuals in households with young children (four years old or younger); and military. Participants will review and provide their opinions and interpretations of the creative originals stimuli intended to motivate response in their audience group.

Table 1: Proposed Number Recruited and Completes by Audience for Online Qualitative Sessions

Audience	Description	Number Recruited	Completes
Diverse Mass	Includes participants from a variety of demographic background that reflect the English-speaking general population in the U.S. to the best degree possible	100	50
Black/African Americans	Self-identify as Non-Hispanic Black/African American	32	16
Hispanic, English-preferred	Self-identify as Hispanic, Latino, or Spanish Speaks English	32	16
Lesbian, Gay, Bisexual, Transgender, Queer (LGBTQ)	Self-identify as LGBTQ	32	16
People with young children in the household	Live with at least one child age four or younger	32	16
Military	Previously served or currently in active military service	32	16
TOTAL		260	130
Note: Each “sample” will include participants representing a mix of age, gender, education, and region

Participants will review a mix of image (static) ad, video (animatic), and radio ad. The quantity and type of ads will vary by audience.

Activity Guide

This section provides details on the activities that participants will be asked to complete in the online qualitative sessions. Team Y&R includes three days of activities for the five-day sessions to allow participants more time to complete the study. Each day, participants will review a mix of three video, image, or radio originals intended for their audience group and discuss elements that stand out, either positively or negatively, to them. At the end, participants will share how what they have seen might have changed their thoughts about the decennial census.

Please keep in mind that the images presented in this guide are for illustrative purposes only; they are intended to exemplify what the participants will see and are not associated with this project or the U.S Census Bureau.

Introduction – Home Page

The following introduction will appear on the home page of every participant, which is the first thing participants will see upon entering the online qualitative session.

Welcome to the [STUDY NAME]!

This is a study sponsored by the U.S Census Bureau. You can validate that this survey is a legitimate federally-approved information collection using the U.S. Office of Management and Budget approval number 0607-0978, expiring on 08/31/2020.

Our discussion will be about the U.S. decennial census. We’d like to start by sharing some information about what the U.S. decennial census is to be sure we’re all thinking about the same thing. The U.S. census is the count of all the people who live in the United States. It happens every 10 years, which is why it is called the decennial census. The Census plans to ask questions such as how many people live at your address or place and their age, gender, race, ethnicity, relationships, and citizenship. The next census is in 2020. For the rest of the study, please think about this when we say “the census.”

We look forward to getting to know you and understanding your perspectives. Your thoughts and opinions are very important to us – there are no right or wrong answers, and your individual responses won’t be shared outside of the research team.

To best understand you, we set up an interactive website where a moderator may ask you questions based on your responses to aid our understanding of your experiences and point of view. Please check back into the site at least three times before the study ends on Friday, April 12, 2019 for new activities and to answer any follow-up questions the moderator may have asked. The information you provide to the web portal will not be shared or used for promotional purposes under any circumstances.

Again, thank you for participating in the [NAME] study – we look forward to hearing what you have to say!

Figure 2: Sample Home Page and Welcome Message

EXAMPLE

Description of Activities

Activity A: Static (Image) Advertisement Review

Activity Description: Participants will mark up static advertisements, such as ones seen in print or on billboards. Participants will be able to select a variety of reactions and pinpoint where on the ad they are reacting. Reaction icons include:

	I like this			This surprised me
	I don’t like this			This makes me happy
	This confuses me

Upon clicking an area on the piece and selecting the desired reaction, participants will then be prompted to explain what they are reacting to and why.

Shown to Participants:

For today’s activity, we would like to understand your opinions about some draft advertisements that the Census Bureau may use to encourage people to participate in the 2020 Census. These are not final, and may be later developed into complete advertisements that you might see in a newspaper, magazine, or on a billboard.

You will be asked some questions about these draft advertisements, and please keep in mind that these are work-in-progress, not final advertisements. Once you have completed today’s activity, the moderator may have follow-up questions for you. Please check back to see those.

/* NEXT PAGE */

Click on the image and select a marker to show what you liked, disliked, or found confusing about the advertisement.

Figure 3: Sample Image Markup Activity

*Note – the reaction icons will be the same as mentioned above

Below you will see some questions about the advertisement you just viewed. Please complete your responses by filling in the blank spaces.

Figure 4: Sample Discussion Activity

EXAMPLE

Questions for this activity may include:

Metric	Question
Overall Reaction	First, let’s start general. Tell me what caught your attention overall.
Main Message	In your own words, what do you think the ad is trying to convey?
	What questions, if any, come to mind when you see this?
	What did you learn, if anything, from the ad?
	Who do you think this ad is trying to reach? Someone like you or someone else? Describe that person.
Emotion	How does this ad make you feel?
	Does it make you feel differently?
Clarity	What, if anything, was confusing about the ad?
Opinion	How, if at all, does it change your view of the census?
Relevance	Is the ad useful or relevant to you in any sense?
Concerns	Do you have any concerns or doubts after seeing this ad?
Suggestion	What, if anything, is missing from this ad? Moderator to limit “art direction” comments.

/* NEXT PAGE */

Activity B: Animatic (Video) Advertisement Review

Activity Description: Participants will provide open-end responses of the advertisements to be used on TV or in a digital setting. Participants will view the video/animatic once. They will then be shown instructions to watch the video a second time and think about specific parts of the video to which they have (or do not have) a variety of reactions. After, they will be prompted with open-end boxes to say what they liked, disliked, were surprised by, were confused by, or what made them happy.

Shown to Participants:

For today’s exercise, we’d like you to review and give your opinions on some draft video advertisements that the Census Bureau may use for the 2020 Census. We will show you a video ad and then ask some follow-up questions.

Keep in mind this is a draft advertisement that the Census Bureau may use to encourage people to participate in the 2020 Census. This may be developed into more a complete advertisements you might see on TV. It’s really expensive to develop ads, so getting your feedback early on is important. Please keep in mind that this is an idea that would be polished. In fact, the video ad you are about to see is made of rough sketches that are animated and is not like an ad you may be accustomed to seeing on TV.

Once you have completed today’s activity, the moderator may have follow-up questions for you. Please make sure to check back to see those.

First, we’d like to ask you to watch the video ad below (60 seconds). Please make sure that the sound is not disabled on your device. /* NEXT BUTTON NOT VISIBLE UNTIL FULL VIDEO IS VIEWED */

Now we will show you the video ad again. Please think about specific parts of that ad that you like, dislike, or have any other reactions to. You will be asked to share your thoughts about and emotional reaction to the ad on the next page. /* NEXT BUTTON NOT VISIBLE UNTIL FULL VIDEO IS VIEWED */

Figure 6: Sample Video Response Activity

What did you like about the ad?	Response box
What did you not like about the ad?	Response box
Was there anything about the ad that confused you? Please describe.	Response box
Was there anything about the ad that surprised you? Please describe.	Response box
Was there anything about the ad that made you happy? Please describe.	Response box

/* NEXT PAGE */

Below you will see some more questions about the advertisement you just viewed. Please complete your responses by filling in the blank spaces.

Figure 7: Sample Discussion Activity

EXAMPLE

Questions for this activity may include:

Metric	Question
Overall Reaction	First, let’s start general. Tell me what caught your attention overall.
Main Message	In your own words, what do you think the ad is trying to convey?
	What questions, if any, come to mind when you see this?
	What did you learn, if anything, from the ad?
	Who do you think this ad is trying to reach? Someone like you or someone else? Describe that person.
Emotion	How does this ad make you feel?
	Does it make you feel differently?
Clarity	What, if anything, was confusing about the ad?
Opinion	How, if at all, does it change your view of the census?
Relevance	Is the ad useful or relevant to you in any sense?
Concerns	Do you have any concerns or doubts after seeing this ad?
Suggestion	What, if anything, is missing from this ad? Moderator to limit “art direction” comments.

/* NEXT PAGE */

Activity C: Radio Advertisement Review

Activity Description: Participants will provide open-end responses of the advertisements to be used on radio. Participants will listen to the audio once. They will then be shown instructions to listen to the audio a second time and think about specific parts of the audio to which they have (or do not have) a variety of reactions. After, they will be prompted with open-end boxes to say what they liked, disliked, were surprised by, were confused by, or what made them happy.

Shown to Participants:

For today’s exercise, we’d like you to review and give your opinions on some draft radio advertisements that the Census Bureau may use for the 2020 Census. We will show you a radio ad and then ask some follow-up questions.

Keep in mind this is a draft advertisement that the Census Bureau may use to encourage people to participate in the 2020 Census. This may be developed into more a complete advertisements you might hear on the radio. It’s really expensive to develop ads, so getting your feedback early on is important. Please keep in mind that this is an idea that would be polished.

Once you have completed today’s activity, the moderator may have follow-up questions for you. Please make sure to check back to see those.

First, we’d like to ask you to listen to the audio (60 seconds). Please make sure that the sound is not disabled on your device. /* NEXT BUTTON NOT VISIBLE UNTIL FULL RADIO AD IS COMPLETE */

Now we will show you the radio ad again. Please think about specific parts of that ad that you like, dislike, or have any other reactions to. You will be asked to share your thoughts about and emotional reaction to the ad on the next page. /* NEXT BUTTON NOT VISIBLE UNTIL FULL RADIO AD IS VIEWED */

Figure 8: Sample Radio Response Activity

What did you like about the ad?	Response box
What did you not like about the ad?	Response box
Was there anything about the ad that confused you? Please describe.	Response box
Was there anything about the ad that surprised you? Please describe.	Response box
Was there anything about the ad that made you happy? Please describe.	Response box

/* NEXT PAGE */

Below you will see some more questions about the advertisement you just listened to. Please complete your responses by filling in the blank spaces.

Figure 9: Sample Discussion Activity

EXAMPLE

Questions for this activity may include:

Metric	Question
Overall Reaction	First, let’s start general. Tell me what caught your attention overall.
Main Message	In your own words, what do you think the ad is trying to convey?
	What questions, if any, come to mind when you see this?
	What did you learn, if anything, from the ad?
	Who do you think this ad is trying to reach? Someone like you or someone else? Describe that person.
Emotion	How does this ad make you feel?
	Does it make you feel differently?
Clarity	What, if anything, was confusing about the ad?
Opinion	How, if at all, does it change your view of the census?
Relevance	Is the ad useful or relevant to you in any sense?
Concerns	Do you have any concerns or doubts after seeing this ad?
Suggestion	What, if anything, is missing from this ad? Moderator to limit “art direction” comments.

/* NEXT PAGE */

Activity D: Take-Away Message about the Census

Activity Description: After being exposed to all the ads, respondents will reflect on what they’ve learned about the census during the online qualitative session and their take-away message about the 2020 Census.

Shown to participants:

## APPEAR ON FINAL DAY OF AD REVIEW##

For today’s exercise, we would like you to reflect on what you have learned about the census during the last five days that you did not necessarily know before. Please check back to see if the moderator have follow-up questions for you.

Figure 11: Sample Reflection Activity

EXAMPLE

Questions for this activity may include:

Metric	Question
Post Reaction	After what you have seen, how do you feel about the census?
	What would you tell your families, friends, and colleagues to encourage them to fill out the 2020 Census?
Interest	Is there anything else that you would like to share that we haven’t touched on during the study?