(OMB Control No. 0990-0476)
Current Events Tracker (CET)
The CET will involve cross-sectional weekly surveys. A different sample of adults will be invited to participate in the survey each week. A total of 1,000 respondents are expected to complete the survey each week. The survey will be conducted each week and will track questions including COVID-19 vaccine uptake and intentions, perceptions of COVID-19 vaccines, attitudes toward institutions managing the coronavirus response, sources of information used for awareness, and preventive measures taken.
The population of interest for this study is U.S. adults, aged 18 and older. The survey will be conducted using an online probability-based survey panel, the Ipsos KnowledgePanel. Every week for two years, a sample of U.S. adults aged 18 and over will be selected from the Ipsos KnowledgePanel to participate in the Weekend Ipsos Omnibus survey, which will include the CET questions developed for this study. Ipsos fields their omnibus survey as a way to address the information needs of multiple organizations through a single survey, thus reducing respondents’ time and burden by having participants respond to just one set of questions instead of multiple surveys. The survey vehicle is designed to be flexible to accommodate many different clients. Often the questions fielded as part of the omnibus are topical in nature and could cover topics such as views of events in the news; however, the additional clients and survey questions included in the omnibus survey over the next two years is not known at this time. The CET questions are expected to comprise the majority of the questions in the survey from week to week.
The Ipsos KnowledgePanel contains approximately 60,000 panel members and is nationally representative. Members of the KnowledgePanel were invited to participate in the panel through address-based sampling (ABS) of U.S. households The KnowledgePanel includes hard-to-reach populations and households without landline telephones, without internet access, or whose dominant language is Spanish. Ipsos weights the entire KnowledgePanel according to benchmarks from the March 2020 U.S. Census Bureau Current Population Survey (CPS) so that the weighted distribution of the KnowledgePanel reflects the U.S. adult population. The geodemographic dimensions used for weighting the KnowledgePanel include, but are not limited to, the following dimensions: age, race/ethnicity, education census region, household income, homeownership status, metropolitan area, Hispanic origin, and language dominance (with respect to the Spanish language).
Foundational Focus Groups, Interviews, and/or Dyads
HHS COVID-19 Public Education Campaign foundational focus groups, interviews, and/or dyads will be conducted remotely using teleconference software. Foundational focus groups will be conducted in rounds of 18 focus groups (with 4-6 participants per group). Specific participant eligibility criteria may vary by effort, depending on the needs of the Campaign, but participants will primarily include individuals who comprise the “movable middle,” those who are undecided about getting a COVID-19 vaccine (for themselves or for their children) but are open to the idea; the group excludes those who do not see merit in vaccines in general or who say they would never get the COVID-19 vaccine for themselves or for their children. These sessions may be segmented based on participants’ race/ethnicity, age, gender, primary language spoken, geographic location, and/or vaccination attitudes and status. As an example, for the first round, nine groups were conducted with the general population (a mix of genders and race/ethnicities) segmented by age; three groups with Black/African American adults (mixed gender) segmented by age; three groups with Hispanic/Latinx adults (mixed gender) segmented by age and primary language; and three groups with American Indian/Alaska Native adults, segmented by gender. Participants are grouped based on their self-reported demographics. Participants recruited specifically for American Indian/Alaska Native sessions are asked about their tribal enrollment and/or affiliation.
A copy of the screener, which outlines the exact criteria that must be met for participants to qualify for the first round of focus groups, wass provided with the initial submission. Specific demographic criteria used for recruiting or segmenting groups may vary slightly from round to round. ASPA will provide updated screeners to OMB for review.
Vendors will recruit participants from their panel databases via invitation emails and a phone screener to participate in the sessions.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
The copy testing survey will involve cross-sectional surveys of 1,500 sample members who comprise the target audiences before each Wave of creative assets are aired. Up to 36 waves of surveys will be fielded throughout the duration of the Campaign. The survey measures perceptions of COVID-19 vaccines, intended behaviors related to the COVID-19 vaccines, and the perceived effectiveness of advertisements viewed.
Each Message Matrix Survey will involve a cross-sectional survey of up to 4,000 respondents who comprise the target audiences of the campaign. Up to 9 rounds of data collection will be conducted over the course of the Campaign. The survey measures reactions to possible campaign messages.
Each Creative Testing Survey or Experiment will recruit up to 3,000 respondents who comprise the target audiences of the campaign. Up to 6 rounds of data collection will be conducted over the course of the Campaign. Surveys will assess various features of the Campaign content, such as the perceived effectiveness of messages, the influence of material format (e.g., video vs. static image ad), likelihood of engaging in desired behaviors, and changes in COVID-related knowledge after viewing Campaign materials.
The population of interest for these studies is U.S. adults or adolescents, aged 13 and older, who have not yet received a COVID-19 vaccine or updated vaccine. We expect most waves to be conducted with adults 18 years and older, but there is potential for one wave to be conducted with adolescents 13 years and older. The surveys will be conducted using an online, nonprobability-based, opt-in survey panel managed by Prodege. Prodege offers a panel of over 17 million members and maintains a database of over 1,000 demographic and behavioral attributers that can be used for sample targeting. Members of this panel agree to have their information stored and shared for the purposes of recruiting for research studies.
Statistical Methodology, Estimation, and Degree of Accuracy
Current Events Tracker (CET)
Each wave of the CET survey will be selected using a stratified random sample from Ipsos KnowledgePanel. Strata will be based on key demographic attributes. The sample sizes in each stratum will be determined based on the expected response rate for the strata with the goal of having a final dataset that is representative of the United States adult population.
The CET will be weighted to account for the probability of inclusion in the KnowledgePanel, probability of selection into the CET, the probability of responding to the survey, and any nonresponse bias that may occur using this methodology. Weights will then be calibrated to Census benchmarks for key demographics, such as age, gender, region, education, ethnicity, and race.
The length of the survey is expected to be about 10 minutes to complete the questions asked on behalf of ASPA. Note that Ipsos uses only completed surveys.
Foundational Focus Groups, Interviews, and/or Dyads
The foundational focus groups, interviews, and dyads will be conducted using qualitative data gathering techniques that do not require statistical metrics of group accuracy. The discussions will be analyzed for thematic findings and exemplar quotes or discussion to include in final reports.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
Participants in each wave of the copy testing survey, messaging matrix survey, and creative testing surveys and experiments will be selected using a stratified random sample from Prodege’s panel. Strata will be based on key demographic variables of interest in each wave. Recruitment efforts will be adjusted in each wave based on expected response rates and target demographics. The copy testing survey will be weighted according to Census benchmarks for key demographics.
Current Events Tracker (CET)
The CET will be a self-administered web survey. The web survey will be optimized to ensure that survey participants can complete the questionnaire on mobile devices (e.g., cell phones, tablets, etc.). For the CET, participants will be recruited to take the survey from the Ipsos KnowledgePanel every week. Ipsos will identify a sample of panel members to take the survey based on the demographic information they have provided Ipsos (e.g., age, race, ethnicity). The only eligibility requirement to complete the survey is age (18 years and older) and language preference (English). Since Ipsos includes this information as part of their panel members’ data, no screening of participants is necessary to determine eligibility for the survey. Approximately 2,857 panel members will be invited to take the survey each week to achieve the desired number of completed surveys. Ipsos currently fields an online omnibus survey (Weekend Omni) to 1,000 U.S. adults in their panel every week. Beginning in January 2021, the panel vendor will incorporate the CET into the KnowledgePanel WeekendOmni. During survey fielding, Ipsos will send email communications to invite and remind participants from its KnowledgePanel to participate in their Weekend Omnibus survey. Approximately 1,000 participants will complete each wave for up to 92 waves, totaling 92,000 participants over approximately two years. Reminders will be sent as needed to reach the targeted 1,000 completed surveys during each fielding. Fielding will last three days per week.
Foundational Focus Groups, Interviews, and/or Dyads
Foundational focus groups, interviews, and dyads will be conducted remotely (with 4-6 participants per focus group, 1 participant per interview, and 2 participants per dyad). Professional vendors will recruit the necessary participants. The vendors will be responsible for recruiting participants for the sessions from their databases and all participants will be required to go through a screener to ensure their eligibility for the sessions. Focus groups, interviews, and dyads will be conducted online via Zoom teleconference. Participants will be required to join the discussion through the link provided to them by the recruiting facility using both audio and video throughout the session. To ensure smooth discussion during the session, any participants who are unable to connect with both audio and video will be provided their incentive but will not be permitted to participate in the session. A trained moderator will guide participants through the discussion, using a combination of probing questions and activities throughout the session. Once the session is complete, the respondent(s) will be dismissed, and the online session will be closed. Each session will be recorded to help with notetaking and transcription of the discussion.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
These surveys will be conducted using an online, nonprobability-based survey panel managed by Prodege. The online surveys will be optimized to ensure that respondents can complete the questionnaire on mobile devices (e.g., cell phones, tablets). The online surveys will use a combination of audio/video clips and static images to assess respondents’ aided Campaign recall, as well as to show them new Campaign content to assess their reactions. For some surveys, respondents may be randomly assigned to an ad/message condition in which they are shown one subset of Campaign advertisements or messages.
Prodege will recruit respondents from their panel database via invitation emails. Respondents will be targeted based on basic demographic variables, such as age, race and ethnicity, in order to achieve a diverse sample that closely reflects the distribution of these demographics within the population of interest. If adolescents are involved in any waves, additional outreach to parents will be included in recruitment efforts. Respondents will also be screened by a set of requirements provided by the first few questions on the survey. These requirements include not working for the U.S. Federal government or media marketing fields, not having participated in a COVID-19 vaccine trial, and not having completed the target behavior (e.g., getting vaccinated or getting an updated vaccine). Those who do not meet the eligibility criteria for the survey will be screened out.
Current Events Tracker (CET)
Ipsos recruits panel members for each study using email for contact. Sample members will receive an initial email invitation and email reminders will be sent to those who do not respond to the initial invitation. To encourage response, sample members who complete the survey will receive an incentive equivalent to $1 in the form of points that panel members can redeem for merchandise or gift cards. The panel recruitment response rate for Ipsos’ KnowledgePanel is 12% and the cooperation rate for each survey wave is expected to be between 35% and 38%. Ipsos will take eligibility for the survey into account when sampling panel members for this survey; therefore, additional screening questions are not necessary.
Foundational Focus Groups, Interviews, and/or Dyads
Foundational focus groups, interviews, and dyads will be conducted remotely, with 4-6 participants per focus group, 1 participant per interview, and 2 participants per dyad, for a total of up to 864 participants over the course of the Campaign. Vendors will recruit 8 participants for each focus group to ensure enough participants attend each group. An incentive of $75 for adults will be provided to those who participate in the sessions to help ensure adequate recruitment of participants and recruit a more diverse group of participants overall. Vendors who recruit participants for the sessions will follow-up with participants prior to the start of their scheduled session to remind them of their participation and ensure adequate show rates for each round of data collection.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
The cooperation rate for each survey is expected to be between 30% and 35%. Potential participants will receive an invitation email from Prodege inviting them to take the survey. If adolescents are involved in any waves, parental permission will be obtained. Participants who complete the survey will receive an incentive equivalent to $5.00 in the form of points that may be redeemed for gift cards to popular retailers (online and offline) or used for donations toward charities.
Current Events Tracker (CET)
The Ipsos Weekend Omnibus typically has a high cooperation rate (exceeding 35%) for a weekly survey. Factors impacting non-response will be assessed after each administration and addressed in survey weighting to reduce non-response bias.
Foundational Focus Groups, Interviews, and/or Dyads
For each round of focus groups, vendors will recruit and enroll 144 potential subjects (8 per group) to be able to complete the focus groups among 72-108 subjects total (4–6 participants per group). Vendors will be instructed to recruit 8 potential subjects for each focus group with the goal of including 4-6 participants per group. This practice is intended to mitigate the impact of “no shows” and ensure a variety of perspectives are represented through the groups. For interviews and dyads, vendors will recruit participants for sessions on a rolling basis until we have conducted the target number of sessions for that round of data collection. Ultimately, the focus groups, interviews, and dyads are meant to be a qualitative means of collecting data and are not designed to be representative of the entire population. Results and findings will be presented in a way that reflects the overall qualitative design and its associated limitations.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
Prodege estimates the surveys will have a high cooperation rate (exceeding 30%) based on their work with similar studies. Factors impacting non-response will be assessed after each administration and addressed in survey weighting to reduce non-response bias. In later waves, estimates may be adjusted to factor in the rate of vaccination amongst the general population, which may impact nonresponse rates if our target sample (i.e., individuals who have not yet received a COVID-19 vaccine) changes.
Current Events Tracker (CET)
The panel vendor will program the survey and will check all aspects of the programming. This will include verifying the text of the instructions, questions, and response options are correct and that the skip patterns and other aspects of survey logic are programmed appropriately.
After fielding, the vendor will eliminate sample members who provided poor quality data by speeding through the survey, “straight-lining,” or skipping most of the questions. FMG will perform an independent evaluation to confirm the decisions to exclude respondents due to poor data quality. After data collection, the vendor will provide FMG with the raw data as collected from the respondents as well as the final data after any necessary editing has been completed. FMG will have two analysts independently verify that data edits are correct, and that the final data are as expected and align with the questionnaire that was used for programming.
Foundational Focus Groups, Interviews, and/or Dyads
Vendors will screen potential participants based on the provided screener to ensure they meet the definition of the “movable middle” and any additional eligibility or subgroup criteria (e.g., race/ethnicity).
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
The panel vendor will program the surveys and will check all aspects of the programming. This will include verifying the text of the instructions, questions, and response options are correct and that the skip patterns and other aspects of survey logic are programmed appropriately. FMG will independently verify these points as well to ensure the questionnaire is programmed correctly.
After fielding, the vendor will eliminate sample members who provided poor quality data by speeding through the survey, “straight-lining,” or skipping most of the questions. FMG will perform an independent evaluation to confirm the decisions to exclude respondents due to poor data quality. After data collection, the vendor will provide FMG with the raw data as collected from the respondents as well as the final data after any necessary editing has been completed. FMG will have two analysts independently verify that data edits are correct, and that the final data are as expected and align with the questionnaire that was used for programming.
Current Events Tracker (CET)
For the CET, our primary research goals include determining the following research questions:
Vaccine Uptake and Confidence: To what degree have audiences already received some or all doses of a COVID-19 vaccine? Do audiences intend to receive a COVID-19 vaccine, and what decisions and perceptions drive their intended behavior?
Vaccine Trust and Information: To what degree do audiences trust the safety and efficacy of the vaccine development process, as well as its public health and political messengers? To what degree do audiences trust individual vaccines, pharmaceutical companies, and/or vaccine development companies? To what degree does politicization or historical episodes affect the audiences’ perceptions of the COVID-19 vaccines? What other sources of information affect audiences’ degree of vaccine confidence?
HHS Trust/Campaign Brand Equity: To what extent do audiences trust HHS as a source of information on COVID-19? How does this compare to other information sources?
Knowledge: To what extent do audiences understand the vaccine development and testing process, vaccine effectiveness, and vaccine safety? To what extent can audiences accurately identify COVID-19 misinformation and rumors?
The CET will serve as a measure for both attitudes and behavior related to COVID-19 vaccines and preventive behaviors among the U. S. population as a whole and key subpopulations. Measures of COVID-19 vaccine uptake and confidence offer both a way to ascertain campaign impact in terms of change in targeted outcomes that are associated with fewer COVID-19 infections, as well as to point campaign leadership to areas in greater need for adjusting messaging content, delivery, or coverage. The CET provides a means of quickly examining the general public’s key attitude positions, health vulnerabilities, and trust in experts to help inform and guide the campaign. Those questions that are fielded consistently from week to week will be examined over time to see how the public’s attitudes are changing during the course of the campaign; however, this survey will not be used to determine campaign-attributable change in attitudes or behaviors. This survey is designed to understand how American’s information needs may change during the course of the campaign, so that messaging can adapt to meet these needs. The flexibility of the chosen omnibus survey will also allow ASPA to collect findings on emergent information needs with a very short lead time, ensuring the Campaign can best respond to current events and milestones without require extensive coordination.
Copies of each wave’s survey are uploaded and include repeated questions, revised questions, and new questions approved through nonsubstantive change requests. New and revised questions are designed to capture emergent research needs and are designed to be changed from week to week as needed. For example, it may be important to understand how current events, such as vaccine developments or rollout, impact vaccine confidence and uptake. These questions will be identified and added to the survey as relevant; we cannot anticipate these questions so must adapt the survey accordingly over time. ASPA will provide OMB with updated questionnaires for subsequent waves of fielding throughout the duration of the study.
Foundational Focus Groups, Interviews, and/or Dyads
The purpose of the foundational focus groups, interviews, and dyads is to explore key audiences’ perceptions and attitudes toward the COVID-19 pandemic and what preventive measures they take daily to avoid viral transmission. The success of the HHS COVID-19 Public Education Campaign depends on the ability to uncover and identify the target audiences’ understanding of the pandemic and their wants, needs, preferred messaging channel and source preferences when it comes to learning more about COVID-19. In addition, we want to better understand their usage of preventive measures to slow the spread of the virus.
A copy of the moderator guide for the first round of focus groups was included as an attachment to the initial submission. The discussion guides will change from round to round based on current events and the environment surrounding COVID-19 and vaccination; however, primary research questions will focus around participants’ experiences and perceptions with COVID-19 and their attitudes, experiences, and choices with respect to the COVID-19 vaccine. Questions will be adjusted based on how the pandemic has changed over time and how the vaccine rollout has progressed. ASPA will provide the final discussion guides for subsequent rounds of data collection to OMB.
The primary purpose of these focus groups, interviews, and dyads will be to inform the HHS COVID-19 Public Education Campaign strategy about audience risk knowledge, perceptions, current behaviors, and barriers and motivators to healthy behaviors (including getting the COVID-19 vaccine). Ultimately, these findings will also help inform the development of messages and materials, which, in turn, will be produced and disseminated to the general public.
Copy Testing Surveys
The primary purpose of the copy testing surveys is to test whether and to what extent the advertising is having the intended effect on the target audience’s attitudes and behavioral intentions. Key measures on the survey will include creative comprehension, resonance, relevance, and attitude and behavior change. Findings from the survey will be used internally to refine and enhance creative assets. Ultimately, these findings will help inform the development of messages and materials, which, in turn, will be disseminated to the general public.
A copy of the survey instrument was included in original submission. We anticipate the survey instrument will be largely unchanged from Wave to Wave, with the key exception of modifying the stimuli that are being tested. ASPA will submit nonsubstantial change requests for the survey as warranted across waves.
Message Matrix Surveys
The purpose of the Messaging Matrix surveys is to evaluate, validate, and prioritize Campaign messages for various target audiences. The surveys test Campaign messages, including CDC-approved facts about COVID-19, to help ASPA understand which messages are most effective at influencing COVID-19-related attitudes and behaviors.
A copy of the survey instrument is included with this submission. We anticipate the survey instrument will be largely unchanged from Wave to Wave, with the key exception of modifying the messages that are being tested. ASPA will submit nonsubstantial change requests for the survey as warranted across waves.
Creative Testing Surveys and Experiments
The purpose of the Creative Testing Surveys and Experiments is to assess participant reactions to various Campaign materials to inform the selection and development of creative concepts, messages, or material format used for campaign outreach to key audiences. Key measures may include perceived effectiveness of messages, the influence of material format (e.g., video vs. static image ad), likelihood of engaging in desired behaviors, and changes in COVID-related knowledge after viewing Campaign materials.
A copy of the survey instrument is included with this submission. We anticipate the survey instrument will be largely unchanged from Wave to Wave, with the key exception of modifying the stimuli that are being tested. ASPA will submit nonsubstantial change requests for the survey as warranted across waves.
Current Events Tracker (CET)
Cognitive testing was conducted on the CET prior to finalizing the survey instrument. The cognitive testing was used to refine the survey questions and improve utility. A total of six individuals participated in the testing and debriefs for the CET instrument. Participants were recruited to ensure diversification by age, gender, race/ethnicity, and educational attainment.
Trained moderators conducted the cognitive interviews virtually using the Zoom platform. Participants used screen sharing to scroll through a mock online survey and were asked to use the think-aloud method to describe their thought process and response selection as they proceeded through the test survey. After completing the survey instrument, the interviewer asked participants a series of debriefing questions that focused on the survey overall and specific survey questions. The debriefing questions focused on items that participants expressed questions or confusion with during completion of the mock survey, as well as specific items and wording that the researchers had identified in advance.
All interviews were recorded to capture participant comments on the survey, and any questions that required revisions. The data collection did not focus on the actual survey instrument responses but rather on how well the questions were understood and how well they conveyed the research intentions. All identifiable information was kept private to the extent allowable by law and destroyed subsequent to the conclusion of all interviews. Revisions to the questionnaire were made based on the results of the survey instrument testing.
Foundational Focus Groups, Interviews, and/or Dyads
Trained moderators/interviewers will conduct the foundational focus groups, interviews, and dyads using the Zoom platform. Moderators/interviewers will participate in trainings provided by the research team prior to the sessions to review in detail project information and materials. They will also receive training in using the online Zoom platform to ensure that they can conduct the sessions effectively with few issues. Dry runs of each round of data collection will be conducted by the moderators/interviewers to become familiar with the discussion guide and activities, making sure that they are comfortable managing the amount of time dedicated to each topic in the guide.
Copy Testing Surveys, Message Matrix Surveys, and Creative Testing Surveys and Experiments
A subset of questions on these surveys was tested prior to finalizing the survey instrument. The cognitive testing was used to refine the survey questions and improve utility. Trained moderators conducted the cognitive interviews virtually using the Zoom platform. Participants used screen sharing to scroll through a mock online survey and were asked to use the think-aloud method to describe their thought process and response selection as they proceeded through the test survey. After completing the survey instrument, the interviewer asked participants a series of debriefing questions that focused on the survey overall and specific survey questions. The debriefing questions focused on items that participants expressed questions or confusion with during completion of the mock survey, as well as specific items and wording that the researchers had identified in advance.
All interviews were recorded to capture participant comments on the survey, and any questions that required revisions. The data collection did not focus on the actual survey instrument responses but rather on how well the questions were understood and how well they conveyed the research intentions. All identifiable information was kept private to the extent allowable by law and destroyed subsequent to the conclusion of all interviews. Revisions to the questionnaire were made based on the results of the survey instrument testing.
The following persons outside of HHS contributed to, reviewed, and/or approved the design, instrumentation and sampling plan:
Name |
Affiliation |
Telephone Number |
Leah Hoffman, MPH |
Fors Marsh Group |
571-444-1694 |
Lindsey Strausser |
Fors Marsh Group |
571-858-3773 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement – Part B |
| Subject | HIP 2.0; Healthy Indiana Plan, CMS; Supporting Statement |
| Author | Centers for Medicare & Medicaid Services |
| File Modified | 0000-00-00 |
| File Created | 2023-07-29 |