ToxFAQs Before and After Testing Protocol

Attachment A: ToxFAQs Before and After Testing Protocol

ToxFAQs Before and After Testing Protocol

Table of Contents

Objectives 1

Participants 1

Methods 2

Final Report and Recommendations 3

Objectives

The Agency for Toxic Substances and Disease Registry (ATSDR) is updating a series of popular ToxFAQs — fact sheets about a toxic chemical. Each new ToxFAQ will feature plain language edits and design updates, helping readers understand and use the information.

ATSDR is testing the before and after versions of the carbon monoxide ToxFAQ (See Attachment B) to evaluate whether participants rate the after version as easier to understand, more useful, and more attractive than the original version. On behalf of ATSDR, CommunicateHealth will evaluate the ToxFAQ before and after versions via an online survey with 250 consumers.

Specifically, ATSDR wants to learn:

1. Will exposure to the after version increase participants’ knowledge about carbon monoxide?  

2. Which version will participants rate as easier to understand, more useful, and better designed?

3. After comparison, which version will participants rate more effective overall?

CommunicateHealth will deliver a topline summary of findings and recommendations to ATSDR. Personally identifiable information will not be collected.

Participants

CommunicateHealth will conduct a Qualtrics online survey with 250 total participants. IT security clearance for the use of the Qualtrics online survey has been granted by the NCEH Information System Security Officer (ISSO). Our desired participants are consumers who represent a mix of education levels, race/ethnicities, income levels, and geographic locations.

Specifically, participants should represent a mix of the following characteristics:

• Age: Mixed, 18 and over

• Gender and Education: Mixed

• Race/Ethnicity: Mixed

• Education Level: Mixed (At least one third of participants cite high school as the highest level of education completed)

• Income: Mixed (At least one half of participants cite an income of $50,000 or less)

• Geography: Mixed

• Special Group Status: Consumers

We will exclude from participation people who:

• Are under the age of 18

• Have participated in any research within the past 6 months

• Conduct market research or work in advertising or public relations

• Are unable to speak and read English well enough to complete the study

Recruitment

Participants will be recruited through a Qualtrics online survey panel using screening parameters based on participant characteristics above. Qualtrics will use their nationwide participant database to identify qualifying participants. Contact with participants is not required when using the nationwide database to identify eligible participants. Participants will not be contacted unless they qualify to receive the email invitation. Qualtrics will send an email invitation to potential participants (Attachment C), which will include the link to begin the survey and information about non-monetary reward points.

Participants will:

• Be a diverse mix, recruited from a national Qualtrics participant panel, including at least half of participants with low income and a third with limited education levels

• Be offered Qualtrics rewards points (small-non-monetary incentives) as a token of appreciation for their participation

• Be asked to review materials and answer questions about them in the 20-minute survey; rewards points have been shown to lead to sufficient recruitment and retention for the tasks required to enable successful completion of the project

This small non-monetary incentive will:

• Be consistent with industry standards for quantitative data collection efforts

• Be given by Qualtrics (not CommunicateHealth or ATSDR) to panel participants for each completed survey they take through Qualtrics

• Accumulate for panel participants as they take additional surveys (unrelated to this study) through Qualtrics, eventually being redeemed for items such as gift cards and subscriptions (e.g., magazines)

• Be needed to receive a sufficient response rate, based on CommunicateHealth’s and Qualtrics’ previous experiences to ensure recruitment and retention for this hard-to-reach population (diverse, low income, and limited education) in similar studies

• Be required by Qualtrics in order to conduct recruitment from their diverse national panel of participants

Methods

ATSDR will conduct before and after testing via a modified randomized control trial (RCT), using the online survey platform Qualtrics.

Participants will be randomized into either the control or intervention arm. Control arm participants will review and answer questions about the before version of the carbon monoxide ToxFAQ. Intervention arm participants will do the same for the after version of the carbon monoxide ToxFAQ. Participants will not be aware of which version they’re reviewing. (See Attachment B for before and after versions of ToxFAQ fact sheets.) All participants will be asked to compare and evaluate side-by-side excerpts of the before and after versions.

The survey will take 20 minutes to complete (Attachment D [text] & E [online]).

The survey will include:

• Consent

• Randomization to control or intervention

• Demographic questions (no identifiable information will be collected)

• Review full, 2-page ToxFAQ before version (control) or after version (intervention)

  • Knowledge questions about ToxFAQ

  • Questions adapted from the Consumer Information Rating Form (CIRF) developed by Krass et al. (2002) on comprehensibility, utility, and design quality

• Side-by-side comparison questions about excerpts from each version

• Thank you

Handling of Data and Records

No individually identifiable information is being collected in this study. The proposed data collection will have little or no effect on the participants’ privacy. Only comments, quotes, and responses from participants will be noted and used as feedback to inform revisions to the messages being tested.

IRB review

This project is classified as non-research and does not require IRB review.

Final Report and Recommendations

CommunicateHealth will analyze feedback from the survey to identify differences in comprehension, utility, and attractiveness between the two versions. Findings will be summarized in a top-line report and inform recommendations on how to improve future ToxFAQ revisions. The results obtained from this feedback will be used internally for ATSDR program improvement and the results will not be published in a public report or presented at public meetings.