Gathering Consumer Feedback on Warning Symbols

TITLE OF INFORMATION COLLECTION:

Gathering Consumer Feedback on Warning Symbols

PURPOSE

This research is designed to assess how well currently used safety symbols/pictograms (“graphical symbols”), and a set of newly developed graphical symbols, effectively communicate hazards posed by some consumer products. This research will follow the open comprehension testing procedures articulated in ANSI Z535.3, the American National Standard Criteria for Safety Symbols. ANSI Z535 is the primary U.S. voluntary standard for guiding the design of signs, colors, and symbols intended to identify and warn against specific hazards and for other accident prevention purposes. For symbols that have been previously designed and validated in accordance with ISO rules of graphical symbols, this research will serve to verify the symbols’ understandability in the United States. Kalsher & Associates, LLC (“the project team”), is collecting the information. U.S. Consumer Product Safety Commission (CPSC) staff will guide and support the project, as needed. The CPSC is authorized under section 5(a) of the CPSA, 15 U.S.C. 2054(a), to conduct research relating to the causes and prevention of injuries or deaths associated with consumer products.

DESCRIPTION OF RESPONDENTS

The project team will recruit a total of 80 participants to participate in (1) cognitive interviews to assess comprehension of the graphical symbols, and/or (2) focus groups to achieve a deeper understanding of the characteristics of the graphical symbols that contribute to, or detract from, their effectiveness in communicating their respective intended messages.

The project team will recruit participants via a variety of means, including emails, fliers posted at strategic locations, and word-of-mouth contact. Research participants will receive $25 for each activity completed. Participants who choose to leave before completing either (or both) of the tasks will be compensated on a pro-rata basis for the work they have completed up to that point.

• Cognitive Interviews. Eighty community volunteers will evaluate the graphical symbols, using the ANSI Z535.3 Open Comprehension Testing procedure (see Appendix B, pp. 24–34). This task should take no more than 1 hour to complete. The project team will present the graphical symbols along with a textual description of the context in which they will be used and where the symbol would be located. The project team may use paper-and-pencil or computer-based testing to determine whether respondents, matched on relevant characteristics, perform differently using the different test administration formats.

• Focus Groups. Members of the project team will conduct approximately four focus groups, each including approximately 10 participants (for a total of 40 participants) and lasting about 1 hour. The focus group discussions will address participant awareness and comprehension of the pictograms being tested, self-reported factors that could potentially impact whether participants would follow the warnings, and suggestions for how to modify the pictograms to improve their effectiveness. The focus group sessions will be audio-recorded to facilitate subsequent efforts to modify/improve graphical symbols that perform poorly. The project team will gather permission to record these sessions as part of the IRB-approved consent procedure. Only members of the project team will listen to the recorded sessions. Please note that the subject of the research is the graphical symbols, not the participants.

GIFTS OR PAYMENTS

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? Yes [$50 for both activities, or $25 for one activity]

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Cognitive Interview Only	40	1 hour	40 hours
Cognitive Interview plus Focus Group	40	2 hours	80 hours
Totals	80		120 hours

FEDERAL COST: The estimated annual cost to the federal government is $54,830.

Total estimated cost to the government for conducting the data collection is as follows:

Number of Participants 80

Total estimated cost of conducting the evaluation sessions $54,830

Average cost per completed Participant, including $25 or $50 stipend $685.38

The estimate is based on the total cost of the awarded research contract, divided by the specified number of completed participants.

APPENDICES: TABLE OF CONTENTS

Appendix A: Cognitive interview instructions / booklets

Appendix B: Focus group protocol

Appendix C: Email invitation to participate in research

Appendix D: Screener survey

Appendix E: Research session confirmation email

Appendix F: Sample consent form – Cognitive interview only

Appendix G: Sample consent form – Cognitive interview and focus group

Appendix A: Cognitive interview instructions/booklets

Cognitive Interview: Facilitator Instructions

“This booklet contains numerous examples of graphical symbols. You might encounter these on product labels, on machines, in workplaces, or in public areas. We want to see how well each of the symbols communicates its intended message. You will be helping us do this by writing down what you think each symbol means.”

“It is very important that you write down exactly what you believe is the intended message. If your answer is too vague or general, we will not be able to determine whether the correct message is getting across.”

“Open your booklet to page one. These are examples of what you will be seeing. Each page will have an illustration of a symbol that you might encounter on a sign or label. There is a blank for you to write down exactly what you believe the symbol means and another blank for you to write down the action that you believe should be taken in response to this symbol.”

“This page shows you an example of both an inadequate and a good answer.”

“The first example is too brief and general. The second example shows a much better answer. It describes the exact meaning of the symbol and it is specific in telling just what actions should be taken. Make sure that all of your answers are clear and precise, like the good example.”

“There are many different kinds of symbols in your booklets. Some may be familiar to you, and others you may never have seen before. Just do the best you can for each, and take an “educated guess,” if you are not sure of the meaning. Remember, the symbols are being tested, not you! When you think about what the symbol might mean, remember that these are typically found on signs or labels that could be encountered on products, on machines, or in public places.”

“It is important that you work alone. Do not talk to anyone or make comments out loud. Work through the booklet one page at a time. Once you finish a page, do not go back over it. There is no time limit, but try not to spend too much time on any particular symbol. If you don’t know the meaning of a symbol, just make your best guess and go on. When you have completed the entire booklet, including the biographical items on the last page, please bring it to me.”

“Are there any questions before we begin?”

“Now please turn the page and begin the symbol booklet. Remember to print your responses clearly.”

COGNITIVE INTERVIEW BOOKLET

Name (please print): ____________________________________________________________

Today’s Date: ____________________________________________

EXAMPLE OF AN INADEQUATE ANSWER
	Context: This symbol appears on appliances and machines used in the home and workplace.
Exactly what do you think this symbol means? Gears and hand
What action should you take in response to this symbol? Be careful
What could happen if you ignore the symbol’s message? Could get hurt
EXAMPLE OF A GOOD ANSWER
	Context: This symbol appears on appliances and machines used in the home and workplace.
Exactly what do you think this symbol means? Caution. Moving Gears. Do not stick hand near machine while it is running.
What action should you take in response to this symbol? I would stay away and not put my hand near the machine until someone stopped it.
What could happen if you ignore the symbol’s message? My fingers or hand could be crushed by the moving gears.

Page 1

INSERT GRAPHICAL SYMBOL HERE	Contextual description for graphical symbols currently in use: Context: This symbol appears on ….. Contextual description for newly developed graphical symbols not yet in use: Context: This symbol might appear on …..
Exactly what do you think this symbol means?
What action should you take in response to this symbol?
What could happen if you ignore the symbol’s message?

Page 2

Biographical Information

Please answer the following questions about yourself:

Age: _________ Sex: ____ Male ____ Female

Education: (check the option that best describes the highest level of education you have attained)

____ Some high school ____ High school degree ____ Some college

____ 2-year college degree ____ 4-year college degree ____ Master’s degree

____ Doctoral or professional degree

____ Other (please specify): _____________________________________________________

Marital Status:

____ Single ____ Married ____ Legally Separated ____ Divorced

Children: ____ Yes ____ No

(If yes: Please indicate the number of children that live in your home, either full-time or part-time, and their ages in the space provided):

Race:

____ Asian ____ Black/African ____ Caucasian

____ Hispanic/Latino ____ Native American ____ Pacific Islander

____ Mixed Race ____ Prefer not to answer

Current occupation: ___________________________________________________

Page X

Appendix B: Focus group protocol

General Description of the Focus Group Data Collection Method and Screening Procedures

Participant Universe and Sampling Methods

Focus group studies are directed group discussions that enable observers to infer the underlying views and assumptions of the group’s participants. Focus group research relies on qualitative methods and is not intended to produce results that can be generalized to the overall population. Results of this research will not be used to make statements representative of the universe of study, to produce statistical descriptions (careful, repeatable measurements), or to generalize the information beyond the scope of the study sample.

For these focus groups, we may use quota sampling to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience (average consumers in the U.S. over the age of 18 years of age, relatively balanced in terms of sex and age groupings). We may also use cluster sampling to form one or more focus groups (e.g., contacting local schools systems and related organizations, such as the local YMCA). Piggybacking upon an initial gathering via a snowballing approach may bring together additional participants we are seeking. We will request basic demographic information from potential participants as a screening mechanism to ensure that we have a relatively balanced sample in terms of sex, age, education and ethnicity.

To reduce the number of no-shows, scheduled focus group participants will be sent a reminder letter and/or telephone call giving the time of the interview and directions to the location. Tracking response rate is not applicable since we will almost certainly attain a nonprobability sample that is not representative of the overall target population.

The focus group sessions will be held in locations that participants travel to by car or short-range public transportation. The project team will perform all data collection and analysis in compliance with OMB standards and guidance, and with the Privacy Act.

Procedures for the Collection of Information

Focus groups are a form of group interview used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process. Here, we will use the focus groups to aid in the development of new graphical symbols and to refine graphical symbols currently in use. The goal is to optimize the ability of the set of symbols to accurately communicate their intended message. A focus group typically is composed of approximately 10 people who have characteristics similar to the target audience or subgroups of the target audience. These sessions are conducted by a moderator who uses a loosely structured discussion outline to keep the sessions on track while allowing participants to talk openly and spontaneously and to change direction as the discussion unfolds and new topics emerge.

In general, the moderator will avoid asking yes/no questions or leading questions and will refrain from giving examples that might cue the right answer. Additionally, the moderator will ask questions about positive aspects of a symbol before turning to (potentially) negative aspects of a symbol. The moderator may proceed with probes (e.g., “What else?” “Does anyone have a different thought?”) and follow-up questions depending on the flow of the discussion.

Test of Procedures or Methods to be Undertaken

Prior to the initial focus group session, we will pilot test the proposed procedures and materials, including the facilitator guide, facilitator script and sample questions (see sections below).

Sample Materials

Focus Group Facilitator Script

Hello, my name is <name>, and I am working on this project with staff of the U.S. Consumer Product Safety Commission. I would like to thank you very much for your participation today. Your input is very valuable to us.

Before we begin, I will give you some background on what we are doing . . .

I am here to ask questions to keep the conversation moving and to get information about how we might redesign graphical safety symbols so they are more effective in communicating the hazards associated with some consumer products. There are no right or wrong answers. All of your feedback is valuable.

Our conversation will be audio-recorded for the purposes of notetaking and referencing, but the audio-recording will remain confidential and listened to by members of our research team only. Once the project ends, we will destroy the audio recordings.

Do you have any questions? Okay, let’s begin.

Note: The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the focus group in a natural and conversational manner, paraphrasing or giving additional explanation, as appropriate. The protocol provides a list of suggested questions to ask, but the facilitator will be allowed the flexibility to reorganize or modify questions, depending on the flow of the discussion.

Sample Focus Group Questions

“The sheet that I’m handing out contains the set of symbols you evaluated in the previous session. Take a moment to look them over. Were any of the symbols particularly easy or difficult to understand? Let’s start with the ones that seemed easy to understand.”

[Allow time for each speaker to state their views.]

“Let’s start with this symbol [select one to focus on].

For symbols that you feel are well understood:

“Did you understand the meaning of the symbol right away, or did you have to think about what it means?”

Did the colors in the symbol help with understanding? [if relevant]

“Can you tell me specific characteristics of the symbol that made it particularly easy for you to understand? Do you have any suggestions for how the symbol could be improved?”

“Whenever possible, simpler is better when it comes to symbol design. The concreteness of the potential hazard is obviously a consideration. For example, physical hazards, such as the hands in gears symbol is often easier to understand, as compared to less concrete hazards, such as radiation. Can you think of ways this symbol could be simplified to eliminate any unnecessary detail?”

“Do you think the symbol will be well understood on its own? Or, is additional supporting text necessary to facilitate correct understanding of the symbol’s meaning? If yes, do you have any suggestions for the supporting text?”

For symbols that you feel were not adequately understood:

“Can anyone tell me what this symbol is intended to communicate?”

[If no participant can correctly state the intended meaning, say: “This symbol was intended to communicate [state the hazard to be communicated].”

“Can you describe the characteristics of the symbol, or the hazard, that made it particularly difficult to understand? Do you have any suggestions for how the symbol could be improved? If needed, do you have any suggestions for text that could accompany the symbol to make it more understandable?”

Appendix C: Email invitation to participate in research

Cognitive Interview Only

<Date>

Subject: Invitation to participate in a research study

Hi <name>,

I am contacting you to invite you to participate in a research study on graphical symbols. A graphical symbol is a visible figure with a particular meaning used to transmit information independently of language. Graphical symbols are often used to convey important messages about product features, directions, or safety. If you decide to participate, you will be asked to evaluate a set of graphical symbols intended to communicate non-obvious hazards associated with some consumer products.

The session will take place on <date> at <time> at <location>. To thank you for your participation, you will receive $25. If you choose to leave the study before completing the task, you will be paid on a pro-rata basis for the work completed.

Please fill out this brief survey <link to screener>, and we will get back to you soon. We are looking forward to speaking with you, and we appreciate your participation!

Sincerely,

<Name>

<Title>

Cognitive Interview and Focus Group

<Date>

Subject: Invitation to participate in a research study

Hi <name>,

I am contacting you to invite you to participate in a research study on graphical symbols. A graphical symbol is a visible figure with a particular meaning used to transmit information independently of language. Graphical symbols are often used to convey important messages about product features, directions, or safety. If you decide to participate, you will be asked to evaluate a set of graphical symbols intended to communicate non-obvious hazards associated with some consumer products and then participate in a focus group discussion of the symbols. In exchange, you will receive $25 for evaluating the symbols, and $25 for your participation in the focus group, for a total of $50. If you choose to leave the study before completing both tasks, you will be paid on a pro-rata basis for the work completed.

Please fill out this brief survey <link to screener>, and we will get back to you soon. We are looking forward to speaking with you, and we appreciate your participation!

Sincerely,

<Name>

<Title>

Appendix D: Screener survey

Please answer the following questions about yourself:

Age: _________ Sex: ____ Male ____ Female

Education: (check the option that best describes the highest level of education you have attained)

____ Some high school ____ High school degree ____ Some college

____ 2-year college degree ____ 4-year college degree ____ Master’s degree

____ Doctoral or professional degree

____ Other (please specify): _____________________________________________________

Marital Status:

____ Single ____ Married ____ Legally Separated ____ Divorced

Children: ____ Yes ____ No

(If yes: Please indicate in the space provided the number of children that live in your home, either full-time or part-time, and their ages):

Race:

____ Asian ____ Black/African ____ Caucasian

____ Hispanic/Latino ____ Native American ____ Pacific Islander

____ Mixed Race ____ Prefer not to answer

Current occupation: ___________________________________________________

Appendix E: Research session confirmation email

Cognitive Interview Only

<Date>

Subject: Your research participation on <date>

Hi <name>,

Thank you so much for agreeing to participate in a research study on graphical symbols. A graphical symbol is a visible figure with a particular meaning used to transmit information independently of language. Graphical symbols are often used to convey important messages about product features, directions, or safety. We greatly appreciate your assistance in helping us assess and refine graphical safety symbols intended to communicate non-obvious hazards associated with some consumer products. The session will take place on <date> at <time> at <location>. The session will take about 1 hour.

As a reminder, you will be evaluating a set of graphical symbols. To thank you for your participation, you will receive $25. If you choose to leave the study before completing the evaluation task, you will be paid on a pro-rata basis for the work completed.

If you have any questions, please call <Name> at <number>. We are looking forward to your participation!

Thank you,

<Name>

<Title>

Cognitive Interview and Focus Group

<Date>

Subject: Your research participation on <date>

Hi <name>,

Thank you so much for agreeing to participate in a research study on graphical symbols. A graphical symbol is a visible figure with a particular meaning used to transmit information independently of language. Graphical symbols are often used to convey important messages about product features, directions, or safety. We greatly appreciate your assistance in helping us assess and refine graphical safety symbols intended to communicate non-obvious hazards associated with some consumer products. The session will take place on <date> at <time> at <location>. The session will take about 2 hours.

As a reminder, you will be evaluating a set of graphical symbols and then participating in a focus group in which we will discuss the symbols in greater detail. To thank you for your participation, you will receive $25 per completed research activity, for a possible total of $50. If you choose to leave the study before completing the tasks, you will be paid on a pro-rata basis for the work completed.

If you have any questions, please call <Name> at <number>. We are looking forward to your participation!

Thank you,

<Name>

<Title>

Appendix F: Sample consent form – Cognitive interview only

Informed Consent Agreement

Please read this consent agreement carefully before you decide to participate in this study.

Purpose of the research study: This research study is designed to assess how well graphical safety symbols effectively communicate non-obvious hazards associated with products and equipment to consumers. This research could be used to improve the effectiveness of graphical symbols to communicate safety-related information to diverse audiences. Given the growing diversity of the U.S. population, in concert with the rapid expansion in global trade, this is an important goal. The graphical symbols that you will be asked to evaluate were selected based on injury data associated with consumer products and equipment and the severity of the non-obvious hazards that threaten consumers.

What you will do in the study: You will be asked to evaluate a set of symbols. Each symbol will include a description of the context in which it may occur. You will answer the following questions about each symbol: “Exactly what do you think this symbol means?” “What actions should you take in response to seeing the symbol?” “What could happen if you ignore the symbol’s message?” It is important to remember that the symbols are being evaluated—not you.

Time required: This task should take no more than 1 hour of your time.

Risks: There are no known risks associated with participation in this study.

Benefits: You will receive $25 each for completing the symbol evaluation task.

Confidentiality/Anonymous data: The information you give in the study will be handled confidentially. Your rating data will be anonymous, which means that your name will not be linked to the data.

Right to withdraw from the study: Your participation in this research is completely voluntary. You may refuse to participate or stop participating at any time and for any reason. If you choose to stop participating before completing the evaluation task, you will be paid on a pro-rata basis for the portion you have completed up to that point.

If you have questions about the study, contact:

Michael J. Kalsher

Department of Cognitive Science

301-F Carnegie Bldg.

110 8^th Street
RPI, Troy, NY 12180
Telephone: 518-276-8267

[email protected]

Chair

Institutional Review Board

Rensselaer Polytechnic Institute

CII 9015

110 8^th Street

Troy, NY 12180

Telephone: 518-276-4873

[email protected]

____________________________________________________________________________________

By signing below, you agree to participate in this research study, and acknowledge that you have read and understood the information provided above.

Print Name: _________________________________________________________________

Signature: _______________________________________________ Date: _____________

Email address: _______________________________________________________________

Appendix G: Sample consent form – Cognitive interview and focus group

Informed Consent Agreement

Please read this consent agreement carefully before you decide to participate in this study.

Purpose of the research study: The purpose of this research is to assess the extent to which graphical safety symbols effectively communicate non-obvious hazards associated with products and equipment to consumers. This research could be used to improve the effectiveness of graphical symbols to communicate safety-related information to diverse audiences. Given the growing diversity of the U.S. population, in concert with the rapid expansion in global trade, this is an important goal. The graphical symbols that you will be asked to evaluate were selected based on injury data associated with consumer products and equipment and the severity of the non-obvious hazards that threaten consumers.

What you will do in the study: You will be asked to evaluate a set of symbols. Each symbol will include a description of the context in which it may occur. For the evaluation task, you will answer the following questions about each symbol: “Exactly what do you think this symbol means?” “What actions should you take in response to seeing the symbol?” “What could happen if you ignore the symbol’s message?”

Afterward, you will participate in a focus group to discuss the symbols in greater detail. The purpose of the focus groups is to gather more detailed information about each symbol to guide efforts to redesign symbols that perform poorly, or conversely, to help us understand why certain symbols perform well. It is important to remember that it is the symbols that are being evaluated—not you.

Time required: Each task should take no more than 1 hour to complete. Participating in both tasks will take about 2 hours of your time.

Risks: There are no known risks associated with participation in this study.

Benefits: You will receive $25 each for each completed task (i.e., the symbol evaluation task and participation in a focus group). If you complete both tasks, you will receive a total of $50.

Confidentiality/Anonymous data: The information you give in the study will be handled confidentially. Your rating data will be anonymous, which means that your name will not be linked to the data. We wish to audio-record the focus group sessions so that these discussions are available to the research team as we work to redesign symbols that perform poorly. Only members of the research team will listen to the audio recordings. The audio recordings will be destroyed once the study is completed.

Right to withdraw from the study: Your participation in this research is completely voluntary. You may refuse to participate, or stop participating at any time, and for any reason. If you choose to stop participating before completing one or both tasks, you will be paid on a pro-rata basis for the portion of the tasks you have completed up to that point.

If you have questions about the study, contact:

Michael J. Kalsher

Department of Cognitive Science

301-F Carnegie Bldg.

110 8^th Street
RPI, Troy, NY 12180
Telephone: 518-276-8267

[email protected]

Chair

Institutional Review Board

Rensselaer Polytechnic Institute

CII 9015

110 8^th Street

Troy, NY 12180

Telephone: 518-276-4873

[email protected]

____________________________________________________________________________________

By signing below, you agree to participate in this research study, and acknowledge that you have read and understood the information provided above.

Print Name: _________________________________________________________________

Signature: _______________________________________________ Date: _____________

Email address: _______________________________________________________________

24