Appendix A: Cognitive Interview Protocol
Evaluation of Recreational Off-Highway Vehicle (“ROV”) Hang Tag
Cognitive Interview Protocol
INTERVIEWER INSTRUCTIONS: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate.
The protocol provides a list of suggested questions to ask and guidelines for specific areas of the ROV hang tag that potentially need to be tested. However, the interview will require a flexible probing approach, selection of relevant probes or modification of the listed probe formulation depending on the participant think-aloud and answers to the survey items. Text written in italics is to be read aloud by the interviewer.
Hello, my name is ______ and I work for EurekaFacts. It’s nice to meet you and thank you very much for helping us out today.
Let me begin by explaining our goal and what you are going to be doing today. You are participating in a study on issues related to recreational off-highway vehicles (“ROV” for short). You may be more familiar with terms such as UTV, side-by-side ATV, or just off-road vehicle. ROVs are off-road vehicles that have side-by-side seating, a steering wheel, and foot pedals. We will not be talking about traditional ATVs, with handlebars and motorcycle-style seating today. The Consumer Product Safety Commission (“CPSC”) proposes that ROVs should have a hang tag attached to the ROV at the time that the ROV is purchased from the dealer. In this study we will explore customers’ opinions about a proposed design of the ROV label. I will ask for your opinion about various features and information being presented to you.
The purpose of this study is to help us make sure that the information provided on the label is clear and understandable for customers like you, and helps them make informed decisions during ROV purchase.
Some key things to remember as we move along:
There are no right or wrong answers. We are not testing you or your knowledge. We are here to evaluate an informational tool.
What we are concerned about are the features, functionality, and usability of the information about ROVs presented to you. This will help provide us with information that will help ROV buyers.
We are also very interested in your thoughts and opinions as we proceed. So please “think aloud” as you look at and read through the provided information. During a “think aloud” I would like you to verbalize everything that you are thinking. Tell me your impressions and thoughts, while you are trying to understand the information we are dealing with, and also if there is anything that you are not sure of, or are confused by.
We are interested in both positive and negative feedback you might have. I did not design the material we will look at so your comments do not affect me personally.
When you would like to talk about something in particular, indicate and point to (either with your finger or the mouse) the section you are talking about.
This interview is being observed by my colleagues, who will be taking notes about your comments, and it will also be audio recorded so they can review the recordings later. No one else will hear or review the recordings except for the researchers on this project.
Neither your name nor any other identifying information about you will be associated with the recordings or any other information you provide in today’s interview.
If at any time you decide you need a break we can do that, or if you do not want to go on, that is your choice and you may stop.
Do you have any questions? If you have any questions at any point, please just ask.
Before we begin, I want to reassure you that we will keep the information you give us confidential. The interview is conducted for research purposes only, and information and comments that you will provide will not be shared directly with Consumer Product Safety Commission (“CPSC”) or any other government agency.
We will work together for about 30 minutes.
Do you have any questions?
Okay, let’s begin.
II. Think Aloud Demonstration and Practice
I’m now going to show you how to think out loud. When I am finished you will get a chance to practice thinking aloud. Since we can’t tell what is going on in your brain, we need you to “think aloud.”
The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do. Just say aloud the words and the thoughts that are in your head.
Let me give you an example (Demonstrate think-aloud with example 1 – interviewer will talk through their decision making process before making a selection).
Example 1:
When you are hungry in the afternoon which of the following choices would you consider buying from a vending machine? (Please check all that apply)
___ Chocolate Bar
___ Potato Chips
___ Granola Bar
___ Animal Crackers
___ Pretzels
___ Soda
Other ___________________________________ (Please specify)
If you are ready, now you will do a think-aloud practice. I’m going to give you a simple question. I want you to read it out loud and think aloud as you figure out your answer. Remember to say out loud whatever you are thinking.
If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are confused (and what is confusing you) and times when you understand what you are doing.
Example 2:
How many windows are in your home?___________
Provide the print out of the label to the participant.
Please take a look at this. I would like you to take some time to read and explore the information provided here and please think out loud as you do so.
If the participant is not verbalizing within 15 seconds, interviewer should offer a verbal “nudge” to remind the participant to keep talking, such as:
What is going on in your head at this time?
What are you thinking?
You seem to be thinking hard—can you tell me what was in your mind right then?
What are you thinking now?
Any other thoughts?
If the participant says something about his/her thoughts but it is difficult to tell exactly what the participant means, or if the participant begins to talk but does not say very much, interviewer can prompt the participant to elaborate, for example:
Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?
If more clarity or a more explicit description of the participant’s thoughts is desirable:
Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?
IV. Post-Think-Aloud Follow-Up Questions (Verbal Probing)
If the think-aloud did not reveal the answers to the questions, then ask a selection of the following probes depending on what the participant has not addressed.
1. Attention
What was your first impression? What was your first thought when you looked at the label?
Does this warning catch your attention? What in particular catches your attention? Or, what about it is not attention catching?
Imagine seeing a label with information attached on a ROV in the store. Would you read the whole text from beginning to end or would you just look at or check a few parts of it? In general, to what extent do you read warning labels on products before you buy them, and how much of that is based on the type of product?
2. Communication
Message:
Tell me in your own words, what information is this label providing?
What message is this warning trying to get across?
What do you think is the main purpose of the label message?
How would you know whether one vehicle is better than another?
Comprehension:
What, if any, part of this label is confusing to you?
Are there words or descriptions that you found hard to understand? If so, point them out to me.
What does the term “lateral acceleration” mean?
Did you guess what this chart (point to the scale) is for? If Yes: What did you think it means? What does it show? If No: Did the text at the bottom help to understand the chart?
What do you think this number in the black box means? What do the numbers underneath the bar mean?
How clearly is the topic explained? (and probe for answer basis)
Point at the symbol of tipped over vehicle: What do you think this picture is of?
Have you seen a label or hang tag similar to this one in any stores? If so: Where have you seen that label/hang tag? If possible, name a specific store, but if not, name a type of store. What type of product was it with?
Thank participant for his/her time.
Before we finish, I’d like to hear any other thoughts you have about the label message and design.
Is there anything else you would like to tell me about working on the task?
Is there anything you would like to ask me about what we did today? [Answer participant questions]
Thank you for helping us to improve the label. |
Appendix B: Round 1 Focus Group Moderator Guide
Evaluation of Recreational Off-Highway Vehicle (“ROV”) Hang Tag
Round 1- Evaluation of Initial Design
Focus Group Discussion
Moderator Guide
Moderator instructions:
This guide will be used to steer discussion to the topic areas to be covered, and the specific questions of interest within each topic area. For the purposes of qualitative research, it is not necessarily intended that these questions be asked exactly as they are worded here. Focus group discussions are more like a conversation than a set of structured questions. The discussion should be as informal as possible and participants should be encouraged to speak openly and freely. The Moderator will also need to make sure that all participants in the group have an opportunity to express their opinions. Because each group of participants may be different, a responsive and flexible approach should be used for the focus group discussion.
I. INTRODUCTION
Hello my name is _______________ and I will lead the discussion today. First, thank you for your time. I work for EurekaFacts, a research firm that has been hired to talk with people like you to improve materials related to ROVs.
During the earlier interviews we have discussed how easy or difficult to understand the information on the ROV label. Now we will discuss topics related to the ROV driving and safety. Let me assure you that this is not a sales meeting of any kind. I don’t have anything to sell you. This is a form of market research and I am interested only in your opinions. This discussion will last about 1.5 hours.
Your views and experience are important, so we would like you to tell us what you think and feel about your experiences and about the ROV label message we show you. It is important that you provide us with your honest opinions and that you understand that we will not make any judgments of you for your opinions. Also, as we are talking about your personal opinions and experiences, everyone may not agree with each other.
Guidelines
To make this session work as a research session, there are some important guidelines to follow:
Please talk in a voice at least as loud as mine.
There are no wrong answers – just different points of view. Say what you believe.
We will observe the no smoking rule during this session.
I may need to move on to another area of discussion sooner than you are ready, and I may have to cut a conversation short in the interest of time. If I interrupt you, it is not personal; I just want to cover everything on the agenda.
Please silence your cell phones or pagers.
Disclosures
Mention microphones, audio and videotaping.
One-way mirror; associates viewing, notes may come in
The session is being taped so I can write an accurate report, not of “who said what,” but "what got said.” Your name will not be used in the report. No one will be able to attribute anything that is said in this room today to a specific individual; your comments will be anonymous.
Permissions
You may excuse yourself at any time to go to the restroom or to get more food or beverages.
Do you have any questions before we begin?
II. RESPONDENTS INTRODUCTIONS/ WARM UP
Before we get started, I’d like to learn a little more about you. Please share with me:
your first name
how you spend your day
favorite past-time.
Moderator introduces himself/herself as well.
III.TOPICS AND QUESTIONS FOR DISCUSSION
Knowledge of and experience with ROVs
How did you hear about ROVs? What do you know about ROVs?
How many of you are ROV owners? How many ROVs do you own? How long ago did you purchase your most recent one?
How many of you have other off-road vehicles, such as ATVs, snowmobiles, or dirt bikes?
How many of you are ROV drivers only? For how long have you driven ROVs?
How/in what capacity do you use ROVs?
Knowledge of and experience with ROV rollover
What do you know about the safety of ROVs?
In your opinion, are ROVs safe? Why do you have that opinion? What is the basis for your opinion?
Before coming here today, have you ever heard about risks associated with ROVs? How about with the risk of ROV rollovers? How did you hear about them?
Have you or someone you know experienced rollover accident? If yes, how did it happen? Why do you think that accident happened? What caused the rollover?
Before coming here today, did you have any knowledge what factors causing rollover accidents?
ROV purchase decision making and behavior:
When you were thinking about buying / deciding to buy a ROV, where did you go to learn more about ROVs?
Thinking back to when you were first considering purchasing an ROV, what were the most important factor(s) in your decision making process?
What specific vehicle characteristics were you looking at initially?
Did you compare between different vehicles (ROV makes)? Why or why not?
For those of you that own an ROV, now that you have one, what are the most important features to you? For those of you that drive an ROV but don’t own one, if you were to buy one, what would be the most important features to you? (Respondents will be asked to write this down to avoid influence bias and then share as a group.)
Identification – personal relevance:
Hand out booklet with ROV label.
Please take a moment to review the tag again.
What is the primary message of this tag? (Respondents will be asked to write their comments first to avoid influence bias and then share as a group.) Are there other messages?
How believable is this warning? What do and don’t you believe? What would make it more believable?
Who do you think this label is aimed at? Why?
Is this message relevant to you personally? Why/why not?
What did you learn, if anything, from reading the label?
Label message effect – call to action
Does this message change how you think about buying ROVs? Please explain.
What about the message in this label, if anything, would make you compare between different vehicles? What part of the message?
How does this message change how you think about the ROV driving?
After reading the label, how safe or unsafe do you think that ROV driving is? Please explain.
Hang tag design impressions and preferences:
Look again at the ROV label. During the interviews we discussed whether the information on the label is easy to understand. Now we will discuss together what you like and/or dislike about this label.
What do you like and/or dislike about the image on this label? Why?
What do you like and/or dislike about the text on this label? Why?
How would you improve the chart on this label?
What are your thoughts on the colors used?
What are your thoughts on the size of the font? The type of font?
What are your thoughts on the size of the tag?
Suggestions for improvement
What would you change to make:
The information on the label easier to understand?
People realize the importance of the information on the tag?
People realize the risk of rollover?
How would you improve the images on this label?
How would you improve the text on this label?
Working as a team, think about the design and the information displayed in the tag. What are the most important changes that need to be made to make this an effective tag?
During this time, the moderator will check with the observers for any additional questions and/or clarifications.
Moderator returns to the focus group room and reviews respondents’ answers and asks observers’ follow up questions.
Moderator thanks respondents for their time and participation and dismisses them.
Appendix C: Round 2 Focus Group Moderator Guide
Evaluation of Recreational Off-Highway Vehicle (“ROV”) Hang Tag
Round 2 – Evaluation of Alternative Designs
Focus Group Discussion
Moderator Guide
Moderator instructions:
This guide will be used to steer discussion to the topic areas to be covered, and the specific questions of interest within each topic area. For the purposes of qualitative research, it is not necessarily intended that these questions be asked exactly as they are worded here. Focus group discussions are more like a conversation than a set of structured questions. The discussion should be as informal as possible and participants should be encouraged to speak openly and freely. The Moderator will also need to make sure that all participants in the group have an opportunity to express their opinions. Because each group of participants may be different, a responsive and flexible approach should be used for the focus group discussion.
I. INTRODUCTION
Hello my name is _______________ and I will lead the discussion today. First, thank you for your time. I work for EurekaFacts, a research firm that has been hired to talk to people like you to improve materials related to ROVs.
During this session we will discuss topics related to the ROVs and design and understandability of the ROV hang tag. Let me assure you that this is not a sales meeting of any kind. I don’t have anything to sell you. This is a form of market research and I am interested only in your opinions. This discussion will last about 2 hours.
Your views and experience are important, so we would like you to tell us what you think and feel about your experiences and about the ROV label message we show you. It is important that you provide us with your honest opinions and that you understand that we will not make any judgments of you for your opinions. Also, as we are talking about your personal opinions and experiences, everyone may not agree with each other.
Guidelines
In order to make this session work as a research session, there are some important guidelines to follow:
Please talk in a voice at least as loud as mine.
There are no wrong answers – just different points of view. Say what you believe.
We will observe the no smoking rule during this session.
I may need to move on to another area of discussion sooner than you are ready, and I may have to cut a conversation short in the interest of time. If I interrupt you, it is not personal; I just want to cover everything on the agenda.
Please silence your cell phones or pagers.
Disclosures
Mention microphones, audio and videotaping.
One-way mirror; associates viewing, notes may come in
The session is being taped so I can write an accurate report, not of “who said what,” but "what got said.” Your name will not be used in the report. No one will be able to attribute anything that is said in this room today to a specific individual; your comments will be anonymous.
Permissions
You may excuse yourself at any time to go to the restroom or to get more food or beverages.
Do you have any questions before we begin?
II. RESPONDENTS INTRODUCTIONS/ WARM UP
Before we get started, I’d like to learn a little more about you. Please share with me:
your first name
how you spend your day
favorite past-time.
Moderator introduces himself/herself as well.
III.TOPICS AND QUESTIONS FOR DISCUSSION
Part 1: Individual evaluation of labels
Moderator will hand out one label at time (in varying order for each respondent and each label will have an identifier, such as letters P, H, R to further eliminate bias).
We will start by looking at some labels and getting your individual responses before we discuss them together.
Evaluation of each label
We are going to review three labels, one at a time. I am going to distribute a label to each of you. Please keep the label face down until everyone has received one and I ask you to turn it over. When you turn it over, please take a moment to look at the label.
On the piece of paper in front of you, please write down your immediate impression.
Write down what is, in your opinion, the primary message (purpose) of the label. Also, underline anything you like, circle any information that is confusing, put an X through any information you do not like.
This will be on a flip chart for easy reference.
Remind participants to remain quiet and not discuss the label messages at this stage.
The order of showing the labels should be varied for each group. This is to minimize the chance of preferences being influenced by the order in which the labels are presented.
Comparison of the labels
Now that we have had a chance to review the three labels, please lay them out in front of you and use the worksheet in front of you to answer the following questions:
Which of the label messages is easiest to understand? Why?
Which one of the labels designs you like the most? Why?
Which one of the labels do you think would be most effective for informing customers about the risk of injury related to ROV driving?
Which one of the labels do you think would be most effective for motivating customers to compare the vehicles before making a purchasing decision?
Part 2: Group Discussion
Now that you’ve had a chance to review the labels on your own, we’ll move to a group discussion about each of the labels you’ve just seen, one at a time. Please give us your own honest opinions, even if they are different from others in the group.
Have participants view each label message, in the same order this time. After viewing each message, prompt the group with the following questions for each topic area.
1. Attention
What was your first impression? What was your first thought when you looked at the label?
Does this label catch your attention? What in particular catches your attention? Or, what about it is not attention catching?
Imagine seeing a label with this information attached on a ROV in the store. Would you read the whole text from beginning to end or would you just look at or check a few parts of it? In general, to what extent do you read label labels on products before you buy them, and how much of that is based on the type of product?
2. Communication – message, comprehension, credibility
Message:
Can you tell me in your own words, what information this label is providing?
What is the primary message of this tag? (Respondents will be asked to write their comments first to avoid influence bias and then share as a group.) Are there other messages?
What do you think is the main purpose of the label message?
How would you know whether one vehicle is better than another?
Comprehension:
What, if any, part of this label is confusing to you?
Are there words or descriptions that you found hard to understand? If so, point them out to me.
What does the term “lateral acceleration” mean?
How clearly is the topic explained? (and probe for answer basis)
If not mentioned previously by participants probe understanding of different elements of the design, pictures and symbols.
Comprehension:
How believable is this warning? What do or don’t you believe? What would make the warning more believable?
3. Identification – personal relevance
Who do you think this label is aimed at? Why?
Is this message relevant to you personally? Why/why not?
What did you learn, if anything, from reading the label?
Label message effect
Does this message change how you think about buying ROVs? Please explain.
What about the message in this label, if anything, would make you compare between different vehicles? What part of the message?
How does this message change how you think about the ROV driving?
After reading the label, how safe or unsafe do you think that ROV driving is? Please explain.
Hang tag design impressions and preferences
What do you like and/or dislike about the image on this label? Why?
What do you like and/or dislike about the text on this label? Why?
What are your thoughts on the colors used?
What are your thoughts on the size of the font? The type of font?
What are your thoughts on the size of the tag?
Suggestions for improvement
What would you change to make:
The information on the label easier to understand?
People realize the importance of the information on the tag?
People realize the risk of rollover?
How would you improve the images on this label?
How would you improve the text on this label?
Working as a team, think about the design and the information displayed in the tag. What are the most important changes that need to be made to make this an effective tag?
During this time, the moderator will check with the observers for any additional questions and/or clarifications.
Moderator returns to the focus group room and reviews respondents’ answers and asks observers’ follow up questions.
Moderator thanks respondents for their time and participation and dismisses them.
Appendix D: Participant Recruitment Screener Script
Evaluation of Recreational Off-Highway Vehicle (ROV) Hang Tag
Screener
The following is intended as a generic version of a screener that can be used for recruiting participants for this research study. It can be used for recruiting face-to-face or by telephone. For each round of focus groups specific quotas will be added to these questions.
Hello, My name is_______________________, and I am calling from EurekaFacts, a research company in Maryland. We are currently recruiting participants for a discussion group regarding Recreational Off-Highway Vehicles (or ROVs for short). I am calling to see if you are interested in participating. If you are interested, you will receive $80 Visa gift card for a two hour discussion group.
Are you interested in participating?
IF NO: Thank you and have a great day/evening.
IF YES: Great! First, I need to ask you a few questions to find out if your background meets the needs of this study.
Are you age 18 or older?
Yes
No TERMINATE
Have you operated a ROV (Recreational Off-Highway Vehicle), sometimes called a side-by-side ATV, within the last 5 years?
Yes
No
Have you purchased a ROV within the last 5 years?
Yes GO TO Q6
No
Do you currently own, or have you owned an ROV within the past 5 years?
Yes
No GO TO Q7
What type or make of ROV do/did you own?
Please specify_________________________ GO TO Q7
What type or make of ROV did you purchase?
Please specify_________________________
Do you plan on purchasing an ROV in the near future?
Yes
No Go to Q9
(Programming note: If Q2, Q3, Q4 and Q7=No TERMINATE)
Do you plan on purchasing the ROV for yourself or someone else?
Myself
Someone else (please specify) ________________________
For what purpose do you use or intend to use the ROV? (Select all that apply)
For utility (e.g. rural users, and farmers)
For recreational purposes (e.g. outdoor sports enthusiast)
Other (please specify) __________________________
ASK ONLY IF NOT OBVIOUS: Are you male or female?
Male
Female
How would you consider your primary race or ethnicity?
American Indian or Alaskan Native
Asian
Black or African American
Native Hawaiian or Pacific islander
White or Caucasian
Hispanic or Latino
Other
What is your current age?
18 – 30
31 – 40
41 – 50
51 – 60
61 – 64
65 +
Thinking of everyone in your household who receives income, what is the total yearly income for your household before taxes, including salaries, Social Security, pension, interest, and investment earnings. DO NOT READ THE LIST (RECRUIT A MIX)
Under $25,000
$25,000 to just below $30,000
$30,000 to just below $37,000
$37,000 to just below $45,000
$50,000 to just below $52,000
$52,000 to just below $60,000
$60,000 to just below $67,000
$67,000 to just below $75,000
$75,000 to just below $100,000
$100,000 or more
No answer
What is the highest level of education you have completed? (RECRUIT A MIX)
Less than high school degree
High school graduate/GED
Some college or technical school
College or technical school graduate
Post-graduate
No answer
Thank you for answering these questions. You are eligible to participate in this research project. As I mentioned previously, you have been invited to participate in a small discussion group regarding ROVs. Your participation means that you would participate in two hour discussion. You will receive $80 Visa gift card upon completion of the group. If you are chosen for the discussion you will receive a call confirming the date and time.
Would you still like to participate?
IF YES – Thank you so much for your willingness to participate in the group. I will need to record the following information in order to send you a confirmation letter/email with the directions and instructions for the group.
IF NO – Thank you for taking the time to talk with me today. You do not qualify for the study. Have a great day/evening.
Could you please confirm your contact information?
Full Name:
Phone Number:
Mailing Address:
Email:
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | RSquibb |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |