Usability Testing of the ASA24 Respondent Site
Attachments:
Attachment 1A. Screeners for potential ASA24 participants
Attachment 1C. Usability testing guide
Attachment 1A. Screening Criteria and Survey for Potential ASA24 Subjects
Twenty four (24) individuals are needed to participate in a usability test for the ASA24 Respondent website. Two groups of respondents should be recruited as described below. Note: This document includes questions for both groups of participants. In practice, questionnaires may be separated for ease of use. Questions for both recruitments are color- and title-coded in the material below.
For Group A, there should be twelve (12) individuals who either have low literacy or who are older. A list of potential volunteers who are low literacy individuals may be obtained from a local literacy council. Individuals could then be recruited from that list.
As much as possible the individuals should be divided into half (6) with low literacy and half (6) who are older (defined as 65-75 years).
Individuals should have no previous experience with the ASA24 website.
All participants should have at least intermediate computer/internet experience. However, if finding appropriate individuals with this level of experience is a problem, all participants should be able to use the computer by themselves.
Age, gender, and race should be mixed as well as possible.
These individuals will be tested in the lab.
Group B is twelve (12) Spanish speaking respondents who will be recruited to use the Spanish version of the site.
All participants should have at least intermediate computer/internet experience.
Age, gender, and race should be mixed as well as possible.
Individuals should have no previous experience with the ASA24 website.
These individuals will be tested remotely.
The following table repeats the numbers and a brief description and the numbers of participants who are needed.
Group |
Description of Respondent Participants |
Number needed |
A |
Low literacy |
6 |
A |
Older individuals (65 -75 yrs) |
6 |
B |
Spanish speaking |
12 |
OMB No. 0925-0589-01 Expiry Date:4/30/2014 Public reporting burden for this collection of information is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-01). Do not return the completed form to this address. |
Screening Survey
Hello, my name is _________________, calling for User-Centered Design. We are recruiting individuals to do usability testing of a website used by researchers, dietitians, and others to learn about what people eat. We would like to ask you a few questions to find out if you qualify and would like to participate. This will only take a few minutes of your time and no one will attempt to sell you anything. This is strictly for research purposes. If you are interested and qualify for the study, you will be paid to participate. Can I ask you a few questions?
Group A Participant Screener
Do you or anyone in your household work for:
An Internet or software development company
A usability or market research business/company
(DISMISS if yes to either of the above)
Have you participated in a market research study, such as a focus group or an in-person interview, within the last six months?
Within the last 6 months? (DISMISS)
Over 6 months?
Never?
What is your age group?
17 or less (Dismiss)
18-45
46-64
65 -75 or older (Qualifies as older, we need 6 individuals. Check for literacy too.)
>75 (DISMISS)
What is your present level of education
Did not graduate from High School (May qualify for low literacy)
High School Diploma (May qualify for low literacy)
Some college (DISMISS)
Degree from College (DISMISS)
Some post graduate courses (DISMISS)
Post graduate degree (DISMISS)
Note that the next 4 questions are designed to determine literacy. If individuals are being recruited from a list of low literacy individuals, these questions will not be asked.
Are you comfortable reading the daily newspaper?
Yes [Even if someone says yes, ask question below]
No [If someone says no, assume they have low literacy skills]
What do you typically read in the newspaper? Tell me all that you read.
Headlines
Daily news stories in the paper; Which types? (Use this to verify that they actually read stories which will be a partial determinant of literacy)
___________________________________________________________________________
TV Digest
Comics and Editorial cartoon
Advice Columns
Editorials
Sports scores
Stock market numbers
Other material, Please describe __________________________________________
What was the last material you read just for pleasure? __________________________________________________________________________
When was the last time you read for pleasure?
Today
Yesterday
This week
Last week
This month
A few months ago
Other, Please describe _________________________________________________
[Make a judgment about literacy based on
Education level at a high school diploma or lower can predict low literacy skills,
Actually reading (or not) the newspaper (which requires ~ a fifth grade education) whereas simply reading numbers in the newspaper does not require literacy skills, and
Reading for pleasure within the recent past is a sign of literacy.
If you have difficulty deciding if this individual has low literacy skills, contact UCD for a consultation. Six (6) low literacy skilled individuals are needed.
Consider this individual someone who has low literacy skills.
Do not consider this individual someone who has low literacy skills.
How many hours per week do you use the Internet for business or personal use, excluding checking your e-mail?
Less than 2 hours a week
Between 2-5 hours a week
Between 6-15 hours a week
Over 15 hours a week
How would you rate your Internet experience?
None or almost none (DISMISS)
Novice, but can work on the internet by self
Intermediate
Advanced
Which of the following activities have you done on the Internet? (Check all that apply)
Made a purchase online
Pay bills online
Have worked or created pages on MySpace or Facebook
Downloaded Music or Videos
Looked for information about a hobby or interest
Looked at information from a local, state or government web site
Looked for current news, sports, or weather
DISMISS if respondent has not done at least one of the activities above.
[Do not ask but] Record gender
Male
Female
Group B, Participant Screener
This material will be presented in Spanish to potential participants.
Do you or anyone in your household work for:
An Internet or software development company
A usability or market research business/company
(DISMISS if yes to either of the above)
Have you ever used the National Cancer Institute’s application called ASA24? This is a website that prompts you to record everything that you ate and drank the day before.
Yes (DISMISS)
No
Have you participated in a market research study, such as a focus group or an in-person interview, within the last six months?
Within the last 6 months? (DISMISS)
Over 6 months
Never?
Do you have any trouble reading Spanish?
Yes [If someone says yes, DISMISS]
No
What is your age group?
19 or less (DISMISS)
20-34
35-49
50 or older (DISMISS)
What is your present level of education
Did not graduate from High School
High School Diploma
Some college
Degree from College
Some post graduate courses
Post graduate degree
How many hours per week do you use the Internet for business or personal use, excluding checking your e-mail?
Less than 2 hours a week
Between 2-5 hours a week
Between 6-15 hours a week
Over 15 hours a week
How would you rate your Internet experience?
None or almost none (DISMISS, if they are a true novice)
Intermediate
Advanced
Which of the following activities have you done on the Internet? (Check all that apply)
Made a purchase online
Pay bills online
Have worked or created pages on MySpace or Facebook
Downloaded Music or Videos
Looked for information about a hobby or interest
Looked at information from a local, state or government web site
Looked for current news, sports, or weather
DISMISS if respondent has not done at least two of the activities above.
[Do not ask but] Record gender
Male
Female
Thank you for taking the time to answer my questions. We would like to invite you to participate in the usability test. For the test, you will be asked to use a computer by yourself and work with a website and answer some questions? This will take about an hour.
For both Group A’s: We would like you to participate at the National Cancer Institute Usability Lab in Rockville, MD. The sessions will be held on [date] at [time] at the lab at 6116 Executive Blvd. Room 400 in Rockville, MD. We will email you a confirmation & directions, and we will call you the day before to confirm your attendance. Do you have a way to get to the lab? For in-lab participation, we can offer you a stipend of $100 as a thank you for your time.
For Group B: We will have you test the website remotely using conferencing software. The sessions will be held on [date] at [time]. For remote participation, we can offer you a stipend of $50 as a thank you for your time.
For Groups A and B:
Respondent's name ______________________________________
Respondent's availability __________________________________
Respondent’s email address _______________________________
Telephone number _______________________________________
For Group B:
Respondent’s mailing address ___________________________________
Attachment 1C:
Facilitator’s Guide for Usability Testing of the
ASA 24 Respondent Site
Submitted by:
User-Centered Design, Inc.
20548 Deerwatch Place
Ashburn, VA 20147
(703) 729-2370
OMB No. 0925-0589-01 Expiry Date:4/30/2014 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-01). Do not return the completed form to this address. |
[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be asked as written. The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the session. While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order. The facilitator may allow the order of the tasks to change in order to let the process flow naturally. ]
[Administer the informed consent] Gain verbal consent is testing is being done remotely. (??)
Thank you for agreeing to participate in this study. Do you have any questions for me before we get started?
There are a few things I’d like you to keep in mind while we work. The first is, this is not a test of you or your knowledge. This is an evaluation of the design of the website you will be working with today. I will be trying to evaluate the design through your eyes. I’ll give you some things I’d like you to try to do and then see if it is clear to you what you should to do. Mostly I will just watch you work but I’m also here to answer any questions, although I may let you keep working on your own before I answer you. At times, I may ask you some direct questions; this is not to test you but to find out if the website is easy to follow. In other words, I am testing the website, not you.
The second thing I would like you to keep in mind is that I didn’t design what you will be working with today so don’t worry about hurting my feelings. I want you to feel free to praise or criticize the design as you see fit. If there are areas of concern or things you don’t like, I want you to feel free to talk about them. That way, we might be able to figure out a way to make the website better.
Finally, I’d like you to know that I have one of my colleagues with us today taking notes for me. There [may be/are] other people who are observing today as well. Don’t worry about them. You and I will work on this together and they’ll just watch.
[To verify that we know actual information about the participants (and their background) and to get them comfortable with talking in this strange situation, we’ll have them talk about subjects that they are comfortable with.]
Today the website we’ll be looking at is a way of asking you questions and letting you answer without much typing. Have you ever filled out a form on the internet? What sort of forms?
Participant talks…..
This will actually be about what you ate over a 24 hour period. It’s called a dietary recall. Have you ever participated in a dietary recall process before? [If yes] Tell me about it. [If no] Tell me what you think it would be like.
Participant talks…..
First, I’d like you to look at the website. Please describe your first impressions of this “look”.
Now I’m going to ask you to do go through the website and enter the information that is requested. Feel free to ask any questions that come to mind as you work but I might not answer right away to see if the design lets you find the answer on your own. Also, if it feels comfortable, tell me what you think about each of the entries that you make.
Areas/items to watch and/or talk about:
Penguin (general impressions)
Instructions
Changing font size
Turning penguin on/off
Meal name
Time
Location
TV
Eating alone
Going to next screen
Threefold screen
Browsing food categories
Searching for food and drinks
Moving an item over to My Foods and Drinks
Editing/copying/deleting foods or drinks
Editing/deleting meals
What to do if can’t find item in categories (no match found)
What to do when done entering all meals and snacks
Meal gap questions
Multi-ingredient queries
Answering questions regarding the details of reported food
Using the portion selectors for foods and drinks (report amount consumed, not amount taken or served)
Using the additions screen – common additions, searching
Review of food
Forgotten food
Supplements
Find, record, amount of supplements
Specific tasks to be checked:
Quit the application and re-access the material previously entered
Duplicate check (if doesn’t occur naturally, force this)
Add multi ingredient (sandwich, hamburger, etc. )
Add a new supplement
Use the Make Changes button
Add forgotten foods Add additions to already-reported foods
Great, now I’d like to ask you a few questions.
Most websites take awhile to master. How hard or easy was it for you to [use| learn] this website today? Do you think that others with your background will have about the same degree of difficulty that you did?
What are the things you liked most about the way the website looks and works?
What things did you like least about the way the website looks and works?
Was there a time when you were stuck and didn’t quite know what to do next? What would have helped?
How did you feel about the use of the penguin to provide instructions to you?
What about the way in which the website asked about the foods you ate: Did that seem ok, or did it bother you?
Do you have any suggestions for changes that would improve the website that we haven’t already covered already?
OK, we’re done. Are there any questions you would like to ask me about the website or about today’s experience that I did not ask you about?
[Give stipend for in-person testing and for remote participants, it will be mailed.]
Thanks again for your participation.
Criteria and Screening Survey for ASA24
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | DEPARTMENT OF HEALTH & HUMAN SERVICES |
| Author | krs0 |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |