1 #1_ASA 24 Screener and Guide

Questionnaire Cognitive Interviewing and Pretesting (NCI)

#1 Attach 1A and 1C Screener & Guide

Bundled Website Usability Testings #1 (ASA24) and #2 (GEM)

OMB: 0925-0589

Document [docx]
Download: docx | pdf




Usability Testing of the ASA24 Respondent Site


Attachments:


Attachment 1A. Screeners for potential ASA24 participants


Attachment 1C. Usability testing guide



Attachment 1A. Screening Criteria and Survey for Potential ASA24 Subjects


Twenty four (24) individuals are needed to participate in a usability test for the ASA24 Respondent website. Two groups of respondents should be recruited as described below. Note: This document includes questions for both groups of participants. In practice, questionnaires may be separated for ease of use. Questions for both recruitments are color- and title-coded in the material below.


For Group A, there should be twelve (12) individuals who either have low literacy or who are older. A list of potential volunteers who are low literacy individuals may be obtained from a local literacy council. Individuals could then be recruited from that list.

  • As much as possible the individuals should be divided into half (6) with low literacy and half (6) who are older (defined as 65-75 years).

  • Individuals should have no previous experience with the ASA24 website.

  • All participants should have at least intermediate computer/internet experience. However, if finding appropriate individuals with this level of experience is a problem, all participants should be able to use the computer by themselves.

  • Age, gender, and race should be mixed as well as possible.

  • These individuals will be tested in the lab.



Group B is twelve (12) Spanish speaking respondents who will be recruited to use the Spanish version of the site.

  • All participants should have at least intermediate computer/internet experience.

  • Age, gender, and race should be mixed as well as possible.

  • Individuals should have no previous experience with the ASA24 website.

  • These individuals will be tested remotely.



The following table repeats the numbers and a brief description and the numbers of participants who are needed.

Group

Description of Respondent Participants

Number needed

A

Low literacy

6

A

Older individuals (65 -75 yrs)

6

B

Spanish speaking

12






OMB No. 0925-0589-01

Expiry Date:4/30/2014

Public reporting burden for this collection of information is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-01). Do not return the completed form to this address.



Screening Survey


Hello, my name is _________________, calling for User-Centered Design. We are recruiting individuals to do usability testing of a website used by researchers, dietitians, and others to learn about what people eat. We would like to ask you a few questions to find out if you qualify and would like to participate. This will only take a few minutes of your time and no one will attempt to sell you anything. This is strictly for research purposes. If you are interested and qualify for the study, you will be paid to participate. Can I ask you a few questions?



Group A Participant Screener


  1. Do you or anyone in your household work for:


An Internet or software development company

A usability or market research business/company


(DISMISS if yes to either of the above)


  1. Have you participated in a market research study, such as a focus group or an in-person interview, within the last six months?


Within the last 6 months? (DISMISS)

Over 6 months?

Never?


  1. What is your age group?


17 or less (Dismiss)

18-45

46-64

65 -75 or older (Qualifies as older, we need 6 individuals. Check for literacy too.)

>75 (DISMISS)


  1. What is your present level of education


Did not graduate from High School (May qualify for low literacy)

High School Diploma (May qualify for low literacy)

Some college (DISMISS)

Degree from College (DISMISS)

Some post graduate courses (DISMISS)

Post graduate degree (DISMISS)


Note that the next 4 questions are designed to determine literacy. If individuals are being recruited from a list of low literacy individuals, these questions will not be asked.


  1. Are you comfortable reading the daily newspaper?


Yes [Even if someone says yes, ask question below]

No [If someone says no, assume they have low literacy skills]




  1. What do you typically read in the newspaper? Tell me all that you read.


Headlines

Daily news stories in the paper; Which types? (Use this to verify that they actually read stories which will be a partial determinant of literacy)

___________________________________________________________________________


TV Digest

Comics and Editorial cartoon

Advice Columns

Editorials

Sports scores

Stock market numbers

Other material, Please describe __________________________________________


  1. What was the last material you read just for pleasure? __________________________________________________________________________


  1. When was the last time you read for pleasure?


Today

Yesterday

This week

Last week

This month

A few months ago

Other, Please describe _________________________________________________


[Make a judgment about literacy based on

  • Education level at a high school diploma or lower can predict low literacy skills,

  • Actually reading (or not) the newspaper (which requires ~ a fifth grade education) whereas simply reading numbers in the newspaper does not require literacy skills, and

  • Reading for pleasure within the recent past is a sign of literacy.


If you have difficulty deciding if this individual has low literacy skills, contact UCD for a consultation. Six (6) low literacy skilled individuals are needed.


Consider this individual someone who has low literacy skills.

Do not consider this individual someone who has low literacy skills.




  1. How many hours per week do you use the Internet for business or personal use, excluding checking your e-mail?


Less than 2 hours a week

Between 2-5 hours a week

Between 6-15 hours a week

Over 15 hours a week




  1. How would you rate your Internet experience?


None or almost none (DISMISS)

Novice, but can work on the internet by self

Intermediate

Advanced


  1. Which of the following activities have you done on the Internet? (Check all that apply)


Made a purchase online

Pay bills online

Have worked or created pages on MySpace or Facebook

Downloaded Music or Videos

Looked for information about a hobby or interest

Looked at information from a local, state or government web site

Looked for current news, sports, or weather


DISMISS if respondent has not done at least one of the activities above.


  1. [Do not ask but] Record gender


Male

Female




Group B, Participant Screener


This material will be presented in Spanish to potential participants.


  1. Do you or anyone in your household work for:


An Internet or software development company

A usability or market research business/company


(DISMISS if yes to either of the above)


  1. Have you ever used the National Cancer Institute’s application called ASA24? This is a website that prompts you to record everything that you ate and drank the day before.


Yes (DISMISS)

No


  1. Have you participated in a market research study, such as a focus group or an in-person interview, within the last six months?


Within the last 6 months? (DISMISS)

Over 6 months

Never?





  1. Do you have any trouble reading Spanish?


Yes [If someone says yes, DISMISS]

No


  1. What is your age group?


19 or less (DISMISS)

20-34

35-49

50 or older (DISMISS)


  1. What is your present level of education


Did not graduate from High School

High School Diploma

Some college

Degree from College

Some post graduate courses

Post graduate degree


  1. How many hours per week do you use the Internet for business or personal use, excluding checking your e-mail?


Less than 2 hours a week

Between 2-5 hours a week

Between 6-15 hours a week

Over 15 hours a week


  1. How would you rate your Internet experience?


None or almost none (DISMISS, if they are a true novice)

Intermediate

Advanced


  1. Which of the following activities have you done on the Internet? (Check all that apply)


Made a purchase online

Pay bills online

Have worked or created pages on MySpace or Facebook

Downloaded Music or Videos

Looked for information about a hobby or interest

Looked at information from a local, state or government web site

Looked for current news, sports, or weather


DISMISS if respondent has not done at least two of the activities above.


  1. [Do not ask but] Record gender


Male

Female


Thank you for taking the time to answer my questions. We would like to invite you to participate in the usability test. For the test, you will be asked to use a computer by yourself and work with a website and answer some questions? This will take about an hour.


For both Group A’s: We would like you to participate at the National Cancer Institute Usability Lab in Rockville, MD. The sessions will be held on [date] at [time] at the lab at 6116 Executive Blvd. Room 400 in Rockville, MD. We will email you a confirmation & directions, and we will call you the day before to confirm your attendance. Do you have a way to get to the lab? For in-lab participation, we can offer you a stipend of $100 as a thank you for your time.


For Group B: We will have you test the website remotely using conferencing software. The sessions will be held on [date] at [time]. For remote participation, we can offer you a stipend of $50 as a thank you for your time.




For Groups A and B:


Respondent's name ______________________________________


Respondent's availability __________________________________


Respondent’s email address _______________________________


Telephone number _______________________________________




For Group B:


Respondent’s mailing address ___________________________________






Attachment 1C:


Facilitator’s Guide for Usability Testing of the

ASA 24 Respondent Site



Submitted by:

User-Centered Design, Inc.

20548 Deerwatch Place

Ashburn, VA 20147

(703) 729-2370

www.user-centereddesign.com





























OMB No. 0925-0589-01

Expiry Date:4/30/2014

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-01). Do not return the completed form to this address.


[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be asked as written. The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the session. While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order. The facilitator may allow the order of the tasks to change in order to let the process flow naturally. ]


Pre-Test

[Administer the informed consent] Gain verbal consent is testing is being done remotely. (??)


Introduction

Thank you for agreeing to participate in this study. Do you have any questions for me before we get started?


There are a few things I’d like you to keep in mind while we work. The first is, this is not a test of you or your knowledge. This is an evaluation of the design of the website you will be working with today. I will be trying to evaluate the design through your eyes. I’ll give you some things I’d like you to try to do and then see if it is clear to you what you should to do. Mostly I will just watch you work but I’m also here to answer any questions, although I may let you keep working on your own before I answer you. At times, I may ask you some direct questions; this is not to test you but to find out if the website is easy to follow. In other words, I am testing the website, not you.


The second thing I would like you to keep in mind is that I didn’t design what you will be working with today so don’t worry about hurting my feelings. I want you to feel free to praise or criticize the design as you see fit. If there are areas of concern or things you don’t like, I want you to feel free to talk about them. That way, we might be able to figure out a way to make the website better.

Finally, I’d like you to know that I have one of my colleagues with us today taking notes for me. There [may be/are] other people who are observing today as well. Don’t worry about them. You and I will work on this together and they’ll just watch.


[To verify that we know actual information about the participants (and their background) and to get them comfortable with talking in this strange situation, we’ll have them talk about subjects that they are comfortable with.]


Today the website we’ll be looking at is a way of asking you questions and letting you answer without much typing. Have you ever filled out a form on the internet? What sort of forms?


Participant talks…..


This will actually be about what you ate over a 24 hour period. It’s called a dietary recall. Have you ever participated in a dietary recall process before? [If yes] Tell me about it. [If no] Tell me what you think it would be like.


Participant talks…..


Tasks

First, I’d like you to look at the website. Please describe your first impressions of this “look”.

Now I’m going to ask you to do go through the website and enter the information that is requested. Feel free to ask any questions that come to mind as you work but I might not answer right away to see if the design lets you find the answer on your own. Also, if it feels comfortable, tell me what you think about each of the entries that you make.

I’m going to watch as you go through the application and I may ask you questions. Meanwhile, I’d like you to tell me if it’s easy or difficult for you to see what you should do and then what you should do next. Your username and password for this will be _____ and _____. [Also give them the name and password written on a piece of paper.]

Areas/items to watch and/or talk about:

Penguin (general impressions)

Instructions

Changing font size

Turning penguin on/off

Meal name

Time

Location

TV

Eating alone

Going to next screen

Threefold screen

Browsing food categories

Searching for food and drinks

Moving an item over to My Foods and Drinks

Editing/copying/deleting foods or drinks

Editing/deleting meals

What to do if can’t find item in categories (no match found)

What to do when done entering all meals and snacks

Meal gap questions

Multi-ingredient queries

Answering questions regarding the details of reported food

Using the portion selectors for foods and drinks (report amount consumed, not amount taken or served)

Using the additions screen – common additions, searching

Review of food

Forgotten food

Supplements

Find, record, amount of supplements


Specific tasks to be checked:

  • Quit the application and re-access the material previously entered

  • Duplicate check (if doesn’t occur naturally, force this)

  • Add multi ingredient (sandwich, hamburger, etc. )

  • Add a new supplement

  • Use the Make Changes button

  • Add forgotten foods Add additions to already-reported foods


Post Test Interview Questions


Great, now I’d like to ask you a few questions.

  1. Most websites take awhile to master. How hard or easy was it for you to [use| learn] this website today? Do you think that others with your background will have about the same degree of difficulty that you did?

  2. What are the things you liked most about the way the website looks and works?

  3. What things did you like least about the way the website looks and works?

  4. Was there a time when you were stuck and didn’t quite know what to do next? What would have helped?

  5. How did you feel about the use of the penguin to provide instructions to you?

  6. What about the way in which the website asked about the foods you ate: Did that seem ok, or did it bother you?

  7. Do you have any suggestions for changes that would improve the website that we haven’t already covered already?


Wrap up


OK, we’re done. Are there any questions you would like to ask me about the website or about today’s experience that I did not ask you about?


[Give stipend for in-person testing and for remote participants, it will be mailed.]


Thanks again for your participation.


Criteria and Screening Survey for ASA24

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleDEPARTMENT OF HEALTH & HUMAN SERVICES
Authorkrs0
File Modified0000-00-00
File Created2021-02-01

© 2024 OMB.report | Privacy Policy