End User Survey

Form Approved

OMB No. 0920-0840

Expiration Date 02/29/2016

“Informing the Development of Mobile Apps for HIV Prevention, Treatment, & Care”

2k. End User Survey

Public reporting burden of this collection of information is estimated to average 30 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; Attn: OMB-PRA (0920-0840)

6. Perceived Ease of Use and Potential Usefulness Questionnaire

Item	Strongly disagree	Disagree	Neutral	Agree	Strongly Agree
I find the app easy to use.	1	2	3	4	5
Learning to operate the app is easy for me.	1	2	3	4	5
Interaction with the app is difficult.	1	2	3	4	5
I find it easy to get the app to do what I want it to do.	1	2	3	4	5
The app is flexible to interact with.	1	2	3	4	5
It is easy for me to remember how to perform tasks using the app.	1	2	3	4	5
Interacting with the app requires a lot of mental effort.	1	2	3	4	5
My interaction with the app is clear and understandable.	1	2	3	4	5
Using the app would give me greater control over my HIV care/ testing.	1	2	3	4	5
The app would enable me to complete tasks related to my HIV care/ prevention more quickly.	1	2	3	4	5
The app would support critical aspects related to my HIV care/ prevention.	1	2	3	4	5
Use of the app would increase the effectiveness of my HIV care/ testing.	1	2	3	4	5
Using the app would allow me to accomplish more work than would otherwise be possible.	1	2	3	4	5
Using the system would make it easier for me to maintain my health.	1	2	3	4	5
Use of the app would significantly increase the quality of documentation.	1	2	3	4	5
Use of the app would increase the quality of my care for the same amount of effort.	1	2	3	4	5
I would find the app useful in my life.	1	2	3	4	5

Demographics:

• Duration of computer use (years)

< 1 year

1-2 years

2-3 years

3-4 years

>5 years

• Place of use of computer

At work only

At home only

Both home and at work

Others _____________

• How often do you use Internet?

Everyday

2-3 times/ week

Once a week

Every other week

Once a month

Other_______

7. Heuristic Evaluation Form

1. Visibility of System Status

The system should always keep user informed about what is going on, through appropriate feedback within reasonable time.

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
1.1	Does every screen have a title or header that describes its contents?	 Yes  No  NA
1.2	Is there visual feedback in menus or dialog boxes about which choices are selectable?	 Yes  No  NA
1.3	Is there a clear indication of the current location?	 Yes  No  NA
1.4	Is the menu-naming terminology consistent with the user's task domain?	 Yes  No  NA
1.5	Does the system provide visibility: that is, by looking, can the user tell the state of the system and the alternatives for action?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

2. Match between System and the Real World

The system should speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order. 

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
2.1	Are the section headings and sub-sections in each screen ordered in the most logical way?	 Yes  No  NA
2.2	Is there a natural sequence to the menu choices for a data item?	 Yes  No  NA
2.3	Are all the words/concepts and phrases used in each screen familiar to users?	 Yes  No  NA
		 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

3. User Control and Freedom

Users should be free to select and sequence tasks (when appropriate), rather than having the system does this for them. Users will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Users should make their own decisions regarding the costs of exiting current work.

 I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
3.1	Is there a clear exit on each document screen?	 Yes  No  NA
3.2	Are all screens accessible across the system?	 Yes  No  NA
3.3	Is there an "undo" function?	 Yes  No  NA
3.4	Can users reduce data entry time by copying and modifying existing data?	 Yes  No  NA
3.5	Do users have the option of either clicking on menu items with a mouse or using a keyboard shortcut?	 Yes  No  NA
3.6	Can users easily move forward and backward between fields?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

4. Consistency and Standards

Users should not have to wonder whether different words, situations, or actions mean the same thing.

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
4.1	Have formatting standards been followed consistently in all screens within the system?	 Yes  No  NA
4.2	Are abbreviations clearly explained?	 Yes  No  NA
4.3	Are there salient visual cues to identify the active screen?	 Yes  No  NA
4.4	Is vertical scrolling possible in each screen?	 Yes  No  NA
4.5	Are there no more than four to seven colors, and are they far apart along the visible spectrum?	 Yes  No  NA
4.6	Is the most important information placed at the beginning of the form?	 Yes  No  NA
4.7	Are names consistent, both within each tab and across the system, in grammatical style and terminology?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

5. Help Users Recognize, Diagnose, and Recover From Errors

Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

I. Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
5.1	If prompts are used, are they brief and unambiguous?	 Yes  No  NA
5.2	Do error messages use plain language to describe the nature of the problem and to suggest a way of solving it?	 Yes  No  NA
5.3	Do all error messages in the site use consistent grammatical style, form, terminology, and abbreviations?	 Yes  No  NA
5.4	If an error is detected in a data entry field, does the system place the cursor in that field or highlight the error?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

6. Error Prevention

Even better than good error messages is a careful design which prevents a problem from occurring in the first place.

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
6.1	Have dots or underscores been used to indicate field length?	 Yes  No  NA
6.2	Are the sections, sub-sections, and data elements logical, distinctive, and mutually exclusive?	 Yes  No  NA
6.3	Is navigation between screens simple and visible?	 Yes  No  NA
6.4	Does the system prevent users from making errors in a data entry field?	 Yes  No  NA
6.5	Do fields in data entry screens and dialog boxes contain default values when appropriate?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

7. Recognition Rather Than Recall

Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
7.1	Are prompts, cues, and messages placed where the eye is likely to be looking on the screen?	 Yes  No  NA
7.2	Is white space used to create symmetry and lead the eye in the appropriate direction?	 Yes  No  NA
7.3	Have items been grouped into logical zones, and have headings been used to distinguish between zones?	 Yes  No  NA
7.4	Are optional data entry fields clearly marked?	 Yes  No  NA
7.5	Is color highlighting used to get the user's attention?	 Yes  No  NA
7.6	Is color highlighting used to indicate that an item has been selected?	 Yes  No  NA
7.7	Is color coding consistent throughout the system?	 Yes  No  NA
7.8	Can the user easily locate data?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

8. Flexibility and Efficiency of Use

The Web site should offer users a number of options when it comes to finding content on the site. Users should be able to achieve their goals in an efficient manner.

I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
8.1	If menu lists are short (seven items or fewer), can users select an item by moving the cursor?	 Yes  No  NA
8.2	Are there ways of helping users to find content on the site? (such as hyperlinks, or alphabetical index)	 Yes  No  NA
8.3	If the system uses a pointing device, do users have the option of either clicking on fields or using a keyboard shortcut?	 Yes  No  NA
8.4	On data entry screens, do users have the option of either clicking directly on a field or using a keyboard shortcut?	 Yes  No  NA
8.5	On menus, do users have the option of either clicking directly on a menu item or using a keyboard shortcut?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

9. Aesthetic and Minimalist Design

Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

 I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
9.1	Is only (and all) information essential to decision making displayed on the screen?	 Yes  No  NA
9.2	Have large objects, bold fonts, and simple areas been used to distinguish sections?	 Yes  No  NA
9.3	Are field labels brief, familiar, and descriptive?	 Yes  No  NA
9.4	Is the visual layout well designed?	 Yes  No  NA
9.5	Are there any unnecessary data elements in each screen?	 Yes  No  NA
9.6	Is each lower-level sub-section/data item associated with only one higher-level section?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

10. Help and Documentation

Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

 I.  Please check your response for the individual items related to this usability factor:

#	Usability Factor	Response	Comments
10.1	Is there a help function; for example, a key labeled HELP or a special menu?	 Yes  No  NA
10.2	Is help information provided that can be easily searched and followed?	 Yes  No  NA
10.3	In help page, is the visual layout well designed?	 Yes  No  NA
10.4	Is the interface of help page consistent with the interfaces of the site it supports?	 Yes  No  NA
10.5	Can users easily switch between help and their work?	 Yes  No  NA
10.6	Can users resume work where they left off after accessing help?	 Yes  No  NA

II. Please circle the overall severity rating for this usability factor:

No Usability Problem	Cosmetic Problem Only	Minor Usability Problem	Major Usability Problem	Usability Catastrophe
0	1	2	3	4

III. If you have other comments, please specify.

8. The Post-Study System Usability Questionnaire (PSSUQ)

Administration and Scoring. Give the PSSUQ to participants after they have completed all the scenarios in a usability study. You can calculate four scores from the responses to the PSSUQ items: the overall satisfaction score (OVERALL), system usefulness (SYSUSE), information quality (INFOQUAL) and interface quality (INTERQUAL). Refer to Appendix Table 1 in the next section of this appendix for the current scoring rules of the PSSUQ.

Instructions and Items. The questionnaire's instructions and items are:

This questionnaire, which starts on the following page, gives you an opportunity to tell us your reactions to the system you used. Your responses will help us understand what aspects of the system you are particularly concerned about and the aspects that satisfy you. To as great a degree as possible, think about all the tasks that you have done with the system while you answer these questions.

Please read each statement and indicate how strongly you agree or disagree with the statement by circling a number on the scale. If a statement does not apply to you, circle N/A. Please write comments to elaborate on your answers.

After you have completed this questionnaire, I'll go over your answers with you to make sure I understand all of your responses.

Thank you!

1. Overall, I am satisfied with how easy it is to use this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
2. It was simple to use this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
3. I could effectively complete the tasks and scenarios using this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
4. I was able to complete the tasks and scenarios quickly using this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
5. I was able to efficiently complete the tasks and scenarios using this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
6. I felt comfortable using this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:

7. It was easy to learn to use this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
8. I believe I could become productive quickly using this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
9. The system gave error messages that clearly told me how to fix problems.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
10. Whenever I made a mistake using the system, I could recover easily and quickly.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
11. The information (such as on-line help, on-screen messages and other documentation) provided with this system was clear.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:

12. It was easy to find the information I needed.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
13. The information provided for the system was easy to understand.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
14. The information was effective in helping me complete the tasks and scenarios.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
15. The organization of information on the system screens was clear.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:

Note: The interface includes those items that you use to interact with the system. For example, some components of the interface are the keyboard, the mouse, the screens (including their use of graphics and language).
16. The interface of this system was pleasant.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
17. I liked using the interface of this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
18. This system has all the functions and capabilities I expect it to have.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS:
19. Overall, I am satisfied with this system.
STRONGLY AGREE	1	2	3	4	5	6	7	STRONGLY DISAGREE
COMMENTS: