2 #2_GEM Attach 2C 2D Survey and Guide

Evaluation of the Grid Enabled Measures (GEM) Website

OMB#: 0925-0589-02

Attachment 2C: Participant Survey

Attachment 2D: Facilitator Interview Guide

Attachment 2C: Participant Survey

The GEM web site serves several audience groups and it will be helpful to us to understand how you fit into those audience groups before participation in a usability test of the site. The following questions are intended to help us understand your background.

OMB No: 0925-0589-02

Expiry Date: 4/30/2014

Public reporting burden for this collection of information is estimated to average 5 minutes total, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN; PRA (0925-0589-02). Do not return the completed form to this address.

Use an X, bold, or the (Word) highlighter to mark your responses.

What is your background? [check all that apply] Communication Economics Education Epidemiology Evaluation Informatics/Library Science Psychology Public Health Social Sciences Statistics Other (please specify__________): Which positions describe you? [check all that apply] Program director at federal government Program director at non-profit organization Scientist Evaluator Academic researcher Student/ Trainee/ Fellow Administrative support staff Other (please specify):

Which of the following describe your typical work activities? [check all that apply] Program Planning/Development Program Implementation Writing articles for peer-reviewed scientific journals Writing Grants Writing Reports for internal planning Collecting/Analyzing Data Health Education Directing Clinical Services Informing Policy Advocacy Administrative Other (please specify): How would you describe your level of research experience? Novice Expert Have you used the GEM Website before? Yes No

OMB No. 0925-0589-02

Expiry Date: 4/30/2014

Attachment 2D: Facilitator Interview Guide

Usability Testing of the

GEM Database Website

Public reporting burden for this collection of information is estimated to average 60 minutes total, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN; PRA (0925-0589-02). Do not return the completed form to this address.

Interview Guide, First and Second Rounds of Usability Testing

[This material is only a guide for the website usability contractor (still to be determined) as they are conducting interviews with individuals about the Grid Enabled Measures Database Website to determine their information needs related to standardized measures, harmonized data, etc.  These questions are simply a guide and may not be asked using exactly these words or in exactly this order.  These questions will be used as necessary but will primarily follow the natural flow of the conversation.  For example, it is possible that based on the flow of conversation, not all of the questions will need to be asked.  Alternatively, some of the follow up questions may be answered naturally without being asked.] (from 0925-0589-05, approved last year for the website usability).

Intro: Thank you very much for taking part in usability testing for the GEM website, which is being designed by the National Cancer Institute.

[New User Only] I am going to give you a chance to explore the GEM website and then I’d like to ask you a few questions that will help us to identify how the website can be improved, both in terms of how it is designed, and the content of the website. Next, I am going to ask you to perform a few common website tasks and follow up with questions about these tasks. This whole process will take about an hour to complete.

While I will be asking you specific questions during this session, as you go through the website and complete tasks, please say out loud any thoughts you have about the website, including its look, feel, navigatability, and ease of use. The National Cancer Institute’s goal is to make this site better for you as a (researcher/policy maker/clinician), and your feedback will help us to achieve that goal.

There are two important things you should keep in mind before we get started:

• First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like to discuss them with you to see if we can find a way to make the website better.

• Second, we are evaluating the website, not you so you cannot make any mistakes. The website is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the website – not with you.

Do you have any questions for me before we get started? Ok, wonderful.

[Previous User Only] I am going to ask you about your opinion of the website and if it meets our program’s goals and to perform a few common website tasks. I will then follow up with questions about these tasks to help us to identify how the website can be improved, both in terms of how it is designed, and the content of the website. This whole process will take about an hour to complete.

While I will be asking you specific questions during this session, as you go through the website and complete tasks, please say out loud any thoughts you have about the website, including its look, feel, navigatability, and ease of use. The National Cancer Institute’s goal is to make this site better for you as a (researcher/policy maker/clinician), and your feedback will help us to achieve that goal.

There are two important things you should keep in mind before we get started:

• First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like to discuss them with you to see if we can find a way to make the website better.

• Second, we are evaluating the website, not you so you cannot make any mistakes. The website is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the website – not with you.

Do you have any questions for me before we get started? Ok, wonderful.

[New Users Only] Now, I would like you to go to https://www.gem-beta.org/ and just explore the site for about 5 minutes. Remember to please state out loud any thoughts you have about the site.

[ After 5 Minutes of exploring and comments] Great, now that you have gotten acquainted with the website…

1. What did you anticipate you would find on the GEM website before going through it [New Users Only]?

1. To what extent was the content of the website similar to or different from what you expected to find?

2. What are your initial thoughts about the goal(s) of the site? What do you think is its purpose?

[Previous Users Only] Now, I would like you to go to https://www.gem-beta.org/ and I am going to ask you to complete some website tasks and answer some questions. Remember to please state out loud any thoughts you have about the website as we go along.

Tasks:

3. Find information about GEM: How would you find out basic information about GEM and the services the website offers? Where would you go first?

4. Read/edit/add information about constructs, measures and datasets

Ask the user to read and edit information about existing constructs, measures and datasets. Ask the user to add information about a new construct and/or measure.

5. Comment on constructs

Ask the user to add a comment to an existing construct.

6. Comment on measures

Ask the user to add a comment to an existing measure.

7. Rate measures

Ask the users to add a rating to an existing measure.

8. Upload measures

Ask the user to explain how he/she would upload an existing measure into GEM

9. Upload data

Ask the user to explain how he/she would upload an existing dataset into GEM

10. Access data

Ask the user to explain how he/she would access data through GEM.

2. [FACILITATOR NOTE, NOT STATED OUT LOUD] Does the user understand how to use the interactive functions of GEM?

Follow-up questions for Users:

3. Does the website successfully communicate the purpose and goals of GEM?

4. How easily and quickly were you able to navigate to a desired page?

5. Does the GEM website clearly communicate how users are able to share harmonized data and the rationale for doing this?

6. Do you currently share data with other researchers? (Yes/No)

7. If you do share data, how do you share it (CD/DVD, FTP etc.)? With whom do you share the data and are they at your institution or other institutions?

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

8. What issues come to mind when you think about sharing data (e.g., confidentiality concerns; IRB concerns)?

9. PLEASE TAKE A MOMENT TO VISIT a caBIG WEBSITE THAT IS A RESOURCE FOR THOSE INTERESTED IN SHARING DATA: https://cabig-kc.nci.nih.gov/DSIC/KC/index.php/Main_Page; make sure to review this tool: https://cabig-kc.nci.nih.gov/DSIC/KC/index.php/Electronic_Data_Sharing_and_Security_Framework

We are thinking of using this tool as a resource for those who share data through GEM.

Do you think this kind of application would be useful to you in sharing data? [choose one answer]

Yes

Perhaps

No

10. What functions of the GEM website are most beneficial to you?

1. Ability to read information about constructs and measures and data?

2. Ability to add/edit information about constructs, measures and data?

3. Ability to share and download research measures?

4. Ability to access data?

5. Ability to share data?

11. To what extent would your research and information needs being met by the website? If its needs are not being met, how could the website be revised to better meet these needs?

12. What do you think of ‘My GEM’? Would you use this resource to subscribe to a construct or measure? Why or why not?

13. Were there any resources that you hoped to find, that were not present, or that you couldn’t locate? If so, what were they?

a. What additional resources would you like to see included on the website?

14. Before we wrap up:

a. Do you have any additional feedback about the design and functionality of the website?

b. Do you have any additional feedback about what you’d like to see in terms of additional or different content or functionality for the website?

c. Is there any other feedback you want to give, about any of the topics we talked about today?

Thank you for sharing your feedback about the GEM website. Your feedback will help us to enhance the design and content of the site. Do you have any questions for me about GEM?

Interview Guide, Second and Third Rounds of Usability Testing

[Testing will be done iteratively with three different groups of individuals with changes to the site being made between testings based on user feedback. Thus, interview questions for successive testing iterations will be very similar to previous ones asked (see below) but will also include tailored questions based on what we learn from previous rounds of usability testing]

Intro: Thank you very much for taking part in usability testing for the GEM database website, being designed by the National Cancer Institute. Now that you’ve explored the GEM website, I’d like to ask you a few questions that will help us to identify how the website can be improved, both in terms of how it is designed, and the content of the website. The National Cancer Institute’s goal is to make this site better for you as a ________(Select one of the following, based on the role of the interviewee: researcher, policy person etc.)

1. Have you used the GEM website before today?

If yes, how often do you use it, and what did you use it for?

If no, after the exposure to the website through usability testing today, do you think you would use it in the near future? What would you use it for?

2. How do you like the website so far? What components of the website do you consider to be most useful to your work? Please explain.

3. Have you added any information about constructs or measures or downloaded a measure to the website?

• If Yes, could you please tell us more about your experience with adding information or downloading or uploading a measure? What are some aspects we can improve for you to add information or measures?

• If No, how likely are you to add information or measures download or to this site in the future? Why or why not?

4. How do you feel about the [potentially new features such as expert blogs, listserv, discussion boards, news section] of the website? How frequently do you access or participate in these forms of social media? In what ways do you think these web features can be improved?

5. How likely would you be to share data using GEM? Why or why not?

6. Do you have any additional comments about today's session or any additional advice to us as we move forward in the development of the next iteration of the GEM website?

Thank you for sharing your feedback about GEM. Your feedback will help us to enhance the design and content of the site. Do you have any questions for me about GEM?

10