Moderated Usability Test

Form Approved

OMB No. 0990-0379

Exp. Date 08/31/2023

Moderated Usability Test Sessions

Moderated usability tests will be conducted with potential applicants and are comprised of two parts. Each part has a menu of NOFO components that the facilitator can choose from when hosting a session.

The two parts are:

1. Cloze tests + content questions to measure comprehension

2. Comparative usability tests to measure readability/ease of use

Estimated time to complete: 60 minutes

Part 1

Cloze tests + content questions [30 min]

In this activity, users will complete a cloze test for each NOFO program summary, and then answer follow up moderator questions. Cloze tests measure how easy content is to comprehend. In these cloze tests, we’ll replace every 6^th word with a blank space and ask readers to fill in the most logical answer. A score of 60% correct or higher is ideal.

Suggested menu of options for cloze tests:

• NOFO prototype program summary paragraphs

• NOFO prototype eligibility paragraphs

• NOFO prototype registration instructions

• NOFO prototype submission requirements

• All of the above re-written in a more conversation or formal tone

Script:

• Thank you for taking the time to speak with us today. As we mentioned in our email, we are working to simplify grant announcements, so they are easier to understand and so applicants have a better chance to succeed.

• Your feedback will be valuable in informing the redesign of the grant announcement and help ensure it is accessible for future applicants.

• Your candid feedback is important and anything you share with us is anonymous and will not be tied to your name or organization. You don’t have to answer any questions you don’t feel comfortable answering, and you can end this conversation at any point. Nothing you share with us will impact any current or future application.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0379. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

• We also want to compensate you for your time today and would like to pay you $40.

• This sixty-minute session has two parts.

• Let’s start with the first part. I’m going to show you a paragraph that is missing words and I want you to take three minutes to write in the words that are missing. After you complete the activity, I’ll show you the paragraph with the correct words and ask you a few questions.

• Please remember that we’re testing the ability of this writing to properly convey a message; we’re not testing you.

• Here is the paragraph with words missing. Please fill in the blanks as best you can and let me know when you’ve finished.

Program Summary

The Children’s Bureau seeks trauma-_________ interventions to address trauma in _________ and youth with complex mental, ___________, and health conditions. Funding supports __________ to implement interventions that are ___________ and developmentally responsive, achieve demonstrable ____________ in well-being for children and ________, and share lessons learned.

• Here is the paragraph without missing words. Let me know when you’re done reading.

Program Summary

The Children’s Bureau seeks trauma-informed interventions to address trauma in children and youth with complex mental, behavioral, and health conditions. Funding supports projects to implement interventions that are culturally and developmentally responsive, achieve demonstrable improvements in well-being for children and youth, and share lessons learned.

• What did you think about the information in this paragraph?

• What was easy to answer, and why?

• What was difficult to answer, and why?

• If you could change anything about the information in this paragraph, what would it be?

• If you had to explain this funding opportunity to a child, what would you say?

Repeat with additional cloze tests, document cloze test scores and responses.

Part 2

Comparative usability tests [30 min]

These tests will compare different designs with users to measure performance (as best we can) and preference.

The designs we test will use the same content so that we can evaluate the effectiveness of the designs themselves.

Suggested menu of options for comparative usability tests:

• Opportunity overview page

• Submission instructions page

• Breadcrumb tools

• Divider pages

• Matching requirement formulas

• Table designs

• Page with iconography versus one without

• Formatting (landscape vs portrait)

Script:

Let’s move on to the second part. I’m going to show you several different designs, one at a time, and ask you to complete tasks and answer questions.

This is an opportunity overview page for a federal grant announcement. Take a few minutes to read it.

Watch for struggle points during tasks and ask follow up questions.

• Task 1: Find the title of this grant announcement.

• Task 2: Find the opportunity number for this grant announcement.

• Task 3: Find the eligibility information.

• Task 4: Find the completion date.

• How would you rate this design on a scale from 1 to 10 with 1 being very poor and 10 being very good?

• Why did you choose that rating?

• How would you describe this design?

• What’s working well?

• What’s not working well?

• If you could improve one thing, what would it be?

Repeat with additional designs:

Continue comparative usability tests with additional design components.

Thank you again for your time. If you have any questions you can email [email protected].