Usability Study for Design of the NAEP Public Website 2013

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Interview and User Testing Instruments

Request for Clearance for Usability Study for Design of the NAEP Public Website 2013

OMB# 1880-0542

January 31, 2021

Table of Contents

1 Interview Questionnaires 2

2 Directions and sample content list for card sort 4

3 User Scenarios 6

4 System Usability Scale 7

1. Interview Questionnaires

Interviews will be conducted to learn what type of information users would want to find on the NAEP public website. Interviews will be based on the questions below. They will be open-ended and, if a user brings up new content that would be of interest to them, the interviewer will probe to get more information about that content. Because user reactions are unknowable in advance, some probing questions will be devised during the interview and are not included in this list of questions.

Adult Questionnaire

Hello ____________________,

As you know, my associates and I are conducting interviews to find out what information we can offer to people on the NAEP website that would be the most useful to them. The NAEP program conducts assessments around the country to provide information about what America’s students know and can do in various subject areas, such as mathematics, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

NAEP provides results on subject-matter achievement, instructional experiences, and school environment for populations of students (e.g., all fourth-graders) and groups within those populations (e.g., female students, Hispanic students). NAEP does not provide scores for individual students or schools.

If you have used the NAEP website in the past, we are interested in your experiences using the site and the ways in which you think we could improve the experience for people such as yourself. If you have not used the NAEP website in the past, we are interested in knowing what kind of education and assessment sites you do visit, and why.

1. Have you ever visited the NAEP public site?

   1. If “yes”, continue with “Current Visitors” questionnaire. If “no”, skip to “Potential Visitors” questionnaire

Current Visitors Questionnaire

1. How often or frequently do you visit the NAEP site?

2. What kind of information have you sought on the NAEP website?

3. Did you find what you were looking for?

4. What made it easy or hard to find information?

5. What other information would you want to find there?

6. What other sites do you visit for this type of information?

7. What do you like about those sites?

8. What would you improve on the NAEP public site?

Potential Visitors Questionnaire

1. Do you surf the web for education-related information?

2. What type of information is of interest to you?

3. What sites do you regularly visit for education information?

4. What would/do you use education assessment information for?

5. Here’s what type of info we have on the NAEP public website..(long-term trends, population breakdowns, assessment literacy, sample items) Is this of interest to you? If so, why??

6. What else would be of interest to you?

7. How did you learn about some of your favorite education-related websites

8. What would you improve on the NAEP public site?

Student Questionnaire

Scenario: “We administer tests to students across the country to see how different groups of students perform on tests of English, math, and other subjects. We want students to be comfortable taking these tests. We want them to know what to expect and to not be nervous. We want students who will be taking our tests to use our website to get information about the tests and see what they are like before they actually take the test.”

Questions:

1. First of all, we want to know how much time you spend on the internet. Do you think you spend less than one hour a week, more than 5 hours a week, or somewhere in between?

2. If you found out you were going to be taking one of our tests, would you want to get more information about it, or would you not worry about it?

3. If your teacher gave you a link to a website about the tests, would you check it out?

4. What kind of information would you want to know about these tests we are giving?

5. Can you think of something we could put on the website to make you comfortable and not nervous about our tests?

6. What kinds of things would you be worried about if you were going to take one of our tests?

7. Have you ever taken practice tests before?

8. Would you want to take a practice test for the tests we are giving?

9. If we had games on our site, would that make you more likely to visit the site?

2. Directions and sample content list for card sort

Participants will be given a list of content and will be asked to group that content as they see fit. The mechanism is “drag and drop,” so they will click on a content item and drag it into a pile of content they perceive to be similar. These “piles” will form the basis of our information architecture.

Sample directions:

Step 1

Take a quick look at the list of items to the left.

We'd like you to sort them into groups that make sense to you.

There is no right or wrong answer, just do what comes naturally.

Step 2

When you're ready, drag an item from the left to create your first group.

Step 3 [presented after first item is placed on the left side of the screen]

If you already know what you'd like to call this group, click the title to rename it now. If not, you can do this later.

Step 4

Now you can either put more items into this group, or make more groups. It's up to you!

When you're done, click "Finished" (top right).

“Ask NAEP" Transcripts	NAEP Technical Documentation
2009 Inclusion Study	NAEP Technical Documentation
Activities for Grades 4 and 8	National and State NAEP
Activities for Grades 8 and 12	National NAEP
Additional Links for Grades 8 and 12	Parent FAQs
Analysis and Scaling	Participation is Important
Assessment Frameworks	Participation Rate Requirements
Authorization and Oversight	Performance Levels - Math
Comparing NAEP and State Proficiency Standards	Performance Levels - Reading
Current Activities	Processing Assessment Materials
Data Collection	Questionnaires
Database	Research With NAEP Data
Design Goals	Sample Design
Development of the Assessment Bridge Study	Sample Design Frequently Asked Questions
District Profiles	Sample Letter from Participating Schools to Parents
Educator Update	Sample Questions
Educators' Frequently Asked Questions	Scale Scores
Effects of Exclusion	Schedule of Assessments
History of NAEP Contractors	Scoring
How do States Join?	Setting Achievement Levels
How NAEP Is Administered	Split-Sample Design
How Results are Reported	State Comparisons Tool
How Schools and Students Are Selected	State NAEP
How to Use NAEP Frameworks	State Profiles
Inclusion, Exclusion, Accommodation	Status of Achievement Levels
Instruments	Student FAQs
Interpreting the Results	TEL Assessment
Item Development	Test Yourself
Item Maps	The Importance of NAEP
Item Scoring Process	Training and Conferences
Long-Term Trend and Main NAEP	Urban Districts (TUDA)
NAEP and International Assessments	Weighting
NAEP and No Child Left Behind	What is Measured?
NAEP Data Explorer	What the Mathematics Assessment Measures
NAEP e-Library	What the Reading Assessment Measures
NAEP History and Background	Who is Assessed?
NAEP Questions Tool	User Scenarios

3. User Scenarios

During the user testing session, participants will be asked to perform certain tasks while visiting a “mock up” of the redesigned NAEP public website. The actual tasks will be determined by the interactive elements created for the website in response to the information gleaned from the user interviews (phase 1). Examples of the type of tasks typically used for user testing are as follows:

General Directions

You will be using the NAEP website to perform several tasks. You will be given enough information to complete the task, but will not be guided step-by-step through the process. Our goal is to see if the information presented on the website is sufficient for you to complete your tasks.

While you are navigating through the website, we would like you to verbalize what you are doing, and why. For example, you might say something like, “I am going to click on this link that says ‘studies’ because I think that is where I will find information about the writing assessment.” If you get frustrated trying to find or do something, that is important information that we would like you to verbalize. Some tasks may not be achievable. In these cases, the research associate will ask you to move on to the next task after a few attempts.

After each set of tasks, you will be asked to rate the ease of use for those tasks, using the following rating scale:

1	2	3	4	5
Very Difficult	Somewhat Difficult	Neither easy nor difficult	Somewhat Easy	Very Easy

Task 1. Let’s say you want to look at a sample question from the 2012 Economic assessment. Show me how you would go about doing that.

Task 2. Our next task is to download data from the 2011 Writing assessment. Show me how you would find and download those data.

Task 3. We have listed the courses that were considered in the High School Transcript Study to represent “Advanced Science.” Please find that information and tell me two of the courses that meet that definition.

Task 4. NAEP is planning a TEL assessment. Please find information about this assessment in order to answer the following questions:

1. What grade will be used for the first NAEP TEL assessment?

2. What three interconnected areas of technology and engineering literacy are measured?

3. When was the TEL pilot study conducted?

Task 5. Find out how much time a classroom teacher is expected to spend helping students get ready for NAEP.

Task 6. You think I might be interested in looking at a sample question booklet. Please find this information and e-mail the link to me.

---

Comments: “Do you have any feedback for us about this task? Things you like or don’t like? Suggestions for improvement?

4. System Usability Scale

Each adult participant will be asked to take the System Usability Scale (SUS) after completion of the third phase of the study. The SUS consists of 10 survey questions and provides a norm-referenced comparison among a wide range of user interface designs.

I think that I would like to use this system frequently.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the system unnecessarily complex.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think the system is easy to use.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think that I would need the support of a technical person to be able to use this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the various functions in this system were well integrated.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think there is too much inconsistency in this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I would imagine that most people would learn to use this system very quickly.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the system very cumbersome to use.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I felt very confident using the system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I needed to learn a lot of things before I could get going with this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐