Usability Study for Design of the NAEP Public Website 2013

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

NAEP Website Usability Study 2013 Volume II

Usability Study for Design of the NAEP Public Website 2013

OMB: 1880-0542

Document [docx]
Download: docx | pdf

National Center for Education Statistics

National Assessment of Educational Progress



Volume II

Interview and User Testing Instruments



Request for Clearance for Usability Study for Design of the NAEP Public Website 2013



OMB# 1880-0542




January 31, 2021


Table of Contents


  1. Interview Questionnaires

Interviews will be conducted to learn what type of information users would want to find on the NAEP public website. Interviews will be based on the questions below. They will be open-ended and, if a user brings up new content that would be of interest to them, the interviewer will probe to get more information about that content. Because user reactions are unknowable in advance, some probing questions will be devised during the interview and are not included in this list of questions.


Adult Questionnaire

Hello ____________________,

As you know, my associates and I are conducting interviews to find out what information we can offer to people on the NAEP website that would be the most useful to them. The NAEP program conducts assessments around the country to provide information about what America’s students know and can do in various subject areas, such as mathematics, reading, science, writing, the arts, civics, economics, geography, and U.S. history.

NAEP provides results on subject-matter achievement, instructional experiences, and school environment for populations of students (e.g., all fourth-graders) and groups within those populations (e.g., female students, Hispanic students). NAEP does not provide scores for individual students or schools.

If you have used the NAEP website in the past, we are interested in your experiences using the site and the ways in which you think we could improve the experience for people such as yourself. If you have not used the NAEP website in the past, we are interested in knowing what kind of education and assessment sites you do visit, and why.

  1. Have you ever visited the NAEP public site?

    1. If “yes”, continue with “Current Visitors” questionnaire. If “no”, skip to “Potential Visitors” questionnaire



Current Visitors Questionnaire

  1. How often or frequently do you visit the NAEP site?

  2. What kind of information have you sought on the NAEP website?

  3. Did you find what you were looking for?

  4. What made it easy or hard to find information?

  5. What other information would you want to find there?

  6. What other sites do you visit for this type of information?

  7. What do you like about those sites?

  8. What would you improve on the NAEP public site?



Potential Visitors Questionnaire

  1. Do you surf the web for education-related information?

  2. What type of information is of interest to you?

  3. What sites do you regularly visit for education information?

  4. What would/do you use education assessment information for?

  5. Here’s what type of info we have on the NAEP public website..(long-term trends, population breakdowns, assessment literacy, sample items) Is this of interest to you? If so, why??

  6. What else would be of interest to you?

  7. How did you learn about some of your favorite education-related websites

  8. What would you improve on the NAEP public site?



Student Questionnaire

Scenario: “We administer tests to students across the country to see how different groups of students perform on tests of English, math, and other subjects. We want students to be comfortable taking these tests. We want them to know what to expect and to not be nervous. We want students who will be taking our tests to use our website to get information about the tests and see what they are like before they actually take the test.”

Questions:

  1. First of all, we want to know how much time you spend on the internet. Do you think you spend less than one hour a week, more than 5 hours a week, or somewhere in between?

  2. If you found out you were going to be taking one of our tests, would you want to get more information about it, or would you not worry about it?

  3. If your teacher gave you a link to a website about the tests, would you check it out?

  4. What kind of information would you want to know about these tests we are giving?

  5. Can you think of something we could put on the website to make you comfortable and not nervous about our tests?

  6. What kinds of things would you be worried about if you were going to take one of our tests?

  7. Have you ever taken practice tests before?

  8. Would you want to take a practice test for the tests we are giving?

  9. If we had games on our site, would that make you more likely to visit the site?

  1. Directions and sample content list for card sort

Participants will be given a list of content and will be asked to group that content as they see fit. The mechanism is “drag and drop,” so they will click on a content item and drag it into a pile of content they perceive to be similar. These “piles” will form the basis of our information architecture.



Sample directions:

Step 1

Take a quick look at the list of items to the left.

We'd like you to sort them into groups that make sense to you.

There is no right or wrong answer, just do what comes naturally.


Step 2

When you're ready, drag an item from the left to create your first group.


Step 3 [presented after first item is placed on the left side of the screen]

If you already know what you'd like to call this group, click the title to rename it now. If not, you can do this later.


Step 4

Now you can either put more items into this group, or make more groups. It's up to you!

When you're done, click "Finished" (top right).


“Ask NAEP" Transcripts

NAEP Technical Documentation

2009 Inclusion Study

NAEP Technical Documentation

Activities for Grades 4 and 8

National and State NAEP

Activities for Grades 8 and 12

National NAEP

Additional Links for Grades 8 and 12

Parent FAQs

Analysis and Scaling

Participation is Important

Assessment Frameworks

Participation Rate Requirements

Authorization and Oversight

Performance Levels - Math

Comparing NAEP and State Proficiency Standards

Performance Levels - Reading

Current Activities

Processing Assessment Materials

Data Collection

Questionnaires

Database

Research With NAEP Data

Design Goals

Sample Design

Development of the Assessment Bridge Study

Sample Design Frequently Asked Questions

District Profiles

Sample Letter from Participating Schools to Parents

Educator Update

Sample Questions

Educators' Frequently Asked Questions

Scale Scores

Effects of Exclusion

Schedule of Assessments

History of NAEP Contractors

Scoring

How do States Join?

Setting Achievement Levels

How NAEP Is Administered

Split-Sample Design

How Results are Reported

State Comparisons Tool

How Schools and Students Are Selected

State NAEP

How to Use NAEP Frameworks

State Profiles

Inclusion, Exclusion, Accommodation

Status of Achievement Levels

Instruments

Student FAQs

Interpreting the Results

TEL Assessment

Item Development

Test Yourself

Item Maps

The Importance of NAEP

Item Scoring Process

Training and Conferences

Long-Term Trend and Main NAEP

Urban Districts (TUDA)

NAEP and International Assessments

Weighting

NAEP and No Child Left Behind

What is Measured?

NAEP Data Explorer

What the Mathematics Assessment Measures

NAEP e-Library

What the Reading Assessment Measures

NAEP History and Background

Who is Assessed?

NAEP Questions Tool

User Scenarios

  1. User Scenarios

During the user testing session, participants will be asked to perform certain tasks while visiting a “mock up” of the redesigned NAEP public website. The actual tasks will be determined by the interactive elements created for the website in response to the information gleaned from the user interviews (phase 1). Examples of the type of tasks typically used for user testing are as follows:

General Directions

You will be using the NAEP website to perform several tasks. You will be given enough information to complete the task, but will not be guided step-by-step through the process. Our goal is to see if the information presented on the website is sufficient for you to complete your tasks.

While you are navigating through the website, we would like you to verbalize what you are doing, and why. For example, you might say something like, “I am going to click on this link that says ‘studies’ because I think that is where I will find information about the writing assessment.” If you get frustrated trying to find or do something, that is important information that we would like you to verbalize. Some tasks may not be achievable. In these cases, the research associate will ask you to move on to the next task after a few attempts.

After each set of tasks, you will be asked to rate the ease of use for those tasks, using the following rating scale:

1

2

3

4

5

Very Difficult

Somewhat Difficult

Neither easy nor difficult

Somewhat Easy

Very Easy

Task 1. Let’s say you want to look at a sample question from the 2012 Economic assessment. Show me how you would go about doing that.

Task 2. Our next task is to download data from the 2011 Writing assessment. Show me how you would find and download those data.

Task 3. We have listed the courses that were considered in the High School Transcript Study to represent “Advanced Science.” Please find that information and tell me two of the courses that meet that definition.

Task 4. NAEP is planning a TEL assessment. Please find information about this assessment in order to answer the following questions:

  1. What grade will be used for the first NAEP TEL assessment?

  2. What three interconnected areas of technology and engineering literacy are measured?

  3. When was the TEL pilot study conducted?

Task 5. Find out how much time a classroom teacher is expected to spend helping students get ready for NAEP.

Task 6. You think I might be interested in looking at a sample question booklet. Please find this information and e-mail the link to me.

---

Comments: “Do you have any feedback for us about this task? Things you like or don’t like? Suggestions for improvement?

  1. System Usability Scale

Each adult participant will be asked to take the System Usability Scale (SUS) after completion of the third phase of the study. The SUS consists of 10 survey questions and provides a norm-referenced comparison among a wide range of user interface designs.

  1. I think that I would like to use this system frequently.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I found the system unnecessarily complex.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I think the system is easy to use.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I think that I would need the support of a technical person to be able to use this system.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I found the various functions in this system were well integrated.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I think there is too much inconsistency in this system.

Strongly
Agree
1


2


3


4

Strongly Disagree
5






  1. I would imagine that most people would learn to use this system very quickly.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I found the system very cumbersome to use.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I felt very confident using the system.

Strongly
Agree
1


2


3


4

Strongly Disagree
5

  1. I needed to learn a lot of things before I could get going with this system.

Strongly
Agree
1


2


3


4

Strongly Disagree
5




File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleOMB Apprioval App, Vol II
Authorjoconnell
File Modified0000-00-00
File Created2021-01-31

© 2024 OMB.report | Privacy Policy