NAEP 2011 Technology Writing Assessment Usability Study

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume 2

Tryout Survey Instruments

for

System Small Scale Usability Study

Request for Clearance for System Small Scale Usability Study for the 2010 NAEP Writing Assessment Pilot

OMB# 1850-0803

(Generic Clearance for Cognitive, Pilot, and Field Test Studies)

Student Grade 8 Hardware Evaluation

Study for Computer-Delivered Pilot Writing Assessment in 2010

The amount of time estimated to complete these forms is 60 minutes.

March 27, 2009

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 1990 K Street N.W., Washington, D.C. 20006-5651.

A project of the Institute of Education Sciences.

The information you provide will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and other applicable Federal laws, your responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of up to $250,000, or both if he or she willfully discloses ANY identifiable information about you.

OMB No. 1850-0803 Approval Expires 09/30/2010

Rate the Sample Writing Test Program

Please circle the number that most clearly describes how you feel about each statement. Write in any comments you have below each statement.

1. Using the laptop to perform the test was:

1 2 3 4 5

Very Easy Easy Neither Easy Difficult Very

Nor Difficult Difficult

Comment: ______________________________________________________________________________

______________________________________________________________________________

2. Reading the words on the screen was:

1 2 3 4 5

Very Easy Easy Neither Easy Difficult Very

Nor Difficult Difficult

Comment:

______________________________________________________________________________

______________________________________________________________________________

3. Typing on the keyboard was:

1 2 3 4 5

Very Easy Easy Neither Easy Difficult Very

Nor Difficult Difficult

Comment: ______________________________________________________________________________

______________________________________________________________________________

4. Understanding what to do for the writing test was:

1 2 3 4 5

Very Easy Easy Neither Easy Difficult Very

Nor Difficult Difficult

Comment: ______________________________________________________________________________

______________________________________________________________________________

5. Doing what I needed to do for the writing test was:

1 2 3 4 5

Very Easy Easy Neither Easy Difficult Very

Nor Difficult Difficult

Comment:

______________________________________________________________________________

______________________________________________________________________________

6. The thing I liked best about taking a writing test using this program was:

______________________________________________________________________________

______________________________________________________________________________

7. The thing I liked least about taking a writing test using this program was:

______________________________________________________________________________

______________________________________________________________________________

8. For each statement below, please put a checkmark in one of the boxes from Strongly Agree to Strongly Disagree.

Opinion	Strongly Agree	Agree	No Opinion	Disagree	Strongly Disagree
I liked the “skin” (color scheme) choices.
It was easy to make the text size bigger or smaller.
It was easy to listen to the text being read by the computer.
I was able to understand the question and see a picture or video.
I was able to highlight words.
I was able to open and close the sections /areas of the screen I needed to use without problems.
I liked being able to use the pre-writing area.
It was easy to answer the test question in the writing area.
I already knew how to use a word processor.
I figured out how to use the word processor.
I already knew how to use a spell checker.
I figured out how to use the spell checker.
I liked being able to turn the timer (clock) off.
I knew my progress during the test and was able to plan my time well.
I had enough time to write a response.

Fulcrum IT TWA Prototype Usability Study
Student Tasks and Facilitator Observations
	Purpose of the prototype/system performance test:	1. Validate mental model, navigation, and page flow	3. Uncover design and technology issues
		2. Evaluate user interaction with page-level controls and features	4. Determine effectiveness of content, editorial style, and graphics, and functionality of features	* Rating Scale 0 = no problems 1 = some problems 2 = many problems 3 = could not do
Task No.	Key Area	Task / Question	Comments/ Problems	User Performance Rating *
1	Login	Sign in using assigned student number
2	Tutorial	View tutorial
3	Prompt	Begin Prompt #1
4	Use features
4a		Open / close panes
4b		Change skin (color scheme)
4c		Change font size
4d		Use text-to-speech
4e		Magnify text
4f		Highlight text
4g		Get Spanish version
4h		Other
4i		Other
5	Use Planning pane	Type in panel
6	Use Response pane	Type in panel
6a		Copy/paste text
6b		Format text
6c		Use thesaurus
7		Use Spell check
8	Submit prompt
9	Proceed to section with background questions and complete
10	Submit background questions
11	Log off

Fulcrum IT TWA Prototype Usability Study
System Operations
				* Rating Scale 0 = no problems 1 = some problems 2 = many problems 3 = did not work
	Scenario	Task / Question	Comments/ Problems	System Performance Rating *
A	Laptops fully connected between student and administrator machine	Can administrator monitor testing; do student machines write data back to admin machine throughout test?
B	Laptops fully disconnected with administrator retrieving data via USB from the student machine and putting it on the administrator machine	Can administrator begin and end test without connectivity; specifically retrieve data from student machines at the end?
C	Laptops fully connected between student and administrator machine; several machines lose connection and then are re-connected	Does system notify administrator of lost connectivity; can administrator re-connect and retrieve all data successfully from student machines?
D	Laptops begin fully connected, lose connection and do not recover it for the remainder of the assessment	Does system notify administrator of lost connectivity; can administrator retrieve all data successfully from student machines?