NAEP Grade 4 Writing Computer-Based Assessment Usability Study, &The Assessment and Teaching of 21st Century Skills (ATC21S) Project Phase 2.2 Cognitive Interviews

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

Volume II

Tryout Survey Instruments

for

Small Scale Usability Study for Grade 4 Writing Computer-Based Assessment

OMB# 1850-0803

(Generic Clearance for Cognitive, Pilot and Field Test Studies)

Student Grade 4 Usability Evaluation of Writing Computer-Based Assessment

The amount of time estimated to complete these forms is 75 minutes.

January 24, 2011

(revised February 15, 2011)

Contents

Part I – Welcome Script and Background Information 1

Part II – Student Oral Survey on Keyboarding and Computer Use 3

Part III – Student Paper Design Preference 5

Part III-A – Paper Designs 8

Part IV-A1 – Student Tasks on Interface Prototypes 11

Part IV-A1a – Background Questions 14

Part IV-A2 – Student Survey 15

Part IV-B1 – Student Tasks on Interface Prototypes 18

Part IV-B2 – Student Survey 22

Part V – Student Design Comparison 25

Part VI –Typing Test 26

Part I – Welcome Script and Background Information

Grade 4 WCBA Prototype Usability Study Script

Text written in italics is suggested content with which the Facilitator should be thoroughly familiar in advance. The Facilitator should project a warm and reassuring manner toward the students to develop a friendly rapport and as such should use conversational language throughout.

Introduction:

Hello, my name is ______ and I work for Fulcrum IT, which is conducting a study for the National Center of Education Statistics (NCES) of the US Department of Education. This is _______ from _______ (observer affiliation.) It’s nice to meet you and thank you for helping us with our computers today.

Today you are going to be looking at different computer programs. This is part of a study to see which type of program works best for students in your grade if they were to take a writing test on laptop computers. One of the programs you will see is being used for writing tests by the National Assessment of Education Progress, also called NAEP for short, with 8th- graders and 12th-graders. We want to see how we might use it or change it for 4th-graders like you.

But you are not here to take a test today. You are helping us to try out the program. We’ll be looking to see how well you can find your way and operate the program, as well as what you think about how it looks or anything you like or don’t like. We won’t be telling anyone how you did or what you thought, but we will use your opinions and results without your name, along with the ideas and answers of other students, to make changes and improve the system for 4th-graders all across the country.

Do you have any questions so far?

Ok, I’m going to be the one talking to you and observing you, and ______ is also going to be observing closely and writing notes. We will start with me asking you a few questions, and then you will look at some pictures on paper before going on the computer. Soon you will be able to type on the laptop computer and then answer some other questions for us. Again, we are not testing you, we will be watching you test the program!

If at any time you decide you do not want to continue, that is your choice and you may stop and go back to class.

Okay, let’s begin.

Ref #	Facilitator Questions	Observer Notes
	Part I — Background Information
101	Name of Fulcrum Facilitator
102	Name/Affiliation of Observer
103	Date
104	Student ID’s number
105	Student’s Gender	 Male  Female
106	Name of School
107	School District
108	School Location	 Metro  Suburb  Rural
109	Special circumstances that may have affected the interview

Part II – Student Oral Survey on Keyboarding and Computer Use

Ref #	Facilitator Questions	Observer Notes
	Part II — Student Oral Survey
201	Do you like computers?	 Yes  No
202	Do you have a computer at home that you can use?	 Yes  No
203	How often do you use it?	 Once a month  Every other week  Every week  A couple of times a week  Every day  Other _____________________
204	Have you ever used a laptop computer?	 Yes  No
205	Do you use a computer at school?	 Yes  No
206	If so, how often?	 Once a month  Every other week  Every week  A couple of times a week  Every day  Other _____________________
207	What do you like to do on computers?	 Play games  Watch videos  Go on the Internet  Do homework  Other ______________________________
208	Have you had any typing or keyboarding lessons?	 Yes  No
209	Have you written a story, essay, or letter on a computer?	 Yes  No
210	If so, how many times (or how often)?	 1 time  2 or more times  Once a month  Every other week  Every week  A couple of times a week  Every day  Other _____________________
211	If you used a computer for writing, have you used any of these tools? If so, how often do you use them when you write on a computer?	Spell Check  A lot of times  Sometimes  Never Cut/copy/or paste  A lot of times  Sometimes  Never Backspace or delete  A lot of times  Sometimes  Never Bold/italics/underline  A lot of times  Sometimes  Never Indent  A lot of times  Sometimes  Never Undo or redo  A lot of times  Sometimes  Never Thesaurus  A lot of times  Sometimes  Never
212	Have you written an email or chat message on a computer?	 Yes  No
	If so, how many times (or how often)?	 1 time  2 or more times  Once a month  Every other week  Every week  A couple of times a week  Every day  Other _____________________
213	Have you ever taken a test or practice test on a computer?	 Yes  No
214	If so, how many times (or how often)?	 1 time  2 or more times  Once a month  Every other week  Every week  A couple of times a week  Every day  Other _____________________

Part III – Student Paper Design Preference

Ref #	Facilitator Question	Observer Notes
301 .	Now I’m going to show you some paper pictures of what some computer screens look like. I’m going to ask you some questions about what you are seeing and what you think things are on the paper. Ready? [Show Design A, B, or C in random order for each student; note which design.] (Refer to Part III-A on pp. 7-8 of this submission.) This is a picture of a computer screen. Can you tell me what you could do on this screen? [Let student tell as much as possible before prompting with the following suggestions.] [Point to a particular button or feature and say,] What could you do with this? [Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.] Is there anything more you can tell me about that? Why do you think that? What about the screen made you think that way? Have you seen something else like this on the computer? If so, what and where?	During the Paper Design Preference phase, observers should note or probe for the following design, tools, navigation, and word processing features for each sample design : Volume control Text-to-speech control Highlighter Zoom Timer Open/close buttons Next button Cut Copy Paste Undo Redo Bold Italicize Underscore Indent Spell check Thesaurus Comments on navigational features Comments on which design they felt was easier or clearer Comments on whether and how did the different layouts make a difference in their choice All other comments
	[When done with first design, put it aside and do the same for the second design.]
302	[Show second design; note which design.] This is a picture of a computer screen. Can you tell me what you could do on this screen? [Let student tell as much as possible before prompting with the following suggested prompts.] [Point to a particular button or feature and say,] What could you do with this? [Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.] Is there anything more you can tell me about that? Why do you think that? What about the screen made you think that way? Have you seen something else like this on the computer? If so, what and where?	Note same types of observations as listed in the observer notes sections of 301.
	[When done with second design, put it aside and do the same for the third design.]
303	[Show third design; note which design.] This is a picture of a computer screen. Can you tell me what you could do on this screen? [Let student tell as much as possible before prompting with the following suggested prompts.] [Point to a particular button or feature and say,] What could you do with this? [Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.] Is there anything more you can tell me about that? Why do you think that? What about the screen made you think that way? Have you seen something else like this on the computer? If so, what and where?	Note same types of observations as listed in the observer notes sections of 301.
	Now I’m going to bring all three pictures back out together so you can see them at the same time. Of the three screens, which one looks the most like a computer you have used before to write on the computer? Which one do you like the best? I am going to have you look at two of these screens on the laptop computer. Which one do you want to try first on the real laptop?

Part III-A – Paper Designs

This paper version of Design A will be used to elicit student preference of text editor design.

This paper version of Design B will be used to elicit student preference of text editor design.

This paper version of Design C will be used to elicit student preference of text editor design.

Part IV-A1 – Student Tasks on Interface Prototypes

Ref #	Facilitator Questions	Observer Notes
	PART IV-A: Student Tasks on the Interface Prototypes.
400-A	[Open the laptop and log the student in. Open the interface the student chose as 1st choice. Allow the student to freely explore the application. During the free exploration, ask the student questions about why he clicked on something, what he thought a button would do and whether it did what he thought (expectations), and how does he think it should work. The Observer will use the separate note sheet to record performance success or difficulty and other observations by answering the questions in the Observer Notes column.]
401-A	[Interface opens ] Can you tell or show me what you can do here? Which color do you like best for the screen? Can you tell or show me what you would do to begin the writing test?	Was the student able to change the skin color? What skin color did the student leave it on? Was the student able to navigate to the Prompt screen using the Next button?
402-A	ZOOM Do you see a way to make any of the words larger or smaller on the screen? Show me how you would do it.	If the student used the zoom feature, what text size level did the student keep for the longest time? Which area(s) did the student focus on when using the text enlarging feature?
403-A	HIGHLIGHT Do you see a way to highlight some words on this screen? Show me how you would do it.	Did the student select text in the prompt either to highlight or use text-to-speech? Did the student try to click the highlighter first and then try to highlight text (i.e. the opposite of how it currently works)?
404-A	TIMER Do you know whether this is a timed test? How do you know? Can you tell me how much time is left in the test?	Did the student check the timer display and leave it on? Did the student turn off the timer display when it came on automatically?
405-A	WORD PROCESSING PANEL Show me how you would begin writing to respond to this prompt.	Was the student able to find/ open the Write panel?
406-A	Can you type some words for me? How about a sentence? [Wait as student types.] Can you show me how you would change or delete a word? Can you show me how you would make a word or some words stand out to be more noticeable?	Does the student look at the keyboard while typing? Does the student have to look for each letter, for a lot of letters, or only for a few letters? Did the student use, delete, back space, cut copy, paste (indicate which, if any)? Did the student use bold, italics or underscore (indicate which, if any)?
407-A	Do you know how to indent the start of a paragraph? Show me how you would do it. Do you know how you would check the spelling of the words in your sentence? Show me how you would do it. Do you know how you would find another word to use in place of something you want to say? Show me how you would do it	Did the student use the indent key on the tools menu, or the tab key? Did the student right-mouse click or use the tools menu? Did the student use the Thesaurus on the tools menu?
408-A	IF STUDENT REACHES A HOLDING PAGE What happened when you clicked the Next button? What did you think would happen?
409-A	BACKGROUND QUESTIONS (refer to Part IV-A1a) In this section, you are going to answer some questions about yourself. When you are ready, you can begin.
410-A	RADIO BUTTONS	Is student able to choose an answer by clicking radio buttons?
411-A	RESET BUTTON / CLEAR BUTTON Can you tell me what this button does? Is there anything else you can do on this page?
412-A	NO ANSWER POPUP Can you tell me whether you can click NEXT without answering a question? Try it and see what happens. Can you tell me what this message means and what you should do?

Part IV-A1a – Background Questions

These are the four background questions that will be used in each design, as shown in Design A. The design and location of the functions will change in Design B and C to determine the most effective placement of the Reset button and whether the No Answer popup message is effective.

Part IV-A2 – Student Survey

What Do You Think?

Circle one number from 1 to 5 for each statement below that best describes what you think. You can write anything you want below each statement.

1. Did you like using the laptop to type your writing?

1	2	3	4	5
Liked a lot	Liked some	No opinion	Didn’t like some	Didn’t like at all

What I think: ______________________________________________________________________________

______________________________________________________________________________

2. How easy or hard was it to read the directions?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

3. How easy or hard was it to type your answer?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

4. How easy or hard was it to zoom to make the words larger?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

5. How easy or hard was it to listen to the computer read the words?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

6. How easy or hard was it to highlight some words?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

Part IV-B1 – Student Tasks on Interface Prototypes

Ref #	Facilitator Questions	Observer Notes
	PART IV-B: Student Tasks on the Interface Prototypes.
400-B	[Open the laptop and log the student in. Open the interface chosen by the facilitator as the 2nd prototype. Allow the student to freely explore the application. During the free exploration, ask the student questions about why he clicked on something, what he thought a button would do and whether it did what he thought (expectations), and how does he think it should work. The Observer will use the separate note sheet to record performance success or difficulty and other observations by answering the questions in the Observer Notes column.
401-B	[Interface opens ] Can you tell or show me what you can do here? Which color do you like best for the screen? Can you tell or show me what you would do to begin the writing test?	Was the student able to change the skin color? What skin color did the student leave it on? Was the student able to navigate to the Prompt screen using the Next button?
402-B	ZOOM Do you see a way to make any of the words larger or smaller on the screen? Show me how you would do it.	If the student used the zoom feature, what text size level did the student keep for the longest time? Which area(s) did the student focus on when using the text enlarging feature?
403-B	HIGHLIGHT Do you see a way to highlight some words on this screen? Show me how you would do it.	Did the student select text in the prompt either to highlight or use text-to-speech? Did the student try to click the highlighter first and then try to highlight text (i.e. the opposite of how it currently works)?
404-B	TIMER Do you know whether this is a timed test? How do you know? Can you tell me how much time is left in the test?	Did the student check the timer display and leave it on? Did the student turn off the timer display when it came on automatically?
405-B	WORD PROCESSING PANEL Show me how you would begin writing to respond to this prompt.	Was the student able to find/ open the Write panel?
406-B	Can you type some words for me? How about a sentence? [Wait as student types.] Can you show me how you would change or delete a word? Can you show me how you would make a word or some words stand out to be more noticeable?	Does the student look at the keyboard while typing? Does the student have to look for each letter, for a lot of letters, or only for a few letters? Did the student use, delete, back space, cut copy, paste (indicate which, if any)? Did the student use bold, italics or underscore (indicate which, if any)?
407-B	Do you know how to indent the start of a paragraph? Show me how you would do it. Do you know how you would check the spelling of the words in your sentence? Show me how you would do it. Do you know how you would find another word to use in place of something you want to say? Show me how you would do it ) Have you used any of these before? If so, how many times or how often do you use them when you write on a computer? Spell check Thesaurus Cut/copy/paste Bold/italicize/underscore Backspace Delete Indent	Did the student use the indent key on the tools menu, or the tab key? Did the student right-mouse click or use the tools menu? Did the student use the Thesaurus on the tools menu?
408-B	IF STUDENT REACHES A HOLDING PAGE What happened when you clicked the Next button? What did you think would happen?
409-B	BACKGROUND QUESTIONS (refer to Part IV-A1a, on p. 13 of this submission) In this section, you are going to answer some questions about yourself. Do not worry if they are the same questions you have already answered because we are we are not keeping track of the answers. When you are ready, you can begin.
410-B	RADIO BUTTONS	Is student able to choose an answer by clicking radio buttons?
411-B	RESET BUTTON / CLEAR BUTTON Can you tell me what this button does? Is there anything else you can do on this page?
412-B	NO ANSWER POPUP Can you tell me whether you can click NEXT without answering a question? Try it and see what happens. Can you tell me what this message means and what you should do?

Part IV-B2 – Student Survey

What Do You Think?

Circle one number from 1 to 5 for each statement below that best describes what you think. You can write anything you want below each statement.

1. Did you like using the laptop to type your writing?

1	2	3	4	5
Liked a lot	Liked some	No opinion	Didn’t like some	Didn’t like at all

What I think: ______________________________________________________________________________

______________________________________________________________________________

2. How easy or hard was it to read the directions?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

3. How easy or hard was it to type your answer?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

4. How easy or hard was it to zoom to make the words larger?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

5. How easy or hard was it to listen to the computer read the words?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

6. How easy or hard was it to highlight some words?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

7. If you used it, how easy or hard was it to make your words bold or italic?

1	2	3	4	5
Liked a lot	Liked some	No opinion	Didn’t like some	Didn’t like at all

What I think: ______________________________________________________________________________

______________________________________________________________________________

8. If you used it, how easy or hard was it to use the Spell Checker?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

9. How easy or hard was it to answer the background questions?

1	2	3	4	5
Very easy	Easy	No opinion	Hard	Very hard

What I think: ______________________________________________________________________________

______________________________________________________________________________

Part V – Student Design Comparison

Compare the Writing Programs

Tell us below how you feel about the writing computer programs you used today.

For each question, choose which program you liked best by putting a checkmark under its number.

Your Opinion	Program #1	Program #2
Which program had the best directions?
Which program had the best reading section?
Which program had the best writing section?
Which program was easiest to use?
Which did you like better?

What was something you really liked about either of the programs?

Program #1:

______________________________________________________________________________

Program #2:

______________________________________________________________________________

Was there something you really did not like about either of the programs?

Program #1:

______________________________________________________________________________

Program #2:

______________________________________________________________________________

Part VI –Typing Test

Facilitator: Now we’d like to have you type a paragraph by typing it in this typing program.

I’m going to open the program on the laptop and enter your number. You will type as best as you can for 5 minutes. Look at the words at the top of the screen and type them in the box underneath. Don’t stop and try to correct any mistakes you make, just keep typing as well as you can for 5 minutes.

When you are ready, press the button to begin.

Typing test program sample image:

Complete typing passage:

People who lived 200 years ago did not have the things we have today. Back then, they did not see cars on the streets or planes in the air.

They did not have cell phones or music players that fit in their pockets. They did not have TV to watch or computers to read or type on.

What would it be like to live in that time? With no car, you would walk to the store to buy food. You would walk or ride a bike to see a friend. To go on a long trip, you would ride a horse, sit in a buggy, or take a train. Most people did not travel very far at all.

With no phone or computer, it would take longer to send a note to a friend or to your family. You would write letters with a pen and some ink, then you would have to mail the letter. It could take days or weeks to get where you sent it!

With no TV, you might spend more time reading books or playing outside. You would go to a play or a concert to hear music.

A lot of people like to think the slower pace of life 200 years ago would be a nice way to live. But other people do not think they would like to live with the old, slower ways of travel, of visiting with friends or family, or of having fun.

Some people think instead they would like to live in the future. What kinds of changes do you think there will be 200 years from now?

Do you think we will still travel in cars and planes, or in something else? How will we listen to music or send notes to our friends? What kinds of games will we play? There can be a lot of changes in the next 200 years, just as there have been in the last 200 years.