NAEP Grade 4 Writing Computer-Based Assessment Usability Study, &The Assessment and Teaching of 21st Century Skills (ATC21S) Project Phase 2.2 Cognitive Interviews

System Clearance for Cognitive, Pilot and Field Test Studies

NAEP Grade 4 WCBA Usability-Vol 2

NAEP Grade 4 Writing Computer-Based Assessment Usability Study, &The Assessment and Teaching of 21st Century Skills (ATC21S) Project Phase 2.2 Cognitive Interviews

OMB: 1850-0803

Document [docx]
Download: docx | pdf

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS



Volume II


Tryout Survey Instruments

for

Small Scale Usability Study for Grade 4 Writing Computer-Based Assessment





OMB# 1850-0803

(Generic Clearance for Cognitive, Pilot and Field Test Studies)



Student Grade 4 Usability Evaluation of Writing Computer-Based Assessment



Shape1






The amount of time estimated to complete these forms is 75 minutes.




January 24, 2011

(revised February 15, 2011)

Contents





Part I – Welcome Script and Background Information


Grade 4 WCBA Prototype Usability Study Script


Text written in italics is suggested content with which the Facilitator should be thoroughly familiar in advance. The Facilitator should project a warm and reassuring manner toward the students to develop a friendly rapport and as such should use conversational language throughout.


Introduction:

Hello, my name is ______ and I work for Fulcrum IT, which is conducting a study for the National Center of Education Statistics (NCES) of the US Department of Education. This is _______ from _______ (observer affiliation.) It’s nice to meet you and thank you for helping us with our computers today.

Today you are going to be looking at different computer programs. This is part of a study to see which type of program works best for students in your grade if they were to take a writing test on laptop computers. One of the programs you will see is being used for writing tests by the National Assessment of Education Progress, also called NAEP for short, with 8th- graders and 12th-graders. We want to see how we might use it or change it for 4th-graders like you.

But you are not here to take a test today. You are helping us to try out the program. We’ll be looking to see how well you can find your way and operate the program, as well as what you think about how it looks or anything you like or don’t like. We won’t be telling anyone how you did or what you thought, but we will use your opinions and results without your name, along with the ideas and answers of other students, to make changes and improve the system for 4th-graders all across the country.

Do you have any questions so far?

Ok, I’m going to be the one talking to you and observing you, and ______ is also going to be observing closely and writing notes. We will start with me asking you a few questions, and then you will look at some pictures on paper before going on the computer. Soon you will be able to type on the laptop computer and then answer some other questions for us. Again, we are not testing you, we will be watching you test the program!

If at any time you decide you do not want to continue, that is your choice and you may stop and go back to class.

Okay, let’s begin.










Ref #

Facilitator Questions

Observer Notes


Part I — Background Information


101

Name of Fulcrum Facilitator


102

Name/Affiliation of Observer


103

Date


104

Student ID’s number


105

Student’s Gender

Male Female

106

Name of School


107

School District


108

School Location

Metro Suburb Rural

109

Special circumstances that may have affected the interview


Part II – Student Oral Survey on Keyboarding and Computer Use

Ref #

Facilitator Questions

Observer Notes


Part II — Student Oral Survey


201

Do you like computers?

Yes No

202

Do you have a computer at home that you can use?

Yes No

203

How often do you use it?

Once a month Every other week Every week A couple of times a week

Every day Other _____________________

204

Have you ever used a laptop computer?

Yes No

205

Do you use a computer at school?

Yes No

206

If so, how often?

Once a month Every other week Every week A couple of times a week

Every day Other _____________________

207

What do you like to do on computers?

Play games Watch videos Go on the Internet Do homework

Other ______________________________

208

Have you had any typing or keyboarding lessons?

Yes No

209

Have you written a story, essay, or letter on a computer?

Yes No

210

If so, how many times (or how often)?

1 time 2 or more times

Once a month Every other week Every week A couple of times a week

Every day Other _____________________

211

If you used a computer for writing, have you used any of these tools? If so, how often do you use them when you write on a computer?




Spell Check A lot of times Sometimes Never

Cut/copy/or paste A lot of times Sometimes Never

Backspace or delete A lot of times Sometimes Never

Bold/italics/underline A lot of times Sometimes Never

Indent A lot of times Sometimes Never

Undo or redo A lot of times Sometimes Never

Thesaurus A lot of times Sometimes Never


212

Have you written an email or chat message on a computer?

Yes No


If so, how many times (or how often)?

1 time 2 or more times

Once a month Every other week Every week A couple of times a week

Every day Other _____________________

213

Have you ever taken a test or practice test on a computer?

Yes No

214

If so, how many times (or how often)?

1 time 2 or more times

Once a month Every other week Every week A couple of times a week

Every day Other _____________________

Part III – Student Paper Design Preference

Ref #

Facilitator Question

Observer Notes









301















.

Now I’m going to show you some paper pictures of what some computer screens look like. I’m going to ask you some questions about what you are seeing and what you think things are on the paper. Ready?

[Show Design A, B, or C in random order for each student; note which design.] (Refer to Part III-A on pp. 7-8 of this submission.)

This is a picture of a computer screen. Can you tell me what you could do on this screen?

[Let student tell as much as possible before prompting with the following suggestions.]

[Point to a particular button or feature and say,] What could you do with this?

[Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.]

Is there anything more you can tell me about that?

Why do you think that?

What about the screen made you think that way?

Have you seen something else like this on the computer? If so, what and where?

During the Paper Design Preference phase, observers should note or probe for the following design, tools, navigation, and word processing features for each sample design :

  • Volume control

  • Text-to-speech control

  • Highlighter

  • Zoom

  • Timer

  • Open/close buttons

  • Next button

  • Cut

  • Copy

  • Paste

  • Undo

  • Redo

  • Bold

  • Italicize

  • Underscore

  • Indent

  • Spell check

  • Thesaurus

  • Comments on navigational features

  • Comments on which design they felt was easier or clearer

  • Comments on whether and how did the different layouts make a difference in their choice

  • All other comments











[When done with first design, put it aside and do the same for the second design.]





302

[Show second design; note which design.]

This is a picture of a computer screen. Can you tell me what you could do on this screen?

[Let student tell as much as possible before prompting with the following suggested prompts.]

[Point to a particular button or feature and say,] What could you do with this?

[Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.]

Is there anything more you can tell me about that?

Why do you think that?

What about the screen made you think that way?

Have you seen something else like this on the computer? If so, what and where?


Note same types of observations as listed in the observer notes sections of 301.


[When done with second design, put it aside and do the same for the third design.]



303

[Show third design; note which design.]

This is a picture of a computer screen. Can you tell me what you could do on this screen?

[Let student tell as much as possible before prompting with the following suggested prompts.]

[Point to a particular button or feature and say,] What could you do with this?

[Repeat above as needed for screen features. Also, add from the following questions to develop the student’s thinking further.]

Is there anything more you can tell me about that?

Why do you think that?

What about the screen made you think that way?

Have you seen something else like this on the computer? If so, what and where?



Note same types of observations as listed in the observer notes sections of 301.


Now I’m going to bring all three pictures back out together so you can see them at the same time. Of the three screens, which one looks the most like a computer you have used before to write on the computer?

Which one do you like the best?

I am going to have you look at two of these screens on the laptop computer. Which one do you want to try first on the real laptop?




Part III-A – Paper Designs

Shape2


This paper version of Design A will be used to elicit student preference of text editor design.


Shape3

This paper version of Design B will be used to elicit student preference of text editor design.


Shape4

This paper version of Design C will be used to elicit student preference of text editor design.




Part IV-A1 Student Tasks on Interface Prototypes

Ref #

Facilitator Questions

Observer Notes


PART IV-A: Student Tasks on the Interface Prototypes.


400-A

[Open the laptop and log the student in. Open the interface the student chose as 1st choice.

Allow the student to freely explore the application. During the free exploration, ask the student questions about why he clicked on something, what he thought a button would do and whether it did what he thought (expectations), and how does he think it should work.

The Observer will use the separate note sheet to record performance success or difficulty and other observations by answering the questions in the Observer Notes column.]








401-A

[Interface opens ]


Can you tell or show me what you can do here?


Which color do you like best for the screen?


Can you tell or show me what you would do to begin the writing test?




Was the student able to change the skin color?

What skin color did the student leave it on?

Was the student able to navigate to the Prompt screen using the Next button?


402-A

ZOOM

Do you see a way to make any of the words larger or smaller on the screen?

Show me how you would do it.

If the student used the zoom feature, what text size level did the student keep for the longest time? Which area(s) did the student focus on when using the text enlarging feature?



403-A

HIGHLIGHT

Do you see a way to highlight some words on this screen? Show me how you would do it.


Did the student select text in the prompt either to highlight or use text-to-speech?

Did the student try to click the highlighter first and then try to highlight text (i.e. the opposite of how it currently works)?


404-A

TIMER

Do you know whether this is a timed test? How do you know?

Can you tell me how much time is left in the test?


Did the student check the timer display and leave it on? Did the student turn off the timer display when it came on automatically?



405-A

WORD PROCESSING PANEL

Show me how you would begin writing to respond to this prompt.


Was the student able to find/ open the Write panel?

406-A

Can you type some words for me?


How about a sentence? [Wait as student types.]


Can you show me how you would change or delete a word?



Can you show me how you would make a word or some words stand out to be more noticeable?



Does the student look at the keyboard while typing? Does the student have to look for each letter, for a lot of letters, or only for a few letters?



Did the student use, delete, back space, cut copy, paste (indicate which, if any)?



Did the student use bold, italics or underscore (indicate which, if any)?

407-A

Do you know how to indent the start of a paragraph?

Show me how you would do it.


Do you know how you would check the spelling of the words in your sentence? Show me how you would do it.


Do you know how you would find another word to use in place of something you want to say? Show me how you would do it



Did the student use the indent key on the tools menu, or the tab key?

Did the student right-mouse click or use the tools menu?



Did the student use the Thesaurus on the tools menu?


408-A

IF STUDENT REACHES A HOLDING PAGE

What happened when you clicked the Next button?

What did you think would happen?



409-A

BACKGROUND QUESTIONS (refer to Part IV-A1a)

In this section, you are going to answer some questions about yourself. When you are ready, you can begin.


410-A

RADIO BUTTONS

Is student able to choose an answer by clicking radio buttons?

411-A

RESET BUTTON / CLEAR BUTTON

Can you tell me what this button does?

Is there anything else you can do on this page?


412-A

NO ANSWER POPUP

Can you tell me whether you can click NEXT without answering a question? Try it and see what happens.

Can you tell me what this message means and what you should do?






Part IV-A1a Background Questions

These are the four background questions that will be used in each design, as shown in Design A. The design and location of the functions will change in Design B and C to determine the most effective placement of the Reset button and whether the No Answer popup message is effective.

Shape5

Shape6

Part IV-A2 Student Survey


What Do You Think?

Circle one number from 1 to 5 for each statement below that best describes what you think. You can write anything you want below each statement.


1. Did you like using the laptop to type your writing?


1


2


3


4


5


Shape7


Shape8


Shape9


Shape10


Shape11

Liked a lot

Liked some

No opinion

Didn’t like some

Didn’t like at all


What I think: ______________________________________________________________________________


______________________________________________________________________________


2. How easy or hard was it to read the directions?


1


2


3


4


5


Shape12


Shape13


Shape14


Shape15


Shape16

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


3. How easy or hard was it to type your answer?


1


2


3


4


5


Shape17


Shape18


Shape19


Shape20


Shape21

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________

4. How easy or hard was it to zoom to make the words larger?


1


2


3


4


5


Shape22


Shape23


Shape24


Shape25


Shape26

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


5. How easy or hard was it to listen to the computer read the words?


1


2


3


4


5


Shape27


Shape28


Shape29


Shape30


Shape31

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


6. How easy or hard was it to highlight some words?


1


2


3


4


5


Shape32


Shape33


Shape34


Shape35


Shape36

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


Part IV-B1 Student Tasks on Interface Prototypes


Ref #

Facilitator Questions

Observer Notes


PART IV-B: Student Tasks on the Interface Prototypes.


400-B

[Open the laptop and log the student in. Open the interface chosen by the facilitator as the 2nd prototype.

Allow the student to freely explore the application. During the free exploration, ask the student questions about why he clicked on something, what he thought a button would do and whether it did what he thought (expectations), and how does he think it should work.

The Observer will use the separate note sheet to record performance success or difficulty and other observations by answering the questions in the Observer Notes column.








401-B

[Interface opens ]


Can you tell or show me what you can do here?



Which color do you like best for the screen?


Can you tell or show me what you would do to begin the writing test?





Was the student able to change the skin color?

What skin color did the student leave it on?

Was the student able to navigate to the Prompt screen using the Next button?


402-B

ZOOM

Do you see a way to make any of the words larger or smaller on the screen?

Show me how you would do it.

If the student used the zoom feature, what text size level did the student keep for the longest time? Which area(s) did the student focus on when using the text enlarging feature?



403-B

HIGHLIGHT

Do you see a way to highlight some words on this screen? Show me how you would do it.


Did the student select text in the prompt either to highlight or use text-to-speech?

Did the student try to click the highlighter first and then try to highlight text (i.e. the opposite of how it currently works)?


404-B

TIMER

Do you know whether this is a timed test? How do you know?

Can you tell me how much time is left in the test?


Did the student check the timer display and leave it on? Did the student turn off the timer display when it came on automatically?



405-B

WORD PROCESSING PANEL

Show me how you would begin writing to respond to this prompt.


Was the student able to find/ open the Write panel?

406-B

Can you type some words for me?


How about a sentence? [Wait as student types.]


Can you show me how you would change or delete a word?


Can you show me how you would make a word or some words stand out to be more noticeable?

Does the student look at the keyboard while typing? Does the student have to look for each letter, for a lot of letters, or only for a few letters?



Did the student use, delete, back space, cut copy, paste (indicate which, if any)?



Did the student use bold, italics or underscore (indicate which, if any)?

407-B

Do you know how to indent the start of a paragraph?

Show me how you would do it.


Do you know how you would check the spelling of the words in your sentence? Show me how you would do it.


Do you know how you would find another word to use in place of something you want to say? Show me how you would do it )


Have you used any of these before? If so, how many times or how often do you use them when you write on a computer?


Spell check

Thesaurus

Cut/copy/paste

Bold/italicize/underscore

Backspace

Delete

Indent

Did the student use the indent key on the tools menu, or the tab key?



Did the student right-mouse click or use the tools menu?



Did the student use the Thesaurus on the tools menu?






408-B

IF STUDENT REACHES A HOLDING PAGE

What happened when you clicked the Next button?

What did you think would happen?



409-B

BACKGROUND QUESTIONS (refer to Part IV-A1a, on p. 13 of this submission)

In this section, you are going to answer some questions about yourself. Do not worry if they are the same questions you have already answered because we are we are not keeping track of the answers. When you are ready, you can begin.


410-B

RADIO BUTTONS

Is student able to choose an answer by clicking radio buttons?

411-B

RESET BUTTON / CLEAR BUTTON

Can you tell me what this button does?

Is there anything else you can do on this page?


412-B

NO ANSWER POPUP

Can you tell me whether you can click NEXT without answering a question? Try it and see what happens.

Can you tell me what this message means and what you should do?



Part IV-B2 Student Survey


What Do You Think?

Circle one number from 1 to 5 for each statement below that best describes what you think. You can write anything you want below each statement.


1. Did you like using the laptop to type your writing?


1


2


3


4


5


Shape37


Shape38


Shape39


Shape40


Shape41

Liked a lot

Liked some

No opinion

Didn’t like some

Didn’t like at all


What I think: ______________________________________________________________________________


______________________________________________________________________________


2. How easy or hard was it to read the directions?


1


2


3


4


5


Shape42


Shape43


Shape44


Shape45


Shape46

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


3. How easy or hard was it to type your answer?


1


2


3


4


5


Shape47


Shape48


Shape49


Shape50


Shape51

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________

4. How easy or hard was it to zoom to make the words larger?


1


2


3


4


5


Shape52


Shape53


Shape54


Shape55


Shape56

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


5. How easy or hard was it to listen to the computer read the words?


1


2


3


4


5


Shape57


Shape58


Shape59


Shape60


Shape61

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


6. How easy or hard was it to highlight some words?


1


2


3


4


5


Shape62


Shape63


Shape64


Shape65


Shape66

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


7. If you used it, how easy or hard was it to make your words bold or italic?


1


2


3


4


5


Shape67


Shape68


Shape69


Shape70


Shape71

Liked a lot

Liked some

No opinion

Didn’t like some

Didn’t like at all


What I think: ______________________________________________________________________________


______________________________________________________________________________


8. If you used it, how easy or hard was it to use the Spell Checker?


1


2


3


4


5


Shape72


Shape73


Shape74


Shape75


Shape76

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________


9. How easy or hard was it to answer the background questions?


1


2


3


4


5


Shape77


Shape78


Shape79


Shape80


Shape81

Very easy

Easy

No opinion

Hard

Very hard


What I think: ______________________________________________________________________________


______________________________________________________________________________



Part V Student Design Comparison


Compare the Writing Programs


Tell us below how you feel about the writing computer programs you used today.


For each question, choose which program you liked best by putting a checkmark under its number.


Your Opinion

Program #1

Program #2

Which program had the best directions?



Which program had the best reading section?



Which program had the best writing section?



Which program was easiest to use?



Which did you like better?





What was something you really liked about either of the programs?


Program #1:

______________________________________________________________________________


Program #2:

______________________________________________________________________________



Was there something you really did not like about either of the programs?


Program #1:

______________________________________________________________________________


Program #2:

______________________________________________________________________________




Part VI Typing Test


Facilitator: Now we’d like to have you type a paragraph by typing it in this typing program.


I’m going to open the program on the laptop and enter your number. You will type as best as you can for 5 minutes. Look at the words at the top of the screen and type them in the box underneath. Don’t stop and try to correct any mistakes you make, just keep typing as well as you can for 5 minutes.


When you are ready, press the button to begin.



Typing test program sample image:


Shape82



Complete typing passage:


People who lived 200 years ago did not have the things we have today. Back then, they did not see cars on the streets or planes in the air.


They did not have cell phones or music players that fit in their pockets. They did not have TV to watch or computers to read or type on.


What would it be like to live in that time? With no car, you would walk to the store to buy food. You would walk or ride a bike to see a friend. To go on a long trip, you would ride a horse, sit in a buggy, or take a train. Most people did not travel very far at all.


With no phone or computer, it would take longer to send a note to a friend or to your family. You would write letters with a pen and some ink, then you would have to mail the letter. It could take days or weeks to get where you sent it!


With no TV, you might spend more time reading books or playing outside. You would go to a play or a concert to hear music.


A lot of people like to think the slower pace of life 200 years ago would be a nice way to live. But other people do not think they would like to live with the old, slower ways of travel, of visiting with friends or family, or of having fun.


Some people think instead they would like to live in the future. What kinds of changes do you think there will be 200 years from now?


Do you think we will still travel in cars and planes, or in something else? How will we listen to music or send notes to our friends? What kinds of games will we play? There can be a lot of changes in the next 200 years, just as there have been in the last 200 years.


































File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleSurvey Instruments
AuthorLori Burgess
File Modified0000-00-00
File Created2021-02-02

© 2024 OMB.report | Privacy Policy