National Assessment of Educational Progress (NAEP) Cognitive Items Cognitive Interview Studies 2011

NCES Cognitive, Pilot, and Field Test Studies System

NAEP Cog Items-Cog Labs Vol 2

National Assessment of Educational Progress (NAEP) Cognitive Items Cognitive Interview Studies 2011

OMB: 1850-0803

Document [docx]
Download: docx | pdf

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS



Volume II

Cognitive Interview Protocols



2011 Cognitive Interview Studies of NAEP Cognitive Items





OMB# 1850-0803 v.45

(Generic Clearance for Cognitive, Pilot, and Field Test Studies)



Shape1














March 2, 2011

Table of Contents


  1. Paperwork Burden Statement


The Paperwork Reduction Act and OMB confidentiality texts are indicated below. Appropriate sections of this information are included in the consent forms and letters contained in the appendices for Volume I. The entire statements will be included on the front of the test materials used in the interviews.


Grade 4 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 60 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].


Grade 8 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 1.5 hours including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.


Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].


  1. Cognitive Laboratory Script - Writing


Text written in italics is to be read aloud by the interviewer.


NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. Text highlighted in red or placed in brackets is generic text that will be tailored to suit the particular task being studied.



Introduction:

Hello, my name is ______ and I work for <Company Name>. It’s nice to meet you and thank you very much for helping us out today.


Create small talk to build rapport with the student by asking a question, such as:

  • What is your favorite subject in school?

    • If student responds with “language arts/reading/writing,” follow up with: Good, then I think you’ll enjoy what we are going to be doing today.

    • If student responds with another subject, respond accordingly (e.g., if the student responds with “science,” ask them what they are studying in their science class).


Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study about grade 4 writing tasks, which is part of the National Assessment of Educational Progress, or NAEP for short, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will be doing one writing task.


It’s okay if you do not know how to do the writing task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.


If at any time you decide you do not want to go on, that is your choice and you may stop.


This new test is on the computer. The instructions about how to do the test are also on the computer.


This interview is being recorded so we can review the tapes later. What you say may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573)..


You’ll be going through the instructions first. After you do this, you will do a writing task.


We will work together for about an hour.


Do you have any questions?


Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue with the writing tutorial. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.


OK, let’s begin.

  1. Writing Interface Tutorial

All students will go through the tutorial on the computer to enable them to do the tasks. The script for this part is below.


This is the first time this test has been done for your grade on the computer. There are directions to show you how you to use the computer and some of the tools you have for writing.


You are going to go through the directions. This will take just a few minutes.


Start the tutorial for the student.


At the conclusion of the tutorial:


Do you have any questions before we continue?





  1. Think Aloud Examples – Writing


4a. Think Aloud Modeling Script

You are going to be doing a writing task in a way that may be different from what you are used to. Instead of working quietly, I want you to read the task out loud and then think out loud by telling me what you are thinking as you work on the task. That includes any planning you may do for your writing, and anything you are thinking as you come up with ideas and write.


I will give you an example of the think-aloud process. Then I will give you a chance to practice it.


You will not be graded on anything you say while you are thinking aloud. There are no wrong thoughts, and everything you think and say is important to us.


Now I’m going to show you how to think out loud. When I am finished you will get a chance to practice thinking aloud.


Think Aloud Explanation:


Since we can’t tell what is going in your brain, we need you to “think aloud.” 


Let me give you an example about making a piece of toast this morning.  If I were thinking aloud while I got the toast ready, it would sound something like this.


What do I want...ok, I want toast. What kind? Maybe this kind with raisins…I love raisins. OK, first plug in the toaster oven, turn it on, close it. OK, done. Hmm (reading the bread package). How come they always have to advertise using this thing about low fat? Wait…serving size 1 piece of bread, Calories: 130… does not seem so bad…Vitamin A 0%, Vitamin C 0%, Iron 8%.  So those are good. (Look up at the corner of the ceiling) Wait! Forgot to put the bread in the toaster! OK, OK, put bread in toaster…ow, hot, hot! Do I want butter or…(Ding) Open the toaster, don’t burn myself…I think I want butter, yeah. I’m really hungry…OK, toast is ready, have butter…this is good.”


The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do.  Just say out loud the words that are in your head. 


 If I don’t hear you speaking, I’ll ask you to keep talking.  I’m telling you that now so you won’t think I am criticizing the way you are working.  I need to hear times when you are thinking in your head that you are confused and times when you are not.


4b. Student Practice


You may need to remind the student to talk aloud as he/she works through the question. If necessary, use the “Think-Aloud Hints” shown below to prompt the student, but be careful not to lead the student.


The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.



Now you will do a think-aloud practice. I’m going to ask you a question and you think aloud as you figure out your answer. Remember to say out loud whatever you are thinking. The question is:


How many windows are in the house or apartment where you live?”


Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.



4c. Additional Interviewer Model


[Use only if you feel the student needs to observe another think aloud before moving on to the actual questions.]


Shape2


[Remember: the italicized text should not be read word for word. You should be familiar enough with its contents to demonstrate a think aloud in a natural and conversational manner.] Hand the student a copy of the item.


  1. Read both the question and the answer choices aloud.

  2. Make sense of the question:

  • The question is asking me which of the following could be the length of the pencil I use in school.

  1. Analyze each option:

  • All the options have the number 6 in them, but the units are all different.

  • Option A says 6 feet, which doesn’t make sense because that would be too big.

  • Option B says 6 pounds, [refer back to question] but the question asks about the length of the pencil and pounds are about weight.

  • Option C says 6 ounces, which is also weight.

  • Option D says 6 inches, [Look at your pencil] which would make sense for the length of a pencil.

  • My answer is D. [Fill in oval for option D on question.]


When you answer the questions I’d like you to say out loud everything that you’re thinking.


Do you have any questions before we continue? [Answer any questions the student may ask.]


Okay, let’s move on to the actual writing task.


Because the information you provide is so important to us, I am going to be taking notes while you think aloud and do the writing task. Remember, you are not getting a grade on the task you do today.

4d. Think-Aloud Hints for the Interviewer


If the student is silent for 5 or more seconds, use the following as a guideline for encouraging the student to say their thoughts, or to help him/her elaborate their responses.


We’re interested in capturing all the participant’s mental processes while doing the task. Your goal is to have the participant speak aloud all his or her thoughts while doing the task. Grade 4 students may pose special challenges, as it may be hard for them to both work on a task and say their thoughts aloud at the same time. Do your best to encourage them, but be aware that you may learn more during the retrospective interview.


If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouths.” Don’t ask questions that lead the participant’s response or otherwise make them rush or change their approach to writing—such as planning before they write. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:


  • What are you thinking now?

  • Any other thoughts?

  • [If there is stimulus]: I see you are looking at the video/pictures/text. What are you thinking?

  • I see you have stopped writing. What are you thinking?

  • I see you are reading the brochure. What are you thinking?


NOTE: If the student simply cannot think aloud and write at the same time (judged by their ongoing difficulty doing so, and discomfort with doing do so), let them write, and then rely on the retrospective interview for information about the student’s reaction to the task.





  1. Generic Writing Protocol

Writing Cognitive Lab

NAEP Writing

2011 Grade 4


Question Identifier: Task Number

Note: The preliminary information should be recorded for a task.

Ref #

Question/Prompt

Response


Preliminary Information


1

Name of Interviewer



2

Date



3

Student ID


Record student’s unique ID number here.

4

Student’s Gender

Male Female

5

Special circumstances that may have affected the interview



6

Do you like studying language arts/reading/writing?

Yes No



Now, you’re going to do a writing task.


Instead of just reading the task and doing it, I’d like you to “think aloud” as you do the task. All that means is to just read the task out loud and then say what you’re thinking as you do the task. Please tell me whatever is going through your mind as you do the task.


As you saw in the directions on the computer, some tasks have illustrations, or photos, or videos. You may have a task like that.


You will be writing on the computer. You will have 25 minutes to do the task. You will not be graded on what you write.


There is no right or wrong way to think aloud. I want to know whatever you are thinking as you read the task and look at any videos or pictures. I want to know whatever you are thinking as you do the task.


I will also have a few questions after you have finished writing. Your thoughts about these tasks will help NAEP make the tasks better.


So, do you have any questions before we start?


Give the student paper and pencil and the planning and revising brochure.


Ok, first, read these task directions out loud.


[The directions that will be displayed on the screen are shown below.]

You are now ready to complete a writing task. You may use the paper and pencil you have been given to plan your writing, or you may plan on the computer. You may want to use suggestions from the Ideas for Planning and Reviewing Your Writing brochure you have been given.

 

Be sure to use the panel labeled "Write" to write your response. Some tools, like spell check and thesaurus, are available to you. You may use these tools if you wish, but you do not have to use them.

 

Click NEXT to continue.


After the student clicks NEXT, continue with the Task and associated questions.




Ref #

Question/Prompt

Response


PHASE 1: Student reads task aloud


7

Please read the task aloud.


OR, If stimulus that can be read is present:


Please read this task and all the other words that are presented aloud.

Record your observations when the student reads the question.






8

Without looking back at the task— I’d like you to tell me what the task wants you to do.


If there are visual or multimedia stimuli or other components:


Without looking back at the task— I’d like you to tell me about the pictures or video or presentation.


IF student does not indicate audience in his or previous comments, AND the task indicates an audience, ask: Describe who you are supposed to be writing for or to.




PHASE 2: Student thinks aloud while doing task


9

Now please go ahead and start work, telling me everything you are thinking.


I will be taking notes while you are working and talking.




If necessary, remind the student to say aloud what his/her thought processes are while working. Use the “Think-Aloud Hints” as needed.

Record student’s think-aloud comments here.







10

Does the student plan?



Does the student use the brochure?





Yes No



Yes No


11

How would you rate the student’s level of motivation while working on the task?

High Moderate Low

12

Based on your observations, did the student demonstrate understanding of the task?

Yes No


Student demonstrated some understanding.

13

[If task features drag and drop images; if not, move to Phase 3] Based on your observations, did the student demonstrate understanding of using drag and drop images in his or her writing?

Yes No


Student demonstrated some understanding.


PHASE 3: Interview with student


14

If a student in your class didn’t understand the task, how would you explain it?


If a stimulus is present, point to the stimulus and ask:


How would you explain this image, text, video or presentation to another student?




15

If the task features a presentation, text, or video with information the student can be expected to use in his or her response (versus passive stimulus):


What did you learn from this video/presentation/text?


16

Before you started writing, what did you think about?


If a stimulus is present, point to the stimulus and ask:


Did you think about this picture/video/text/presentation before you started writing? If yes: Describe what you thought.



17

Did you plan what you wrote? If yes: Tell me what you thought about when you planned.


If a stimulus is present, point to the stimulus and ask:


Did you think about this picture/video/text/presentation when you planned? If yes: Describe what you thought.



18

If task features drag and drop images, point to the images and ask:


What did you think you were supposed to do with these pictures?



Did you use them in your writing?


If yes: Tell me what you thought about when you used them in your writing.




19

Did you change your mind about what you wanted to say after you started to write? If yes: Tell me what you thought about when you did that.


If a stimulus is present, point to the stimulus and ask:


Did you think about this picture/video/text/presentation when you changed anything? If yes: Describe what you thought.



20

Is there anything that you think could make this task easier to understand?


If Yes, What is it?


Were there any words you did not understand in the task?


If Yes, What were they?


Yes No



Record student’s suggestions.



Yes No





21

Have you done tasks like this in school or on other tests?


If Yes, Tell me about them.


If not already answered and student had drag and drop images, ask:


Do you use pictures like this in writing you do in school or at home?


If Yes, Tell me about them.


Yes No






Yes No

22

Have you studied writing in school?



What kinds of feedback does your teacher give you about your writing? If student does not know word “feedback,” say “comments” from the teacher.

Yes No


Thank you! We are all done.






Behavioral Coding Hints for Interviewer


It is important to note when certain behaviors occur that could be evidence of problems with a task. When the following behaviors occur, note the occurrence and when it occurred:


  • Misreading or rereading a word or words in the task

  • Skipping a word or words in the task


Debriefing and Thank You


Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.


Is there anything else you would like to tell me about working on the task?



Thank participant for his/her time. Provide gift card.

  1. Cognitive Laboratory Script - TEL


Text written in italics is to be read aloud by the interviewer.


NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. Text highlighted in red or placed in brackets is generic text that will be tailored to suit the particular task being studied.



Introduction:

Hello, my name is ______ and I work for <Company Name>. It’s nice to meet you and thank you very much for helping us out today.


Create small talk to build rapport with the student by asking a question, such as:

  • What is your favorite subject in school?

    • If student responds with “science” or “social studies” follow up with: Good, then I think you’ll enjoy what we are going to be doing today.

    • If student responds with another subject, respond accordingly (e.g., if the student responds with “English” ask them what they are currently studying in their English class).


Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study for a new test of technology and engineering literacy, which is part of the National Assessment of Educational Progress, or NAEP for short, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 throughout the United States. You will be answering a small group of test questions for 8th graders.


It’s okay if you do not know how to answer the questions. I will not be grading your work today, and no one will know that it was you who did the questions. I’m not evaluating you. The goal is to see if the test questions are good test questions and I need your help to do this. Please try to do your best.


If at any time you decide you do not want to go on, that is your choice and you may stop.



This interview is being recorded so researchers can review the tapes later. Your information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).


We will work together for about an < hour/hour and a half>.


Do you have any questions?


Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue with the think-aloud training. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.


Okay, let’s begin.

  1. Think Aloud Examples – TEL


7a. Think Aloud Modeling Script


You are going to be doing questions and tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to read each part of the task out loud and then think out loud by telling me what you are thinking as you read and work on the task.


In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it.


You will not be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.


I’m now going to show you how to think out loud. When I am finished you will get a chance to practice answering a question thinking aloud.



Think Aloud Explanation:


Since we can’t tell what is going in your brain, we need you to “think aloud.” 


Let me give you an example about making a piece of toast this morning.  If I were thinking aloud while I got the toast ready, it would sound something like this.


Bread drawer.  Muffin or white bread?  White bread. Where is the knife?  Oh, the sink.  Cut.  Plug in the toaster oven, turn it on, close it. OK, Medium.  Hmm (reading the white bread package). Serving size 1 piece of bread, Calories: 130; Calories from fat 10 – I guess that’s not so bad.  Vitamin A 0%, Vitamin C 0%, Calcium 8%, Iron 8%  So aren’t there B vitamins in bread?  Hmm. (Look up at the corner of the ceiling) Wait! I forgot to put the bread in the toaster! OK, OK, put bread in toaster. Do I want butter or…(Ding) Open the toaster, don’t burn yourself! Use the knife to pull out the bread. Table, sit down, eat.”


The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do.  Just say aloud the words that are in your head. 


 If I don’t hear you speaking, I’ll ask you to keep talking.  I’m telling you that now so you won’t think I am criticizing the way you are working.  I need to hear times when you are confused (and what is confusing you) and times when you understand what you are doing.



7b. Student Practice

You may need to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, use the “Think-Aloud Hints” shown below to prompt the student but be careful not to lead the student.


The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.



Now you will do a think-aloud practice. I’m going to ask you a simple question and you think aloud as you figure out your answer. Remember to say out loud whatever you are thinking. The question is:


How many windows are in the house or apartment where you live?”


Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.



7c. Additional Interviewer Model


[Use only if you feel the student needs to observe another think aloud before moving on to the actual questions.]


Shape3


[Remember: the italicized text should not be read word for word. You should be familiar enough with its contents to demonstrate a think aloud in a natural and conversational manner.] Hand the student a copy of the item.


1. Read both the question and the answer choices aloud.

2. Make sense of the question:

  • The question is asking me which of the following could be the length of the pencil I use in school.

3. Analyze each option:

  • All the options have the number 6 in them, but the units are all different.

  • Option A says 6 feet, which doesn’t make sense because that would be too big.

  • Option B says 6 pounds, [refer back to question] but the question asks about the length of the pencil and pounds are a unit of weight.

  • Option C says 6 ounces, which is also a unit of weight.

  • Option D says 6 inches, [Look at your pencil] which would make sense for the length of a pencil.

  • My answer is D. [Fill in oval for option D on question.]


When you answer the questions I’d like you to say aloud everything that you’re thinking.


Do you have any questions before we continue? [Answer any questions the student may ask.]


Okay, let’s move on to the actual test questions.


Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions. Remember, you are not receiving a grade on the questions and tasks you do today.

7d. Think-Aloud Hints for the Interviewer

If the student is silent for 5 or more seconds, use the following as a guideline for encouraging the student to say their thoughts, or to help him/her elaborate their responses.


We’re interested in capturing all the participant’s mental processes while answering questions or doing tasks. Your goal is to have the participant speak aloud all his or her thoughts while doing the questions and tasks. If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouths.” Don’t ask questions that lead the participant’s response or otherwise make them rush or change their approach. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:


  • What are you thinking now?

  • Any other thoughts?

  • I see you are looking at the picture/figure/chart/text, etc... What are you thinking?


Also, if it seems a student is hung up on something, it’s important to note their frustration and when it occurred.




  1. Generic TEL Protocol

Cognitive Labs

NAEP TEL

2011 Grade 8


Question Identifier: Question/Task Number

Note: The preliminary information should be recorded for each question/task in this set. Ask student questions 1-8 only once. Scenario-based assessment tasks consist of a series of embedded assessment items; questions 9-22 are to be repeated for each embedded item or other subsection of the task.



Ref #

Question/Prompt

Response


Preliminary Information


1

Name of interviewer



2

Date



3

Student ID


Record student’s unique ID number here.

4

Student’s Gender

Male Female

5

Name of School



6

School District



7

Special circumstances that may have affected the interview



8

Do you like studying subject?

Yes No


Now, you’re going to be given [1, 2, 3] task(s) to work on.


Instead of just reading the task and doing it, I’d like you to “think aloud” as you do the task.


[modify as appropriate for the particular computer prototype/mockup:] When we are finished making the test, students will take it on computers. However, we aren’t finished making the test yet, so some parts of the tasks don’t work yet. When we come to one of those parts, you will have to write down your answers on a sheet of paper, or we may ask you to write down your answers using a separate program (e.g. a spreadsheet, word processor, or other software meant to provide functionality that will be built into the final versions of items).


You will have [10, 20, 30, 40] minutes to do this task. You will not be graded on what you do during the task.


There is no right or wrong way to think aloud. I want to know whatever you are thinking as you read each part of the task.


I will also have a few questions after you have finished working on the task. Your thoughts will help us make the tasks better.


So, do you have any questions before we start?



Ref #

Question/Prompt

Response


PHASE 1: Student reads each part of the task aloud


9

[for text-based questions]

Please read this question [and the answer choices] aloud.


[for multimedia-based questions]


Please look at this part of the task and read any words that you see aloud.

Record your observations when the student reads the question.







PHASE 2: Student thinks-aloud while doing each part of the task


10

Now please do this part of the task, telling me everything you are thinking.

I will be taking notes while you are working on the task.


If necessary, remind the student to say aloud what his/her thought processes are while answering the question. Use the “Think-Aloud Hints” as needed.

Record student’s think-aloud comments here.







Additional comments are recorded on interviewer’s copy of the question.

11

[for an embedded selected response item]

Record the option(s) selected by the student.




[for an embedded constructed response item]

Record the student’s response.



Student did not select an option.




Student did not respond.

12

Did the student appear to be certain of his/her answer?

Yes No


Student did not respond.

13

How would you rate the student’s level of motivation while working on this question?

High Moderate Low

14

Based on your observations, did the student demonstrate understanding of the TEL concepts in this question?

Yes No


Student demonstrated some understanding.


PHASE 3: Interview with student for each part of the task


15

[for selected response item]


Are there any other answers you thought about choosing?


If so, why?

Record other response options.


None





16

[If the question contains a stimulus or interactive component]

Follow up with this question only if the student has not already addressed it during the think-aloud. Point to the stimulus or interactive element and ask:



How do you think an action you took or this element on the screen helped you answer the question or do this part of the task?


17

If student indicates uncertainty about what a given task element means, ask:


Did you guess what this element means or is for? If Yes: What did you think it means or is for?


And/or


Did not knowing the meaning or purpose of this element keep you from answering the question?


18

Are there words or symbols in the question that you found hard to understand?


If Yes, Which ones?

Yes No



List unfamiliar words or symbols:





[If yes, for each unfamiliar word/symbol, ask:]

What does [term/symbol] mean in your own words?


19

Did you have any other problems understanding what you were supposed to do on this part of the task?




20

How would you explain what this question or part of the task means to a student in your class who doesn’t understand the question or task?





21

Is there anything that you think could make this question or part of the task clearer?


If Yes, what is it?


Yes No


Record student’s suggestions.




22

Have you answered questions or done activities like this one in any of your classes in school or outside of school? if so: Where have you done activities like this one?


And/or


Have you studied this topic in school?


If Yes, in what class?


Yes No

Record any courses student mentions.






Yes No


Thank you. Let’s move on to the next question.


[For last question in the form]

Thank you for helping us improve our test questions for other students.







Behavioral Coding Hints for Interviewer


It is important to note when certain behaviors occur that could be evidence of problems with a task. When the following behaviors occur, note the occurrence and when it occurred:


  • Misreading or rereading a word or words in the task

  • Skipping a word or words in the task


Debriefing and Thank You


Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.


Is there anything else you would like to tell me about working on the task?



Thank participant for his/her time. Provide gift card.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleNATIONAL ASSESSMENT OF
Authorjoconnell
File Modified0000-00-00
File Created2021-02-01

© 2024 OMB.report | Privacy Policy