Review of 2007 NAEP Mathematics Items

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to average 3 to 4 hours including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 1990 K Street N.W., Washington, D.C. 20006-5651.

A project of the Institute of Education Sciences.

The information you provide will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and other applicable Federal laws, your responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of up to $250,000, or both if he or she willfully discloses ANY identifiable information about you.

OMB No. 1850-0803 Approval Expires 09/30/2010

Item Review Data Collection Script

Data collection is done in small sessions with five or six teachers at the same time. There will be two trained interviewers and an observer (from the American Institutes for Research) for each session. The following is the process of the data collection:

1. Prior to Beginning the Interview:

Arrive early and follow the school’s procedure for signing yourself in at the administrative office.
If possible, ask to be taken to the location that the data collection will be done before the teachers arrive.
Set up the work space so that
- All 5-6 teachers will be able to see you across the table or desk, and
- All of your paperwork and materials (pencils, ancillary materials, test forms, interviewer booklet, etc.) are ready when the teachers arrive.

2. The interviewers introduce themselves and the purpose of the study. Consent and confidentiality forms are signed and collected.

Text written in italics is to be read aloud by the interviewers.

NOTE: The interviewers should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate.

Introduction:

Hello, my name is ______ and I work for ____. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study involving sample mathematics test questions for students in grades 4 and 8 in Puerto Rico. This study is being conducted at schools all across the island.

I am here to find out how 4th/8th grade teachers evaluate the quality and appropriateness of mathematics test questions on a number of dimensions, and to hear your suggestions for how the test questions can be improved.

The goal of this study is to identify potential problems with test questions and find out ways to improve them. To achieve this goal, you are going to be reviewing 50 to 60 test questions. I will explain more about the review process later. Also, after your review is complete, I will ask you to identify the most problematic mathematics test questions in your review, and then give you the opportunity to make additional comments or suggestions on how to improve them. The entire process should take 3 to 4 hours total.

I want to assure you that all information obtained today will be kept confidential and will only be used for the purposes of this study. We will not use your name or school name and will not attribute any quotes specifically to you. After you complete the survey you will be compensated with a gift card.

In this study, you will have access to secure and confidential test materials. Therefore we will ask you to sign a confidentiality form before we begin. In addition, I’d like to ask you to sign a consent form indicating you are willing to participate in this study. [Hand out consent forms and confidentiality forms]. They outline some of the issues I’ve just mentioned with regard to anonymity and confidentiality. Please take a minute to read them and let me know if you have any questions. [Collect the signed consent and confidentiality forms and answer any questions.]

If at any time during the interview you decide you do not want to continue, that is your choice and you may stop.

Before we continue, do you have any questions? [Answer any questions the teacher may ask.]

Okay, let’s begin.

3. Next, the interviewers give each teacher a booklet that includes mathematics items they will rate later.

Before we do the survey, I would like you to take about 30 minutes to become familiar with the mathematics questions you are going to work with today. Please take your time and read over these questions and imagine a typical 4th/8th grade student reading and answering them. Feel free to solve the questions if this will help you, though you are not required to mark anything in the booklets. I will collect these booklets when you are done but we will not use any information you have written on these booklets. If you have any important notes you’d like us to read, please use the second set of booklets I will distribute in 35 minutes to record those.

[Hand out the booklets with mathematics items.]

After about 30 minutes, Thank you, I will now collect your booklets.

[Collect all booklets.]

4. Next, the interviewers distribute a handout that includes the survey (Item Review Tool) the teachers will use in rating test questions. The handout includes the 11 statements contained in the Item Review Tool along with short descriptions for each statement (attached, p.10-12). The purpose is to make sure that all the teachers have the same understanding of each statement in the Item Review Tool.

I will now distribute a handout that includes 11 statements in the survey you will later be using to rate mathematics items. A brief description for each statement is provided in this survey. These descriptions will not be included in the booklets that include the test questions that you will be rating. The booklets will contain only the underlined statements. Keep this handout in case you need to refer to it while rating the test questions.

[Hand out the Item Review Tool with descriptions]

Let’s read these statements one at a time.

[Read first statement along with its description.]

Do you have a question about what we mean by mathematical terminology?

[Respond to teachers’ questions. However, if the teachers ask what constitutes an unfamiliar terminology remind them that it is up to them to decide what is unfamiliar.]

[Read second statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions. If the teachers ask what constitutes an unfamiliar visual remind them that it is up to them to decide what is unfamiliar.]

[Read third statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions.]

[Read fourth statement along with its description.]

We want you to make your own judgment as to what skills and knowledge are measured in each mathematics question you will be rating. Is this clear to everyone?

[Respond to teachers’ questions.]

[Read fifth statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions.]

[Read sixth statement along with its description.]

Is this statement clear to everyone? Do you have a question about what we mean by context?

[Respond to teachers’ questions. If the teachers ask what constitutes an unfamiliar context remind them that it is up to them to decide what is unfamiliar.]

[Read seventh statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions.]

[Read eighth statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions.]

[Read ninth statement along with its description.]

Is this statement clear to everyone?

[Respond to teachers’ questions.]

[Read tenth statement along with its description.]

Is this statement clear to everyone? This statement applies to open ended questions only.

[Respond to teachers’ questions.]

[Read eleventh statement along with its description.]

As you see, this question is worded differently compared to previous ones. The rating scale is different and the statement is a positive statement. Is this statement clear to everyone?

5. Next, the interviewers distribute the booklets that contain the mathematics items to be reviewed and the Item Review Tool. Each booklet will contain 50 to 60 mathematics items and the Item Review Tool will be printed right after the mathematics items each time (see pages 13 and 14 for an example mathematics item and the Item Review Tool). Each booklet has a unique Booklet ID. The booklet also contains a Teacher Background Questionnaire at the end that asks basic information such as number of years of teaching experience and highest degree attained (attached, p.15).

I am now going to hand out a booklet containing the mathematics test questions you will be reviewing.

Take a look at each question on the left and use the rating scale on the right to rate the question. It is very important that you rate each mathematics test question on all statements. If you are not sure about what a given statement means, refer back to the handout where an explanation is provided for each statement.

We would like to hear your independent judgment in this process. Therefore please do not talk to one another during this process; do not ask questions to others or discuss your ratings with one another. If you have questions please approach me and ask your question to me in private.

You will see that at the end of your booklet there is a section titled “Grade 4/8 Mathematics Teacher Background Questions.” Please complete this section after you are done with your ratings. Feel free to leave any questions blank in this section if you do not feel comfortable answering them. You will have about 2 hours to complete this part of the study.

When you are finished please wait patiently while your colleagues complete their ratings.

[Hand out the booklets that contain the test items to be reviewed, the Item Review Tool, and the Teacher Background Questionnaire.. Make sure the teachers follow the instructions. Do not talk to the teachers unless they ask you a question. Feel free to kindly warn them if they do not follow the instructions (for example, if they are talking to each other). About half way through this section of the study ask the teachers how they are doing. How is everybody doing? We spent about one hour so far and you have about another hour to complete your ratings. Continue only when all participants are finished with their ratings.]

6. When everyone is finished with their ratings and Teacher Background Questionnaire, the interviewers hand out a short questionnaire that asks the teachers if they recognize any of the items they reviewed. If yes, they are to indicate the ID of each item and the source that they thought the item was coming from (local assessment, NAEP etc.) (attached, p.14-15). This information is recorded to assess the degree to which the rating process was truly blind. Next, the interviewer asks the teachers to identify four or more mathematics items that they found the most problematic. They record the IDs of those items on the same questionnaire and take a 15-minute break. During the break, the interviewers identify which of these items are NAEP items. After the break, the interviewers distribute a copy of the NAEP items identified as most problematic by the group. The interviewers ask the teachers to comment and discuss issues they found in these items, one item at a time. They are also asked to suggest modifications to these items if applicable. The interviewers record the summary of the discussion. At the end, the interviewers thank the teachers for their participation and distribute gift cards.

Thank you for completing your review. Now I am now going to hand out a questionnaire which asks you if you recognized any of the mathematics questions you reviewed today. If there are any questions you recognized please list both their IDs and the source from which you recognized it on the front page. Let me know if you need additional space and/or additional paper to write on.

[Allow teachers enough time to fill out the questionnaire.]

Please take a look at the mathematics test questions in your booklet and identify four or more mathematics questions you found most problematic and list their IDs. Record the IDs of these items on the back page of the same questionnaire. We will discuss the issues you identified in these questions after the break.

[After everyone is finished, We will now take a 10 minute break. Collect all the questionnaires. See which problematic IDs are NAEP IDs using the list provided.]

Now, you will discuss the mathematics questions that you found problematic with each other. You will have five minutes to discuss each question. Think about the issues you found with each test question and think of ways of modifying them without changing the knowledge and skills it assesses. Please choose a spokesperson to summarize and report what you discuss.

[Hand out a question, ask who will be the spokesperson for this question. Allow the teachers to discuss. Make sure their discussion is around the main questions above (issues with the item, and ways of modifying the item). After five minutes is up for a question ask the following: OK, now let’s list the problems you found with this question. Record on your laptop exactly what the responses are. If different teachers have different answers record their answers separately. Make sure the teachers’ answers are as specific as possible. For example, “I just don’t like this question” will not be an acceptable answer. If the teachers’ responses are not specific enough then ask the teachers to be more specific. For example, if they say “This is a difficult question; that’s why I find it problematic,” then ask them what makes the question difficult. Next ask the teachers if the question can be modified.]

Is it possible to modify this question without changing the mathematical knowledge and skills it is assessing? If yes, how would you modify this question? Can you suggest exact wording? [Again, record different opinions and make sure the answers are as specific as possible. For example, “I would make this question clearer” would not be a specific enough answer. In a case like this, ask the teachers how they would make the question clearer. Try to get an exact wording from the teachers. In other words, ask them what the modified question would read, if possible. Repeat this for each problematic NAEP question the teachers identified. Cover as many questions as you can without going over the four-hour period dedicated for the study (from the initial introduction to the end). Thank the teachers for their participation and hand out gift cards.]

Item Review Tool with Descriptions

The question includes mathematical terms that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

Indicate the unfamiliar term(s) if your rating is (1) or (2): ____________________

Examples of mathematical terms: Integer, function, equation, vector, matrix, triangle, parallel. A mathematical term would not be familiar for a student if it has not been introduced in the classroom at his/her grade level.

The question includes visuals (graphs, tables, charts, figures or diagrams) that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

A student would not be familiar with a visual if it either has not been introduced at this grade level or other types of visuals are more commonly used in classrooms or in assessments. Geometric shapes (triangles, squares, lines representing angles) should not be regarded as visuals unless they are part of a graph, table, chart, figure, or diagram.

The question includes visuals (graphs, tables, charts, figures or diagrams) that are unnecessarily complex for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

A visual representation might be complex for a student because there is excessive or unnecessary information presented in the visual. Geometric shapes (triangles, squares, lines representing angles) should not be regarded as visuals unless they are part of a graph, table, chart, figure, or diagram.

The question assesses mathematical knowledge or skills NOT taught in the classroom for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

A student might have had very little to no opportunity in the classrooms to learn the mathematical knowledge or skills assessed in the question at this grade level.

The calculations needed to solve the problem are unnecessarily demanding for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The question either requires too many calculations or calculations involving complicated operations.

The context used in the question is unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The context in a mathematics question is essentially the story around the problem. For example, if the question is simply asking the student to add 2 plus 3 there would be no context in this question. If the question says Jane went to the grocery store and bought 2 apples and 3 mangos, the question is presented with a context.

A student might not be familiar with the context in the question because of two main reasons: (1) He/she cannot relate to the story in the question. Example: the question talks about subway stops and the student lives in a city where there is no subway system. (2) He/she is familiar with the context itself but he is not used to seeing it in mathematic questions.

The question includes non-mathematical words/phrases that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

Indicate the unfamiliar word(s) if your rating is (1) or (2): ____________________

Some of the words or groups of words (phrases) in the question might not be familiar to the students at this grade level.

The language used in the question is unnecessarily complex for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

Unnecessarily long sentences or confusing sentence structures can obstruct students’ understanding of the question. The question could have been posed more simply.

The reading in the question is unnecessarily heavy for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The reading required in the question would be unnecessarily heavy if a student cannot reach the correct answer because the question is too long. The question tests reading ability in addition to mathematical knowledge.

The level of detail required in the answer is not clear to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

It is not clear what exactly the student needs to demonstrate in his/her answer in order to obtain full credit. This statement applies to open-ended questions only.

I would use this question to assess my students:

(1) Strongly Agree (2) Agree (3) Disagree (4) Strongly Disagree

The question is grade appropriate. I would use this question without any modifications to assess my students.

Sample Item

There will be 58 people at a breakfast and each person will eat 2 eggs. There are 12 eggs in each carton. How many cartons of eggs will be needed for the breakfast?

A) 9

B) 10

C) 72

D) 116

Item ID #: 11223344

Item Review Tool

Carefully read the mathematics question. Think about how your students might approach this question. Think about the challenges they might face in understanding this question or solving this problem. Now, rate this question on each of the following statements. Indicate your rating by circling your answers.

The question includes mathematical terms that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

Indicate the unfamiliar term(s) if your rating is (1) or (2): ____________________

The question includes visuals (graphs, tables, charts, figures or diagrams) that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The question includes visuals (graphs, tables, charts, figures or diagrams) that are unnecessarily complex for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The question assesses mathematical knowledge or skills NOT taught in the classroom for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The calculations needed to solve the problem are unnecessarily demanding for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The context used in the question is unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The question includes non-mathematical words/phrases that are unfamiliar to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

Indicate the unfamiliar word(s) if your rating is (1) or (2): ____________________

The language used in the question is unnecessarily complex for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The reading in the question is unnecessarily heavy for:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

The level of detail required in the answer is not clear to:

(1) Nearly all of my students (2) Most of my students (3) Some of my students (4) Almost none of my students

I would use this question to assess my students:

(1) Strongly Agree (2) Agree (3) Disagree (4) Strongly Disagree

Teacher Background Questionnaire

	Question/Prompt	Response
1	Date
2	Booklet ID
3	Gender	 Male  Female
4	Name of School
5	School District
6	School Location	 Metro  North  Inland  South
7	How many years have you taught 4^th/8^thgrade Mathematics?
8	Were you teaching 4^th/8^th grade Mathematics in the academic year 2006-2007?	 Yes  No

Questionnaire about Reviewed Mathematics Questions

1. Did you recognize any of the mathematics questions you reviewed today? That is, do you recall seeing any of these questions before?

Yes ( ) No ( )

Skip to the next question if your answer is No.

2. If you recognize any of the mathematics questions you reviewed please list their IDs (the number at the lower right corner of the page the item is printed on) below. Indicate what you think the sources of these items are (only the ones you recognized). Draw an arrow to connect the ID and the source.

ID number of question(s) you recognized	Source of this question

3. Take a look at the mathematics test questions in your booklet and identify four or more mathematics questions you found most problematic and list their IDs below. You will find the IDs printed at the lower right corner of each page the test questions appear on. We will discuss the issues you identified in these test questions after the break.

ID number of question(s) you found most problematic. List four or more.

File Type	application/msword
File Title	NATIONAL ASSESSMENT OF
Author	Elizabeth Kulik
Last Modified By	#Administrator
File Modified	2009-05-20
File Created	2009-05-20