Supporting Statement and Materials

This material is being submitted under the generic Institute of Education Sciences (IES) clearance agreement (OMB #1850-0803 v.8) that was approved in July 2007. This generic clearance provides for the National Center of Educational Statistics (NCES) to conduct various procedures (field tests, cognitive interviews) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments.

Background

The National Assessment of Educational Progress (NAEP) Mathematics assessments were administered in Puerto Rico in 2003, 2005, and 2007 at grades 4 and 8. Results showed not only that performance was lower in Puerto Rico than in the nation and other jurisdictions, but also that the patterns of student performance in this jurisdiction were different than those in other NAEP jurisdictions. While earlier investigations have focused on item translation/adaptation, item fit, and alternative scaling and linking models, it would be useful for the NAEP program to extend the previous statistical research and examine in greater detail students’ understanding of the cognitive items in the assessment.

Cognitive interviews should allow us to better understand students’ comprehension of NAEP mathematics items, including:

whether the student understands the Spanish version of the item as intended;
whether the student encounters any unfamiliar words, contexts, or stimulus materials (e.g., geometric figures, graphs, etc.); and
how the student approaches and solves the item.

The interviews would not focus on whether the student answers the items correctly, but rather on acquiring information that could potentially inform item development, adaptation, and translation activities for future items. To this end, NCES proposes to carry out a series of one-on-one interviews in 2009 with grade 4 and grade 8 students from public schools in Puerto Rico.

Design and Context

Sampling Information

During the two weeks in which the interviews will be conducted, approximately 17 schools in 16 school districts throughout Puerto Rico will be included to yield a sufficient range of academic levels and regional representation among both students and schools. ASPIRA, Inc. of Puerto Rico (see description in Section 4) and the Puerto Rico Department of Education (via the NAEP state coordinator) have worked to draft a stratified distribution for a sample selection of schools. The following characteristics have been taken into consideration:

schools in urban/rural zones;
schools located -on the north, south, east, and west sides of the island (including coastal and mountain regions);
high performance/low performance schools;
schools under “Plan de Mejoramiento” (Improvement Plan, which are schools that have not achieved adequate yearly progress); and,
school level (i.e., the grades that are included in the school).

Once schools are chosen based on these characteristics, a random sampling of approximately 150–160 students, within grades 4 and 8, will be chosen, balancing both gender and achievement. Students with disabilities will not be considered in the sample selection.

Item Information

Items to be included in the study will be representative of both the content of the NAEP Mathematics assessment (i.e., all five content areas: number properties and operations, measurement, geometry, data analysis and probability, and algebra) and of the different item types (i.e., multiple-choice and constructed-response). A selection of mathematics items that were administered as a part of the 2007 NAEP assessment in Puerto Rico—equivalent to two blocks—will be included in this study. In addition, items with various types of stimuli (i.e., geometric figures, graphs, etc.) will be included. Finally, the statistical item fit information obtained from the Task Order Component study “In-Depth Analysis of the Puerto Rican NAEP Data” will be considered during the item selection process so that items that both fit and did not fit the NAEP statistical model are included in the study.

There will be 35 items at each grade, assembled into seven 5-item mini-tests. Each student will participate in only one of the seven mini-tests. It is estimated that each interview will last approximately 50–60 minutes total, including the time needed to work through the items and answer protocol questions. Approximately 10–12 students will provide feedback for each item. Schools will be offered an incentive to participate, and students will be compensated for their time.

Cognitive Interview Information

The template for the cognitive interviews is contained in Volume II of this submittal (pages 1–18). The template includes:

welcome/thank you/introductory remarks,
an item for interviewer modeling of the think-aloud process,
an item for a practice think-aloud activity for the student,
a generic version of the item scripts that will be customized for each of the items in the study, and
closing remarks/thanks.

The template contains the majority of the interview questions. However, there will also be specific questions that apply to each particular cognitive item. Volume II includes one specific grade 4 example (pages 19–23 of Volume II) and one specific grade 8 example (pages 24–28 of Volume II) which demonstrate the types of specific cognitive interview questions that will be asked. For example, the grade 4 interview contains such questions as “What do you think ‘right angle turn’ means in this question?” and “How do you think this picture helps you answer the question?”

Analysis Information

Following the field activity, ASPIRA will provide qualitative data to ETS (Educational Testing Service, an NCES contractor) in Excel spreadsheets. Using spreadsheets will enable the data to be manipulated in different ways, e.g., to group data for similar types of items together, thus providing additional insights on student performance. The summary results and information will be translated into English for use in the analyses and final report.

The qualitative data will be reviewed to determine the presence of patterns (e.g., the ways in which students are responding to items in certain content areas, items of a certain format, and items with particular types of stimulus material) that could potentially inform the revision of the items to make them more accessible to students in Puerto Rico. Data collected from the field regarding how students are interpreting the items and which mathematical and nonmathematical terms are unfamiliar to students will be examined to provide a better understanding of any presentation-related barriers to the items that might exist. Querying the students on their content knowledge and instructional experience with the mathematical content in the item may also contribute to a more complete picture of how they are interacting with the items.

Translation Information

The following translation activities will occur as part of the study:

From English into Spanish, translations done at ETS

Item-specific protocols
Generic protocol
Supplemental documentation (e.g., letters to schools, consent form for parents, confidentiality agreement, etc.)
Interview training materials, including presentation and interview scripts
Items not previously administered in Spanish. Note, the cognitive items went through the translation review cycle, which includes internal reviews, committee review, and external Translation Verification Review carried out by SLTI (Second Language Testing Incorporated—see section 5).

From Spanish into English, translation done at SLTI

Spreadsheet of interview notes and student responses. Note, the Translation Verification Review will be carried out by ETS.

Cognitive Interview Process

ASPIRA, Inc. of Puerto Rico (http://www.aspirapr.org/) will facilitate the field activity, including recruitment of interviewers. ASPIRA is a well-respected national nonprofit organization supporting educational programs in Puerto Rico; it has the requisite infrastructure within the local educational community and the experience necessary to facilitate communication at the local level that will lend efficiency to the overall process. See Appendix D for a sample script confirming school participation and confirmation and thank you letters for the schools.

The ETS Translation Team will work with ASPIRA to train the interviewers in the implementation of the protocols to be used with students in the field. ETS staff will also act as observers during selected interviews at schools in various districts in Puerto Rico. Protocols for the interviews have been developed based on NAEP materials currently in use, with appropriate adaptations for the mathematical context. These protocols, along with all necessary training materials, will be translated into Spanish. As part of the interview process, the session will be audio recorded. The audio recording may serve as a tool to clarify student responses during the interview and will not be transcribed or otherwise used in the course of analysis.

The interviews will focus on the following two questions:

Does the Spanish translation effectively communicate to the student the same construct being assessed as the original English language item?
Has the student studied the particular topic covered by the item in mathematics classes?

The interviewer will explain to the student at the outset that the aim of the interview is to collect information on how students solve mathematics problems, not on whether they answer the problems correctly. The entire interview will be conducted in Spanish.

The interviews will follow these steps:

The interviewer will first ask the student to read the item aloud. One purpose of this step is to permit the interviewer to determine whether the student is having difficulty reading any of the words in the item.
The interviewer will ask the student to work through the item. This will give the student an opportunity to think about the item before the interviewer proceeds with further questioning.
After the student has produced an answer or response for the item, the interviewer will begin to collect information through questioning the student directly on areas such as comprehension of the item (including what was being asked in the item), stimulus material (e.g., geometric figures, graphs, etc.) provided, whether the student encountered any unfamiliar words, how the student solved the item, and whether the content was familiar. The findings from the item fit analysis will help define the exact nature of the questions asked of the students.
The interviewer, using his/her judgment based on experience, will note pertinent aspects of the interview process, such as the student’s level of motivation and any special circumstances that might affect the interview.
The interviewer will also query the student about the perceived level of difficulty of the item, and what made the item difficult or easy.
In addition, the interviewer will ask the student whether there was enough information to be able to answer the item. If not, the interviewer will ask what additional information would enable the student to answer it.
As the student is providing information during the session, the interviewer will be recording responses on the interview form. Volume II of this submittal contains a generic example of the interview form on pages 11-18, a specific grade 4 example on pages 19-23, and a specific grade 8 example on pages 24-28. The interview form combines coding schemes and written responses, which will facilitate efficient analysis of the information upon completion of the data collection phase of the study.
This process will not focus on whether the student produced a correct or incorrect answer for each question, but instead, on how that answer was determined (i.e., how the question was interpreted, the thinking process engaged in, etc.). As part of the process, the interviewer will be asked to record the student’s answer and whether it was correct, but that information will not be shared with the student. Students will not be given the correct answers by the interviewer as this may impact any subsequent questions or interviews.

Consultations Outside the Agency

The ASPIRA Association of Puerto Rico (described in Section 4) will facilitate the field activity, including recruitment of interviewers. The executive director of ASPIRA is Adalexis Ríos.

Second Language Testing, Inc. (SLTI) is an independent agency that will provide translation verification, as it has done for other NAEP assessments. The president of SLTI is Charles Stansfield.

Assurance of Confidentiality

Participation is voluntary. Written consent will be obtained from legal guardians of minor students before interviews are conducted. In addition, students will be given a consent form to ensure that they are aware of their information is confidential (See Appendices A and B for parent and student forms, respectively.) No personally identifiable information will be gathered from either schools or students. Students will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form, The student consent forms and parent permission slips, which include the student name, will be kept separately from the student interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is released.

The interviews will be audio recorded. The only identification included on the audio files will be the student ID. The audio files will be secured in a locked cabinet for the duration of the study and will be destroyed after the final report is released.

Parents and students will be provided with the following confidentiality pledge: The information you provide will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107–347 and other applicable Federal laws, your responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of up to $250,000, or both if he or she willfully discloses ANY identifiable information about you.

Test security will be assured at the administrator, interviewer, and student levels. The interviewers must sign a confidentiality and test security agreement. (See Appendix C for the interviewer test security agreement.)

Justification for Sensitive Questions

Throughout the interview protocol development process, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have identified and eliminated potential bias in questions.

In addition, the cognitive mathematics item development process included sensitivity reviews before use in previously administered assessments.

Estimate of Hour Burden

Schools will need to select and recruit students and provide a room in which the interviews can occur. Each student will participate in a one hour cognitive interview. Volume II contains both a generic (on pages 11-18) and specific example (one at grade 4 and one at grade 8) cognitive interview forms.

The estimated respondent burden follows:

Respondent	Hours per respondent	Number of respondents	Total Hours
Schools (for student recruitment)	2	17	34
Grade 4 student	1	75-80	75-80
Grade 8 student	1	75-80	75-80
Totals		167-177	184-194

Estimate of Costs for Recruiting and Paying Respondents

Students will receive educational materials worth approximately $5 for appreciation of their participation. Participating schools will receive a $50 gift card to Borders bookstore for appreciation of their participation. These amounts are consistent with NCES guidelines.

Cost to Federal Government

For conducting the cognitive interviews, translation, and related travel, the costs are approximately $72,894. The following table provides the overall project cost estimates:

TOC Puerto Rico Cognitive Labs February 2009

	Cost
1 Staff Costs
Subtotal	143,248

2 Subcontract Costs
Translation & Review (SLTI + Committees)	26,946
ASPIRA subcontract	35,000
Training Costs	6,300
Observation Costs	4,648
Subtotal	72,894

3 Other Project Materials
Digital Recorders	350
Incentives for Student and School Participation	2,600
Photocopy/Shipping	3,000
Subtotal	5,950

Total Cost of Task Without Fee	222,092
Fixed Fee	6,484
Award Fee	10,807
Total Cost of Task	239,383

Schedule^*

Activity	Dates
Draft interview protocols, developed in English, submitted to NCES for review and comment	January 21, 2009
Comments due to ETS from NCES on the draft protocols	January 28, 2009
Final interview protocols, developed in English	February 4, 2009
Training materials developed in English	January 2009
Interview protocols translated into Spanish	January–February 2009
Preparation of materials for Office of Management and Budget (OMB) review	February 5–19, 2009
Materials submitted to NCES for review prior to OMB submission	February 24, 2009
Training materials translated into Spanish	February 2009
Training ASPIRA interviewers in Puerto Rico	March 16–20, 2009
Interviews in Puerto Rico schools	March 30–April 3, 2009 April 13–April 17, 2009
Data compilation in Spanish	April 2009
Translation of data into English	May 2009
Draft summary report submitted to NCES	July 24, 2009
NCES returns comments to ETS	August 14, 2009
Final summary report submitted to NCES	August 28, 2009
Monthly progress reports	Monthly, January 2009–August 2009

* Please note that weeks indicated reflect dates available for the activity as per the Puerto Rico school calendar.

Volume II of this submission includes the corresponding cognitive interview protocols to be used in the study.

Appendix A–Legal Guardian Permission Slip

<DATE>

Dear Parent or Guardian:

Between <DATES>, 2009, ASPIRA representatives, on behalf of the National Assessment of Educational Progress (NAEP), will meet students at <SCHOOL> to conduct interviews about mathematics tests. The purpose of this study is not to test students, but to listen to their ideas for improvement of the test questions.

Each representative will meet with a student in a one-on-one interview, which will last approximately one hour. These interviews will be audiotaped. During that time, the student will be asked to explain what he or she thinks about the test questions. Your child will not be identified by name, nor will any score be assigned to his or her responses.

These interviews are an important stage of the NAEP test development process. The comments from the students we interview help us to improve the questions to make the best test possible. Your child will be given a small token of appreciation at the conclusion of the interview.

The information your child provides will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107–347 and other applicable Federal laws, your child’s responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of up to $250,000, or both if he or she willfully discloses ANY identifiable information about your child.

We hope that you will give your consent for your son or daughter to participate in the NAEP interviews by signing this form and returning it to the school. Without your signed consent, the student will not be able to participate in the study.

Sincerely,

Amy R. Dresher

NAEP Deputy Director

Educational Testing Service

____________________________________________________________

I hereby give my permission for ___________________________ to

(Student’s name)

participate in the NAEP Mathematics question tryouts.

Signed: _____________________________ Date: ________________

(Parent or Guardian)

Appendix B–Student Consent Form

<DATE>

Dear Student:

An ASPIRA representative will meet with you today for a validation study of the mathematics assessment of the National Assessment of Educational Progress (NAEP). The purpose of this study is to improve the test questions. Your ideas are important, and the representative will listen closely to you. You will not be judged, and you will not receive a grade for your answers or comments.

Other students like you will also participate in one-on-one sessions that will last approximately one hour. These sessions will be audiotaped. During that time, you will be given a series of math questions to answer, and you will be asked to explain what you think about the test questions as you work through them. You will not be identified by name, and your responses will not be scored.

The information you provide will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107–347 and other applicable Federal laws, your responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of up to $250,000, or both if he or she willfully discloses ANY identifiable information about you.

These interviews are an important stage of the NAEP test development process. Your comments, as well as those from other students we interview, help us to improve the questions to make the best test possible.

We thank you for your voluntary participation in the NAEP validation study, and ask you to please sign the form below.

Thank you,

Amy R. Dresher

NAEP Deputy Director

Educational Testing Service

____________________________________________________________

I hereby accept to participate voluntarily in this interview.

_____________________________

Name

_____________________________ _____________________________

Signature Date

Appendix C–Interviewer Confidentiality Agreement

Agreement: Item and Test Security Requirements for the National Assessment of Educational Progress (NAEP)

Under this agreement, you will either be reviewing, creating, or handling secure test material belonging to the National Assessment of Educational Progress and the National Center for Education Statistics (NCES) in the United States Department of Education. These test materials are confidential and may not be discussed with any person who has not signed this agreement. These materials may not be copied, published, announced, or in any other way made public, and you may not grant permission for anyone else to do so. If you are creating secure test material under this agreement, for purposes of copyright, it is understood and agreed that any material produced will be considered a work made for hire. As such, ETS or its designee will be the sole and exclusive owner of the materials you produce and will hold copyright thereon in the United States and elsewhere throughout the world, and you will retain no rights in such material.

By signing this agreement, you acknowledge that the test materials that are the subject of this agreement constitute proprietary and confidential materials of the United States Department of Education. You further understand that any disclosure, unauthorized use, or reproduction of these materials would damage the confidentiality of NAEP. Furthermore, by entering into this agreement, you are entering into a special relationship of trust with NAEP, wherein you agree to comply with the following:

Confidential test materials must be used only for the sole purpose of performing professional services for ETS. Any other use of these materials would require the prior written consent of ETS.
Confidential test materials must be kept in a locked container when they are not being used.
Confidential test materials must not be shared with any parties who have not signed this agreement and without the prior written authorization of ETS.
Confidential test materials must not be copied in whole or in part, and all copies of these materials must be returned promptly to ETS or its designee upon the completion of task.
In the event that confidential test materials are stolen or lost, or in the event that an unauthorized party has gained access to these materials, ETS or its designee must be notified immediately.

ACCEPTED AND AGREED TO:

Name: ______________________________

Signature: ___________________________

Date: _______________________________

Appendix D–Contact with Participating Schools

Script for Confirmation Call with Schools

Good morning/afternoon. May I please speak to <name of principal or contact person> about the Puerto Rico Cognitive Lab Study [Estudio de Validación de la Prueba NAEP]?

Good morning/afternoon. My name is <name> from ASPIRA. I understand from my colleague, <name of person making initial contact>, that students, teachers and a school administrator from your school have agreed to participate in the Puerto Rico Cognitive Lab Study [Estudio de Validación de la Prueba NAEP]. Thank you very much for participating in this important study. Do you have any additional questions at this time?

I’m calling to confirm a day that I can visit your school and conduct the following interviews: <1 or 2 students at grade 4, 1 or 2 students at grade 8.>

I would like to visit your school on <date>. Would it be possible for me to come on that day? I plan to arrive at your school at least ½ hour before the first interview. Can you please tell me what time the first interview could take place? Will someone in the office be there at <time you want to arrive> to show me the room where the interviews will take place?

Is there a particular place I need to park? May I please have the school address? Are there any special circumstances I should know about the directions to your school (closed streets or new one-way streets)?

The school will be rewarded with a gift card as a thank you for your participation. In addition, each student will receive a small token of appreciation at the completion of the interview. Who would be receiving the gift card for the school? {Make a note of this.} And as a reminder, I will need to have a signed consent form from a parent of each participating student prior to interviewing the student. I will be unable to interview any student for whom we don’t have a consent form. I will pick these up from the school on the morning of the interviews.

I will be back in touch with you if I have any further questions. Please do not hesitate to call me at <phone number> or send an email <email address> should you have any additional questions.

Thanks again for agreeing to participate in this project. I look forward to meeting you and others at your school on <date and time>.

Confirmation Letter to School

<Date>

Dear <Name>:

Thank you for agreeing to help refine mathematics questions for the National Assessment of Educational Progress (NAEP). This letter includes details about the visit to your school.

An ASPIRA representative(s), on behalf of NAEP, will visit your school on <date>, for the purpose of working with fourth- and eighth-grade students. Two observers from ETS, María Martiniello and Sylvia Ledesma, will also be present during the interviews. <Name of ASPIRA representative(s)> will arrive at the school at approximately <time> and will need to work with each of the <NUMBER> students for approximately one hour. Therefore, two simultaneous one-on-one sessions will be conducted over approximately <NUMBER> hours. While pairs of students could be in the same space during these sessions, the preferred approach would be to have simultaneous sessions occur in different rooms to minimize distractions.

Parental permission forms are enclosed. Please make additional copies if needed. If you have any questions, please feel free to call us at <PHONE NUMBER>. Thank you again for your assistance with this important project. We look forward to working with the students.

Sincerely,

Adalexis Ríos

ASPIRA

Thank You Letter to School

<DATE>

Dear <Name>:

I would like to thank you, on behalf of the National Assessment of Educational Progress (NAEP), for your school’s participation in a study that will help our assessment development process. Because of the assistance of schools like yours, the NAEP program will be able to learn from the direct feedback of students.

Our interviewers found the sessions extremely useful and productive. They found the one-on-one format to be enjoyable and informative; the students involved were open, candid, and insightful. Our staff also found your school to be gracious in allowing them into your busy school day, and supportive in providing interviewing space, students, and general assistance.

Again, I would like to thank you and, in further appreciation, present your school with the enclosed gift certificate from Borders.

Best wishes for continued success to you and your students.

Sincerely,

Adalexis Ríos

ASPIRA

File Type	application/msword
File Title	Volume I:
Author	Temp_MHolte
Last Modified By	rochelle w martinez
File Modified	2009-03-20
File Created	2009-03-20

Supporting Statement and Materials

Vol 1 PR Cog Lab rev 3-19-09.doc

System Clearance for Cognitive, Pilot and Field Test Studies