justification

This material is being submitted under the generic Institute of Education Sciences (IES) clearance agreement for cognitive labs and pilot tests (OMB #1850-0803 v.27). This generic clearance provides for the National Center of Educational Statistics (NCES) to conduct various procedures (such as field tests and cognitive interviews) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments.

Background and Study Rationale

This study investigates the effect of the new computer-based writing assessment audiovisual (AV) prompts at grades 8 and 12 on student motivation and performance and the effectiveness of the writing task tutorial.

The study design is the same at both grades and takes into account when, during the writing task, effects of the AV prompt stimuli (graphics, audio, or video) may occur and how to identify them. The effects of prompt stimuli can occur during three subtasks: (1) interpretation of the writing task, (2) planning of the writing response, and (3) writing of the response. The cognitive interview study is designed to determine how the writing prompts that incorporate AV may affect performance during each of the subtasks. The methods take into account that a student may be aware of and able to report some AV effects, but not others.^¹

This is largely an observational study. Although the sample will include a mix of student characteristics and prompt types, it does not explicitly measure differences by those conditions. The data being collected will consist mainly of verbal reports in response to probes or from think-aloud tasks, in addition to usability task performance and volunteered comments on the tutorial and the prompts. These largely qualitative measures will provide insight into the effects the prompts have on comprehension of the writing tasks, on planning the writing tasks and, to some extent, on the writing.

The purpose of this study is to identify features of the writing task tutorial and of the AV prompts that may be beneficial or that may be problematic in order to inform current and future item development.

Study Design and Context

Research Questions

The questions are structured so that they can be addressed by the findings from the study’s qualitative methods research. The questions are all aspects of the overarching objective to investigate the effects of prompts generally and the features of prompts related to the observed effects.

The main research questions are:

To what extent does the tutorial enhance student performance?
Do any prompts adversely affect motivation and performance?
Do problematic prompts have features in common?
Do students react more positively to some prompts than others?
Do students find that some prompts are more helpful than others?

The study design first elicits information from the students with minimal changes in assessment procedures and is followed by interview questions which focus more directly on the prompts themselves. The design has four main components:

student use of the writing task tutorial;
effect of prompts on student performance on writing tasks;
examination of student responses two writing tasks using think-aloud protocols; and
session debriefing.

Performance of the Writing Task Tutorial

The goal of the tutorial component of the protocol is to present the available features and the process involved in the writing task in order to maximize each student’s performance.

At each grade, the sample of 20 students will be randomly divided into two equal groups. Both groups will go through the tutorial and be asked a short set of questions about their assessment of the tutorial.

One group will then be asked to complete a set of tasks explained in the tutorial (such as having sections of text read aloud, opening the writing panel, etc.), on an AV prompt while thinking aloud. This is to assess their understanding of the tutorial and to gauge the usability of the different features available for the AV prompts. While this testing assesses what a student retains shortly after completing the tutorial, the act of focusing on the usability tasks may, in itself, make the student more comfortable with the computer features and potentially enhance the effectiveness of the AV prompts for the student.

The other group will complete the two writing tasks before taking the usability test, so that their performance will not be affected.

Effect of Prompts on Two Writing Tasks

Two writing tasks—with a combination of prompts balanced across the sample—will be presented to the students. Before beginning to write, students will be asked their understanding of the task. After each writing task, debriefing probes will elicit reactions to the prompt and to the writing task.

A set of generic probes will allow some comparison of ways in which the additional media affects student performance. Prompt-specific probes will increase understanding of how particular features of a probe may affect task performance.

In order to limit the interview and writing sessions to 90 minutes in total, the time on each writing task is reduced from 30 to 15 minutes. However, because the writing tasks will be scored, the study is designed to minimize other changes that could interfere with writing performance. Cognitive interview techniques (e.g., thinking aloud or responding to probes) will not be used during the writing time.

Examination of Two Additional, Untimed Think-Aloud Prompts

In this component, each student will be shown (in random order) two additional prompts and asked to think aloud as they prepare for the writing task described (but will not actually do the writing task). After completing plans for each writing task, the student will be asked for his or her impressions of the prompt and the writing task, and then asked a set of probes about the prompt.

After the think-aloud section on both prompts, the student will be asked, “If you had your choice of writing tasks, which one would you choose? And why?”

Session Debriefing

The last part of the session will be a debriefing to elicit any additional comments from the student.

Representative Sample of U.S. 8th- and 12th-Graders

The sample size will be 20 students per grade. This will permit inclusion of multiple conditions related to the nature of the prompt-stimulus (text, AV, and AV removed) and the characteristics of the students (race, gender, and some rough measure of SES).

Twenty students at each grade would allow for

a mix of race/ethnicity, with 5 students in each of 4 demographic groups;
two SES groups; and
multiple tests of all of the task stimuli that include AV stimuli.

Cognitive Interview Information

Template for the Cognitive Interviews

The cognitive interviews are designed to determine whether and how student performance is impacted by the addition of graphics, audio, and video elements to prompts for the 2011 Writing Assessment.

The interviewer will explain the goals of the session to the student. The aim is to understand how students process the writing task and how the graphic, audio, or video prompts affect task comprehension or influence plans for writing.

Interviewers will use several different cognitive interviewing techniques, including general, think-aloud, and prompt-specific probes, observation, and debriefing questions.

After the tutorial, half of the students will do two writing tasks, as noted and described above. These tasks will use a subset of the 2011 writing prompts, along with some audio and AV prompts from which the audio or AV feature has been removed. The prompts will be randomly allocated to booklets for each subsample of students.

The cognitive interview probes for these tasks will be a combination of generic probes and probes written for the particular prompts that were administered.

Units of Analysis

The key unit of analysis is the writing prompt. Single prompts and groups of similar prompts (see categorization, below) will be analyzed across students within grade. Depending on the within-grade results, it may be possible to combine certain sets of prompts across grades.

The types of data collected about the prompts will include

think-aloud verbal reports;
responses to generic probes;
responses to prompt-specific probes;
additional volunteered student comments; and
debriefing questions.

A coding frame will be developed for the responses to think-aloud questions and other verbal reports. The frame will be designed to identify and code reflection of verbal responses to the following areas:

feelings of correct and incorrect task comprehension;
aspects of a prompt that are helpful, unhelpful, or detrimental to task comprehension;
task reflecting the student’s interest or lack of interest, enthusiasm or discouragement, linkage or no linkage of the expression to some aspect of the prompt;
information in the prompt employed in planning the writing task; and
information from the prompt that would be included in the writing.

The draft coding frame will be modified and supplemented based on reviewing the initial cognitive interviews. The focus will be on developing codes (and instructions for coding) that relate to the research questions.

Other units of analysis will be defined as sections (possibly single screens or related screens, along with the related verbal information) in the tutorial. The data for these units will include:

think-aloud verbal reports;
measures of performance of usability tasks response to debriefing questions;
volunteered comments; and
time spent on each task.

Analysis Plan

The general analysis approach will be to compile the different types of data in spreadsheets and other formats to facilitate identification of patterns of responses for specific prompts or categories of prompts, for example, patterns of counts of verbal report codes and of responses to probes or debriefing questions.

Each type of data for a prompt will be examined both independently and in conjunction with other data for the prompt in order to determine whether a feature or an effect of a prompt is observed across multiple measures and/or across administrations of the prompt.

This approach will ensure that the data are analyzed in a way that is thorough and that will address the main research questions, as follows:

To what extent does the tutorial enhance student performance?
Do any prompts adversely affect motivation and performance?
Do problematic prompts have features in common?
Do students react more positively to some prompts than others?
Do students find that some prompts are more helpful than others?

The 2011 Writing Framework notes the following three general characteristics of writing that will be examined in the students’ responses:

development of ideas;
organization of ideas; and
language facility and conventions.

Information from the think-aloud, probe, and debriefing questions about planning the writing tasks will provide indirect evidence of the effect of the prompts on the three characteristics of writing.

The verbal reports will address the following questions about the prompt-stimuli effects listed in the framework’s Writing Specifications (pp. 16-17):

Are stimuli easy to comprehend?
Do stimuli have clear relationships to the task topic?
Do stimuli provide information students can use in their responses?
Do students use information or ideas from the stimuli?
Do stimuli contain extraneous information?

The study will also partially address the above issues.

Because the sample size is insufficient to provide results for individual prompts, we will review the salient features of each prompt with the objective of categorizing them into a small number of general types so that the results can help inform decisions about which types of prompts are most effective or, conversely, least detrimental.

A more detailed analysis plan will be developed as part of the project. Draft and final reports of the findings will be delivered.

Cognitive Interview Process

Abt Associates, Inc. (see section 6) will ensure that qualified interviewers are available to conduct the interviews with students. Interviewers will be trained on the cognitive interviewing techniques of the protocol. Development of interview protocols will be based on the generic protocol structure described in volume II. The interviews will focus on how the addition to the writing prompts of AV elements, i.e., graphics, audio, or video, will affect 8th- and 12th-grade students taking the NAEP writing assessment.

Recruitment

UserWorks will recruit participants for the study via email through their Washington, D.C. metropolitan area participant database of volunteers and through other available lists. Parents of students expressing interest in participation will be contacted for parental permission. Potential student participants will be screened by telephone according to a screening questionnaire mutually agreed upon by UserWorks and Abt Associates. After scheduling, the participants for the sessions will be mailed or e-mailed directions to the study location and asked to reply by e-mail or phone to acknowledge receipt of the directions. On the day prior to the study, participants will be called to confirm their appointments.

Participant Characteristics

Forty participants for this study will be recruited in order to comply with the following criteria in response to a screening interview:

20 eighth-grade students;
20 twelfth-grade students;
50:50 male to female ratio;
mix of race;
mix of socio-economic background; and
public or private school students.

Study Research Design Structure

The structure of the study design, as described above, is summarized in the following chart.

Stage	Potential effect	Methods	Data or measures
Tutorial for writing tasks Tutorial usability Presentation of 2 prompts (randomized) Planning response	Support task performance Encouragement or not Reaction to and use of stimulus Understanding of task	Think-aloud with probing Cognitive interview with retrospective report and limited probing	Verbal reports Verbal reports Answers to probes
Writing to task	Quality of response	Observation	Time on task Length of response
Post-writing	N/A	Debriefing questions	Answers and volunteered comments
Presentation of 2 additional prompts both with AV stimuli (no actual writing task)	Encouragement or not Reaction to and use of stimulus Understanding of task	Cognitive interview with concurrent report and probing Choice of prompts and reason for preferences	Verbal reports Selection Answers and volunteered comments
Final debriefing	N/A	Debriefing questions	Answers and volunteered comments

Sample Design

The sample size will be 20 students per grade. This will permit inclusion of multiple conditions related to the nature of the prompt-stimulus (text with graphic/photo, with audio-and-video, or with audio-and-video removed). It will also allow for the inclusion of the desired student characteristics (race, gender, or some rough measure of SES).

Twenty students at each grade would allow for

a mix of race/ethnicity, with 5 students in each of 4 demographic groups;
two SES groups; and
multiple tests of all of the task prompts that include AV stimuli.

Sample Selection and Allocation of Prompts to Booklets

Each student will be administered a booklet consisting of 4 prompts. The student will complete the writing task for the first set of two prompts, but the student will not complete the writing task for the second set of two prompts as they will be think-aloud prompts. There will be 5 booklets at each grade level.

The prompts are treated as three types (excluding prompts that consist only of text), for the purposes of the cognitive testing procedures.

Sample Selection Table

Distribution of assessment prompts by grade

Type of prompt	8th grade	12th grade	Total probes
Audio-and-video (AV)	10	9	19
Audio (A)	5	0	5
Audio-and-video removed (-AV)	4	6	10
Audio removed (-A)	2	N/A	2
Illustrations, photos, graphics (G)	6	9	15

Text: The prompt consists of text and different combinations of illustrations, photos or graphics

AV: The prompt consists of text and audio-and-video

-AV: An AV prompt with audio-and-video removed or audio removed

A: The prompt consists of text and audio

-A: The prompt consists of text with audio removed

G: The prompt consists of text and a graphic image

Categorization of Prompts: Distribution and Type of Prompt

It was necessary to select a subsample of prompts for the cognitive interview study based on several factors. These factors included (1) the total sample size of students, (2) maintaining a reasonable number of administrations of each prompt chosen for cognitive testing, and (3) the need to create additional prompts with the AV feature removed.

Prior to selection, the audio-and-video-only prompts and audio-only prompts were sorted into three categories based on a judgment of the contribution of the stimuli to the writing task. The first category of prompts is composed of those in which stimuli provide information essential to the task, e.g., the 8th-grade school calendar task and the 12th-grade video game controversy task. The second category is composed of those in which stimuli may be of marginal use—perhaps by suggesting examples that could be used in the essay—but that are not essential (for example, the 8th-grade good volunteer project and the 12th-grade extracurricular activities task). The last category is prompts in which stimuli that contribute little or no information can be used in the task (though they may, of course, have other effects, such as on motivation). For example, some 8th-grade audio-only prompts simply read all or parts of text appearing on the screen (merely automating a reading feature available in the system), and one 12th-grade prompt shows only a theater audience applauding.

Prompts for testing were selected from each of the three categories, with an emphasis on the first two. Prompts selected for removal of the audio-and-video feature or audio feature were selected only from the last two categories. The selection was purposeful, not random, to permit selection of prompts best representing their category or that had a feature that was judged important to test. A small number of prompts with text only and some type of non-dynamic visual were included as a control condition. The controls were somewhat undersampled to increase the amount of testing of the audio-and-video and audio prompts, the research priority.

Distribution of Prompts

The allocation, by grade, of the writing prompts to the two writing tasks and to the two think alouds is shown in the following chart:

Sample Allocation of Types of Prompts

	Writing tasks	n	Think aloud	n
8th grade	AV	5	AV	2
	-AV	2	-AV	3
	A	1	A	1
	-A	1	-A	1
	G	1	G	3

	Writing tasks	n	Think aloud	n
12th grade	AV	5	AV	3
	-AV	3	-AV	4
	A	N/A	A	N/A
	-A	N/A	-A	N/A
	G	2	G	3

Text: The prompt consists of text and different combinations of illustrations, photos or graphics

AV The prompt consists of text and audio-and-video

-AV An AV prompt with audio-and-video removed or audio removed

A The prompt consists of text and audio

-A The prompt consists of text with audio removed

G The prompt consists of text and a graphic image

Prompt Selection

The next two charts show the allocation of specific prompts to 5 booklets at each grade, with the prompts that will also be administered with the audio-and-video feature removed shown with their numbers in bold.

8th Grade Booklets

	Booklet A	Booklet B	Booklet C	Booklet D	Booklet E
1st Writing Task	AV	G	AV	A	AV
	Prompt 1	Prompt 2	Prompt 3	Prompt 4	Prompt 5
	99887	99294	97412	96709	97410

2nd Writing Task	-A	AV	-AV	AV	-AV
	Prompt 6	Prompt 7	Prompt 8	Prompt 9	Prompt 10
	97792	96047	97410	97413	96047

	Booklet A	Booklet B	Booklet C	Booklet D	Booklet E
1st Think Aloud	-AV	A	-A	G	G
	Prompt 9	Prompt 6	Prompt 4	Prompt 11	Prompt 12
	97413	97792	96709	99336	96040

2nd Think Aloud	G	-AV	AV	-AV	AV
	Prompt 13 96348	Prompt 3 97412	Prompt 14 99888	Prompt 15 92609	Prompt 15 92609

Text: The prompt consists of text and different combinations of illustrations, photos or graphics

AV The prompt consists of text and audio-and-video

-AV An AV prompt with audio-and-video removed or audio removed

A The prompt consists of text and audio

-A The prompt consists of text with audio removed

G The prompt consists of text and a graphic image

12th Grade Booklets

	Booklet F	Booklet G	Booklet H	Booklet I	Booklet J
1st Writing Task	AV	G	AV	AV	AV
	Prompt 1	Prompt 2	Prompt 3	Prompt 4	Prompt 5
	94077	95608	94106	94565	99825

2nd Writing Task	G	AV	-AV	-AV	-AV
	Prompt 6	Prompt 7	Prompt 8	Prompt 9	Prompt 1
	94132	128646	94516	95553	94077

	Booklet F	Booklet G	Booklet H	Booklet I	Booklet J
1st Think Aloud	-AV	AV	G	AV	G
	Prompt 3	Prompt 9	Prompt 10	Prompt 8	Prompt 11
	94106	95553	94955	94516	95540

2nd Think Aloud	AV	-AV	G	-AV	-AV
	Prompt 12 93680	Prompt 13 94565	Prompt 14 94852	Prompt 7 128646	Prompt 15 93680

Text: The prompt consists of text and different combinations of illustrations, photos or graphics

AV The prompt consists of text and audio-and-video

-AV An AV prompt with audio-and-video removed or audio removed

A The prompt consists of text and audio

-A The prompt consists of text with audio removed

G The prompt consists of text and a graphic image

Consultants outside the Agency

NAEP Education Statistics Services Institute (NAEP ESSI)

The NAEP Education Statistics Services Institute (NAEP ESSI) supports the U.S. Department of Education by providing technical assistance, research and development support for NCES related to NAEP. Staffs at NAEP ESSI perform a range of functions including leading technical support for the quality assurance project and assessment development, conducting technical reviews of NAEP publications, assisting in background questionnaire validity studies, coordinating special studies related to cognitive items and framework comparisons, and supporting item development, scoring, and other technical aspects of NAEP.

Abt Associates

Abt Associates is a large, established for-profit government and business research and consulting firm. Abt Associates is working as a subcontractor for NAEP ESSI on this project and provides expert development, testing, and refinement of data collection tools to ensure their reliability and validity. Located in Bethesda, Maryland, Abt has a Cognitive Testing Laboratory facility which offers a range of cognitive interviewing and usability testing services.

UserWorks

UserWorks is a local company that specializes in recruiting for focus groups, cognitive interviews and other research. Abt has worked with them regularly for several years.

Consultants

Victorine Hocker serves as a consultant at NAEP ESSI. She has extensive background in scoring in NAEP as well as content area expertise in reading and writing.

Johnny Blair, a Senior Scientist at Abt Associates, is a specialist in survey research design, cognitive interviewing, and usability testing. He has extensive experience in survey methodology, statistics, demography, instrument design, data analysis, and evaluation. He has published extensively on issues such as cognitive interview data quality, optimizing cognitive pretest sample sizes, and customizing cognitive interview protocols.

Laura Burns, a cognitive interviewer, is a graduate of the Joint Program in Survey Methodology, University of Maryland, and has several years of experience conducting cognitive interviews. During her graduate study, she was a research assistant in the Abt Cognitive Testing Laboratory.

Kristi Hanson, a cognitive interviewer, has over ten years of experience conducting cognitive and other qualitative interviews for several companies in the DC area.

Assurance of Confidentiality

NCES has policies and procedures that ensure privacy, security, and confidentiality, in compliance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and Education Sciences Reform Act (Public Law 107-110, 20 U.S.C. §9622). This legislation ensures that security and confidentiality policies and procedures of all NCES studies, including the NAEP project, are in compliance with the Privacy Act of 1974 and its amendments, NCES confidentiality procedures, and the Department of Education ADP Security Manual. The federal authority mandating NAEP in Section 9622 of US Code 20 requires the confidentiality of personally identifiable information:

(A) IN GENERAL—The Commissioner for Education Statistics shall ensure that all personally identifiable information about students, their academic achievement, and their families, and that information with respect to individual schools, remains confidential, in accordance with section 552a of title 5.

(B) PROHIBITION—The Assessment Board, the Commissioner for Education Statistics, and any contractor or subcontractor shall not maintain any system of records containing a student's name, birth information, Social Security number, or parents' name or names, or any other personally identifiable information.

Participation is voluntary. Written consent will be obtained from legal guardians of minor students before interviews are conducted. In addition, students will be given an assent form to ensure that they are aware that their information is confidential (see appendixes A and B for student and parent forms, respectively). No personally identifiable information will be gathered from either schools or students. Students will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all student materials together. The student ID will not be linked to the student name in any way or form. The student assent forms and parent permission slips, which include the student name, will be separated from the student interview files and kept in a locked cabinet for the duration of the study and will be destroyed after the final report is released.

The interviews will be recorded. The only identification included on the files will be the student ID. The audio files will be secured in a locked cabinet for the duration of the study and will be destroyed after the final report is released.

Students and parents will be provided with the following confidentiality pledge on the front of the test booklet:

Abt Associates, Inc. is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

Test security will be assured at the administrator, interviewer, and student levels. The interviewer and researcher must each sign a confidentiality agreement. (See appendixes C and D for the interviewer and researcher confidentiality agreements.)

Justification for Sensitive Questions

Throughout the interview protocol development process, effort has been made to avoid asking for information that might be considered sensitive or offensive. Reviewers have identified and eliminated potential bias in questions.

In addition, the cognitive writing item development process included sensitivity reviews before use in previously administered assessments.

Estimate of Hour Burden

A two-stage recruitment effort will be conducted via email and phone. Initial parent contact and response via email is estimated at .05 hours. The follow-up phone call to parents of potential participants and to student participants is estimated at 9 minutes or .15 hours per family. Each interview is expected to take 90 minutes or 1.5 hours. The estimated respondent burden follows:

Respondent	Hours per respondent	Number of respondents	Number of responses	Total Hours
Parent and Student Recruitment
Initial contact	.05	500	500	25
Follow-up via phone	.15	100	100	15
Interviews
Grade 8 students	1.5	20	20	30
Grade 12 students	1.5	20	20	30

Total Burden Hours		500	640	100

Estimate of Costs for Recruiting and Paying Respondents

UserWorks has agreed to recruit respondents. Each participating student will receive a $30 gift card in compensation for time and effort. Because the study will take place outside of regular academic school hours, the monetary incentive is aimed at ensuring participation and motivation on behalf of the students. In addition, we are offering a check of $25 per parent to help offset the travel/transportation costs to bring the participating student to the cognitive laboratory site.

Cost to Federal Government

The following table provides the overall project cost estimates:

Activity	Provider	Estimated Cost
Design, prepare, recruit, conduct cog labs, scoring, analysis, and reporting	NAEP Education Statistics Services Institute (NESSI)	$237,095
Item development–related activities	ETS	$6,000
Configure laptops per the requirements of the Cog Lab study, decouple the two tutorials from the 2010 Writing Computer based assessment, and provide the observable data and raw scoring data after the study	Fulcrum	$7,872
Totals		$250,967

Study Schedule

Activity		Dates
Preparing study design	Review by NCES	April 1, 2010
	Make final revisions	April 15, 2010
Prepare interview protocols	Review by NCES	April 1, 2010
	Make final revisions	April 15, 2010
Item selection	Select NAEP items	April 1, 2010
OMB	Prepare OMB package	April 15–23, 2010
OMB Submission		May 1, 2010
Data collection	Recruit participants	June 1–June 9, 2010
	Data collection	June 10–25, 2010
Data preparation	Enter data into spreadsheets and coding	June 10–July 2, 2009
Data analysis and report	Analysis of data	July 5–14, 2010
	Final study report	August 15, 2010

Volume II of this submission includes the cognitive interview scripts.

1 A small pretest study (< 10 students) will be conducted prior to the writing cognitive interview study to provide feedback on the logistics and procedures in preparation of the actual cognitive interview study described here and to help develop the coding frame for the verbal reports, and to train interviewers and observers.

File Type	application/msword
File Title	April 2 DRAFT
Author	BlairJ
Last Modified By	Ricardo Martinez
File Modified	2010-05-22
File Created	2010-05-22