National Center for Education Statistics
National Assessment of Educational Progress
National Assessment of Educational Progress (NAEP) Engagement Augmentation Study
Volume I
Supporting Statement
OMB# 1850-0803 v.279
December 2020
1) Submittal-Related Information 3
2) Background and Study Rationale 3
3) Recruitment and Data Collection 4
4) Consultations Outside the Agency 7
5) Justification for Sensitive Questions 7
6) Assurance of Confidentiality 7
7) Estimate of Hourly Burden 8
8) Estimate of Costs for Paying Respondents 7
9) Costs to Federal Government 8
Volume II — Protocols
Appendices — recruitment and communication materials.
This material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to conduct various procedures (such as field tests and cognitive interviews) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments.
The National Assessment of Educational Progress (NAEP) is a federally authorized survey, by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622), of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.
Since students do not receive a NAEP score, the assessment is considered “low-stakes” for students. Due to the low-stakes assessment context in NAEP, there are long-standing questions about the level of engagement and effort of the participating students and, consequently, about the validity of the reported results. For assessment results to be valid, students should be sufficiently engaged with the assessment tasks to reflect their knowledge and skills accurately. In October 2018, ACT, Inc. (ACT) submitted a proposal to investigate student engagement in NAEP and test potential enhancements to increase student engagement. The proposal was accepted for a cooperative agreement with NCES and American Institutes for Research (AIR), which began in 2019. As part of the cooperative agreement NCES shared six pilot Interactive Computer Tasks (ICT) in Science for Grades 4 and 8, which the ACT project team analyzed to identify potential concerns for student engagement (see NAEP documentation on sample tasks; explore sample tasks at https://www.nationsreportcard.gov/science_2009/ict_tasks.aspx). Following the analysis, the ACT team met with NCES and AIR and identified one Grade 8 task to target for enhancement to be developed and assessed with a group of students in cognitive interviews. Modifications to the original task/items include the following key aspects:
Rephrase the prompt to give more agency to students in learning through data collection and designing the investigation
Reduce reading demand and time spent (while not substantially changing the evidence for student understanding)
Increase metacognitive reflection by providing indirect feedback
Allow students to plan in order to increase agency and metacognitive reflection on their actions
This submittal requests clearance for virtual cognitive interviews of one digitally delivered pilot Science ICT from the tasks shared with ACT, consisting of multiple choice and constructed response items and the corresponding modified task that ACT has developed to try to increase student engagement.
Included in the submittal are:
Volume I — supporting statement that describes the design, data collection, burden, cost, and schedules of the study
Volume II — protocols and questions used in the cognitive interviews
Appendices — recruitment and communication materials.
ACT is the lead contractor for the virtual cognitive interviews. An overview of the virtual cognitive interviews is presented on the following page.
In virtual cognitive interviews (often referred to as a cognitive laboratory study or cog lab), an interviewer uses a semi-structured protocol in a one-on-one interview (see Volume II for protocols). Cognitive interview studies are largely aimed to provide qualitative data collection based on verbal reports in response to questions. For this study students will complete one or two versions of a NAEP Science ICT pilot assessment task:
Task A: The original pilot task presented using Unity through Internet Explorer (hosted on the local drive of the ACT laptop and made available to a student through Zoom remote control feature that allows the student to take control of the interviewers’ screen in a meeting1)
Task B: An augmented version of the task presented using a similar interface through PowerPoint with clickable buttons for multiple choice items and textboxes for constructed response items (hosted on the same ACT laptop and made available to a student via Zoom).
After completing each task, the interviewer will ask the student a few questions to gather information about student reactions to the task. The largely qualitative data collected will be mainly verbal reports in response to these questions, in addition to volunteered comments. Quantitative data will also be collected on student response and process data while completing the tasks.
Participating students will be randomly assigned into one of the following three groups in order to address possible effects of recall and fatigue:
Group AB (experimental condition): Students complete Task A, respond to questions from the interviewer on the task, followed by Task B and interview questions on the second task.
Group BA (experimental condition): Students complete Task B, respond to questions from the interviewer on the task, followed by Task A and interview questions on the second task.
Group B (control condition): Students complete Task B and respond to questions from the interviewer on the task,
The objective of this cognitive interview study is to assess the nature of student engagement and motivation during different types of activities and assessment items within the NAEP Science ICT. The original and augmented tasks are each estimated to take students approximately 15-minutes. The information gained from this study will be used by ACT to generate reports and recommendations to share with NCES and AIR as part of the cooperative agreement.
ACT will recruit a sample of 87 students in Grade 8 for one-round of virtual cognitive interviews led by the Project Director and the key personnel experienced in conducting cognitive interviews in the Programme for International Student Assessment (PISA) and projects at ACT. Although the sample will include a mix of student characteristics, as listed below, the results will not explicitly measure differences by these characteristics and will not aim to be nationally representative:
A mix of race/ethnicity (Hispanic or Latino, White, Black or African American, Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander), and
A mix of gender.
Ultimately targeting a sample of 75 students for this study (Group AB=30, Group BA=30, Group B=15), the project team will apply an inclusive approach and will extend the participation to 87 students to account for possible technical difficulties that might arise during the virtual cognitive interviews and solutions to these difficulties (see Option B in Volume II, for further details). Data from students who experienced technical difficulties during the virtual cognitive labs will not be included in the main analysis of the study and will be used as appropriate to enrich the sample of student responses (i.e., responses provided by the student prior to the occurrence of technical difficulty).
Students will be recruited with parents’ consent through an advertisement (see Appendix A) posted in one or more of the following:
Classified ads (e.g., local newspapers, Craigslist),
Social media sites (e.g., Facebook, NextDoor, Instagram), and
Newsletters through mailing lists to individuals and organizations (e.g., schools, after-school programs) with previous relationships with ACT.
Recruitment materials will include a direct link to a webpage with the screening survey included, and also provides contact information to reach ACT research staff via email and phone for further information, if needed. During this communication, the parent/guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails (see Appendix B for script). After confirmation that participants are qualified, willing, and available to participate in the research project, they will receive a confirmation e-mail/letter that includes a copy of the consent form (see Appendix C). Because all participating students will be under the age of 18, a parent/guardian will be provided documents for informed consent (see Appendix D) in advance of participation. After the interview, the parent/guardian of the student who participated will receive a thank you email that contains the participants’ e-gift card information (see Appendix E). See Appendix A through E for all recruitment and communication materials.
The template for the virtual cognitive interviews is contained in Volume II of this submittal. The template includes:
welcome, thank you, introductory remarks,
generic version of the cognitive interview scripts, and
closing remarks, thanks for participating.
ACT will conduct the virtual cognitive interviews based on the protocol structures described in Volume II. Protocols for the interviews have been developed based on cognitive lab materials from NCES as well as prior methods used within ACT, with appropriate adaptations for the purpose of this study.
Following parents’ consent (written and then verbal), student participants will join via Zoom link from their homes. They will be welcomed, introduced to the interviewer, and told they are there to help answer questions about how students work through assessment tasks. Student participants will be reassured that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C. §9573 and 6 U.S.C. §151]. At this time, participants will also be reminded that the virtual interview will be audio- and screen-recorded. No video recording of the participants and no names will be collected during Zoom virtual interviews. The students will be asked for their verbal consent before the interview continues.
The interviews will focus on the following questions:
Can the student determine what is being asked of her/him at each point in the task?
What is the student’s overall impression of the task, especially with regard to engagement (e.g., relevance, authenticity, cognitive complexity, self-assessment)? Is the content and/or context understandable and interesting?
Is the student directing her/his attention appropriately at each stage of the task (where is the student’s focus directed at each stage)?
The interviews will include general questions about the clarity of the items following the protocol (see Volume II).
The interviews will follow these steps:
The interviewer will ask the student to complete the first ICT task assigned to the group.
After the student has completed the first ICT task, or after the allotted amount of time has passed (15-minutes), the interviewer will begin to collect information through questioning the student directly. Initially, the student will be asked to report on their engagement and enjoyment of the task. Additional specific questions may follow based on the student’s responses to the open-ended questions.
The interviewer will ask the student to complete the second version of the ICT task (for Groups AB and BA only).
After the student has completed the second ICT task, or after the allotted amount of time has passed (15-minutes), the interviewer will begin to collect information through questioning the student directly. The interviewer, using their judgment based on experience, will note pertinent aspects of the interview process, such any special circumstances that might affect the interview.
As the student is providing information during the session, the interviewer will take notes to record data during the interview. The generic script developed by ACT will provide consistency in gathering data and will aid in data analysis (see example generic scripts in Volume II).This process will not focus on whether the student produced a correct or incorrect answer for each question, but instead, on how the student engaged with the task as they worked through the simulations and items.
As part of the process, student’s response data will be captured by using Zoom recording that captures screen display and audio in support of data analysis. No video recording of the participants and no names will be collected via Zoom. Students will not be given the correct answers by the interviewer as this may impact any subsequent questions or interviews. At the analysis stage, student response data may be used to inform judgments of engagement in the context of process data (e.g., fast response times with correct versus incorrect responses could reflect an easy item versus low effort), but will not be used to estimate the student’s science knowledge.
After the interview concludes, the interviewer will input their notes from the session within 48 hours and submit them to the interview manager.
For the cognitive interview data collections, the key units of analysis will be at the level of task as well as scene/item. The types of data collected about the tasks and items will include: responses to general questions about student reactions to the item; responses to targeted questions specific to the item or task; additional volunteered participant comments; and process and/or response data to items within the tasks.
The deliverable from the analyses of the cognitive interviews will be in the format of a report. The report will contain results for each item and a summary of results. The report will be delivered along with Excel files containing student responses ACT collected in interviewer notes. The general analysis approach will be to compile the data gathered through the interviewer note-taking sheet to present the data at the item-level. The presentation of the report will facilitate the identification of patterns of responses for each item.
The report will be shared with AIR and NCES.
AIR is an established not-for-profit research organization. AIR is coordinating this project between ACT and NCES. AIR headquarters is in Arlington, VA.
Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.
Each participating student will receive a $25 electronic gift card as a thank you for his or her time and effort. A generic gift card (e.g., Visa gift card) that can be used anywhere credit cards are accepted is the recommended incentive.
Participants will be notified that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573 and 6 U.S.C. §151)].
Written consent will be obtained from legal guardians of all students before interviews are conducted. Interview sessions will be screen and audio recorded via Zoom to assist with post-interview note-taking. Information about screen and audio recording will be included in the parent consent form (see Appendix D). No video recording of the participants and no names will be collected via Zoom. Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study and after the completion in a locked file cabinet, and will be destroyed based on guidance from NCES. Screen and audio recordings will be stored in a restricted-access folder in ACT’s network and will be deleted after the project concludes.
Based on the proposed outreach and recruitment methods, we estimate initial respondent burden at 0.15 hours through email and screening survey. We are estimating a recruitment of 97 participants for the target 87 students to participate and then 75 usable cognitive interviews. Interviews will take place via Zoom and will last no more than 75-minutes per student, including timing for students to ensure their ability to join the virtual interview (for example, testing their interview link or joining their interview a few minutes early).
Table 1 details the estimated burden for the cognitive interviews.
Table 1. Estimate of Hourly Burden
Respondent |
Number of respondents |
Number of responses |
Hours per respondent |
Total hours (rounded up) |
Parent or Legal Guardian for Student Recruitment |
||||
Recruitment, screening, and inviting |
97 |
97 |
0.15 |
15 |
Participation (Cognitive Interviews) |
||||
Students |
87 |
87 |
1.25 |
109 |
Total Burden |
184 |
184 |
|
124 |
The anticipated cost of $2,175 of the electronic gift cards is part of the approved budget in this cooperative agreement. Gift cards will be purchased and provided to participating students by ACT.
The total anticipated cost of labor hours for ACT research staff associated with recruitment, training, interviews, analysis and reporting is $93,333. Combining the two means that the total estimated cost to the federal government is $95,508.
The timeline for the virtual cognitive interviews is listed in Table 2 on the following page. We estimate up to 4 weeks to complete recruitment for the interviews and up to 9 weeks to complete the 87 interviews. Following this timeline, interviews would conclude in March 2021. Data analysis and report are anticipated to be completed and submitted to NCES in May 2021.
Table 2. Tentative Project Schedule
Activity |
Dates |
Recruitment |
December 2020 |
Cognitive Interviews |
January 2021-March 2021 |
Analysis and Report Submitted |
April 2021-May 2021 |
1 For technical details on Zoom remote control feature, see: https://support.zoom.us/hc/en-us/articles/201362673-Requesting-or-giving-remote-control
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Vanessa Simmering |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |