Volume 1 NAEP 2022 SocSci & Econ Cog Labs

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP) 2022

Social Science and Economics Questionnaire Cognitive Interviews

OMB# 1850-0803 v.241

November 2018

Table of Contents

1) Submittal-Related Information 2

2) Background and Study Rationale 2

3) Recruitment and Data Collection 3

4) Consultations outside the Agency 5

5) Justification for Sensitive Questions 5

6) Paying Respondents 5

7) Assurance of Confidentiality 5

8) Estimate of Hourly Burden 6

9) Costs to Federal Government 6

10) Project Schedule 7

Attachments:

Volume II – Protocols

Appendices – Communication Materials

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey, by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622), of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.

This request is to conduct cognitive interviews with students, teachers, department heads, and school administrators to test survey questionnaire items planned for the 2022 NAEP Social Science and Economics Assessments. Various forms of pretesting, including cognitive interviews, are used to obtain data about new assessment and survey items during the NAEP item development process and the results are used to enhance the efficiency of data collection instruments’ development. Pretesting before piloting helps to identify and eliminate potential issues with items and tasks, often resulting in fewer challenges in scoring and analysis, and leading to higher pilot item survival rates.

Cognitive interviews allow for gathering of qualitative data about how participants work through items and for probing potential sources of construct irrelevance. The objective is to determine whether items appear to elicit targeted knowledge and skills and/or reduce construct irrelevance in the form of either evidence that can be scored or qualitative data consisting of participant responses and reactions.

The 2022 Core, Civics, Geography, U.S. History, and Economics Survey Questionnaires aim to capture data related to important subject-specific and non-subject-specific (Core) contextual factors for student achievement. Table 1 contains the possible areas of focus for the development of upcoming NAEP survey questionnaires.

Table 1. Core Modules and Subject-Specific Issues.

	Core	Civics	Geography	U.S. History	Economics
Module 1/Issue 1	Socioeconomic Status (SES)	Resources for Learning and Instruction	Resources for Learning and Instruction	Resources for Learning and Instruction	Resources for Learning and Instruction
Module 2/Issue 2	Technology Use	Organization and Instruction	Organization and Instruction	Organization and Instruction	Organization and Instruction
Module 3/Issue 3	Perseverance	Teacher Preparation	Teacher Preparation	Teacher Preparation	Student Factors
Module 4/Issue 4	Enjoyment of Difficult Problems	Student Factors	Student Factors	Student Factors	n/a
Module 5/Issue 5	School Climate	n/a	n/a	n/a	n/a

New items related to the issues for core and subject areas have been developed for potential inclusion in the NAEP Core, Civics, Geography, U.S. History, and Economics Survey Questionnaires, and for pre-testing in cognitive interviews.

In cognitive interviews, an interviewer uses a structured protocol in a one-on-one interview drawing on methods from cognitive science. A verbal probing technique will be used for this cognitive interview study, where the interviewer asks probing questions, as necessary, to explore issues that have been identified a priori as being of particular interest. This interview technique has proven to be productive in previous NAEP studies and will be the primary approach in the NAEP cognitive interviews in this study.

Cognitive interview studies produce largely qualitative data in the form of verbalizations made by participants in response to the interviewer probes. Some informal observations of behavior are also gathered, given that typically, in addition to the interviewer, a second observer is present. Behavioral observations may include such things as nonverbal indicators of affect, suggesting emotional states such as frustration or engagement, and interactions with the task, such as ineffectual or repeated actions suggesting misunderstanding or usability issues.

Cognitive interviews are important given that they help us identify potential problems with items, as well as help inform how to improve items. The main purpose of the described here cognitive interviews study is to:

1. Identify potential problems with the items (i.e., ensure the item is understood by all participants, and confirm items are not sensitive in nature or make the participant uncomfortable);

2. Find ways to improve wording of draft items where possible.

The results from this study will also be used to inform which items should be administered during the 2020 pilot assessments for the 2022 Core, Civics, Geography, U.S. History, and Economics Survey Questionnaires. Volume I of this submittal provides descriptions of the design and sampling, as well as burden, cost, and schedule information for the study. Volume II provides the protocols, items, and probes for the core and subject-specific cognitive interviews. The appendices provide notifications, consent forms, screening checklists, informational flyers, phone scripts, and thank you materials.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics

EurekaFacts, an NCES subcontractor for NAEP, will recruit a maximum of 30 students, five teachers, five department heads, and five school administrators to participate in cognitive interviews. Participants will be recruited in the District of Columbia, Maryland, and Virginia areas. In addition, EurekaFacts may also recruit teachers and school administrators in other states represented in their database. No more than three students will be recruited per school, and no more than one teacher or school administrator will be recruited per school.

Although the sample will include a mix of participant characteristics, the results will not explicitly measure differences by those characteristics. Participants will be recruited to obtain the following criteria:

Students:

• Students who are enrolled in 8^th or 12^th grade for the 2018-2019 school year;¹

• A mix of gender;

• A mix of race/ethnicity (Black, Asian, White, Hispanic, etc.);

• A mix of socioeconomic backgrounds; and

• A mix of English language learner statuses

Teachers, department heads, school administrators:

• A mix of school sizes; and

• A mix of school demographics

Please note that SES will be given a higher priority than other respondent characteristics when recruiting while also ensuring sufficient balance of other criteria. The subcontractor will document the information collected in the screeners using a tracking sheet, which will be used to determine the targeted sample including diversification on key characteristics (see Appendix X). Additionally, it should be noted that the sample is not large enough to support subgroup analyses. Table 2 summarizes the numbers and types of cognitive interviews that are planned. A minimum number of five respondents per subgroup is recommended to identify major problems with an item and for a meaningful analysis of data from exploratory cognitive interviews.² Students will be oversampled to better ensure identification of confusion or sensitivity issues.

Table 2. Sample Size for Core and Subject-Specific Cognitive Interviews³

Respondent Group	Grade 8	Grade 12	Total
Students	10	20	30
Teachers	5	n/a	5
Department Heads	n/a	5	5
School Administrators (Non-Charter School)	--	5	5
Overall Total	15	30	45

While EurekaFacts will use various outreach methods (see Appendices) to recruit participants, the bulk of the recruitment will be conducted by telephone and will be based on acquisition of targeted mailing lists containing residential addresses and landline and cellular telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students 18 years of age or over (see Appendix G); parents of students under 18 years of age (see Appendix D); and teachers/department heads/school administrators (see Appendix R) will be screened to ensure that the recruited individuals meet the criteria for participation in the study (i.e., that the participants are from the targeted demographic groups outlined above). When recruiting participants, EurekaFacts staff will first speak to the student (if 18 years or older); the parent/legal guardian of an interested minor; or the teacher/department head/school administrator before starting the screening process. During this communication, the participant or the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that an individual is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written, informed parental consent and consent of participants age 18 or over (see Appendices L, M, and U) will be obtained for all respondents who are interested in participating in the study.

One-hour (60 minutes)⁴ cognitive interviews will be conducted with students, teachers, department heads, and school administrators (specifically principals). All student cognitive interviews and the majority of teacher, department head, and school administrator cognitive interviews will be conducted in person (teacher/department head/administrator interviews may be conducted via phone or WebEx if needed).

Data Collection Process

Cognitive interviews will take place at a range of suitable venues. EurekaFacts will conduct cognitive interviews at their Rockville, Maryland site or other sites such as after-school activity organizations and community-based organizations. In all cases, a suitable environment, such as a quiet room, will be used to conduct the interviews, and there will be more than one adult present. Each cognitive interview session will last 60 minutes.

Each participant will first be welcomed by staff, introduced to the interviewer and the observer, and told that s/he is there to help answer research questions about how people answer survey questions. Then, the interviewers will explain the cognitive interview process.

Protocols for cognitive interviews will include probes for use as students work through item sets, and also probes for use after students finish answering items (see Volume II). Probes will include a combination of pre-planned questions identified before the session and ad hoc questions that the interviewer identifies as important from observations during the interview, such as clarifications or expansions on points raised by the student. For example, if a student paused for a long time over a particular item, appeared to be frustrated at any point, or indicated an “aha” moment, the interviewer might probe these kinds of observations further, to find out what was going on. To minimize the burden on the student, efforts will be made to limit the number of verbal probes that can be used in any one session or in relation to any set of items. The welcome script, cognitive interview instructions, and hints for the interviewers are provided in Volume II.

The cognitive interviews will be audio recorded. Interviewers will also record their own notes separately, such as behaviors (e.g., “the participant appeared confused”), questions posed by students, and observations of how long various items take to complete.

The types of data collected will include:

• student reactions to and responses to items;

• behavioral data (e.g., actions observable from interviewer notes, process data, screen-captures, gaze patterns where collected);

• responses to generic questions;

• responses to targeted questions specific to the item(s);

• additional volunteered participant comments; and

• answers to debriefing questions.

Analysis Plan

After the session, the notes and audio recording will be summarized to report main findings and illustrative statements that will be analyzed by the NAEP questionnaire development team. The cognitive interview results in 2019 will be used to improve the tested items and inform which items should be administered during the 2020 pilots.

4. Consultations outside the Agency

Educational Testing Service (ETS) is the item development, data analysis, and reporting contractor for NAEP and will develop the items, analyze results, and draft a report with results. EurekaFacts, a research and consulting firm based in Rockville, MD, a subcontractor for ETS, will recruit participants and administer the cognitive interviews. The NAEP State Coordinators serve as the liaisons between state education agencies and NAEP, coordinating NAEP activities within their respective states. NAEP State Coordinators from selected states may provide leads for potential participants for this study—including states where interviews are conducted remotely with teachers, department heads, and school administrators.

5. Justification for Sensitive Questions

Throughout the item and interview protocols development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondent

To encourage participation in a 60-minute cognitive interview session, a $25 gift card from a major credit card company will be offered to each student who participates as a thank you for his or her time and effort. If a parent or guardian provides transportation for a student, they will be offered a $25 gift card from a major credit card company to thank them for bringing the participating student to and from the testing site. To recruit the needed teachers and school administrators, especially given their busy schedules and the short duration of the study, we will offer them a $60 gift card from a major credit card company for participating in a cognitive interview session, as a thank you for their time and effort.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, participants will be notified that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Before students, teachers, department heads, or school administrators can participate in the study, written consent will be obtained from participants 18 years of age or older and from the parents or legal guardians of students less than 18 years of age. Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Cognitive interviews may be audio recorded. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.⁵ All cognitive interviews will be scheduled for no more than one hour (60 minutes). Table 3 details the estimated burden for the survey questionnaire cognitive interview processes.

Table 3. Hourly Burden for Students, Teachers, Department Heads, and School Administrators for Core and Subject-Specific Cognitive Interviews

Respondent	Number of Respondents	Number of Responses	Hours per Respondent	Total Hours
Student Recruitment via Teachers and Staff
Initial contact	30	30	0.05	2
Follow-up and identify students	20*	20	1.0	20
Sub-Total	30	50		22
Parent or Legal Guardian for Student Recruitment
Initial contact	150	150	0.05	8
Follow-up via phone	100*	100	0.15	15
Consent and confirmation	50*	50	0.15	8
Sub-Total	150	300		31
Teacher, Department Head, and School Administrator Recruitment
Initial contact	120	120	0.05	6
Follow-up via phone or e-mail	80*	80	0.15	12
Consent and confirmation	40*	40	0.15	6
Sub-Total	120	240		24
Participation (Cognitive Interviews)
Students	30	30	1.0	30
Teachers	5	5	1.0	5
Department Heads	5	5	1.0	5
School Administrators	5	5	1.0	5
Sub-Total	45	45		45
Total Burden	345	635		122

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

9. Costs to Federal Government

The total cost of the study is $275,000 as detailed in Table 4.

Table 4. Costs to the Federal Government

Activity	Provider	Estimated Cost
Design and prepare for cognitive interviews; analyze findings and prepare report	ETS	$ 25,000
Prepare for and administer cognitive interviews (including recruitment, incentive costs, data collection, analysis, and reporting)	EurekaFacts	$ 250,000

10. Project Schedule

The schedule for this study, including all activities, is provided in Table 5.

Table 5. Project Schedule

Activity Each activity includes recruitment, data collection, and analyses	Dates
Cognitive interviews	November 2018January 2019
Cognitive interview report submitted	March 2019

1 Please note that this study seeks to recruit five students who have taken (or are currently taking) economics, and a minimum of three is required.

2 Roach, A. T., & Sato, E. (2009). White paper: Cognitive interview methods in reading test design and development for alternate assessments based on modified academic achievement standards (AA-MAS). Dover, NH: Measured Progress and Menlo Park, CA: SRI International.

3 There are no teacher surveys for grade 12.

4 Please note that the 60 minutes includes time for introductions, conducting the interview, debriefing, and/or time for additional questions/feedback from the participants.

5 Based on our experiences in other similar NAEP studies, the estimated attrition rates for direct participant recruitment are 33 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 40 percent from confirmation to participation for students and 62.5 percent from confirmation to participation for teachers, department heads, and school administrators. The estimated attrition rate for the initial principal contact for student identification is 33 percent from contact to follow-up.