Justification

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct cognitive interviews to test new data collection procedures and survey items with participants in the foundational phase of the National Study of Special Education Spending. Participants will include up to 60 special education teachers or other staff knowledgeable about students’ individualized education programs (IEPs) and up to 30 interviews with local education agency (LEA) administrators (e.g., directors, assistant directors, coordinators, student services coordinators, or special education directors). Cognitive interviews will focus on developing items for two surveys that measure the types and amounts of special education services provided to students and the resources used to provide those services. The interviews also will focus on identifying best reporters for survey topics, recognizing salient themes to use in drafting communication materials, and collecting feedback about a student needs assessment embedded as part of the special education teacher–student resource survey. Interviews will inform the creation of final survey instruments for a future pilot test and will begin in January 2024.

Background

The field has a critical need for current information about what is spent to provide special education and related services to students with disabilities (SWDs) and about the sources of funding used to pay for this spending. Nearly 7 million SWDs—about 14% of students enrolled in public K–12 education—receive special education services under the Individuals with Disabilities Education Act (IDEA; NCES, n.d.). Providing special education involves a considerable educational investment. In 2021, the federal government allocated more than $13 billion in IDEA grant awards to states to provide special education services to school-age students (U.S. Department of Education, 2023). However, policymakers do not have access to accurate, up-to-date information about how this money is spent to educate SWDs (National Council on Disability, 2018). Federal policymakers need national spending estimates so that they can make informed decisions about how much to appropriate for IDEA, including full funding.

Recurring national school finance surveys conducted by the U.S. Department of Education (ED) do not collect data in sufficient detail to develop national estimates of special education spending (Griffith, 2016; Kolbe, 2019). Instead, the most recent spending estimates are derived from the Special Education Expenditure Project (SEEP), conducted by the American Institutes for Research^® (AIR^®) for ED during the 1999–2000 school year. Now more than 20 years old, the SEEP estimates are insufficient to guide contemporary policy decisions. Since SEEP, significant changes, including the following, have occurred in the special education landscape that affect what is spent to educate SWDs:

Changes to the prevalence and characteristics of the SWDs who receive special education services and their overall level of need (National Center for Education Statistics, 2023)
Shifts in where and how SWDs are served, particularly with respect to the amount of time SWDs spend in general education classrooms (McFarland et al., 2019)
Updated federal and state policies and procedures that prioritize early intervention and shared responsibility between general and special education for educating SWDs (Bailey, 2019, 2021)
Increased expectations that IEPs meet the substantive standard set by the Endrew F. v. Douglas County School District (Re-1, 137 S. Ct. 988; Library of Congress, n.d.) and are aligned to general education content standards (Harr-Robins et al., 2013)
Opportunities for SWDs to attend charter schools (Rhim et al., 2019)

With the time elapsed since SEEP and changes in the special education landscape, we need a new study that generates up-to-date estimates for what is spent to educate SWDs and that addresses policy-relevant questions. The cognitive interviews included in this OMB package lay the foundation for this study by refining and testing approaches to collecting valid and reliable information about what states, districts, and schools spend to provide special education services to SWDs. Two survey instruments will be the subject of interviews, the Special Education Teacher Student Resource Survey and the LEA Student Resource Survey (Attachment 1). The study will use results from the three rounds of cognitive interviews to create final survey questionnaires for a future pilot test. Interviews also will inform future recruitment activities and identify initial points of contacts for survey response.

Design and Context

AIR will use cognitive interviews to develop survey items that measure the types and amounts of services provided to special education students and the resources used to provide those services. Cognitive interviews are intensive, one-on-one interviews, during which the interviewer asks the respondent to read the material and think aloud as they reflect on the information read and answer survey questions. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments and responding to scenarios.

The instruments that will be tested in the cognitive interviews contain items from several sources: instruments from the prior study of special education spending, the SEEP; newly developed items to measure the types and amounts of services students receive; and items assessing student need, adapted from existing instruments, such as the ABILITIES Index (Simeonsson & Bailey, 1991). Because of changes in special education programming since the 1999–2000 school year, the SEEP survey instruments do not include all the needed items to measure spending on special education services and include some items that are no longer applicable. In addition, there are concerns about whether some of the SEEP items will collect valid and reliable data in a future collection, especially those that require providers to recall information about services delivered. AIR conducted an item-by-item review of the SEEP instruments and selected candidate items for a future study and dropped items that were no longer relevant or necessary. To fill gaps in the SEEP items, AIR developed new items that will be needed for a future collection. The SEEP survey instrument for special education teachers also embedded a subset of items from an existing instrument that measures student need, the ABILITIES Index, which AIR is also reviewing for relevance. The SEEP items, the items assessing student need, and the newly developed items need to be tested with current educators and revised, as necessary.

Attachment 1 includes a table of draft survey items that will be tested in the cognitive interviews. Attachment 1 also includes a student needs assessment as part of the special education teacher–student resource survey. The AIR study team currently proposes the ABILITIES Index, which was used in SEEP, but we may also test an alternative needs assessment. Additional items may be pulled as needed across interview rounds from the SEEP instruments found at https://www.csef-air.org/about_seep_instruments.html.

AIR will conduct interviews in three rounds, with different foci for each round (see Exhibit 1). Attachment 2 includes the interview protocols that will be used for the interviews with teachers and administrators for each round.

Exhibit 1. Summary of Cognitive Interview Rounds Foci

Shape1

Recruitment and Data Collection

AIR will recruit cognitive interview participants for the special education teacher–student resource survey and the LEA student resource survey through convenience sampling, relying on existing contacts with LEAs and through personal and professional contacts.

Respondents for recruitment will include the following:

Special education teacher–student resource survey: A student case manager or primary provider(s) with knowledge of the special education and related services an SWD receives and with knowledge of who the best reporters are for the survey topics
LEA student resource survey: LEA student services coordinators or special education directors

The goal of the recruitment efforts is to identify respondents from a range of contexts that may use different terminology to talk about the same special education services. Participants will be recruited from a range of locales, grades taught, and experiences with students of different disability types (Exhibit 2).

Exhibit 2. Expected Number of Cognitive Interview Participants by Survey and Key Recruitment Characteristics

	Number of Cognitive Interview Participants
Key Recruitment Characteristics	Special Education Teacher–Student Resource Survey	LEA Student Resource Survey
Total	60	30
Locale
City, large	12	6
City, midsize	12	6
Suburb, large	12	6
Suburb, midsize	12	6
Rural	12	6
School type (for special education teacher–student resource survey)
Regular public school	48	NA
Charter school	12	NA
Disability category groupings ^a
High-need disabilities^b	18	NA
Low-need disabilities^c	18	NA
Sensory or physical disability^d	24	NA
Grade
K–5	18	NA
6–8	18	NA
9–12	24	NA

Note. NA = Not applicable.

^aBroad disability categories will be used to ensure that interviews with teachers include focal students with a range of needs. Specific disability categories will be collected during the interviews.

^bAutism, intellectual disability, multiple disabilities, traumatic brain injury

^cSpecific learning disability, speech/language impairment, other health impairment, emotional/behavioral disorder

^dVisual impairment, including blindness, deaf-blindness, deafness, hearing impairment, orthopedic impairment

The study team will email each potential respondent and provide information about the interviews, their purpose, and the time commitment. The interviews will be conducted remotely via video conference (using the web-conferencing platform Zoom). If a respondent prefers that the interview be conducted over the phone, we will accommodate that preference, but we will encourage video conferencing because nonverbal cues are important when conducting cognitive interviews.

Estimated Respondent Burden

Exhibit 3 details the respondent burden for the cognitive interviews. The recruitment screener will require 3 minutes per recruitment screening. We anticipate eight screenings will be required per eligible participant (thus, an estimated 720 screenings to yield 90 participants). Interview respondents will participate in a 15-minute preinterview activity to set up the technology for the virtual interview and a 1-hour cognitive interview. These respondent activities will result in an estimated total of 148.5 hours of respondent burden for this study.

Exhibit 3. Estimated Response Burden

Activity	Number of Respondents	Minutes per Respondent	Total Burden in Hours
Recruitment screener	720	3	36
Preinterview activities	90	15	22.5
Cognitive interviews	90	60	90
Total	720	NA	148.5

Note. NA = Not applicable.

Estimate of Costs for Recruiting and Paying Respondents

There is no direct cost to respondents. To encourage respondents to participate in the interviews and to thank them for their time, AIR will offer each respondent a $50 Amazon e-gift card. During the recruitment screener, AIR will inform the participant that they will receive payment if they successfully participate in the cognitive interviews. At the end of the interview, AIR will email the e-gift card and acknowledgement form. The proposed amount for these payments ($50) is within the incentive guidelines outlined in the March 22, 2005 memo “Guidelines for Incentives for NCEE Evaluation Studies” prepared for OMB and is consistent with the updated incentive guidance for the REL program outlined by NCEE’s Commissioner in spring 2022.

Cost to Federal Government

The total cost to the federal government for conducting the cognitive and usability interviews will be $190,772. This cost includes training, recruitment, conducting the interviews, analysis, and reporting conducted by AIR.

Assurance of Confidentiality

Participation is voluntary, and respondents will read a confidentiality statement prior to beginning the interview and completing the draft survey items. In addition, participants will receive, via email, a consent form that explains the purpose and duration of the interview. Participants will sign the consent form electronically prior to their interview. No personally identifiable information will be maintained after the cognitive interview analyses are completed.

Participants will be assigned a unique identifier, which will be created solely for data file management and used to keep all participant materials together. The interviews will be audio- and video-recorded. The recorded files will be secured for the duration of the full study and will be destroyed after the final reports are completed.

Key project staff may observe interviews by silently listening to and watching the Zoom interview. The project staff will not be visible to the participant, but the interviewer will inform the participant that someone is observing the interview. At the end of the interview, the interviewer will give the observer the option to ask the participant questions.

Justification for Sensitive Questions

The survey instruments do not include questions that might be considered sensitive.

Project Schedule

The project schedule (Exhibit 4) calls for recruitment to begin as soon as OMB approval is received. Interviewing is expected to be completed within 6 months of OMB approval.

Exhibit 4. Schedule for Cognitive Interviews

Activity	Start Date	End Date
Recruitment and scheduling of participants	1/2/2024	5/1/2024
Cognitive interviews: Round 1	1/2/2024	2/21/2024
Cognitive interviews: Round 2	2/26/2024	4/3/2024
Cognitive interviews: Round 3	4/8/2024	5/10/2024
Summary of refinement and testing of approaches	4/8/2024	5/31/2024

References

Bailey, T. R. (2019, September 20). Is MTSS the new RTI? Depends on where you live. American Institutes for Research. https://mtss4success.org/blog/mtss-new-rti-depends-where-you-live

Bailey, T. R. (2021, August 26). We’ve got this! Lessons learned after 20 years of MTSS implementation. Presented at the MTSS and PBIS First Virtual Conference, Atlanta, GA.

Griffith, M. (2016, July 27). Do we spend too much on special education? EdNote. https://ednote.ecs.org/do-we-spend-too-much-on-special-education-in-this-country/

Harr-Robins, J., Song, M., Hurlburt, S., Pruce, C., Danielson, L., & Garet, M. (2013). The inclusion of students with disabilities in school accountability systems: An update (NCEE 2013-4017). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/pubs/20134017/pdf/20134017.pdf

Kolbe, T. (2019). Funding special education: Charting a path that confronts complexity and crafts coherence. National Education Policy Center. http://nepc.colorado.edu/publication/special-ed

Library of Congress. (n.d.). National Center for Learning Disabilities (NCLD). [Web Archive]. https://www.loc.gov/item/lcwaN0018878/

McFarland, J., Hussar, B., Zhang, J., Wang, X., Wang, K., Hein, S., … Barmer, A. (2019). The condition of education 2019 (NCES 2019-144). U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019144

National Center for Education Statistics. (n.d.). Fast facts: Students with disabilities. U.S. Department of Education, Institute of Education Sciences. https://nces.ed.gov/fastfacts/display.asp?id=64

National Center for Education Statistics. (2023). Students with disabilities. U.S. Department of Education, Institute of Education Sciences. https://nces.ed.gov/programs/coe/indicator/cgg

National Council on Disability. (2018). Broken promises: The underfunding of IDEA (IDEA Series). https://ncd.gov/sites/default/files/NCD_BrokenPromises_508.pdf

Rhim, L. M., Kothari, S., & Lancet, S. (2019). Key trends in special education in charter schools in 2015–2016: Secondary analysis of the Civil Rights Data Collection. National Center for Special Education in Charter Schools. https://eric.ed.gov/?id=ED604728

Simeonsson, R.J., & Bailey, D.B. (1991). The ABILITIES Index. Frank Porter Graham Child Development Center, University of North Carolina at Chapel Hill. https://fpg.unc.edu/publications/abilities-index

U.S. Department of Education. (2023, October 2). Fiscal years 2022–2024 state tables for the U.S. Department of Education. https://www2.ed.gov/about/overview/budget/statetables/index.html

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	McQuiggan, Meghan
File Modified	0000-00-00
File Created	2024-07-22