Eval of Preschool Spec Ed Practices Phase I_Supporting Statement Part A 12-22-14

D
RAFT REPORT

Supporting Statement for the Paperwork Reduction Act: Submission for the Evaluation of Preschool Special Education Practices, Phase I – PART A

November 24, 2014

Submitted to:

Institute of Education Sciences

555 New Jersey Ave., NW, Room 502J

Washington, DC 20208

Project Officer: Yumiko Sekino
Contract Number: ED-IES-14-C001

Submitted by:

Mathematica Policy Research
600 Alexander Park, Suite 100

Princeton, NJ 08540
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director: Cheri Vogel
Reference Number: 40346.052

CONTENTS

SUPPORTING STATEMENT PART A. JUSTIFICATION FOR THE STUDY 1

A. Justification 2

A1. Circumstances necessitating collection of information 2

A2. How, by whom, and for what purposes the information will be used 4

A3. Use of technology to reduce burden 8

A4. Efforts to avoid duplication 8

A5. Methods to minimize burden on small entities 9

A6. Consequences of not collecting data 9

A7. Special circumstances 9

A8. Federal Register Announcement and Consultation

A9. Payments or gifts 10

A10. Assurances of confidentiality 10

A11. Justification for sensitive questions 11

A12. Estimates of hours burden 12

A13. Estimates of cost burden to respondents 12

A14. Annualized cost to the federal government 12

A15. Reasons for program changes or adjustments 12

A16. Plans for tabulation and publication of results 13

A17. Approval not to display the expiration date for OMB approval 18

A18. Explanation of exceptions 18

TABLES

A.1. Research questions, study objectives, and data sources 6

A.2. Evaluation of preschool special education practices, phase I, data collection plan 7

A.3 Burden associated with data collection 12

A.4. Timeline for data collection and reporting 17

APPENDICES

Appendix A Mathematica Confidentiality Pledge

Appendix B Advance letter to the Chief State School Officer

Appendix C Advance letter to the State Section 619 coordinator

Appendix D Advance letter to the district special education coordinator

Appendix E State Section 619 Coordinator Survey Instrument

Appendix F District Preschool Special Education Coordinator Survey Instrument

SUPPORTING STATEMENT PART A. JUSTIFICATION FOR THE STUDY

The U.S. Department of Education (ED) is requesting Office of Management and Budget (OMB) approval for survey data collection as part of the Evaluation of Preschool Special Education Practices, Phase I. The main objective of the Phase I study is to assess the feasibility of conducting a large-scale randomized controlled trial (RCT) evaluation of one or more curricula or interventions that are used with preschool children with disabilities to promote their learning of language, literacy, social-emotional skills, and/or appropriate behavioral skills for school. The secondary objective of the Phase I study is to provide educators and policymakers with nationally representative descriptive information about current preschool special education programs. If the RCT is deemed feasible and ED decides to exercise the Phase II option, a separate OMB package will be submitted for the RCT.

The feasibility assessment will consider the core features of an evaluation design, including the following:

• Curricula and/or interventions to be evaluated.

• Study context and participants.

• Key design elements, such as the counterfactual condition, unit of assignment, target minimum detectable effects (MDEs), sample size, and data collection plans.

Data to inform the feasibility assessment will be obtained through surveys of school district preschool special education coordinators and state Section 619 coordinators, the subject of the current submission, as well as extant data collection and an evidence review. Extant and survey data will provide information about preschool special education programs and the curricula and interventions that are available and supported by them to identify potential target districts for a possible RCT evaluation. Specifically, extant and survey data will describe the context in which curricula and interventions are delivered as well as provide information to make decisions about key design elements. Mathematica will collect the extant data. Survey respondents will not be asked to provide or confirm the data collected by Mathematica. The evidence review will identify promising curricula and interventions for preschool children with disabilities and features about their implementation in schools.

The study’s overarching research question is whether there are promising curricula and interventions for preschool children with disabilities for which a large-scale effectiveness trial would be feasible and add value to the field. The survey data collection will address the following ten specific research questions that represent critical information gaps for making a feasibility assessment (that is, needed information that cannot be obtained through either the evidence review or extant data):

1. What curricula and interventions are available and supported for use with preschool children with disabilities to promote learning of language, literacy, social-emotional skills, and/or appropriate behavioral skills for school?

2. How are decisions to adopt curricula and interventions made?

3. What agencies, programs, and settings serve preschool children with disabilities?

4. What is the structure of programs that serve preschool children with disabilities?

5. What resources support providing services to preschool children with disabilities?

6. What are the characteristics of staff who deliver services to preschool children with disabilities?

7. What are turnover rates for staff who deliver services to preschool children with disabilities?

8. What are eligibility rules for preschool special education curricula and interventions?

9. What are the enrollment characteristics of preschool children with disabilities and classrooms that include these children?

10. What curricula and interventions for children ages 3 to 5 with disabilities might be suitable for study in a large-scale RCT?

Preschool special education coordinators in school districts and state Section 619 coordinators will provide information to address all but the last question, which will be addressed through the evidence review. The district survey will be administered in a nationally representative sample of 1,200 school districts serving preschool children with disabilities. It will be administered as a 60-minute web survey. The state survey will be administered in all 50 states and the District of Columbia as a 30-minute editable PDF survey. Data collection for each survey will begin in April 2015.

Information obtained as part of the data collection for Phase I will be used to develop a publicly available report for a wide audience of policymakers and educators. If the Institute of Education Sciences (IES) decides to sponsor an RCT following the feasibility assessment, the project team will conduct recruitment and random assignment and will work with curriculum/intervention developers to train treatment group teachers. A separate OMB package will be submitted that requests clearance for data collection activities for the RCT. The RCT would be completed under Phase II of the Evaluation of Preschool Special Education Practices.

Please see the time schedule in Table A.4 for a timeline of the entire project, including the option for developing plans for the RCT, and conducting recruiting and training.

A. Justification

A1. Circumstances necessitating collection of information

Beginning in 1968 with the passage of Public Law 90-538, the Handicapped Children’s Early Education Assistance Act, federal legislation has focused on early education for young children with disabilities. In 1975, Public Law 94-142, known as the Education of All Handicapped Children Act (EHA), was passed. Among its provisions, this landmark legislation guaranteed a free, appropriate, public education (FAPE) and mandated the development of individualized education programs (IEPs) for eligible children based on the results of a nondiscriminatory assessment. In addition, EHA required that IEP services be carried out in the least-restrictive environment (LRE).

Although Public Law 92-142 did not mandate that states extend FAPE, IEP, and LRE provisions to preschool-age children with disabilities, the law acknowledged the importance of services for this age group. Financial incentives were provided under EHA for states to serve children as young as 3 years of age, but no mandate was in place. In the 1986 reauthorization of EHA (Public Law 99-457), federal funds were allocated for two programs to serve young children with disabilities. One was the preschool program for children 3 to 5 years of age with disabilities under Part B, Section 619 of EHA. The second was a new program (Part H, now known as Part C) for eligible infants and toddlers (ages birth through 2 years) and their families. States were given five years, until 1991, to implement fully both programs. In 1990, Public Law 101-476 renamed EHA as the Individuals with Disabilities Education Act (IDEA). In the ensuing years, there have been successive reauthorizations of IDEA, with the most recent occurring in 2004 (Public Law 108-476).

More than 750,000 children ages 3 to 5 were identified for IDEA services in 2012,¹ and the total federal appropriation for Section 619 in both 2013 and 2014 exceeded $350 million.² Data from several studies conducted by IES have shown increases in the number and percentage of preschool children served under IDEA each year.

The 2004 amendments to IDEA stipulated children with disabilities should make progress in the general (preschool) education curriculum and improve their academic and development outcomes. IDEA 2004 does not require IEPs for preschool children with disabilities to be “standards based.”³ More recently, in 2012, the Office of Special Education Programs announced it is revising its accountability system from a system focused primarily on compliance to one that balances compliance with a greater emphasis on results. The accountability system is known as results-driven accountability (RDA).⁴ In the context of RDA, it will be important to increase the capacity of states and local educational agencies to implement, scale up, and sustain evidence-based practices to improve results for preschool children with disabilities.⁵

Three outcome areas have been identified under IDEA for preschool children with disabilities: (a) positive social-emotional skills (including social relationships), (b) acquisition and use of knowledge and skills (including early language/communication and early literacy), and (c) use of appropriate behaviors to meet one’s needs.⁶

Despite the size and longevity of the Section 619 program as well as an expanded focus to make progress in the general (preschool) education curriculum and improve (and report on) outcomes, very little information has been gathered about the specific curricula or interventions being delivered to preschool children with disabilities in different early childhood settings. Even less is known about whether available interventions or curricula when implemented with fidelity will have positive impacts on child outcomes and achieve desired results under RDA.

Previous studies of children with disabilities (the Pre-Elementary Education Longitudinal Study), state and local implementation of IDEA (the IDEA National Implementation Assessment Study),⁷ and administrative data reported at the state and local level for Section 618 are necessary and important, but are insufficient for determining which curricula and interventions are being delivered to and experienced by preschool children with disabilities.⁸ Moreover, it is not currently known to what extent curricula and interventions being delivered to preschool children with disabilities have content aligned with the three child outcome areas or if curricula or interventions being adopted and delivered are evidence based. In 2012, IES funded the Design and IDEA-related Analyses for the National Assessment (DIANA) to assess the feasibility of conducting a rigorous evaluation of an intervention serving preschool children with disabilities served under IDEA. DIANA found that not enough was currently known about existing curricula and interventions being implemented to launch a large-scale rigorous evaluation. In addition, a systematic review of empirical evidence for curricula and interventions both currently in use and those available for use to enhance language, literacy, and social-emotional skills in preschool children with disabilities has not been conducted. Data collected as part of the proposed study will provide foundational information for the design of a large-scale evaluation of preschool special education curricula or interventions. This evaluation is important because although there have been impact studies focused on curricula or interventions for preschool children with disabilities, a large-scale effort informed by a national survey and evidence review has not been undertaken to date.

A2. How, by whom, and for what purposes the information will be used

This study will rely on a new set of surveys and information collected from existing sources, for which there are no additional respondents or burden. Specifically, surveys of all state Section 619 coordinators and a sample of district preschool special education coordinators will be collected on a one-time basis by Mathematica Policy Research, under contract with ED (contract number ED-IES-14-C-0001). The data will be analyzed by Mathematica staff to address the research questions outlined earlier and in Table A.1, and to inform the assessment of the feasibility of an RCT.

The research questions for the feasibility study will be addressed through the following four key activities:

1. Extant data collection. The project team will use available data to limit what must be obtained through new data collection. We will always use the most recently available data; however, these data are likely to lag the current data collection by approximately two years because of the time between when those data are collected and released.

2. District preschool special education coordinator survey. The current study plans call for drawing a nationally representative sample of approximately 1,200 school districts. We plan to oversample large districts because any potential RCT that might follow the feasibility study is likely to focus on school districts with larger preschool special education programs in which assignment to the treatment can be randomized within or across schools. We plan to administer the survey by web and aim for a completion time of no more than 60 minutes. This survey will provide information on both the characteristics of preschool special education programs and the curricula and interventions that have been adopted for use with children attending these programs. The respondent is the district preschool special education coordinator. We anticipate that these coordinators will be able to tell us about the curricula and interventions that districts have adopted for use. Although this is not the same as knowing which curricula and interventions actually are used, the information that district coordinators can provide will help to assess the feasibility of an RCT. Based on our review of several state websites, many states appear to maintain lists of district preschool special education coordinators. We propose to use these state lists to identify survey respondents. If lists of district preschool special education coordinators are not available through state websites we will contact those state Section 619 coordinators to obtain lists of preschool special education coordinators.

3. State Section 619 coordinator survey. This survey will provide a statewide perspective on the program settings in which preschool children with disabilities are served, the state role in supporting preschool special education programs in districts and other entities, and qualifications needed to teach preschool children. The proposed respondent is the state Section 619 coordinator, who coordinates preschool special education programs at the state level. However, state coordinators are far removed from the curricula and interventions adopted by districts and, in our judgment, able to address only a small number of topics relevant to the study. We plan to administer the survey in an electronic format to Section 619 coordinators in all 50 states and the District of Columbia, and we anticipate a completion time of no more than 30 minutes.

4. Evidence review. The evidence review will be the primary source of information about curricula and interventions that could be included in an RCT. The evidence review will identify curricula and interventions with evidence of efficacy and other features (such as supports for implementation) that make them possible candidates for an RCT.

Table A.1. Research questions and data sources

		Source
		Extant data	Survey of district coordinator	Survey of state coordinator	Evidence review
What curricula and interventions are available and supported for use with preschool children with disabilities to promote learning of language, literacy, social-emotional skills, and/or appropriate behavioral skills for school?			X	X
How are decisions to adopt curricula and interventions made?			X	X
What agencies, programs, and settings serve preschool children with disabilities?		X	X	X
What is the structure of programs that serve preschool children with disabilities?			X	X
What resources support providing services to preschool children with disabilities?			X
What are the characteristics of staff that deliver services to preschool children with disabilities?		X	X	X
What are turnover rates for staff that deliver services to preschool children with disabilities?			X
What are eligibility rules for preschool special education curricula and interventions?		X
What are the enrollment characteristics of preschool children with disabilities and classrooms that include these children?		X	X
What curricula and interventions for children ages 3 to 5 with disabilities might be suitable for study in a large-scale RCT?					X

Table A.2 describes more explicitly the links between the research questions, the information sources to answer them, and the timing of data collection.

Table A.2. Evaluation of preschool special education practices, phase I, data collection plan

Respondent	Mode	Timeline	Key data
Extant data collection	District- and state-level administrative records	Fall 2014	Enrollment by setting Number and qualifications of staff State eligibility guidelines Characteristics of children
District preschool special education coordinators	Web survey	Spring 2015	Curricula and interventions that are available and supported for use How decisions to support curricula and interventions are made Description of programs Structure of programs Resource availability Characteristics of staff and staff turnover District eligibility guidelines Enrollment characteristics
Section 619 coordinator	Electronic: editable PDF	Spring 2015	Curricula and interventions that are available and supported for use Structure of programs Training and staff requirements
Evidence review	Not applicable	Fall 2014 to spring 2015	Evidence ratings of promising curricula and interventions for preschool children with disabilities Implementation features

Our main goal in the data collection is to gather important information for assessing the feasibility of an RCT. For example, knowing what curricula and interventions are adopted for use will provide information about the likely counterfactual condition if we conduct an RCT. Developing an understanding about typical rates of staff turnover will inform judgments about both (1) the suitability of curricula and interventions that involve upfront training and (2) how large the study sample has to be to ensure adequate statistical power. Finally, knowing contextual factors such as the numbers of preschool children with disabilities served will help identify the types of districts and schools with enough children to make them well suited to participate in an RCT.

Another goal of the data collection is to gather descriptive information (such as the number of preschool children with disabilities, the settings in which they are served, characteristics of staff, and curricula and interventions available for use in those settings, among other information), which we expect will also be of broader interest to the research community. The survey data will be summarized in a publicly available IES report addressing the research questions described in Table A.1. The data collected in the survey will also be provided as a de-identified restricted-use data set.

A3. Use of technology to reduce burden

The study will use a combination of mechanical and electronic technology to collect data. For each data collection task, the study team has selected the form of technology that will provide reliable information while minimizing respondent burden. Information technology will be heavily used in data collection tasks. Examples include the following:

• An electronic sample management database will be used to house the contact information for the state Section 619 coordinators and district preschool special education coordinators and to track the status of the data collection activities.

• District preschool special education coordinators will participate in a web-based survey.

• The state Section 619 coordinator survey will be administered through an encrypted electronic editable PDF that will be emailed to respondents.

• The study will have a toll-free number and an email address, both of which will be hosted by Mathematica. Staff from Mathematica will field inquiries to the toll-free number and email account on a flow basis across the life of the study.

We will monitor response rates and provide periodic email follow-ups to nonrespondents. We will also use adaptive methodologies to maintain or improve response rates. Four weeks after the survey launch, we will begin telephone calls to nonrespondents. In those calls, we will first remind respondents about completing the self-administered instrument and determine if assistance is needed. We will offer to complete the survey with the respondent by telephone, entering the information into either an editable PDF or into the web survey as appropriate. The brevity of both state and district surveys makes a telephone completion of the entire survey feasible.

A4. Efforts to avoid duplication

The primary data source for the preschool age group is limited to the annual Section 618 state-reported data collected by the Office of Special Education Programs. Under IDEA, state grantees are required to collect and report data annually to the Office of Special Education Programs. IDEA Section 618 state-reported data include, for example, the number of children that each state serves under the IDEA programs and the educational settings in which these children are served. The Section 618 state-reported data provide limited information on the program, services, and practices being implemented within states to preschool-aged children with disabilities or at risk for developmental delay. A review of the evidence available on interventions targeting preschool-age children’s language, literacy, and social-emotional skills also found there to be limited and mixed evidence on the reviewed interventions addressing each of the above skill areas.

The limited information currently available to inform the feasibility of and design options for a study of preschool special education suggests the need for new data collection to describe the preschool programs and services available to and received by young children and the practices being implemented in these programs at the state and district level. Information from this new data collection will provide an important foundation for designing a large-scale evaluation of preschool special education practices.

A5. Methods to minimize burden on small entities

The state survey will not affect small entities. The district survey sample will not burden any districts with fewer than 10 preschool children with disabilities. We are minimizing the burden on the remaining districts with the smallest number of such children by undersampling them.

A6. Consequences of not collecting data

The data collection described in this submission is essential to documenting, on a nationally representative scale, how many preschool children with disabilities are served, in what settings, by what types of staff, and with what available curricula and interventions. This information is not currently known at this level of detail, so the data collection will therefore provide the necessary information to conclude whether or not it is possible to proceed with a rigorous evaluation, and, if so, how to structure such a study. Lacking such an understanding, it will not be possible to identify districts and schools with the capacity and characteristics that would make them potential participants in an RCT. For example, the survey will reveal which districts serve a sufficient number of preschool children with disabilities, and whether individual schools have the authority to choose curricula or interventions to use. The survey will also provide valuable information to researchers and practitioners by describing information about how such services are provided nationally

A7. Special circumstances

There are no special circumstances involved with data collection as part of the Evaluation of Preschool Special Education Practices, Phase I.

A8. Federal Register Announcement and Consultation

1. Federal Register announcement

The 60-day Federal Register notice was published on October 8, 2014, volume 79, page 60844-60845. There were no public comments on the package.

2. Consultation outside the agency

During preparation of the study design and data collection plan for this evaluation,

ED has consulted with 10 members of the project’s Technical Work Group, which met for the first time in April 2014. The table below lists the group members.

William Brown, Ph.D. Professor, Department of Educational Studies University of South Carolina Department of Educational Studies	Khari Garvin Director, North Carolina Head Start State Collaboration Office Office of Early Learning North Carolina Department of Public Instruction
Howard Goldstein, Ph.D. Associate Dean for Research, College of Behavioral & Community Sciences Professor, Communication Sciences & Disorders University of South Florida College of Behavioral & Community Sciences	Charles Greenwood, Ph.D. Professor, Department of Applied Behavioral Science Director, Juniper Gardens Children’s Project University of Kansas Juniper Gardens Children’s Project
Laura Justice, Ph.D. EHE Distinguished Professor, Teaching & Learning Administration Ohio State University College of Education & Human Ecology	Christopher Lonigan, Ph.D. Associate Director, Florida Center for Reading Research Florida State University Department of Psychology
Evelyn Seidenberg Preschool Program Specialist/Preschool Support Instruction Design and Innovation Moore County Schools	Philip Strain, Ph.D. Director, Positive Early Learning Experience (PELE) Center; Professor, Department of Educational Psychology - University of Colorado Denver School of Education & Human Development
Verna Thompson Education Associate, Teaching and Learning & 619 Coordinator Delaware Department of Education	Markay Winston, Ph.D. Chief of the Office of Diverse Learner Supports and Services Chicago Public Schools Office of Diverse Learner Supports and Services

A9. Payments or gifts

State Section 619 and school district special education program coordinators are required to cooperate with evaluations conducted by the U.S. Department of Education and must complete the survey. No incentives will be offered for completion of the survey.

A10. Assurances of confidentiality

Other than the names and contact information for the survey respondents, which is information typically already available in the public domain on state and district websites, no data collected for surveys will contain personally identifiable information. No names and contact information will be released.

The following language will be included on the cover page of district and state surveys under the Notice of Confidentiality:

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. Personally identifiable information about individual respondents will not be released. Please note that data on state policies and practices may be reported by state.

Mathematica and its subcontractors will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Further, personally identifiable data will not be entered into the analysis file; the analysis data records will contain a numeric identifier only. When the results are reported, data will be presented only in aggregate form so that individuals and institutions cannot be identified. The study team will include a statement to this effect with all requests for data and will include a reminder about confidentiality protection in compliance with the legislation. All members of the study team having access to confidential data will be trained on the importance of confidentiality and data security. All data will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required.

The following safeguards are employed to carry out confidentiality assurances during the study:

• All employees at Mathematica and its subcontractors sign a confidentiality pledge emphasizing the importance of confidentiality and describing their obligation to it (Appendix A).

• Access to identifying information about sample members is limited to staff members who have direct responsibility for providing and maintaining sample locating information. At the conclusion of the research, these data are destroyed.

• Identifying information is maintained in separate forms and files, which are linked only by sample identification number.

• Access to the file linking sample identification numbers with the respondents’ IDs and contact information is limited to a small number of individuals who have a need to know this information.

• Access to hard-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

• Computer data files are protected with passwords, and access is limited to specific users. Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use.

The Privacy Act of 1974 applies to this data collection. Mathematica and its subcontractors will make certain that all surveys are held in strict confidence, as described above, and that in no instance will responses be made available except in tabular form. District staff responsible for assisting Mathematica in the data collection will be fully informed of Mathematica’s policies and procedures regarding confidentiality of data.

A11. Justification for sensitive questions

There are no questions of a sensitive nature asked in either of the surveys.

A12. Estimates of hours burden

Beginning in April 2015, surveys will be administered to

• The 50 states and the District of Columbia, and

• 1,200 sampled school districts

In all, responses will be required from 1,251 respondents (51 state officials for the state survey and 1,200 district officials for the district survey).

We estimate it will take state Section 619 coordinators 30 minutes to complete the survey and that it will take 60 minutes for the district preschool special education coordinators. The total burden for the 2015 data collection is 73,530 minutes, or 1,225.5 hours (Table A.3).

A13. Estimates of cost burden to respondents

There is no annualized capital/start-up or ongoing operation and maintenance costs associated with collecting the information.

A14. Annualized cost to the federal government

The estimated average annual cost to the federal government for the study—including designing and administering all collection instruments, processing and analyzing the data, and preparing reports—is $956,868 (the total cost of $2,870,604 divided by the three years of the study).

Table A.3 Burden associated with data collection

Informant/data collection activity	Number of respondents	Minutes per completion	Number of administrations	Burden in minutes	Total burden hours	Total costsa
State
State Section 619 Coordinator Survey	51	30	1	1,530	25.5	$1,107.72
District
District Preschool Special Education Coordinator Survey	1,200	60	1	72,000	1,200	$52,128.00
Total	1,251	90	2	73,530	1,225.5	$53,235.72

^a Assumes an hourly rate of $43.44 for educational administrators—derived from the Bureau of Labor Statistics’ Occupational Employment and Wages for educational administrators, May 2011.

A15. Reasons for program changes or adjustments

This is a new collection. There is a program change resulting in an increase in burden and responses of 1,225.5 hours and 1,251 responses.

A16. Plans for tabulation and publication of results

Data collected from the study’s surveys and information gathered from the evidence review and extant sources will primarily be used to assess the feasibility of conducting an RCT. Because the survey data we plan to collect are likely to be of broader interest to the field, we will also produce a descriptive report on the findings from the surveys. We discuss our analysis and publication plans for both of these purposes in turn.

1. Analysis and publication plans for assessing RCT feasibility

To assess the feasibility of conducting an RCT, we will develop an evaluation design report that describes the core features of three potential design options. These features include the characteristics of candidate interventions; study context and participants; counterfactual condition; key design elements, such as the unit of assignment, target minimum detectable effects (MDEs), sample size, and types of data collection; and training. Below, we describe each of these core features and list types of key decisions that our design report will address and how we will use the information we collect to inform those decisions.

Candidate interventions: Identifying the intervention is related to other design issues, such as (1) substantive focus in terms of outcomes and disabilities; (2) the study settings; and (3) implementation intensity, duration, and cost.

• Decisions for evaluation options: How narrowly or broadly defined the intervention is in intended outcomes and disabilities; whether to prioritize interventions that are more widely used or ones with more evidence of effectiveness from smaller-scale studies; how difficult it would be for schools to implement the intervention after the evaluation ends.

• How extant data, state questionnaires, and/or district surveys inform decisions: They identify how children are being served in agencies, settings, and program structures; indicates implementation prevalence of types of interventions, resources available to districts to support intervention costs, and how districts currently support implementation of similar interventions.

• How evidence review informs decisions: Identifies replicable interventions with promise for improving outcome measures in the domains of interest, indicates settings where it has been implemented, training and implementation requirements, and costs.

Study context and participants. An evaluation must identify the implementation setting and the target population of programs, schools, teachers, and children to be studied.

• Decisions for evaluation options: Setting(s) in which the intervention should be implemented (for example, self-contained and/or integrated classrooms in public schools and/or community-based programs); whether regular classroom teacher or other staff (such as a special education teacher, paraprofessional, or specialist) should implement intervention; whether to include all preschool children in a classroom or only children with certain disabilities; whether to include the entire 3- to 5-year-old age range or focus on a particular level (such as prekindergarten); whether to target children actually identified for special education or establish a high-risk designation in the study using a baseline assessment and also include those children in the study.⁹

• How extant data, state questionnaires, and/or district surveys inform decisions. They describe how children are being served in programs and schools; size of preschool programs; qualifications of preschool program staff; variation in special education eligibility requirements.

• How evidence review informs decisions. Describes implementation settings and characteristics of the population the intervention is intended to serve.

Counterfactual condition: Impacts are interpreted relative to a counterfactual condition, making the characteristics of the counterfactual important for ensuring a meaningful contrast.

• Decisions for evaluation options. Whether to compare a single intervention with training and support to either a business-as-usual set of practices that does not include training and support or to a specific alternative intervention. If several interventions are selected, should they focus on outcome measures in the same or different domains?

• How extant data, state questionnaires, and district surveys inform decisions. They identify the prevalence of types of interventions, which will show the types of interventions used widely and in which districts the counterfactual will be most distinct.

• How evidence review informs decisions. Identifies whether interventions are associated with outcomes in one or more domains; provides detailed information regarding the conceptual model underlying the intervention, which is necessary for understanding what counterfactual condition provides a meaningful contrast to the intervention.

Unit of assignment. The unit of assignment is the level at which randomization is conducted, and units are assigned to the treatment group or the comparison group. The unit of assignment must be logically aligned with the setting in which the intervention is implemented. The unit of assignment can have implications for whether the control group can learn about, and possibly implement, aspects of the intervention, which would contaminate impact estimates.

• Decisions for evaluation options. Whether to conduct randomization at the level of the school, classroom, staff, or student.

• How extant data, state questionnaires, and district surveys inform decisions. They describe the settings in which interventions seem to be commonly implemented. For example, schools could be the units if interventions are implemented by teachers connected to a single school. Staff could be the unit if interventions are implemented by itinerant staff traveling among schools providing direct services (for example, a speech and language therapist providing individual or small-group interventions).

• How evidence review informs decisions. Identifies the setting in which the intervention is intended to be implemented.

Target MDE and size of the study sample. The appropriate MDE for an evaluation depends on the intervention and outcome of interest. The target MDE may be larger for interventions directly aligned with an outcome measure that is reliable and sensitive to change during the preschool year. Evaluations of interventions that are more expensive or more intensive to implement also may target larger MDEs to justify the higher cost. The target MDE has implications for the study sample because, in general, larger samples are needed to achieve smaller MDEs.

• Decisions for evaluation options. Whether to target a smaller MDE, knowing not only that evaluation costs are likely to be higher because larger samples are required, but also that the smaller change in the specified outcome is meaningful and predictive of positive outcomes; which types of outcomes to consider, given that relatively more costly outcomes (such as direct child assessments) tend to have less error and be more stable.

• How extant data, state questionnaires, and/or district surveys inform decisions. They provide information on factors needed to select samples to achieve a target MDE, such as the number of schools per district, classrooms per school, and preschool children per classroom; and the number, age, and distribution of preschool children with disabilities within different disability categories in a district.

• How evidence review informs decisions. Describes intervention effects found in previous studies and for which outcome measures; allows examination of how meaningful previously reported effects would be for policy.

Recruiting and data collection. For an option to be feasible, the study team must be able to recruit districts, schools, teachers, and parents/children, and collect the necessary data for the evaluation.

• Decisions for evaluation options. Balancing the amount of data collection with study objectives and costs; whether to have one or two years of implementation and/or follow-up; whether a two-year option should measure maintenance of effects for children or changes in the quality of implementation (fade-out versus improvements) with a year of experience.

• How extant data, state questionnaires, and/or district surveys inform decisions. They identify districts that meet study eligibility requirements.

• How evidence review informs decisions. Indicates what kinds of outcome measures are sensitive to expected changes in response to the intervention and the length of implementation associated with changes.

Training and implementation. Training and professional development are necessary for an intervention to be implemented as faithfully as possible. Feasible evaluation options must identify how training will be provided to the requisite sample, whether existing training materials are sufficient, and whether piloting of required training should be conducted before large-scale implementation.

• Decisions for evaluation options. Whether to pilot the training; how to balance the intensity of training with cost; whether to conduct trainings at individual schools or centrally; whether the intervention developer should conduct trainings or if a train-the-trainer model is feasible.

• How extant data, state questionnaires, and/or district surveys inform decisions. They describe qualifications of staff delivering preschool services; whether staff turnover during the evaluation is likely to be a concern.

• How evidence review informs decisions. Identifies whether training materials and replicable implementation procedures exist; training costs; qualifications of staff who have implemented the intervention previously; previous documentation on whether the intervention has been implemented with fidelity.

2. Analysis and publication plans for descriptive report of survey findings

In addition to informing the feasibility of conducting an RCT, the data collected by this study will be presented in a descriptive report. Descriptive findings will consist of means, proportions, and standard errors or confidence intervals that take into account the sampling design (for example, some findings may need to be weighted to provide nationally representative information). The report will include descriptions of the following:

• Prevalence of curricula and interventions that are available and supported for use with preschool children with disabilities to promote learning of language, literacy, and/or social-emotional skills/behavior appropriate for school. These are the curricula and interventions that districts make available to teaching staff and support by providing resources such as training and materials.

• How decisions to adopt curricula and interventions are made. We will report the extent to which states have approved lists of interventions and curricula, who decides which interventions and curricula to make available and support, and how much freedom preschool teachers have in selecting which interventions and curricula to use.

• Agencies, programs, and settings that serve preschool children with disabilities. We will report which agencies (for example, school districts, community-based providers) deliver curricula and interventions, the number of children served in different programs (for example the school district’s preschool program, Head Start centers), and the number of children served in different settings (for example, home, inclusive classroom, self-contained classroom).

• Structure of programs that serve preschool children with disabilities. We will report the extent to which children receive services from classroom teachers versus specialists, how services are delivered (for example, small groups, individual pullout), the extent to which children are in full or partial inclusion and the length of the preschool program day.

• Resources that support implementation of curricula and interventions. We will report what training and professional development support is provided to teachers, what support the district provides for the purchase of curriculum/intervention materials, and whether funding is available for implementing promising new curricula or interventions.

• Characteristics of staff that deliver services to preschool children with disabilities. We will report the number and qualifications of full-time equivalent staff working with preschool children with disabilities, the number of full-time equivalent early childhood teachers delivering services to children with disabilities, what qualifications are required to teach preschool children with disabilities, turnover rates, and whether staff are unionized.

• Enrollment in and eligibility for preschool special education services. We will report how states define eligibility for preschool special education programs, how many preschool children with disabilities are served overall, and how many children are served in school-based programs.

3. Time schedule

Table A.4 describes the schedule for activities related to this data collection.

Table A.4. Timeline for data collection and reporting and study option

Study milestone	Milestone date/period of activity
Submit OMB clearance	September 2014
Instrument pre-test (3 states and 9 districts)	September-December 2014
Select and recruit districts	December 2014
OMB approval (estimated)	Late March 2015
Collect extant, state, and district data	November 2014–June 2015
Publish report on descriptive data	Fall 2016
Decision whether to fund the option for developing the RCT	February 2016
Prepare OMB recruitment and addendum packages	Summer/fall 2016
Select, notify, and recruit sample	Winter 2016/spring 2017
Conduct training in selected curriculum or intervention	Summer 2017

A17. Approval not to display the expiration date for OMB approval

Approval not to display the expiration date for OMB approval is not requested.

A18. Explanation of exceptions

No exceptions to the certification statement are requested or required.

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Mathematica^® is a registered trademark
of Mathematica Policy Research, Inc.

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

1 See https://explore.data.gov/Education/2012-IDEA-Part-B-Child-Count-and-Educational-Envir/5t72-4535.

2 See http://www.ed.gov/about/overview/budget/index.html. The federal appropriation for Section 619 did not change from 2013 to 2014.

3 “Standards based” means the inclusion of curricular content from state early learning standards along with services, supports, or specialized instruction to enable a preschool child with a disability to make progress in the general preschool education curriculum.

4 http://www2.ed.gov/about/offices/list/osers/osep/rda/index.html.

5 Overview of the State Systemic Improvement Plan, Part C/619 State Accountability Priority Area. Retrieved from http://ectacenter.org/~calls/2014/ssip/ssip.asp.

6 http://ectacenter.org/eco/pages/fed_req.asp.

7 Bradley, M. C., Daley, T., Levin, M., O’Reilly, R., Parsad, A., Robertson, A., and Werner, A. (2011). IDEA National Assessment Implementation Study (NCEE 2011-4027). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

8 Section 618 of IDEA stipulates states must submit data on their Part C and Part B IDEA programs to the Office of Special Education.

9 For example, a designation could be 1.5 standard deviations below the mean on the outcome of interest.