1810-0618 LEA Survey Part B Supporting Statement

REQUEST FOR CLEARANCE OF PROPOSED STUDY

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

Survey on Use of Funds Under Title II, Part A

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

The Elementary and Secondary Education Act (ESEA), as reauthorized by the Every Student Succeeds Act of 2015 (ESSA), continues to place a major emphasis on teacher quality as a significant factor in improving student achievement. Under the ESEA, Title II, Part A (Supporting Effective Instruction) provides funds to State educational agencies (SEAs) and local educational agencies (LEAs) to support effective instruction through the preparation, training, and recruitment of high-quality teachers, principals, and other school leaders. LEAs are provided Title II, Part A State activities funds for this purpose, allowable uses of which include:

• Developing and implementing evaluation systems for teachers, principals, and other school leaders

• Developing and implementing initiatives to assist in recruiting, hiring, and retaining effective teachers

• Recruiting qualified individuals from other fields to become teachers, principals, or other school leaders

• Reducing class size by recruiting and hiring additional effective teachers

• Providing high-quality, evidence-based professional development for teachers, principals, and other school leaders

• Developing programs to improve the ability of teachers to teach children with disabilities and English learners

• Providing training to assist teachers, principals, and other school leaders with selecting and implementing assessments, and using data from those assessments

• Carrying out in-service training for school personnel

• Providing training to support the identification of gifted and talented students

To gain a better understanding of how LEAs were responding to the high level of Title II, Part A funding and the wide range of activities allowed under ESEA, the U.S. Department of Education (the Department) collected baseline data in 2002-03 from LEAs around the nation to answer the question: “How did districts report spending their federal Teacher Quality funds in 2002-03?” In addition to providing information on what funds LEAs receive and how LEAs use Title II, Part A funds, the Department has used this survey to collect information on the provision of professional development in LEAs. The Department initially collected professional development data through the Consolidated State Performance Report (CSPR), but poor data quality led the Department to collect the data directly from LEAs rather than through the States. To improve data quality and get a better understanding of how States and LEAs are using their funds, the Department is expanding the LEA survey to a state representative sample of 5,000 traditional LEAs and a nationally representative sample of charter school LEAs. Analyses from this data collection will be complementary to the SEA data collection that was approved in a prior OMB submission (Approval #1810-0711).

1. Respondent Universe

The respondent universe consists of two subpopulations of LEAs, one is the population of traditional LEAs and the other is that of charter school LEAs. To survey this respondent universe, a list sampling frame will be used, which will be constructed from the 2016-2017 NCES CCD Public Elementary and Secondary Agency Universe File. Specifically, we will draw the sample from about 16,300 traditional and charter school LEAs in this universe file, of which about 13,400 (or 82%) are classified as traditional LEAs and about 2,900 (18%) as charter school LEAs.

2. Sample Design

Multiple factors, such as respondent burden, data quality, and cost implications, were taken into account when determining the design for this study. We will use a total sample of 5,000 traditional LEAs, in which each of 50 States, the District of Columbia (DC), and Puerto Rico (hereafter, they will be referred to as States) is represented, and a nationally representative sample of 345 charter school LEAs. This sample total of 5,345 LEAs limits respondent burden by including only about 33 percent of all traditional and charter LEAs, while still allowing the Department to produce reliable descriptive statistics such as means, proportions, and totals at both the national and State levels. In past administrations of this survey, a smaller sample size was used because the focus was only on producing estimates that were representative of traditional LEAs at the national level. Now the focus is on producing estimates that are representative of traditional LEAs at both the national and State levels, and to produce estimates that are representative of charter LEAs at the national level. These additional estimation goals require a larger sample. In addition, the Department seeks to disaggregate the survey results by LEA size and poverty level at the national level, so it is important to ensure that the sample adequately represents various levels of LEA size and poverty level (poverty quartiles, in particular). With an increased sample size and a sampling approach that stratifies by LEA size and poverty level, this objective can be met.

The 5,000 traditional LEA sample will be selected from the traditional LEA sampling frame, stratified by State, size, and poverty level. The 345 charter school LEA sample will be selected from the charter LEA sampling frame, stratified by size and poverty level. LEAs will be stratified by poverty (percentage of poor students) using the most recent estimates available from the Census Bureau. To stratify LEAs by State and size (number of students enrolled), we will use data from the 2016-2017 NCES CCD Public Elementary and Secondary Agency Universe File, which is the same file that we are using to establish the sampling frame.

This study has a target response rate of at least 80 percent. This target is in line with actual response rates achieved in past administrations of this survey, which have been 80 percent or higher. The sampling approach that was developed for this survey took into consideration this target response rate.

For the traditional LEAs, we will stratify the frame by State so as to produce more efficient State-level estimates. Within each State, it will further be stratified by LEA size and poverty status. To stratify by LEA size and poverty status in a given State, a minimum of four strata must be created (two size strata crossed by two poverty strata¹). Each stratum must have at least 15 sampled LEAs, which means that there must be at least 60 sampled LEAs in the State. This rule is imposed to prevent fragmentation of strata. States with fewer than 60 LEAs will thus include all of their LEAs in the sample to ensure adequate representation and to minimize the sampling error for that State. In States with a large number of LEAs, there could be eight strata (four size strata crossed by two poverty strata) instead of four. Eight strata are used in a State instead of four, as long as it is still possible to meet the 15 sampled LEA per stratum minimum. Thus, States with a target sample size of at least 120 LEAs will have eight strata, and States with a target sample size of between 60 and 120 LEAs will have four strata. To form the two or four size strata in each State, a sampling method will be used to balance the importance of including large-size LEAs while also including a reasonable number of small-size LEAs. This method involves proportional allocation using the square root of the LEA size of student enrollment. This size measure has been used as a compromise between the optimal design for the continuous variables that requires using the LEA size (without taking the square root) and the optimal design for categorical variables that requires no size measure. After allocating the State sample to strata proportionally to the sums of the size measures, an equal probability sample will be selected from each stratum.

The sample design was selected to satisfy study cost constraints while still enabling us to produce sufficiently reliable estimates of the percentage of LEAs having specified characteristics as well as aggregate measures, such as total dollar amounts allocated for allowable activities under Title II, Part A and total numbers of teachers participating in various professional development activities. This process was used to determine the target total sample size, as well as the specific target sample sizes for each State: the ultimate goal was to have an affordable sample size that produced the most precise estimates at the national and State levels, and that produced estimates that had similar reliability across States. With the target sample sizes, an estimate showing that, for example, 50 percent of traditional LEAs nationally use Title II dollars to reduce class size would have a relative standard error (RSE) of less than 2 percent. Similar summary statistics at the State-level would have a margin of error around 10 percent, which is still a policy-relevant level of precision. The sample allocation to States is presented in Table 1 along with the frame size based on the 2016-2017 NCES CCD data. Note that the total sample sizes are slightly off from the intended sample sizes due to rounding.

Table 1 Frame size and sample allocation for traditional LEAs

STATE	Frame Size	Field Sample Size	Target Sample Size	Number of Strata
ALABAMA	132	94	75	4
ALASKA	53	53	42	1
ARIZONA	224	114	91	4
ARKANSAS	235	116	93	4
CALIFORNIA	995	151	121	8
COLORADO	178	106	85	4
CONNECTICUT	169	104	83	4
DELAWARE	19	19	15	1
DISTRICT OF COLUMBIA	1	1	1	1
FLORIDA	70	68	54	4
GEORGIA	180	106	85	4
HAWAII	1	1	1	1
IDAHO	114	88	70	4
ILLINOIS	855	149	119	8
INDIANA	291	123	98	8
IOWA	338	128	102	8
KANSAS	286	123	98	8
KENTUCKY	173	105	84	4
LOUISIANA	69	67	54	4
MAINE	239	117	93	4
MARYLAND	24	24	19	1
MASSACHUSETTS	238	116	93	4
MICHIGAN	542	140	112	8
MINNESOTA	330	127	102	8
MISSISSIPPI	146	98	78	4
MISSOURI	520	139	111	8
MONTANA	408	133	106	8
NEBRASKA	245	117	94	4
NEVADA	18	18	14	1
NEW HAMPSHIRE	177	105	84	4
NEW JERSEY	546	140	112	8
NEW MEXICO	89	77	62	4
NEW YORK	721	146	117	8
NORTH CAROLINA	115	88	70	4
NORTH DAKOTA	175	105	84	4
OHIO	614	143	114	8
OKLAHOMA	517	139	111	8
OREGON	181	106	85	4
PENNSYLVANIA	500	138	111	8
PUERTO RICO	1	1	1	1
RHODE ISLAND	32	32	26	1
SOUTH CAROLINA	84	75	60	4
SOUTH DAKOTA	151	99	79	4
TENNESSEE	146	98	78	4
TEXAS	1,024	151	121	8
UTAH	41	41	33	1
VERMONT	272	121	97	8
VIRGINIA	130	93	74	4
WASHINGTON	298	124	99	8
WEST VIRGINIA	55	55	44	1
WISCONSIN	422	134	107	8
WYOMING	48	48	38	1
Total	13,432	5,003	4,000

For the sample of charter school LEAs, we want to provide a nationally representative sample of charter school LEAs. This will be accomplished by drawing a new sample from the most recent CCD frame (2016-17) of eligible charter school LEAs. As with the traditional LEA sample, a sampling approach was selected to yield the most precise estimates that are within the study cost constraints. This process results in a target sample size of 345 charter school LEAs, which will be stratified by size and poverty. More strata generally will result in more precise overall estimates, but as with the sampling approach for traditional LEAs, each strata should include at least a certain number of charter LEAs to avoid fragmentation of strata. In this sample, the target number of charter LEAs per stratum is set at 17 minimum, which means that for a total target sample size of 345, there can be about 20 strata. The 20 total strata were formed by crossing five size strata with four poverty strata.² The same method used for the traditional LEA sample will also be used for the charter LEA sample to form the size strata, balancing the importance of including large-size LEAs while also including a reasonable number of small-size LEAs.

With the target sample size, an estimate showing that, for example, 50 percent of charter LEAs nationally use Title II dollars to reduce class size would have a relative standard error (RSE) of 6.7 percent, which is a policy-relevant level of precision. As with the traditional LEA sample, after allocating the total sample to strata proportionally to the sums of the size measures, an equal probability sample will be selected from each stratum.

There are no unusual problems requiring specialized sampling procedures. This data collection will be annual with a new sample each year (it will be a repeated cross-section of LEAs, not a longitudinal sample of LEAs).

Weighting, Imputation, and Variance Estimation. To account for the sampling design and survey non-response (expected to be around 20 percent), we will weight the survey data to ensure that the final sample is sufficiently representative of the target populations. Weighting starts with the base weight (i.e., the inverse of the sampling probability) and adds an adjustment for unit nonresponse. We will use the weighting class method to do the nonresponse weighting adjustment. There are several ways of creating weighting classes. We will use the response propensity score method because it more easily accommodates the rich auxiliary information that is available in the CCD-based sampling frame. We will further adjust this nonresponse-adjusted weight through the post-stratification technique to benchmark the weighted data against total Title II, Part A funding for each state, which is known to further improve the precision of survey estimates.

For item nonresponse for key variables, we will impute using Westat proprietary imputation software AutoImpute. We will use the jackknife variance estimator to estimate the variance of survey estimates.

Once the data are weighted, cleaned, and imputed, we will analyze and report on the data using descriptive statistics and simple cross tabs.

3. Methods for Maximizing the Response Rate

A letter including login information for the online data collection system (Appendix C) will be sent to each LEA. The letter will detail the reasons for selection and requesting cooperation. LEAs will respond to the survey using an online system. If the completed survey is not received by the expected date, a reminder card will be sent to the LEA. Following the reminder card, contractor personnel will resend the letter to nonrespondents and continue to attempt to obtain completed surveys for 3 months after the initial mail-out of the data collection instruments. If the response rate has not reached 80 percent by the final deadline, Westat will search for email addresses and phone numbers of the superintendents of nonresponding LEAs and send a final email reminder and start calling the superintendents. The contractor for this study has a long history of achieving high response rates through repeated follow-ups.

As noted in the prior section, the survey data will be weighted, and unit nonresponse will be handled by nonresponse weighting adjustment using auxiliary variables available in the sampling frame. If, however, the response rate falls below 80 percent, Westat will conduct an analysis to assess the likelihood of nonresponse bias. This analysis will involve using standard techniques to assess how similar estimates of auxiliary variables from the full sample are to estimates for the same variables from the weighted respondent sample.

4. Tests of Procedures and Methods

For the prior Title II, Part A study, the Department consulted with several SEAs and LEAs to determine the feasibility of the data collection. For the revised survey, we conducted a pilot test with three LEAs. In selecting the LEAs to participate in the pilot test, we sought to include LEAs of varying size and location. The purpose of the test was to (1) verify that LEAs will be able to provide information for all of the data items on the data collection instrument and (2) ensure that the burden estimates used in this clearance package are accurate. As a result of this effort, we made several revisions to the wording on the data collection instrument to improve clarity and reduce respondent burden.

5. Consultations on Statistical Aspects of the Design

All sample design development will be provided by Dr. Hyunshik Lee, Senior Statistician in Westat’s Statistical Support Group (301-610-5112). Westat will collect and analyze the information for the Department.

1 Poverty strata are defined by the percentage of poor students in the district. To create two poverty strata for a particular state, that State’s median percentage is used.

2 The four poverty strata will be formed by using the quartiles of district percentage of poor students across all charter LEAs in the nation.

22