School Pulse Panel
OMB# 1850-0975 v.4
Supporting Statement Part B
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
March 2023
revised June 2023
revised August 2023
Contents
B1. Respondent Universe and Sample Design and Estimation 1
B2. Procedures for the Collection of Information 3
B3. Methods to Maximize Response Rates 6
B5. Individuals Responsible for Study Design and Performance 6
The U.S. Census Bureau will collect the School Pulse Panel (SPP) data on behalf of NCES. Data collection will be a self-administered, online survey. The survey is estimated to require, on average, 30 minutes of school staff time. The sampled school will be offered a reimbursement of $200 each month that they complete a survey over the course of 11 months between August 2023 and June 2024. The reimbursement will be paid out monthly in the form of debit cards. If a school district does not permit its schools to receive any form of incentive, the reimbursement will be sent to a point of contact in the district, or the reimbursement will be withheld. Principals, or the school staff most knowledgeable about their school environment and experiences within the school, can complete the school-level survey.
The resulting data will provide aggregate estimates for public schools across the nation. The goal will be to have national representation from 1,200 responding schools in order to report out national estimates. To achieve this, a stratified sample design will be used to select approximately 4,000 U.S. public schools. In addition, reserve samples of replacement schools will be selected to boost the number of responses if a notable number of schools or districts from the initial sample do not respond. The sample is designed to provide national estimates of primary, middle, and high schools and districts, taking into account the type of locale (urbanicity) and racial/ethnic student enrollment.
The sampling frame for the School Pulse Panel is derived from the National Teacher and Principal Survey (NTPS) 2023-24 frame, which itself is largely derived from the Common Core of Data (CCD), the file of public schools supplied annually by state educational agencies to NCES. Only public schools in the 50 states and the District of Columbia will be included in the School Pulse Panel sampling frame. A universe collection (115 schools) from the Outlying Areas (Guam, Northern Marianas, U.S. Virgin Islands, American Samoa) will be administered as well. Certain types of schools are excluded, including newly closed schools, home schools, private schools, and schools with a high grade of kindergarten or lower. Regular public schools, charter schools, alternative schools, special education schools, vocational schools, virtual schools, and schools that have partial or total magnet programs are included in the frame. For sample allocation purposes, strata are defined by instructional level, region, percent students of color enrollment, type of locale (urbanicity), and school size.
Tables 1-3 show the estimated expected distribution of the public-school sampling universe for the School Pulse Panel by school level and region (Table 1), school level and urbanicity (Table 2), and by school level and percent students of color enrollment (Table 3), respectively.
Table 1. Expected respondent universe for the SPP sample, by school level and region, based on the 2023-24 NTPS Frame
Region |
Elementary |
Middle/Combined/Other |
High/Grade 9-11 |
Total |
Northeast |
8,450 |
2,969 |
3,443 |
14,862 |
Midwest |
12,468 |
4,461 |
6,051 |
22,980 |
South |
18,075 |
7,480 |
8,157 |
33,712 |
West |
13,306 |
4,805 |
5,813 |
23,924 |
Total |
52,299 |
19,715 |
23,464 |
95,478 |
Table 2. Expected respondent universe for the SPP sample, by school level and urbanicity, based on the 2023-24 NTPS Frame
Urbanicity |
Elementary |
Middle/Combined/Other |
High/Grade 9-11 |
Total |
City |
15,650 |
5,031 |
6,332 |
27,013 |
Suburb |
18,007 |
6,471 |
6,355 |
30,833 |
Town |
5,823 |
2,942 |
3,687 |
12,452 |
Rural |
12,819 |
5,271 |
7,090 |
25,180 |
Total |
52,299 |
19,715 |
23,464 |
95,478 |
Table 3. Expected respondent universe for the SPP school sample, by school level and percent students of color enrollment, based on the 2023-24 NTPS Frame
Percent Students of Color |
Elementary |
Middle/Combined/Other |
High/Grade 9-11 |
Total |
0 to less than 25 |
15,487 |
5,975 |
7,423 |
28,885 |
25 to less than 50 |
11,220 |
4,508 |
4,435 |
20,163 |
50 to less than 75 |
8,884 |
3,529 |
3,576 |
15,989 |
75+ |
16,708 |
5,703 |
8,030 |
30,441 |
Total |
52,299 |
19,715 |
23,464 |
94,578 |
Sample Selection and Response Rates
A stratified sample design has been used to select approximately 4,000 U.S. public schools. The sample will be designed to provide cross-sectional national estimates of primary, middle/combined, and high schools, as well as districts, taking into account the type of locale (urbanicity), racial/ethnic student enrollment, and region. Note that combined schools will be grouped with middle schools for the purposes of measurement and estimation.
There are two stages of sample selection. A base sample of 4,000 schools has been drawn as an initial stage as well as another reserve sample (second stage) of 4,000 schools with similar characteristics to our base sample. In case we do not get the necessary number of schools to participate out of the base sample, we plan to reach out to the reserve sample schools to complete the panel.
Roughly 115 Outlying Areas schools will be a part of the SPP data collection. It is planned for the collection of these schools to be a universe collection, so they are separated from the sample of stateside schools. All study operations will be identical for the collection from these schools except that OA schools will not receive a physical letter inviting them to participate in the survey each month.
Recruitment for the panel study began in March 2023, where districts require applications and approval for their schools’ participation in the study. For those schools and districts that do not have this requirement, recruitment will begin in June and July of 2023. Learning from the 2021-22 SPP cohort and response patterns, the goal for the 2023-24 cohort will be to obtain commitments from 1,200 schools to participate in the study throughout the duration of the year-long monthly collections (roughly 30 percent response rates).
Sample Design for the School Pulse Panel
The main objective of the School Pulse Panel sampling design is to obtain overall estimates broken out by various school characteristics. For sample allocation and sample selection purposes, strata were defined by instructional level. In addition, region, locale, percent minority enrollment, enrollment size, and charter status were used as implicit stratification variables by sorting schools by these variables within each stratum before sample selection. The explicit stratification and the first three implicit stratification variables (region, locale, and percent minority enrollment) are priorities for evaluation for this panel. The method determined to allocating schools to the different sampling strata is allocate them proportionally to the U.S. public school population.
Weights will be attached to each surveyed school so that the weighted data will represent the population. The final weight for completed cases will be composed of a sampling base weight and an adjustment for nonresponse. Nonresponse weighting adjustment cells for the SPP data will be determined using a categorical search algorithm called Chi-Square Automatic Interaction Detection (CHAID). CHAID begins by identifying the school-level characteristics of interest that are the best predictors of response. It divides the dataset into groups so that the unit response rate within cells is as constant as possible and the unit response rate between cells is as different as possible. The characteristics of interest as predictors of response must be available for both respondents and nonrespondents in order to conduct a CHAID analysis, and, in the case of SPP, will be available through the CCD sampling frame. The final, adjusted weights will be raked so that the sum of the weights matches the number of schools derived from the School Pulse frame.
Standard errors of the estimates will be estimated using jackknife replication. Replicate codes that indicate the computing strata and the half-sample to which each sample unit belongs will be developed, as will the weights for all replicates that were formed in order to calculate variances.
Special contact districts require that a research application be submitted to and reviewed by the district before they will allow schools under their jurisdiction to participate in a study. Districts are identified as “special contact districts” prior to data collection because they were flagged as such during previous cycles of the Schools and Staffing Survey (SASS), NTPS, the School Survey on Crime and Safety (SSOCS), or the 2021-2022 SPP. Special contact districts are also identified during data collection when districts indicate that they will not complete the survey until a research application is submitted, reviewed, and approved.
Once a district is identified as a special contact district, basic information about the district is obtained from the NCES Common Core of Data (CCD). The basic information includes the NCES LEA ID number, district name, city, and state. The next step is to search the district’s website for a point of contact and any information available about the district’s requirements for conducting external research. Some districts identified as being a special contact district from the previous cycle may be incorrect and staff will verify whether a given district has requirements for conducting external research before proceeding.
The following are examples of the type of information that will be gathered from each district’s website in order to prepare a research application for submission to this district:
Name and contact information for the district office or department that reviews applications to conduct external research, and the name and contact information of the person in charge of that office;
Information about review schedules and submission deadlines;
Whether application fees are required, and if so, how much;
Whether a district sponsor is required;
Whether an online application is required, and if so, the link to the application, if possible;
Information about research topics and/or an agenda on which the district is focusing;
The web link to the main research department or office website; and
Research guidelines, instructions, application forms, District Action Plans, Strategic Plan or Goals, if any.
Recruitment staff will contact districts by phone and email to obtain key information not listed on the district’s website, (e.g., requirements for the research application, research application submission deadlines, etc.). OMB approval to contact these districts was obtained in February 2023 (OMB#1850-0975)
SPP staff developed a generic research application that covers the information typically requested in district research applications. Staff will customize the generic research application to each district’s specific requirements that need to be addressed or included in the research application (e.g., how the study addresses key district goals, or inclusion of a district study sponsor), or submit the generic application with minimal changes to districts that do not have specific application requirements.
Using the information obtained from the district website or phone or email exchanges, a district research request packet will be prepared. Each research application will include the following documents, where applicable:
District research application cover letter;
Research application (district-specific or generic, as required by the district);
Study summary;
FAQ document;
Special contact district approval form;
Participant informed consent form (if required by the district);
SPP Project Director’s resume;
Copy of questionnaires; and
Application fee (if required by the district).
Where required or requested, applications will include draft 2023-24 SPP questionnaires. The SPP content that most closely matches the 2023-24 SPP is that fielded in 2022 and may be provided as an exemplar of questionnaires to be administered in school year 2023-24. Additionally, SPP staff will recommend district staff to reference this package to review draft items provided in the item bank in Attachment B. Other information about the study may be required by the district and will be included with the application or provided upon request.
Approximately one week after the application is submitted to the district (either electronically or in hard copy, as required by the district), SPP district recruitment staff will contact the district’s research office to confirm receipt of the package and to ask when the district expects to review the research application and when a decision will be made. If additional information is requested by the district (e.g., the list of sampled schools), recruitment staff will follow up on such requests and will be available to answer any questions the district may have throughout the data collection period.
Some districts charge a fee (~$50-200) to process research application requests, which will be paid as necessary. Special district operations will begin by contacting up to 100 “certainty” special contact districts for which, due to their size, it is certain that at least one school from their jurisdiction will be randomly sampled. Other special contact districts will be contacted after the sample is drawn in late spring of 2023.
The school and district precontact notification includes mailing a two-sided, full-color letter to sampled schools to introduce the survey, to promote survey recognition, and to verify the school’s mailing address.
A school screener survey will be sent via email to principals to obtain point of contact information for the delivery of the monthly data collections. This information will collect principal information, designated point of contact information, and reimbursement mailing address information.
The School Pulse Panel data collection will begin in August of 2023. The U.S. Census Bureau, acting as a contractor for NCES, is responsible for the data collection operation. For each month, each sampled school (or district) will receive a letter and an email notifying them of the survey, which will include information necessary to complete the online questionnaire. Respondents will have a two-week window to respond to the survey. Reminder emails will be sent during the data collection window.
Due to the expedited development of this study and release of data, NCES considers the SPP data collections as experimental. Therefore, NCES data quality standards may not be met and estimates will be released on the SPP dashboard within 5-6 weeks after the end of data collection. This process will be repeated monthly throughout the 2023-24 school year. As of the time of the submission of this package, NCES and the Census Bureau have begun coordinating efforts to determine the potential of moving away from the “experimental” label of this data collection. Specifically, extensive nonresponse bias analyses will be conducted for each monthly collection for the 2021-22 cohort to determine if substantial bias is found as a major concern. It is intended to continue labeling this study as “experimental” until the study has been cleared of this labeling.
The samples of schools will be drawn in late spring of 2023. Many districts (known as “special contact districts”) require research applications to be submitted and reviewed for approval before contact can be made with schools within their districts. Special contact district outreach was cleared in February 2023 (OMB# 1850-0975) and recruitment will begin in March 2023.
School Communication
The School Pulse Panel will be conducted via a self-administered web-based survey instrument. A clerical operation prior to data collection will be conducted to obtain e-mail addresses for all the sampled school principals, and these e-mail addresses are used to contact the school principals for the screener and throughout the monthly data collections. For each month, an initial letter and initial email are sent requesting their participation in the study and to inform schools about reimbursements of $200 a month offered to schools over the course of the school year for their continued participation in the study. Reminder e-mails are sent throughout the two-week data collection window.
A copy of the letters and e-mails to be sent to school principals throughout the SPP data collection are included in appendix A.
During the monthly data collections, Census Bureau staff will initiate phone calls with nonrespondents, reminding them to complete their questionnaire.
Refusal Conversion for Schools That Will Not Participate
If a school expresses strong concerns about confidentiality at any time during data collection, these concerns will be directed to the Census Project Director (and possibly to NCES) for formal assurance. All materials will include the project’s toll-free number. In addition, initial emails will include information about why the participation of each sampled school is important and how respondent information will be protected.
NCES is committed to obtaining a response rate for the full School Pulse Panel survey that allows for reporting national-level estimates on a monthly basis. In general, a key to achieving this is to track the response status of each sampled school, with telephone follow-up, as well as follow-up by e-mail, of those schools that do not respond promptly. To help track response status, survey responses will be monitored through an automated receipt control system. Telephone interviews will be conducted only by Census Bureau interviewers who have received training in general telephone interview techniques as well as specific training for SPP.
As part of the development of the SPP instruments, the monthly instruments will regularly undergo cognitive testing. New items proposed in this package will go through cognitive testing with 20-30 school and district personnel. This testing will occur in late spring and early summer of 2023. For any new items that are developed that are not included in this package, quarterly monthly instruments with the new items will be posted for 30-day comment and may simultaneously undergo cognitive testing with 8-9 school personnel. Change requests reflecting modifications to items based on feedback from cognitive testing will be submitted and approved prior to each monthly collection. In March 2024, we will incorporate a communications experiment to test whether the tailored communications NCES has developed improves response rates over more generic communications. Half of respondents will receive materials that contain detailed information about the topic areas they will be surveyed on, and half will receive materials that do not contain this information (see pp. 51-63 in Appendix A2). The estimated burden has not changed as a result of this addition because respondents will receive only one version of the letter and email via random assignment (communications with information about the monthly collection vs. communications with no information about the monthly collection). These communication materials may be updated at a later time through a change request.
Several key staff responsible for the study design and performance of the School Pulse Panel. They are:
Rebecca Bielamowicz, National Center for Education Statistics
Ryan Iaconelli, National Center for Education Statistics
Chris Chapman, National Center for Education Statistics
Andrew Zukerberg, National Center for Education Statistics
Cassandra Logan, U.S. Census Bureau
Elke McLaren, U.S. Census Bureau
Alfred Meier, U.S. Census Bureau
Kathleen Kephart, U.S. Census Bureau
Jessica Holzberg, U.S. Census Bureau
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kathryn.Chandler |
| File Modified | 0000-00-00 |
| File Created | 2023-09-09 |