Evaluation of Teacher Residency Programs
Part B: Collections of Information Employing Statistical Methods
December 2024
Submitted to: |
Submitted by: |
U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance 550 12th Street, S.W. Washington, DC 20202 Project
Officer: Meredith Bachman |
Mathematica P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Fax: (609) 799-0005 Project Director: Melissa Clark Reference Number: 50911 |
Contents
Part B. Supporting Statement for Paperwork Reduction Act Submission 1
B1. Respondent universe and sampling methods 1
B2. Statistical methods for sample selection and degree of accuracy needed 2
B3. Methods to maximize response rates 3
Tables
Appendices
Appendix A: District data form
Appendix B: TQP grant staff data form
Appendix C: Email notifications for data form
Appendix D: Response to Public Comments from the 60-day Federal Register Notice
The U.S. Department of Education (ED)’s Institute of Education Sciences (IES) requests clearance for data collection activities to evaluate teacher residency programs’ contributions to the teacher workforce. Specifically, this request covers the collection of data to understand the extent to which teacher residency programs funded by ED’s Teacher Quality Partnership (TQP) grants help diversify the teacher pipeline and fill hard-to-staff teaching positions in their partner districts. The Higher Education Act established the TQP program to support partnerships between teacher preparation programs and high-need districts to implement teacher residency programs. The study will collect data on newly hired teachers from the school districts and charter school networks (referred to throughout as “districts”) that partner with TQP grantees. If some districts are unable to provide data on new hires from TQP-funded residency programs, we will collect those data from TQP grantees.
The respondent universe will consist of the 80 school districts and charter networks (referred to as “districts” throughout) that partner with the fiscal year (FY) 2019 Teacher Quality Partnership (TQP) grantees. The TQP grants are ED’s largest source of funding for teacher residency programs. By focusing on the universe of districts that partner with the FY2019 TQP grantees, the study’s findings will provide valuable information for policymakers on the extent to which TQP-funded residency programs are helping districts diversify their teacher workforce and fill hard-to-staff positions. The study will focus specifically on the FY 2019 grantees. The main benefit of focusing on this cohort of grantees is that we can measure outcomes for TQP-funded residency programs in the final year of their grants—ensuring that grantees have had sufficient time to implement and refine their programs. As a result, the data collection will focus on the final cohort of completers from these TQP-funded residency programs.
The study team used two existing sources of information to identify the districts that partner with the FY2019 TQP grantees:
Survey of teacher residency programs. A survey of teacher residency programs from the 2022–2023 school year conducted by Mathematica asked programs to identify their district partners.
FY 2019 TQP grant applications. In cases where a residency program did not provide the names of district or charter partners in the survey we will supplement with information from the grant applications.
Among the 31 FY2019 TQP grantees, 23 responded to the survey, 3 did not respond to the survey, 3 did not meet the study’s definition of teacher residency programs and were deemed ineligible (two of these programs had residents serve as the teacher of record and the other program trained teacher leaders), and 2 were not longer operating. The 23 grantees that responded to the survey identified a total of 75 school district partners. Based on grant applications, the 3 districts that did not respond had 5 district partners, for a total of 80 district partners among the FY2019 TQP grantees.
There is no statistical methodology for the sample selection. The study will collect data from all of the districts that partner with the FY2019 TQP grantees.
To address the research questions, the study team will analyze the aggregate data districts provide about the residency and non-residency new hires in their districts.
The study team will describe the extent to which residency programs help districts diversify the teacher workforce by tabulating aggregate counts of newly hired teacher by race and ethnicity and comparing the percentage of new hires from TQP-funded residency programs who are teachers of color to the percentage of (1) new hires from other routes into teaching who are teachers of color and (2) all teachers in the partners districts who are teachers of color. The study will also compare the the percentage of new hires from TQP-funded residency programs who are teachers of color to the percentage of students of color in the partners districts.
To examine the extent to which new hires from TQP-funded residency programs fill hard-to-staff positions in partner districts, the study team will compare the percentage of new hires from TQP-funded residency programs who fill teaching positions that districts view as hard-to-staff compared to the percentage of new hires from other routes. The study team will conduct a similar comparison for the percentage of new hires who fill positions in hard-to-staff schools. For both of these analyses, the study team will focus on the teaching positions and schools that districts describe as hard-to-staff.
To examine the extent to which the TQP grant timeline might have played a role in the study findings, the study team will report the percentage of grantees that continued their TQP-funded residency programs beyond the grant period. Because grantees may have revised the design of their programs after the grant period, we will also report the percentage of grantees that made changes to key aspects of their program design (for grantees that continued their programs). For grantees that did not continue their programs, we will report the percentage that cited specific challenges to continuing their programs.
This section does not apply to the forms because they will be based on a census of the full universe of FY2019 TQP grantees and their partners (with an anticipated response rate of 85 percent). The study will compare the average percentage of new hires from TQP-funded residency programs who are teachers of color or fill hard-to-staff positions with the average percentage for new hires from other routes, using simple descriptive comparisons of means.
The study team does not anticipate any unusual problems that require specialized sampling procedures.
The team will complete the study data collection activities only once.
Developing and testing the form to maximize ease of completion and reduce respondent burden. The study team developed an Excel form for districts (Appendix A) and TQP-funded residency programs (Appendix B) (when needed) to provide the requested data, following processes that have proved successful for similar data collections. The team minimized the length of the district data form to only include the request for the counts of teachers, with no other open- or close-ended questions. To reduce item nonresponse, the form includes programmed checks alerting respondents to out-of-range or inconsistent responses they enter. These checks allow respondents to change their response based on guidance provided in the form or leave their answer and continue to the next question. The study team conducted a pre-test to test and revise the form for clarity, accuracy, length, flow, and wording.
Providing clear and informative emails to district and TQP grantee staff (Appendix C). The study team will send an advance email to the identified contact with an overview of the data collection effort. After the advance email goes out, the study team will follow-up with an invitation email that contains specific instructions on how to access and complete the form. The study team will also send reminder emails during the data collection period to encourage responses. To reassure respondents that the data they provide will be kept confidential, the study team will include a statement on confidentiality and data collection requirements (Education Sciences Reform Act of 2002, Title I, Part E, Section 183) in the forms.
Dealing with nonresponse and reporting errors. The study team will identify nonresponse and reporting errors by checking for complete and reasonable answers as soon as grantees and districts complete their forms and will follow-up with respondents about any errors. The study team will also follow-up with grantees and their partner districts as needed to minimize overall nonresponse and reach the target response rate.
The study team pre-tested the data forms during the 60-day public comment period to confirm the time needed to complete the forms, improve the clarity of questions, and reduce burden. The study team pre-tested the grantee data form with four TQP grantees from prior cohorts and the district data form with six of their district partners. Results from the pre-test informed changes made to the the form for clarity and accuracy, including the wording of questions, response options, and estimated completion time (we reduced the estimated completion time from 4 to 3 hours based on the pre-test).
The pre-test provided an opportunity to test the forms to ensure they function as designed. This included manual testing for the accuracy of any customized text for respondents, correct enforcement of select one versus select multiple options, and other logic. This will increase the accuracy of data collected while minimizing respondent burden.
The following individuals were consulted on the statistical aspects of the study design and on collecting and analyzing data:
Table B.1. Individuals consulted on statistical design
Name |
Title |
Telephone Number |
Mathematica staff |
|
|
Melissa Clark |
Principal researcher |
(609) 750-3193 |
Jeffrey Max |
Principal researcher |
(202) 484-4236 |
Philip Gleason |
Senior fellow |
(202) 264-3443 |
Libby Makowsky |
Senior researcher |
(734) 794-8026 |
IES staff |
|
|
Meredith Bachman |
Research scientist |
(202) 245-7494 |
Tom Wei |
Branch Chief, National Center for Education Evaluation |
(202) 453-7370 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Promise Neighborhoods OMB Part B |
| Author | MATHEMATICA |
| File Modified | 0000-00-00 |
| File Created | 2025-01-08 |