1875-0256TIF Surveys_4249_Supporting Statement B_9 15 10

Request for Clearance of Data Collection Instruments for the

TEACHER INCENTIVE FUND EVALUATION

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

B. Collections of Information Employing Statistical Methods 2

1. Respondent Universe and Sample Selection 2

2. Data Collection 3

3. Methods to Maximize Response Rates 3

4. Protocol Development and Review 3

5. Contact Information 3

Request for Clearance of Data Collection Instruments for the

TEACHER INCENTIVE FUND EVALUATION

II. SUPPORTING Statement For Paperwork Reduction Act Submission

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sample Selection

This revision request is for representative surveys of principals and teachers in all TIF grantee sites in winter 2011 and includes a discussion of the previously approved site visits in fall 2010 and fall 2011. The proposed design for the principal and teacher surveys was developed with the primary goal of providing representative samples of principals and teachers from all sites to capture the full diversity of grantee efforts and measure both national and grantee-level estimates (when possible) with precision.¹

Respondents for the site visits will represent the range of participants and stakeholders in each grantee (key informants). These key informants will include TIF project staff members (e.g., project directors, evaluators), educators (e.g., teachers, principals), administrative staff (e.g., district superintendent, State administrator) and other stakeholders (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials). The study team will work with the TIF project director in each grantee to identify key informants.

Teacher Principal Survey

Below we discuss the sampling design for the principal and teacher surveys, highlighting differences in design. Five principles guide the proposed sample design:

1. The sample will represent the full diversity of the efforts undertaken by the 33 grantees.

2. The sample will allow a national estimate to be measured with precision.

3. The sample will allow grantee-level estimates to be measured with precision where the grantee’s size permits.

4. The sample will yield data from all 33 grantees.

5. The sample shall control data collection costs.

One of the motivations for adding a survey to that evaluation is that it is the only means of collecting comparable data on the workings of the program from all 33 sites. The information currently available suggests the approaches taken across the 33 grantees vary substantially, which causes us to believe grantee-level estimates will significantly enhance our understanding of TIF’s effects. The feasibility report results will be provided to OMB and the proposal for the outcomes study will be discussed with OMB prior to any submission.

As part of the telephone data collection activities described in the previously approved collection (completed), researchers gathered up-to-date information on grantee size and use those figures in the survey sampling design. The numbers presented here do not distinguish between participating eligible teachers or principals and nonparticipating eligible teachers or principals (i.e., eligible teachers who opted out of the TIF project). For many grantees this is not an issue because participation in at least some aspect of TIF is required for all teachers and principals.

Principal Survey

We will aim to obtain random sample of participating principals from all grantees. Because most grantees have a relatively small number of schools (and, therefore, principals), it is not possible to obtain grantee-level estimates of principals’ responses with any degree of precision for most grantees. The sampling is designed instead to provide enough power to conduct comparisons of principals’ responses between two subgroups of grantees (e.g., prior grantee experience with educator performance pay vs. new to performance pay) with relative precision. Additionally, to insure that the resulting estimates are representative of the TIF project as a whole and can facilitate comparisons across a range of potential subgroups, the sample will include at least 15 schools for all grantees (or all schools in the grantee, if that is fewer than 15 schools); and will not sample more than 60 schools from any grantee.

Based on those parameters, Exhibit B1 presents the distribution of principal surveys across the 33 grantees following the procedure described above. Each row refers to a specific grantee. The first column presents the number of schools per grantee which corresponds to the number of principals (assuming one principal per school). Column 2 presents the grantee’s schools/principals as a proportion of the national TIF school/principal population. Column 3 lists the number of principal surveys fielded following the imposition of a minimum sample size of 15 and a maximum sample size of 60. Assuming an 80 percent response rate, column 4 lists the number of completed principal surveys by grantee. The final column shows the implied standard error for a mean of 0.5 and moderate school-level clustering of teacher responses (rho=0.2). Our calculations estimate a standard error of 0.024 for a survey of 526 principals (or a sample of approximately 657 principals assuming a response rate of 80%).

Exhibit B1. Sample Reflecting 500 Completed Principal Surveys

Implied Standard Errors by Site and Nationally for a Mean of 0.5, Moderate School-level Clustering of Rho=0.2

Grantee	(1)	(2)	(3)	(4)	(5)
	Number of schools/principals	Percent of all grantee principals represented	Number of fielded principal surveys	Number of completed surveys (80% response rate)	Implied standard error for mean of 0.5 (rho=0.2)
Grantee 1	26	2.3%	18	14	0.129
Grantee 2	11	1.0%	11	9	0.151
Grantee 3	18	1.6%	18	14	0.129
Grantee 4	132	11.6%	72	58	0.065
Grantee 5	13	1.1%	13	10	0.139
Grantee 6	21	1.8%	18	14	0.129
Grantee 7	4	0.4%	4	3	0.250
Grantee 8	25	2.2%	18	14	0.129
Grantee 9	116	10.2%	67	54	0.067
Grantee 10	36	3.2%	21	17	0.120
Grantee 11	10	0.9%	10	8	0.158
Grantee 12	40	3.5%	23	18	0.114
Grantee 13	10	0.9%	10	8	0.158
Grantee 14	1	0.1%	1	1	0.500
Grantee 15	42	3.7%	25	20	0.112
Grantee 16	16	1.4%	16	13	0.129
Grantee 17	5	0.4%	5	4	0.224
Grantee 18	7	0.6%	7	6	0.189
Grantee 19	25	2.2%	18	14	0.129
Grantee 20	10	0.9%	10	8	0.158
Grantee 21	48	4.2%	28	22	0.104
Grantee 22	8	0.7%	8	6	0.177
Grantee 23	6	0.5%	6	5	0.204
Grantee 24	64	5.6%	37	30	0.090
Grantee 25	23	2.0%	18	14	0.129
Grantee 26	6	0.5%	6	5	0.204
Grantee 27	30	2.6%	18	14	0.129
Grantee 28	17	1.5%	17	14	0.129
Grantee 29	213	18.7%	72	58	0.065
Grantee 30	27	2.4%	18	14	0.129
Grantee 31	27	2.4%	18	14	0.129
Grantee 32	6	0.5%	6	5	0.204
Grantee 33	99	8.7%	18	14	0.129
National Estimates	1,142	100%	657	526	0.024

Source: “Hill Briefing Table” Compiled by U.S. Department of Education

Teacher Survey

We will aim to obtain an equal probability sample of participating teachers within each grantee controlling the number of teachers that would need to be sampled from any individual school. We first select the desired number of schools—for example, 15 or one out of four—with probability proportionate to the school’s number of TIF-eligible teachers (imposing minimum and maximum numbers) and then choose an equal number of teachers among the sampled schools. This strategy assumes that researchers will be given access to roster information on eligible teachers in all participating schools, either directly from the grantees or through ED. Since grantees are required to maintain rosters of TIF-eligible teachers, we assume that they will serve as our primary source of information needed to select teacher survey respondents. To save on costs, we plan first to sample schools and obtain lists of teachers for each selected school. Then we will sample teachers from the lists for those schools.

The distribution of the teacher sample involves first distributing the sample across grantees according to the number of teachers in each site, and then adding additional sample where it could increase the precision of the estimates.

There is substantial range in the number of TIF-eligible teachers across the grantees (ranging from about 30 to over 10,000). To capture the full diversity of grantee efforts, we need to obtain large enough within-grantee samples of teachers. We will therefore sample more teachers in smaller grantees than implied by the national share. In addition, to avoid having too much of the sample in one project, we will sample fewer teachers in the largest grantees (e.g., Dallas) than implied by the national share. This approach will require that we establish a minimum and a maximum within-grantee teacher sample size. Currently, we envision a minimum sample size of 115 (or the entire population from the grantee if there are fewer than 115 teachers) and a maximum sample size of 440 teachers. We expect to receive between 90 and 352 surveys per grantee assuming a roughly 80 percent response rate. To obtain nationally representative estimates, we will assign less weight to teachers from the over-sampled small grantees and more weight to those in the under-sampled large grantees.

In the second-stage, we selectively increase the number of teachers and schools to be sampled in order to increase the precision of grantee-level estimates. Based on the teacher survey sample identified at the end of the first stage, there are four grantees for which we can measure grantee-level estimates with relative precision (e.g., Denver, CO; Hillsborough, FL; Dallas, TX; and Houston, TX). By increasing the number of teachers and schools to be sampled, we are able to add eight more grantees to that list (i.e., standard errors less than or equal to 0.055). It is not possible to achieve standard errors less than or equal to 0.055 in the remaining grantees even if we sample all TIF-eligible teachers in all schools. These projects are not large enough.

The teacher survey will include more than 8,000 teachers to be sampled from approximately 640 schools representing all grantees (except for two that use TIF solely for principal performance pay). Assuming an 80 percent response rate, this would yield 6,483 completed teacher surveys. The large teacher sample gives the proposed design several advantages, including a sample representative of the full diversity of grantee efforts as well as the precise measurement of national and grantee-level estimates (where grantee size permits).

To give a sense of the likely precision of estimates based on an 80 percent completion rate, we calculate the standard errors for an outcome with a mean equal to 0.5 (e.g., half of sampled teachers indicate a statement is true) with a moderate school-level clustering of teacher responses (intraclass correlation of 0.2). This level of clustering provides fairly conservative estimates of the expected standard errors. Should actual clustering be less pronounced than this, the standard errors would shrink, and our estimates would be more precise. (The confidence interval around an estimate would range from two standard errors below to two standard errors above the estimate.) We selected a mean of 0.5 because standard errors are largest in level for a proportion of 0.5 over the range of 0 to 1. Our current calculations estimate a standard error of 0.015 for national estimates and standard errors not exceeding 0.055 in the 12 sites with the largest numbers of TIF-eligible teachers.

Exhibit B2 shows how a sample of 8,104 fielded surveys would be distributed across the 31 grantees (excluding the two that use TIF solely for principal performance pay) following the procedure detailed above. Each row refers to a specific grantee. The first two columns present the number of participating schools and teachers per grantee, respectively. Column 3 presents the grantee’s teachers as a proportion of the national TIF teacher population. Column 4 lists the number of fielded teacher surveys following the imposition of a minimum sample size of 115 and a maximum sample size of 440 and selectively increase the number of sampled teachers in order to decrease grantee-level standard errors. Assuming an 80 percent response rate, column 5 lists the number of completed teacher surveys by grantee. Column 6 lists the number of schools from which the teachers will be sampled. The final column shows the implied standard error for a mean of 0.5 and moderate school-level clustering of teacher responses (rho=0.2).

Exhibit B2. Sample Reflecting 6,500 Completed Teacher Surveys

Implied Standard Errors by Site and Nationally for a Mean of 0.5, Moderate School-level Clustering of Rho=0.2

Grantee	(1)	(2)	(3)	(4)	(5)	(6)	(7)
	Number of schools	Number of teachers	Percent of all grantee teachers represented	Number of fielded teacher surveys	Number of completed surveys 80% response rate	Number of schools with surveys	Implied standard error for mean of 0.5 (rho=0.2)
Grantee 1	26	109	0.2%	109	87	26	0.065
Grantee 2	11	534	1.2%	213	170	11	0.076
Grantee 3	18	736	1.7%	238	190	18	0.062
Grantee 4	132	0	0.0%	0	0
Grantee 5	13	361	0.8%	213	170	13	0.071
Grantee 6	21	400	0.9%	400	320	21	0.055
Grantee 7	4	250	0.6%	163	130	4	0.118
Grantee 8	25	229	0.5%	229	183	25	0.056
Grantee 9	116	7,195	16.2%	440	352	29	0.048
Grantee 10	36	54	0.1%	54	43	36	0.078
Grantee 11	10	1,004	2.3%	188	150	10	0.08
Grantee 12	40	1,240	2.8%	289	232	30	0.05
Grantee 13	10	700	1.6%	188	150	10	0.08
Grantee 14	1	62	0.1%	62	50	1	0.071
Grantee 15	42	100	0.2%	100	80	42	0.061
Grantee 16	16	1,200	2.7%	210	168	16	0.066
Grantee 17	5	280	0.6%	188	150	5	0.106
Grantee 18	7	292	0.7%	188	150	7	0.092
Grantee 19	25	285	0.6%	285	228	25	0.054
Grantee 20	10	262	0.6%	188	150	10	0.08
Grantee 21	48	1,305	2.9%	297	237	30	0.05
Grantee 22	8	70	0.2%	70	56	8	0.099
Grantee 23	11	345	0.8%	213	170	11	0.076
Grantee 24	64	0	0.0%	0	0
Grantee 25	23	3,649	8.2%	660	528	23	0.051
Grantee 26	6	204	0.5%	163	130	6	0.099
Grantee 27	30	1,050	2.4%	278	222	30	0.051
Grantee 28	17	326	0.7%	238	190	17	0.063
Grantee 29	213	10,391	23.4%	440	352	53	0.039
Grantee 30	27	4,187	9.4%	440	352	27	0.049
Grantee 31	27	1200	2.7%	385	308	27	0.05
Grantee 32	6	200	0.4%	163	130	6	0.099
Grantee 33	99	1,980	4.5%	348	278	30	0.049
National Estimates	1,142	44,450	100%	8,104	6,483	642	0.015

Source: “Hill Briefing Table” Compiled by U.S. Department of Education

Site Visits

Respondents for the telephone interviews and both site visits will represent the range of participants and stakeholders in each grantee (key informants). These key informants will include TIF project staff members (e.g., project directors, evaluators), educators (e.g., teachers, principals), administrative staff (e.g., district superintendent, State administrator) and other stakeholders (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials). The study team will work with the TIF project director in each grantee to identify key informants.

We will use the site visits to delve more deeply into the components and system supports that we learned of during the telephone interviews (completed). Our proposed approach is designed to document and explain varying levels of implementation consistency with plans and the authorizing legislation; long-term implementation issues and challenges; and the principal, teacher, and student outcomes of performance pay programs across all grantees.

Site Visit 1

Based on the information we learned in the telephone interviews and a thorough review of extant grantee documents we will produce detailed grantee profiles from which we will select a sample of projects for Site Visit 1 (2010). We are proposing a stratified random sample of active grantees. We will remove any grantee sites in which we find the projects was not implemented (based on conversations with the TIF program office) or in the event that a grantee’s project ceases to operate over the course of this study. Although these non-implemented/discontinued projects would not be included in the case studies, we will capture valuable implementation data from the telephone interviews and will include extensive information on the reasons for any issues with implementation or continued operation.

From the population of active grantees that have implemented plans, we will randomly select four projects where 50 percent or more of the TIF grant is focused on performance pay for improved student learning, four projects implementing broader forms of differentiated compensation (e.g., to attract teachers to hard-to-staff schools), and four projects that have comprehensive systems that combine performance pay for improved student learning and other forms of differentiated compensation (Exhibit B3). This approach is designed to yield a sample that represents the range of implementation fidelity, but does not waste evaluation resources on studying sites where implementation never occurred or was not sustained through the grant period.

Exhibit B3. Stratified Random Sample for Site Visit 1

Performance Pay	Differentiated	Comprehensive
4 grantees	4 grantees	4 grantees

The actual sample will be developed in consultation with the TWG and will depend in part on the distribution of different strategies across the population of eligible projects. If possible, we will also consider urbanicity and strength of local bargaining units as additional sampling criteria.

We are aware that sampling plans that do not include discontinued projects may undermine the strength of inferences about factors associated with project success because the presence or absence of those factors is never investigated in this excluded subset. In fact, this shortcoming plagued the early literature on effective schools. However, gathering extensive information in telephone interviews will likely mitigate this potential weakness. Furthermore, by focusing on those grantees that have enacted their plans, the first round of case studies will be able to document the long-term implementation challenges—an area not well understood in the current research literature on performance pay.

Site Visit 2

The sampling approach we recommend for the second site visit (in 2011) would stratify by project outcome in addition to the strata described for Site Visit 1. This would require conducting the optional outcome analysis prior to the second round of case studies in order to sample based on grantees’ observed outcomes. (Note: The outcome analysis is described in the introduction to Supporting Statement A). This approach would ensure that the evaluation would gather in-depth data on the grantee plans, practices, and contextual factors that lead to a range of student and teacher outcomes. Thus, as part of our commitment to advancing the research base on performance pay, the second round of case studies will focus on explaining outcomes—another neglected area of the research literature. We propose to select a stratified random sample of three high-performing projects and one low-performing project (with performance based on findings from the outcomes study) from each of the three types of grantee programs. Exhibit B4 illustrates one scenario under this approach.

Exhibit B4. Stratified Random Sample for Site Visit 2

	Performance Pay	Differentiated	Comprehensive
High Performing	3 grantees	3 grantees	3 grantees
Low Performing	1 grantee	1 grantee	1 grantee

This strategy will provide comprehensive data on implementation consistency with authorizing legislation and project plans; long-term implementation challenges; and the principal, student, and teacher outcomes associated with performance pay.

At this point the total number of grantees to be included in the site visits is not yet known. It is possible that the same 12 sites are included in both Site Visit 1 and Site Visit 2 or it may be that there is no overlap yielding 24 different grantees over the two sets of site visits.

2. Data Collection

Surveys of Principals and Teachers

Surveys of principals and teachers will be conducted in winter 2011. The surveys will be available both in a paper version and online. The paper version will be distributed to respondents at their schools by a site-liaison (described below). The introductory language of the paper survey respondents will be directed to a link if they prefer to take the survey online and provided a unique ID to enter.

To gather contact information for principals and teachers, both for survey administration and for drawing the teacher sample, our initial strategy shall be to rely on grantee sites to provide staff rosters for participating schools. For survey administration, we will need school addresses, school phone numbers, and individual e-mail addresses if available (to allow for e-mailing a link to complete the survey online). Working with the grantee sites to collect staff rosters is more cost-effective than going to each participating school separately.

To facilitate completion of the teacher surveys, researchers will contact principals (or assistant principals) at each school in the sample to ask them to designate a liaison to assist with teacher survey administration. The liaison will be someone with a good working relationship with the teachers and who will be able to easily and ethically communicate with survey respondents on behalf of the evaluation team. Liaisons will be oriented to the project by researchers from the study team and instructed in the distribution and collection of surveys.

The surveys for each school will be packaged and sent directly to the liaison with a master list and tools to track the survey returns for their school. Each of the surveys will be stuffed in an envelope printed with tracking information. Liaisons will distribute the surveys to the appropriate teachers and instruct them to complete the survey, place it back in the envelope, and seal it with a sticker that is included with each survey. Liaisons will collect the completed surveys and conduct follow-up with their teachers. The surveys will be returned by the liaison in a package rather than by individual teachers.

In addition to distributing and collecting surveys, a primary role of the liaison is follow-up assistance with nonrespondents. This could include in- person contact, mailbox notes, and phone calls. Each liaison will receive a pre-paid stipend, in the form of a gift card, at a rate of $5 per teacher survey for which they are responsible. Given a smaller principal sample, we will not rely on the liaisons for this follow-up. Researchers will conduct follow-up calls directly with principals to generating a high response rate for this population. As mentioned in Supporting Statement A, because the school liaisons are hired by the contractor to act as assistants with distribution and pick-up of the surveys, and follow-up assistance, they are not considered respondents. Therefore, the time spent in these administrative activities is not considered as burden in this collection.

Researchers will develop a database for tracking completed surveys and use it to guide follow-up efforts with nonrespondents. The database will include information regarding the data surveys were received as well as whether it was taken online or submitted on paper.

Site Visits

The sampling issues related to the site visit data collection activities are covered in the previous section and described in the data collection tasks and deliverables described in Exhibit A4.

3. Methods to Maximize Response Rates

Given the support and endorsement of this evaluation by the TIF program office and the grantee’s knowledge of and expected participation in this evaluation, we anticipate cooperation and participation by all grantees. Grantees were made aware of this evaluation in the original application, points were awarded for agreeing to participate, and in fact all proposed participating in the evaluation. In addition, members of local TIF grantee project teams were introduced to the study (and its leaders) at the annual grantee meeting hosted by CECR in the summer of 2009. The TIF program office will also provide support for the study and convey its importance to grantees through their regular communication with grantees. Initial communication for all data collection will be made with the TIF project directors at each grantee.

Surveys of Principals and Teachers

A number of steps have been built into the survey data collection process to maximize response rates for the surveys of principals and teachers. The methods specifically designed to support our telephone interviews and site visits were discussed in the previously approved submission.

• The use of a school-site liaison is a pivotal support for maximizing our response rates. For each school selected into the study, researchers will contact school administrators to identify a liaison to support data collection. The school site liaisons will assist with survey administration with a primary responsibility of follow-up with nonrespondents. Stipends will be offered to liaisons at a rate of $5 per teacher survey for which they are responsible, in the form of a gift card. These stipends will be provided before the surveys are distributed to teachers.

• An on-line version of the survey will be made available to respondents who prefer this mode of responding.

• A 1-800 number and email address will be provided on all of the materials for respondent questions with a researcher response turnaround of no more than one business day.

• A letter of support from ED will be included in the mailing to communicate the importance of the study to TIF participants and encouraging participation. We will also look to the program office at ED and the Center for Educator Compensation Reform (CECR) to reinforce the importance of survey participation in their ongoing communication with grantees.

Timely follow-up is also built-in to yield high response rates including:

• A tracking database will be used to guide follow-up efforts with nonrespondents. Liaisons will be contacted 10 days after the initial mailing with additional follow-up calls made at approximately 10 day intervals.

• Follow-up will include reminder postcards, phone calls by the evaluation team, phone calls and other reminders from the grantee site liaisons, and, where possible, in-person follow-up by the liaisons.

• If necessary researchers will conduct the surveys by phone to increase the response rate.

Site Visits

Project directors will help to identify key informants for site visits. Multiple attempts will be made to reach identified respondents including phone calls, emails, and follow-up by project staff. We will work with the project directors to identify the most appropriate way to gain access to school employees and provide scheduling templates and funding for substitute coverage when necessary. In the rare cases when identified informants cannot be reached or are unavailable during the time period of the site visits we will work with the project directors to identify alternate informants. Informants will be contacted by experienced and well-trained interviewers who will introduce the study by providing relevant background and rationale. In similar studies we have found that interviews such as these provide a venue for respondents to share experiences and contribute to the body of knowledge which motivates many respondents. In addition we have taken the following steps to maximize participation and minimize respondent burden:

• We have worded all data collection instruments as concisely as possible. To the extent possible, we will coordinate data collection activities within the evaluation team in order to ensure that these activities impose a manageable burden on respondents, while yielding data that collectively answer the evaluation questions of most interest to policy-makers and the field.

• Prior to data collection, we will send letters of introduction to project directors informing them of the study and describing all relevant data collection activities. The letters will include: (1) contact information for evaluation team staff who can answer questions about the study, (2) information about OMB clearance, and (3) contact information for the TIF program office.

4. Protocol Development and Review

Surveys of Principals and Teachers

Prior to finalizing the attached protocols, we conducted pilot interviews and solicited feedback from our TWG. During the months of November and December, SRI researchers piloted the teacher and principal surveys with 7 respondents from a range of sites across the country. Teachers and principals who participated were asked to complete the survey independently and to note both the amount of time required to complete the survey and any areas of confusion. Upon completion of the survey, participants were interviewed by researchers who gathered general impressions about the survey and asked detailed questions about several survey items. The surveys were piloted with teachers and principals from both traditional public schools and charter schools.

Teachers and principals provided positive feedback regarding their experiences taking the survey. The survey formatting and instructions were easily understood by respondents. Respondents were able to locate their program on the cover page, understood all questions, and followed instructions for skip patterns. There were no questions that seemed out of place or overly invasive (a potential concern for questions about pay) to teachers and principals. Further, nearly all terms and survey phrases were commonly understood by respondents. Where unclear, we have revised.

We also asked two TWG members, Matthew G. Springer and Ellen B. Goldring, to review the surveys. In addition, SRI, BPA, and the Urban Institute completed phone interviews with key informants in each grantee (as part of the phone interviews data collection task). We have reviewed the interview data to ensure that the surveys reflect our current understanding of the program designs implemented across grantees.

Site Visits

The protocols for the site visits (previously approved) were developed and revised based on pilot interviews and feedback from our TWG. In collaboration with the TIF program office we identified three TIF sites from which to solicit feedback on the three draft interview protocols. We shared the protocols with members of the community in three TIF grantees: (1) Guilford County Schools (NC); (2) Harrison School District Two (CO); and (3) the South Carolina Department of Education (SC). We also sent the protocols to all members of the TWG (Drs. Goldring, King, Loeb, Rice, Schochet, Springer), and requested an extensive review from TWG members Dr. Suzanne Wilson (a qualitative researcher with expertise in teacher development) and Dr. Dan Goldhaber (a researcher with expertise on educator labor markets and performance pay who uses quantitative and qualitative methods in his work).

The results of these reviews suggest that the questions were clear. Reviewers reported that the strategy of using data available from extant program monitoring to customize the protocols reduced possible redundancy between program monitoring and our data collection strategy. Reviewer’s greatest concern was that we find a way to facilitate project directors’ gathering of quantitative data on the award structure and payouts. Reviewers were also concerned about the potential for grantees with pre-existing performance-pay programs to intertwine multiple performance-pay programs in response to questions. To address these issues, we revised the protocols to streamline the gathering of data on project design and implementation of payments (by creating the Award Structure and Payout Form and adding the option of sending us an electronic file with necessary data instead of completing that form). We also modified our initial questions to make clear the distinction, especially when interviewing project directors and other leaders, between TIF and other pre- and co-existing performance-pay projects. The review process and protocol revision were completed prior to submitting the protocols to OMB.

We also found that the training we provided to the small number of researchers conducting pilot interviews (who represent the range in experience of those who will be conducting the actual interviews) was sufficient to enable them to gather necessary data.

5. Contact Information

The contact person at the U.S. Department of Education is Dr. Andrew Abrams. The primary contractor of this study is SRI International, based in Menlo Park, CA. Berkeley Policy Associates, based in Oakland, California and the Urban Institute, based in Washington DC, are the subcontractors. The principal investigator of the study is Dr. Daniel Humphrey and the project director is Dr. H. Alix Gallagher. Data collection will be conducted by researchers from SRI International, Berkeley Policy Associates, and the Urban Institute under the direction of
Dr. Humphrey. The contact information for these individuals is as follows:

Andrew Abrams, Ph.D.

U.S. Department of Education

Policy and Program Studies Services

Phone: 202-401-1232

Email: mailto:A[email protected]

Daniel Humphrey, Ed.D.

SRI International

Phone: (650) 859-4014

E-Mail: [email protected]

H. Alix Gallagher, Ph.D.

SRI International

Phone: (650) 859-3504

E-Mail: [email protected]

1 Some of the grantees are so small that it will be impossible to get precise estimates due to grantee size.