Responses to OMB Questions about the Noyce Program

National Science Foundation

Directorate for Education and Human Resources

Robert Noyce Teacher Scholarship Program

Responses to OMB Questions about the Robert Noyce Teacher Scholarship Program Evaluation– March 2011

Overview of the Noyce Program Evaluation

Division: Division of Undergraduate Education (DUE)

Contractor: Abt Associates, Inc.

Program Purpose:

In 2002 Congress authorized the Robert Noyce Teacher Scholarship Program under the National Science Foundation Authorization Act of 2002 (P.L. 107-368, Sec. 10). The program was reauthorized in the America COMPETES Act (P.L. 110-69, Sec. 7030) and in the Reauthorization of America COMPETES Act in 2010 (P.L. 111-358). Through this Act, the Director of the National Science Foundation (NSF) is authorized to “carry out a program to award grants to institutions of higher education (or consortia of such institutions) to provide scholarships, stipends, and programming designed to recruit and train mathematics and science teachers.” In the America COMPETES Act, Congress expanded the program to include summer internships for prospective pre-service students, as well as Teaching Fellowships for STEM career-changers preparing to become teachers, and Master Teaching Fellowships for current science and mathematics teachers preparing to become Master Teachers. The Noyce program received additional funding from the American Recovery and Reinvestment Act of 2009. Recipients of Noyce Scholarships, Stipends and Fellowships are required to teach in high-need school districts. Within each funded project, the project leadership team is expected to include both STEM discipline faculty and education faculty working in collaboration with school districts and Master K-12 Teachers.

Specifically, the long-term performance goals of the program are to:

(1) Encourage talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high need districts and

(2) Retain talented mathematics and science teachers in these settings.

A prior third-party comprehensive evaluation was completed in Spring 2010 by Dr. Frances Lawrenz, University of Minnesota. Evaluation reports and other documents are available at the Noyce Program Evaluation Website

(http://www.cehd.umn.edu/EdPsych/NOYCE/default.html)

Major Evaluation Questions:

The program evaluation has five major evaluation questions each with a set of sub-questions. Each major evaluation question is associated with a research design and type of analysis. Each sub-question is aligned with data sources (e.g., primary data collection and/or extant data).

1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?

2. How do stakeholders perceive the Noyce award and Noyce recipients?

3. What are the characteristics of the schools in which Noyce recipients teach?

4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?

5. What is the impact of Noyce on teacher recruitment and retention and on student achievement?

Research Design/Methods

As described above, the evaluation addresses five questions or focuses. The evaluators collect information on the implementation of the program and describe the relationship among program characteristics, financial incentives, and teacher plans to enter or stay in teaching and teacher leadership roles. The evaluators examine the impact of the Noyce grant on an IHE’s production of STEM certified teachers who teach in high-need districts. In addition, the evaluators plan to conduct a substudy using extant data to examine the impact of the Noyce Program on student achievement in math and/or science for students who have been taught by Noyce teachers. .

The evaluation of the Noyce program includes two types of research design with three types of analyses:

• A descriptive research design using descriptive analyses is used to answer Questions 1, 2, and 3.   The study uses surveys and interviews (1) to describe types of activities for recruitment, selection, preparation, and support of recipients; stakeholder perceptions; and schools in which Noyce recipients teach and (2) to examine the relationships among types of support, activities, types of recipients, retention in teaching, and involvement in leadership roles.

• A descriptive research design is also used to answer Question 4.  This portion of the evaluation uses relational analyses to answer the questions and uses statistical techniques (e.g., multiple regression) to statistically control for other factors (i.e., climate of schools, district hiring practices).

A quasi-experimental design is used to answer Question 5.  This portion of the program evaluation uses impact analyses to answer the question. To assess the impact of the program, the study uses a “difference of differences” quasi-experimental design that also could be described as a short interrupted time series with matched comparison groups designed to assess the impact of the program on teacher recruitment, retention, and student achievement. This component of the evaluation utilizes extant data, such as data collected by the Noyce program monitoring system, state longitudinal teacher certification and employment data from a sample of states, and student achievement data from a sample of districts. The addition of the collection and analysis of student achievement data is a modification to the original contract. To support this effort, a contract modification will be issued by NSF.

(See Overview Exhibit 1: Research Questions by Data Sources for the alignment of major research questions with sub-questions)

Ongoing Annual Program Monitoring

This data collection activity is designed to track the extent to which Noyce awards meet the objectives of the program and to provide extant data for the evaluation. Managed by ICF Macro, this information is used to administer and monitor the progress of the program through Noyce-supported projects in the various institutions. At the project level, PIs are required to collect and report a standard set of information regarding their Noyce projects on an annual basis. PIs report information on post-secondary institutions, school districts, scholarship recipients, stipend recipients, fellowship recipients, internships, and post-scholarship/stipend/fellowship follow-up. The Noyce Program Monitoring is a data resource for addressing Evaluation Questions 2, 4, and 5.

Expected Contributions of NOYCE Program Evaluation:

Accountability: The Program Evaluation will allow NOYCE to specify the impact of the program on the recruitment of talented science, technology, engineering, and mathematics (STEM) majors and professionals to become science and mathematics K-12 teachers in high-need districts and the retention of talented mathematics and science teachers in these settings.

Program Improvement/Learning: The NOYCE Program Officers will utilize findings and information about the process, best practices, and impact of the program to improve the NOYCE program through revising solicitations and in communications to PIs.

Inform/Lead the Field: NOYCE Program Officers will utilize and disseminate the evaluation findings about successful systematic approaches and best practices for university STEM departments and STEM teacher education programs to NOYCE project and non-NOYCE universities and programs that pursue goals similar to those of NOYCE. This evaluation will produce findings and methods that are needed to further the study of the impact of financial incentives on efforts to recruit and retain STEM majors and professionals in teaching in high-need districts. Finally, the findings from the program evaluation could contribute to the knowledge base on STEM teacher recruitment and preparation programs and how best to address the national need for more mathematics and science teachers.

Exhibit 1: Research Questions by Data Sources - Overview of an NSF Program Evaluation – The Robert Noyce Teacher Scholarship Program
Research Question	Data Sources									Research Design/Type of Analyses
	Primary Data Collection					Secondary (Extant) Data Collection
	PIs	STEM Faculty	Principals	Noyce Recipients1	Interns	Demographic Data	Noyce Monitoring Data	Administrative data2
1. What are the goals of Noyce awards and what activities do their teacher preparation programs use to recruit, select, prepare, and support Noyce recipients?										Descriptive Study/Descriptive Analyses
a. What are the goals and objectives of the Noyce awards?	
b. What strategies do Noyce awardees use to recruit and select Noyce candidates?		
c. What activities and supports do teacher preparation programs that have Noyce awards use to prepare Noyce recipients to teach in general, and to teach in high-need schools, in particular?				
d. What activities do teacher preparation programs that have Noyce awards use to support Noyce completers once they are teaching?				
e. What activities do teacher preparation programs that have Noyce awards use to introduce Noyce interns to teaching mathematics and science as a career option?					
2. How do stakeholders perceive the Noyce award and Noyce recipients?										Descriptive Study/Descriptive Analyses
a. What effects do PIs and STEM faculty believe the Noyce award has had on recruitment of STEM teachers, in terms of quantity, quality, and diversity, the retention of those teachers? How do they perceive the burdens imposed by the Noyce award?		
b. What are school principals’ perceptions of benefits and burdens imposed by the Noyce award for their K-12 schools?			
c, How do school principals perceive the qualifications and teaching performance of Noyce recipients?			
d. What are STEM departmental/faculty responsibilities for preparing K-12 mathematics and science teachers? What are STEM faculty/departments perceptions of the effects of the Noyce award on their departments?		
e. How do Noyce recipients perceive the preparation they received from their teacher preparation program in preparing them for teaching? When did Noyce recipients first become interested in teaching? What are the reasons Noyce recipients give for leaving the program or for teaching or not teaching in high-need districts?							
f. How do Noyce interns perceive the influence of their participation in the Noyce Program on their decision to enter teaching and on their interest in math and science? How else has the Noyce Program influenced them?							
3. What are the characteristics of the schools in which Noyce recipients teach?										Descriptive Study/Descriptive Analyses
a. What are the demographic characteristics of schools/districts at which Noyce recipients teach?						
How do Noyce recipients perceive the climate of their schools?				
How do the schools in which Noyce recipients teach work with Noyce IHEs?			
4. What are the relationships between the types of supports, activities, and training that Noyce recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay in teaching and leadership roles?										Descriptive Study/Relational Analyses (multiple regression/statistical control)
a. How are the types of supports/ activities/ training, financial incentives, school/district characteristics, or other personal experiences related to Noyce recipients’ plans to enter and/or remain in teaching and leadership roles?							
5. What is the impact of Noyce on teacher recruitment, retention, and on student achievement?									Quasi-Experimental Study: Impact Analyses Difference of Difference (pre-post with comparison groups design)
a. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers?								
b. How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers that take teaching jobs in high-need districts?								
c. How does an IHE’s receipt of a Noyce grant affect the persistence in teaching in high-need districts among the STEM graduates of its teacher certification program?								
d. Among students in high-need schools, what is the impact of being taught by a teacher who has received a Noyce grant on students’ achievement scores?								
1 Noyce recipient surveys will be administered to Noyce scholars, stipend recipients, and fellows. Noyce Interns will receive a separate survey. 2 State longitudinal teacher certification and employment data from a sample of states and student achievement data from a sample of districts.

Draft Responses to OMB Questions about NSF’s Evaluation of the Robert Noyce Teacher Scholarship Program—March 2011

Responses to OMB Questions:

1. Supporting Statement (A16): We recommend that NSF revise the language in the second to last paragraph pertaining to whether or not NSF will publish product. This statement is alarming given that we, under the PRA, are focused on the “use” of this data. The question for NSF is then, why conduct such a robust survey, if they ultimately will not be releasing survey results?

The results of the program evaluation will be made public. The following paragraph will be substituted for the second to the last paragraph of Section A.16 of the Supporting Statement for Paperwork Reduction Submission.

“The National Science Foundation will make available the executive summary, final report, and other documents describing the findings of the evaluation of the Robert Noyce Teacher Scholarship Program after review and clearance by the NSF. The contracting agency, which will be conducting the program evaluation on behalf of NSF, will submit the report for review by the program, the COTR, and other NSF officials for clearance. NSF will review the quality of the reporting, data analyses, findings, and descriptions of the limitations of the study; ensure that there are no errors in the description of the program and its awardees; and assess compliance with privacy laws, regulations, and policies. To facilitate the timely dissemination of the findings to the public and program stakeholders, NSF is reducing its reliance on formal and traditional publication methods and publication formats. Once the review is completed, the NSF will make public the executive summary, the final report, and other documents describing the findings through a variety of means that are tailored to the information needs of the public and the program stakeholders. In addition, on a case-by-case basis, NSF could support dissemination of information through conferences and publication avenues by the contractor.

2. Methods

a. We are not persuaded that a census of each group is needed. Please suggest a sampling approach for each group, with the possible exception of the PIs, who as grant recipients we understand have more of an obligation to report.

After carefully reviewing available data and considering sampling participants by respondent type, the study team and NSF continue to believe that surveying the census of each respondent group is justified, since sampling is not likely to provide sufficient data about subgroups of interest to our agency. Furthermore, given the relatively small sample sizes of the subgroups, the costs of sampling would lead to a higher cost than surveying the full census of participants, as confidence intervals would need to be calculated and potential biases would need to be taken into consideration when drawing conclusions. Below, we discuss the considerations made in sampling each respondent type.

Principal Investigators (PIs) – Including PIs of projects funded from 2003 through 2009 will allow the study to describe the range of projects being funded and will reflect the diversity of institutions and PIs.

Recipients – There are 10 types of Noyce recipients that can be identified based on annual data entered by PIs; each type represents a different entry point to the teaching profession and certification or teaching status. One question the study will address is whether variation in timing of funding is associated with variation in the outcomes related to retention in teaching. Specifically, in 2009 (the most recent year for which program monitoring data are available), nearly 2,100 individuals were funded at one of four junctures: at two points during their undergraduate careers, shortly after completion of bachelor’s degrees, or at some point long since college graduation (for some career changers). Across the 10 categories shown in Exhibit 1, the cell sizes range from 12 to 543.

Exhibit 1 provides data at an aggregate level; it does not, however, differentiate between individuals funded directly after graduation and “career changers,” who had already been employed in a STEM profession for a number of years, for example. Nor does it differentiate between individuals who have just begun teaching and those who have already filled their required service years. Thus, to distinguish each of the categories needed for subgroup analyses, the cell sizes will be even smaller than those shown in Exhibit 1.

Exhibit 1: Number of Noyce Recipients, by Status
Support Type	In Teacher Preparation Program	Completed Teacher Preparation, Teaching	Completed Teacher Preparation, Not Teaching	Completed Teacher Preparation, Left Noyce	Teacher Preparation Not Completed, Left Noyce
Scholarship (started as undergraduate)	259	222	162	13	64
Stipend (started as post-baccalaureate or career changer)	348	392	543	12	77
Note: Figures are based on annual data entered by PIs in the fall of 2009. Updated figures will be available shortly.

The following paragraphs describe the results of an investigation into the sampling of participants. Assuming a simple random sample and 95% confidence level, Exhibit 2 below displays the sample size needed to achieve various levels of precision.¹ Ideally, a study would be designed to have a precision of 0.030, which is a plus or minus 3 percentage point margin of error.

Exhibit 2. Number of Respondents and Precision
Precision	Sample Size
0.020	2401
0.030	1067
0.035	784
0.040	600
0.045	474
0.050	384

Given these estimates, the only subgroups for which a sample could potentially be drawn are former stipend recipients that have completed the teacher preparation program and who are either teaching or not teaching. Furthermore, most of these recipients would need to be sampled to obtain estimates with reasonable precision, so the likely response rates need to be taken into consideration to obtain these sample sizes.

Some recipients started their Noyce activities up to seven years before the first round of data collection in 2010, and the study may not be able to locate and include such individuals. It is likely that locating and successfully recruiting Noyce recipients will be more difficult for those who have already completed their teacher preparation and are no longer receiving scholarships or stipends. Taken together, these factors suggest that the study needs to survey the census of recipients.

STEM and Education Faculty –The Noyce program focuses on recruiting STEM majors into teaching; consequently the survey of IHE faculty focuses primarily on the impact that participation in Noyce has on STEM faculty, whose departments are responsible for preparing STEM majors.  PIs identified STEM Faculty members to survey about the Noyce Program.  Each PI identified STEM faculty members across multiple STEM disciplines, including:  mathematics, biology, chemistry, physics, geoscience/ environmental sciences, engineering, computer science, and other STEM disciplines.  The sample varies by department size, depth/breadth of STEM departments, involvement in teacher preparation, and other faculty characteristics (e.g., tenure status).   Additionally, a sample of education faculty will be surveyed to provide a broad picture of the impact of the Noyce program on faculty who are traditionally involved in teacher preparation as well as the collaboration between STEM and education faculty. Approximately 3-5 faculty who are/were actively involved in the project at their institution will be invited to participate in the survey.

This group represents a small convenience sample of faculty relative to the population of the STEM and STEM-Ed faculty at each institution.  Convenience samples are associated with potential sampling bias and the likelihood that the sample might not be representative of the entire population.  These concerns limit the types of inferences that can be made on the general population. Taking a sample of this convenience sample might exacerbate these problems. 

K-12 Principals –A primary programmatic goal is to prepare teachers to work in high-need districts and schools. Data from the census of principals where Noyce recipients were recently teaching (in the 2009-10 school year) can inform NSF about whether current teachers are indeed working in high-need settings, how these teachers are performing in their teaching positions, and whether teachers’ supervisors describe the preparation of Noyce recipients as a factor in their hiring and retention. One of the specific questions the study has been asked to address is about the characteristics of the settings in which Noyce recipients teach. Based on the monitoring data that PIs entered in the fall of 2009, Noyce recipients were teaching in approximately 400 schools that vary in size, urbanicity, level (elementary, middle, secondary) in 34 states and the District of Columbia. The study will examine whether and how teacher outcomes vary by type of setting.

A stratified systematic sample of principals could be used for the study. However, given the number of principals in the defined population and the potential for low response rates (in the previous external evaluation of the Noyce Program, the study obtained a 19 percent response rate from district administrators), the evaluation team would not recommend sampling this population.

2. Based on extensive involvement with efforts to obtain state data for a variety of education surveys and evaluations, we are quite concerned about access to, availability of, and quality of the specific state teacher variables that are needed for the impact portion of this evaluation.

The impact analyses on teacher outcomes (research question 5) will seek to address the following questions:

1. The impact of Noyce on the number of teachers certified: How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers?

2. The impact of Noyce on teacher recruitment into high-need districts: How does an IHE’s receipt of a Noyce grant affect its production of certified or licensed STEM teachers that take teaching jobs in high-need districts?

3. The impact of Noyce on teacher retention in high-need districts: How does an IHE’s receipt of a Noyce grant affect the persistence in teaching in high-need districts among the STEM graduates of its teacher certification program? The idea of using state certification and employment data for the study’s impact analysis of the Noyce Program on teacher outcomes was proposed by the study’s Evaluation Advisory Committee. One committee member in particular had collected similar data from New York State for a study comparing teachers trained by the New York City Teaching Fellows program to teachers trained through other programs.

.

Abt staff have explored the availability of state level data, referring to such resources as the Data Quality Campaign (see http://www.dataqualitycampaign.org/), the National Center for Analysis of Longitudinal Data in Education Research (CALDER, see http://www.caldercenter.org/), and state department of education websites, to learn about states-level availability of data on teacher graduation, certification and employment data.

While in recent years more states have begun collecting the needed data elements, we are aware that some states still do not collect these and will not be able to provide the necessary data for the study. However, based on Abt’s previous experience with state data collection and success in obtaining extant state data for other studies, including for the Reading First Implementation Study, NSF is confident that data from some (as yet undetermined) number of states will be available for the study. However, the advice to consider contingency plans is well-taken. If all data are not available from a particular state, the study could potentially collect data on recent STEM graduates of teacher preparation programs from the institutions themselves. Other possible contingency plans are described in the response to question 2.b.iv. below.

1. How will NSF address the concern that those states more willing or able to provide data represent a biased sample of the universe?

OMB is correct that some states will be more willing or able to provide data than others. Ideally, for high external validity, the study would either obtain data on the production of STEM teachers from all Noyce IHEs and their comparison IHE counterparts, or obtain data from a randomly selected sample of IHEs. The proposed design’s reliance on extant data produces both strengths and limitations for the goal of having high external validity. On the positive side, the inexpensive form of data collection (relative to other study designs that would rely on primary data collection) means that data from a particular state would include all Noyce and comparison IHEs in that state. Thus the external validity of the impact of Noyce on IHEs within states where data are available will be very high. On the negative side, the availability of extant data means that the study cannot obtain data from the census of all states or a random sample of states. There are several reasons why, despite this limitation, it is reasonable to assume that the external validity will be reasonably high.

The first is that preliminary indications from Abt’s review of publicly available data are such that the states with the greatest number of Noyce IHE grantees seem able to provide the necessary data. For example, Texas and California, where nearly half of the Noyce recipients are teaching, are likely to have the relevant data. Among all of the states being considered for the study, Abt estimates that it may be possible to obtain data from states where up to two-thirds of Noyce grants have been awarded. Second, Abt expects that it could obtain data from states covering a wide geographic range covering all major regions of the country (see 2.b.iii. below).

Finally, there is no reason to believe that a state’s ability to provide data would be related to the impact of Noyce. Availability and access to data are governed by state and/or district policy or the capability to provide requested data. Thus, it is unlikely that a state’s ability to provide the requested data, or not, would bias the study.

2. How will NSF validate assumptions about stability of the groups over time (especially regarding the effects of education reform, the economy, the availability of statewide data, etc) necessary to use a difference in difference design?

The study’s proposed difference of differences design could also be described as a short interrupted time series with matched comparison groups. This design does not actually require that there be no external factors that vary over time affecting the outcome (e.g., effects of education reform, the economy etc.) but does require an assumption that those external factors affect both the treatment and comparison outcomes in the same way at the same time. In order to ensure that this assumption is plausible, we specify that the matched comparison IHEs will come from the same state as the Noyce grantee IHEs, and therefore will be subject to similar external influences of state-wide education reform, local economic conditions, etc.

3. Please provide information on the geographic spread of the grantees and how this relates to the states NSF would like to engage.

Forty-five states, the District of Columbia, and Puerto Rico have one or more IHEs that have received Noyce funding. The study’s impact analyses will need to be limited to the states with at least 2 IHEs that received grants in different years in order to investigate the relationship between Noyce recipients’ plans to stay in teaching and their participation in project activities as well as the amount of financial support (e.g., scholarship, stipend, fellowship) they received from their IHEs. This will allow the analysis to control statistically for other factors (i.e., effects of education reform, changes in state economic circumstances) that could potentially affect recipients’ plan to stay in teaching other than the recipients’ respective IHE’s Noyce activities and level of financial support. There are 22 such states, and it seems likely that at least 13 states will have extant data that could be included in the study. Sources of extant data include project annual and final reports; Noyce Monitoring Data; and District/State personnel records, student achievement data, and demographic data relevant to this study,

4. Please provide a realistic timetable for negotiating with the states and a discussion of contingency plans.

NSF expects the data collection process will take from one to six months. The procedures for contacting states are described below:

• Send a general outline of the study and data needs to data research and evaluation personnel at state departments of education (state department of education websites typically name the individuals responsible for managing such departments).

• Verify whether the initial contact can answer questions about data elements availability and linkability.

• Follow upto clarify our data needs, ascertain the procedures for requesting the data from the state, obtain a copy of a data codebook, if it is available, and determine the timeline.

• Submit a formal data request form.

• Continue to communicate with state staff as they approve the data request and prepare the necessary data.

It is our expectation that most of the work described in the first four bullets can be accomplished within four to six weeks. The amount of time required for each state to get the necessary approvals and to prepare the data depends on the availability of state staff and the procedures they need to follow.

Abt recognizes that not all states will be able to provide the complete data requested, and will work with states to learn whether obtaining partial data might help (albeit in a more limited way) address some of the study’s needs. For example, Abt’s preliminary investigation of data availability from New Jersey suggests that the state does not maintain administrative data identifying the IHE at which each teacher was trained. Another option in this case would be to request the names of all STEM majors directly from the teacher preparation programs in the IHEs in New Jersey that have received awards. In another state, California, it is not evident whether certification and teacher employment data can be linked. In this case, if state personnel were to indicate that the data could not be linked, the study could potentially conduct a more limited set of analyses – focusing specifically on whether the Noyce program had an impact on the number of teachers certified in a STEM field. Additionally, we could investigate whether data are available for a subset of institutions with Noyce awards. For example, many of the teacher training programs at campuses of California State University and the University of California are conducting their own longitudinal evaluations of teacher outcomes from their programs. Since many of these campuses have Noyce awards, this may provide the study with valuable data.

5. Has NSF reviewed the methodology reports of the National Center for Education Statistics’ new Teacher Compensation Survey efforts?

Yes, NSF‘s study contactor has reviewed the NCES Teacher Compensation Survey,“An Evaluation of the Data from the Teacher Compensation Survey: School Year 2006-07 Research and Development Report” by Cornman, S.Q. et al., September 2010. Data from this cross-sectional survey do not include information on the institutions where teachers received their teacher training, when teachers received their certifications, or whether they have certifications in STEM fields. As a result, these data cannot be used to assess the impact of the program.

6. Please provide a paragraph that NSF will commit to including in the evaluation results that describes the limitations of the methodology.

The following paragraph will be added to the Noyce OMB Package in Section A.16 Plans for Publication, Analysis, and Schedule, as the third to the last paragraph.

The NSF recognizes the importance of explicit acknowledgement of the limitations of the evaluation studies that it funds to ensure that audiences have information to fully interpret findings, conclusions, and recommendations. In all of its reporting, the NSF will describe the limitations of each component of the study, the methodology used, and the results of the study.

Once the study is completed a more specific discussion of the limitations of the study will be developed. However, a discussion and description of some potential limitations are provided in the Appendix A.

3. Questionnaires

a. NSF did not submit any questionnaires; rather it submitted only charts with question topics. We need to see all questionnaires immediately.

To facilitate review of the documents, crosswalks were included in the submission. Questionnaires were submitted behind the appropriate crosswalks. We have included copies of the revised questionnaires, associated crosswalks, and overviews (coversheets).

b. NSF needs to source its questions – specifically, it should be using validated questions, such as from the National Center for Education Statistics surveys on teachers.

Questions and response categories in the Noyce surveys that ask for general information about recent STEM graduates and teachers were adapted from existing national protocols (for example, the National Center for Education Statistics Schools and Staffing Survey 2007-08 (SASS) and the National Science Foundation’s (NSF) 2006 National Survey of Recent College Graduates (NSRCG)). Questions adapted from national protocols ask for the graduates’ educational and employment background and teachers’ perception of school climate.

Other survey questions were developed specially for this study, as they focus explicitly on the types of supports provided by the Robert Noyce Teacher Scholarship program to STEM graduates and professionals and how these supports have affected their plans to teach. Some of these questions were adapted from instruments developed by the University of Minnesota and used successfully in a previous evaluation of the Robert Noyce Teacher Scholarship program (http://www.cehd.umn.edu/EdPsych?NOYCE/default.html).

In response to OMB’s suggestion to consider using validated questions from existing surveys, Abt staff further reviewed the SASS survey and the National Survey of Recent College Graduates (NSRCG) to determine whether additional questions could be adapted or incorporated entirely. Based on this review, questions about teachers’ certification and training information, duration of student teaching and duration, and intensity of teachers’ professional development were incorporated into the updated surveys. See attached documents for copies of the revised surveys.

Exhibit 3: Crosswalk of Items in Noyce Recipient Survey that Will be Adapted from Existing National Surveys

Question # from Noyce Recipient Survey, Module C (OMB Attachment G)	Question Topic	Source (Survey name and year)1	Comments
B1	Noyce recipients' current teaching position	2007-08 SASS Q1	Adapt response options from SASS Q1
B4	Noyce recipients' highest educational status	2006 NSRCG QA7	Adapt response options from NSRCG QA7
C1, C2a & C2b, C3, C4, C5a, C5b	Certification subject area, grade level, & other certifications	2007-08 SASS Q33a, b, c, & d	Adapt SASS Q33a, b, c and d, Table 3
D2b	Undergraduate GPA	2006 NSRCG QA4	Adapt response options from NSRCG QA4
D4	Occupation prior to Noyce program: STEM professional?	Scientists and Engineers Statistical Data System (SESTAT)	Adapted from S&E job categories. http://sestat.nsf.gov/docs/occ03maj.html
E4b	Length of student teaching	2007-08 SASS Q30	Adapt response options from SASS Q30.
G2	Duration and intensity of professional development	2007-08 SASS Q41b	Adapt response options from SASS Q41b.
H4	Courses/grade level taught	2007-08 SASS Q15	Adapt Table 1
H5	School climate	2007-08 SASS Q54, Q55, Q56, Q57	Adapt SASS Q54, Q55, Q56, Q57
I1	Leadership positions	2007-08 SASS Q51	Add SASS Q51e
I3	National Board Certification	2007-08 SASS Q32a, b	Adapt SASS Q32a and b
1 Schools and Staffing Survey 2007-08 (SASS), 2006 National Survey of Recent College Graduate (NSRCG)

In addition to using validated questions from national protocols, revisions were made to the surveys based on feedback from pilot test respondents. The pilot test respondents were drawn from the target populations of PIs, STEM faculty and Noyce recipients. The pilot respondents were asked to comment on the length of time needed to complete the survey, point out any ambiguous question wording, and list any response options they found to be missing or repetitive. Based on their comments, changes were made to some of the response options and the wording of several items was changed to reflect the fact that activities may be associated with a Noyce award rather that the Noyce Program. Additionally, items that had responses on a two-point scale were changed to a 4-point Likert scale.

c. We are not persuaded that the follow up interviews are needed in addition to the surveys. If NSF proposes acceptable sampling to reduce burden on each group, we will consider approving such follow up, but it must be better justified.

For the evaluation of the Noyce program, NSF is particularly interested in learning about which entry points (e.g., when in an individual’s educational or employment history) are most strongly associated with outcomes related to teacher certification, retention in teaching, and retention in teaching in high need districts. The surveys are designed to provide broad information about topics that can effectively be addressed in close-ended formats, such as types of teacher preparation activities in which recipients and other respondents engage, number of years in teaching, and other status and/or frequency indicators. The surveys will yield a picture of particular patterns in the data that indicate which recruitment, support, mentoring, and other activities are associated with positive or negative outcomes. However, surveys are less likely to reveal the nuances and complexity of participants’ “lived experience” in choosing a career path, making changes in a career path, or remaining in a chosen career. Data from interviews can validate survey results (convergent lines of evidence) or suggest other ways of interpreting the patterns found in the data (divergent lines of evidence). Informed by the findings and recommendations from the prior evaluation, the study will field surveys first. Interviews will be used to address research question 4 and follow up on topics or themes identified in the analyses of survey data as potential “turning points” in career development to explore how and why respondents made certain decisions or plan to remain in teaching or seek other employment (or exit the labor force).

Once analyzed, the interviews can yield fine-grained distinctions about how and why decisions were made to enter, complete, and/or exit teacher certification programs, and why recipients have elected to stay/exit the teaching profession. For example, based on the prior evaluation, information gained by conducting follow-up interviews revealed a complex structure of the scholars’ paths to teaching in high needs schools. Several main factors influenced scholars’ decisions to choose teaching as a career, such as educational role models; previous careers and non-K-12 teaching related opportunities; personal background; and hearing about and receiving a Noyce scholarship. Further investigation of these different pathways into teaching could provide insight into needed mechanisms and support structures for fostering and sustaining a STEM workforce in teaching.

Understanding the process of involvement in teacher preparation by STEM faculty is critical to the Noyce program. Interviews with other respondents will similarly provide more nuanced information about how and why they became involved with the Noyce program (for PIs and STEM faculty), and how and why K-12 principals make hiring/class assignments, and other decisions about Noyce recipients in their schools. For example, in the prior evaluation, there were mixed findings about the influence of participation in the Noyce Program on faculty attitudes toward teacher preparation. In light of the legislation establishing the Robert Noyce Teacher Scholarship Program, the “Rising of the Gathering Storm” report, “Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5,” and the “Prepare and Inspire: K-12 Education in Science, Technology, Engineering, and Math (STEM) for America’s Future” PCAST report, information gathered from interviews with STEM and education faculty could provide insight into the involvement of and challenges facing STEM and education faculty in the preparation of science and mathematics teachers.

In combination with initial survey results, findings from the interviews can (1) guide post hoc analyses of the survey data in ways not originally anticipated and (2) draw stronger inferences if findings converge. In addition, interviews can provide valuable information about unanticipated consequences of the Noyce Program. Information gathered through interviews with each respondent type could be used to help improve program operations and provide insight into the complexity involved in the recruitment and retention of talented STEM students, career changes, and master teachers into science and mathematics teaching.

References

National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, Committee on Prospering in the Global Economy of the 21^st Centure: An Agenda for American Science and Technology, Committee on Science, Engineering, and Public Policy. 2007. Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, Washington, DC: National Academies Press.

Members of the 2005 “Rising Above the Gathering Storm” Committee, 2010. Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5, Washington.DC: National Academies Press.

Executive Office of the President, President’s Council of Advisors on Science and Technology, 2010. Prepare and Inspire: K-12 Education in Science, Technology, Engineering, and Math (STEM) for America’s Future, Washington, DC (http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-stem-ed-final.pdf)

Appendix A

Possible text for the description of how the evaluation addresses potential limitations is presented below.

1. Both the descriptive and relational components of the study rely in large part on self-reported survey data. To help mitigate against the social desirability to praise a funding source, the study will be collecting data from multiple respondent types. Response rates may well vary across respondent types so that confidence in the representativeness of the data may also vary. Nonresponse analysis can be used to identify the existence of bias due to unit and item nonresponse.

2. Ideally, to measure the impact of the Noyce Program on entry into teaching in high-need districts, the study would randomly assign some applicants (the treatment group) to receive Noyce support and other Noyce applicants to a non-Noyce-funded control group. Random assignment of individuals to a treatment and control condition eliminates differences between individuals except for the fact that one group participates in Noyce and the other does not. Therefore, any observed differences in the rates of entry into teaching in high-need districts can be causally linked (or attributed) to the Noyce Program. Clearly, random assignment is not feasible in this study as Noyce is a competitive scholarship program. Recipients are selected and awarded scholarships based on qualifications, and school placement is a function of openings and local hiring policies. Causal inferences for the study’s quasi-experimental design rest on 2 key assumptions:

• Any pre-existing time trends within IHEs have been correctly specified in the analysis model. The base model assumes that within an IHE, and absent the receipt of a Noyce grant, the production of STEM certified teachers in the post-Noyce period would have the same mean and variance as had been observed in the pre-Noyce period. [Note that we will be testing for linear trends in the pre-Noyce period, so this text may be altered to reflect the use of baseline linear trend prediction models.]

• Secular changes over time, unrelated to the Noyce Program, will have the same effect on treatment and comparison IHEs’ production of STEM certified teachers (and on the production of STEM certified teacher that teach in high-need districts). If, for example, a state-wide education reform tended to have greater (or smaller) effects on Noyce IHEs than comparison IHEs, this assumption would be violated and results may be biased.

3. The study will be specified to have a given power to detect a Noyce effect on the production of STEM certified teachers equal to X teachers per year. If the true average Noyce effect is smaller than X teachers per year, the study may not have adequate power to detect the effect.

4. While the study will try to ensure that it has high quality data, any errors or omissions in the state data sources used to measure the number of STEM certified teachers produced by IHEs in the pre-Noyce and post-Noyce periods would create measurement error which may reduce our ability to detect an impact.

1 Sample size is calculated as 1.96 ^2(p*(1-p))/ (Precision)^2, where p was set equal to 0.50.

Revised DRAFT response to OMB comments for NSF Evaluation of the Noyce Program 13