OMB Package Section A

Evaluation of the
Robert Noyce Teacher
Scholarship Program

Supporting Statement
for Paperwork
Reduction Submission
Cambridge, MA
Bethesda, MD
Durham, NC
Atlanta, GA

OMB #3145

August 6, 2010

Prepared for
Connie Della-Piana
National Science Foundation
Abt Associates Inc.
55 Wheeler Street
Cambridge, MA 02138-1168

Prepared by
Abt Associates Inc.

Contents
Section A: Justification ................................................................................................................... A-1
Introduction.............................................................................................................................. A-1
A.1. Circumstances Requiring the Collection of Data .......................................................... A-1
A.2. Purposes and Uses of the Data ...................................................................................... A-3
A.3. Use of Information Technology to Reduce Burden ...................................................... A-4
A.4. Efforts to Identify Duplication ...................................................................................... A-4
A.5. Small Business .............................................................................................................. A-4
A.6. Consequences of Not Collecting the Information ......................................................... A-5
A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6... A-5
A.8. Selection of Public Comments and Consultation with People Outside the Agency ..... A-5
A.9. Payments or Gifts to Respondents ................................................................................ A-6
A.10. Assurance of Confidentiality......................................................................................... A-6
A.11. Questions of a Sensitive Nature .................................................................................... A-7
A.12. Estimates of Response Burden ...................................................................................... A-7
A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to
Respondents or Record Keepers ................................................................................. A-11
A.14. Estimates of Costs to the Federal Government ........................................................... A-11
A.15. Changes in Burden ...................................................................................................... A-11
A.16. Plans for Publication, Analysis, and Schedule ............................................................ A-11
A.17. Approval to Not Display Expiration Date................................................................... A-13
A.18. Exceptions to Items 19 of OMB Form 83-I ................................................................ A-13
References.............................................................................................................................. A-14
Section B: Collection of Information Employing Statistical Methods......................................... B-1
B.1. Respondent Universe and Sampling Methods............................................................... B-1
Survey and Interview Samples...................................................................................... B-1
Noyce Recipient Sample ............................................................................................... B-1
Noyce Principal Investigators ....................................................................................... B-2
STEM Faculty ............................................................................................................... B-2
K-12 Principals.............................................................................................................. B-3
B.2. Procedures for the Collection of Information................................................................ B-3
B.3. Methods to Maximize Response Rates and Deal with Nonresponse ............................ B-6
B.4. Test Procedures or Methods.......................................................................................... B-6
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data ............................................................................................................ B-6

Abt Associates Inc.

Contents

i

Appendix A: Research Questions by Data Sources
Appendix B: Impact Analyses Using Extant Data
Appendix C: Letters to Noyce Principal Investigators and Study Fact Sheet
Appendix D: Principal Investigator Survey
Appendix E: STEM Faculty Survey
Appendix F: K-12 Principal Survey
Appendix G: Noyce Recipient Survey
Appendix H: Noyce Intern Survey
Appendix I: Principal Investigator Interview Protocol
Appendix J: STEM Faculty Interview Protocol
Appendix K: K-12 Principal Interview Protocol
Appendix L: Noyce Recipient Interview Protocol for Recipients Still in Teacher Certification
Program
Appendix M: Noyce Recipient Interview Protocol for Recipients Who Completed Teacher
Certification Program

Abt Associates Inc.

Contents

ii

Section A
Justification
Introduction
The National Science Foundation (NSF) requests that the Office of Management and Budget (OMB)
approve, under the Paperwork Reduction Act of 1995, a three-year clearance to evaluate the Robert
Noyce Teacher Scholarship Program (Noyce). The Noyce program operates within NSF’s Division
of Undergraduate Education, and bridges the higher education and the K-12 system. The Noyce
Program encourages talented science, technology, engineering, and mathematics (STEM) majors and
professionals to become K-12 mathematics and science teachers. The program provides funds to
institutions of higher education (IHEs) to support scholarships, stipends, and academic programs for
undergraduate STEM majors and post-baccalaureate STEM students who commit to teaching in highneed K-12 school districts as a condition of receiving financial support.
The program supports two other strands for individuals who have already become STEM and/or
teaching professionals. The Noyce program supports STEM professionals who enroll as NSF
Teaching Fellows in master's degree programs leading to teacher certification by providing academic
courses, professional development, and salary supplements as the Teaching Fellows fulfill a four-year
teaching commitment in a high need school district. This programmatic strand also supports
exemplary math and science teachers, who have master’s degrees, to become Master Teaching
Fellows in high need school districts by providing professional development and salary supplements.

A.1. Circumstances Requiring the Collection of Data
The Robert Noyce Teacher Scholarship Program was first authorized under the National Science
Foundation Authorization Act of 2002 (P.L. 107-368) and reauthorized in 2007 under the America
COMPETES Act (P.L. 110-69).
In fall 2009, the NSF contracted Abt Associates Inc. to conduct a comprehensive evaluation of the
Noyce Program. The Noyce Evaluation is charged with addressing the following five broad research
questions:
1. What are the goals of Noyce awards and what activities do their teacher preparation programs
use to recruit, select, prepare, and support Noyce recipients?
2. How do stakeholders perceive the Noyce award and Noyce recipients?
3. What are the characteristics of the schools in which Noyce recipients teach?
4. What are the relationships between the types of supports, activities, and training that Noyce
recipients receive, the types of Noyce recipients, and the recipients’ plans to go into and stay
in teaching and leadership roles?
5. What is the impact of Noyce on teacher recruitment and retention and on teacher
effectiveness?

Abt Associates Inc.

Section A: Introduction

A-1

To address the first four research questions, the evaluation will collect primary data via surveys and
interviews with Principal Investigators (PIs); faculty from science, technology, and engineering
(STEM) departments; Noyce recipients; and K-12 principals. The evaluation will examine all
recipients supported from all grants funded between 2003 and 2009, to capture all key periods of
interest: recruitment and entry into teacher preparation for undergraduates and STEM professionals;
preparation for, and induction into teaching; teaching during obligatory service years; and teacher
retention in the post-service period. It will also rely on Noyce monitoring data entered annually by
PIs, project reports, and demographic data.
To address the final research question, the evaluation will collect extant longitudinal teacher
employment and certification data from a sample of states. Appendix A displays each of the main
research questions and sub-questions and the data sources that will be used to answer those questions.
Exhibit 1 below summarizes the study’s broad data collection activities and when data are to be
collected.
Exhibit 1: Data Collection Instruments and Schedule
Data Collection Schedule
Data Sources
Extant Data Sources
Noyce monitoring data for 2003-2008 awards
Noyce monitoring data for 2009 awards
State longitudinal teacher employment data (for 1999-2010)
Project proposals and reports
Demographic data about states, districts, and schools

SY 2010-11

SY 2011-12











Surveys (administered fall 2010, fall 2011) 1
Principal Investigators (all)
STEM faculty (sample)
Noyce recipients (all)
K-12 principals (sample)











Interviews (conducted winter 2011, winter 2012) 2
Principal Investigators (sample)
STEM faculty (sample)
Noyce recipients (sample)
K-12 principals (sample)











1

Surveys for 2003-2008 grants will be administered in fall 2010, and surveys for 2009 grants will be administered in
fall 2011.
2
Interviews for 2009 grants will be conducted in winter 2011, and interviews for 2009 grants will be administered in
fall 2012.

Earlier, in 2005, several years after the program begun, NSF awarded a grant to the University of
Minnesota to conduct a primarily descriptive evaluation of the Noyce Program; the study included 66
of the grants that had been awarded through 2006, and the results indicated that the program was
being implemented as intended (Lawrenz et al., 2009). The current evaluation is designed to improve
upon the earlier study in several ways. Specifically, the proposed study will:

Abt Associates Inc.

Section A: Introduction

A-2



Examine all 225 full grants1 awarded between 2003 and 2009, through online surveys of the
census of funded projects and the census of Noyce recipients;



Use survey data collection to describe the early career decisions of Noyce recipients who may
or may not yet have completed the teaching obligations required by the program;



Use survey data collection to explore the relationships between program and recipient
characteristics, financial incentives, and teachers’ plans to enter or continue teaching in highneed districts and pursue leadership roles; and



Use a quasi-experimental design to measure program impacts on teacher recruitment and
retention and on teacher effectiveness.

A.2. Purposes and Uses of the Data
The primary purpose for collection is program evaluation. The data collected will allow NSF to
describe the program components associated with teachers teaching in high-need districts and
pursuing leadership positions for different types of recipients. Describing short- and long-term
impacts of the Noyce Program on recruitment and retention will inform future Noyce Program
funding decisions and contribute to the wider NSF discussion on STEM teacher recruitment and
retention.
It is important to recognize that there are already annual program monitoring data being collected
about the Noyce Program. However, the basic level of information collected through annual
monitoring data is not sufficiently comprehensive to describe the characteristics of Noyce-funded
institutions and the activities supported through the Noyce program, the types of faculty involved,
their recruitment and selection processes, the employment and career decisions made by recipients,
and the schools in which recipients teach. The proposed study will also collect information from
recipients about their background information and initial interest in teaching, the activities they
participated in and their perception of the value of those activities and the climate in the schools in
which they teach. Consequently, the study will conduct surveys and interviews to answer the
specified research questions that cannot be answered through existing sources alone.
The study will obtain extant data from two key sources: (1) a purposively selected sample of Noyce
IHEs themselves and (2) state departments of education and from teacher retirement funds on
certification and employment status. The study will collect information from IHEs on the numbers
and names of STEM and teacher certification graduates over a defined period of time, and will then
match those data (see discussion below on protection of confidential information) to state certification
and employment/pension fund data systems to learn about the employment patterns of STEM
graduates from Noyce institutions.

1

This number does not include supplemental or planning grants, which are excluded from the evaluation.
The total number of grants awarded to date, including supplemental and planning grants, is 249.

Abt Associates Inc.

Section A: Introduction

A-3

A.3. Use of Information Technology to Reduce Burden
To reduce burden, online surveys will be used to collect information from participants. Noyce
Program PIs, STEM Faculty, Noyce Recipients, and K-12 Principals will be asked to complete a webbased survey to learn more about the Noyce program activities and the recipients they support. This
approach has become more commonly used in recent years. NSF tends to favor web-based systems
because they can facilitate respondents’ data entry across computer platforms.
One innovative feature of many of the individual Web systems is that information once entered into
the system can be presented to the respondent for verification, thus reducing the respondent burden.
Another valuable feature is that there can be a thorough editing of all submitted data for
completeness, validity, and consistency. Editing is performed as data are entered. Most invalid data
cannot enter the system, and questionable or incomplete entries are called to respondents' attention
before they are submitted to NSF.
Web-based surveys employ user-friendly features such as automated tabulation, data entry with
custom controls such as checkboxes, data verification with error messages for easy online correction,
standard menus, and predefined charts and graphics. In addition, survey skip patterns automatically
move the respondent forward into the next appropriate section, creating less confusion and
simplifying the survey-taking experience. This approach also allows for easy identification of nonrespondents and facilitates follow-up. All these features facilitate the reporting process, provide
useful and rapid feedback to the data providers, and reduce burden.
Email will be used to send respondents their invitations to complete the survey and follow-up with the
non-respondents to ensure their participation.
To follow up on the information obtained through surveys, the study will conduct telephone
interviews with a sample of Noyce recipients, PIs, STEM Faculty, and K-12 principals to learn about
selected aspects of program implementation in greater detail.

A.4. Efforts to Identify Duplication
This evaluation does not duplicate other NSF efforts. There are no other studies at this point that
examine the long-term impacts of the Noyce program.
Project monitoring data, as described in the previous section, for Noyce will be available to the
contractor and the NSF staff working on this research. Surveys will be constructed to ask about
elements of program participation that are not captured in the Noyce Monitoring System. Telephone
interview protocols will focus on topics that cannot easily be summarized in close-ended survey
responses. Future data collection tasks likewise will also draw on whatever prior program data exists,
thus preventing duplication.

A.5. Small Business
No information for this research will be collected from small businesses.

Abt Associates Inc.

Section A: Introduction

A-4

A.6. Consequences of Not Collecting the Information
If this information is not collected, NSF will be unable to document the initial and longer-term
impacts of the Noyce Program as required by GPRA, nor will it be able to adequately assess program
performance as required by PART, and the Deficit Reduction Act of 2005.

A.7. Special Circumstances Justifying Inconsistencies with
Guidelines in 5 CFR 1320.6
The data collections will comply with 5 CFR 1320.6.

A.8. Selection of Public Comments and Consultation with People
Outside the Agency
8A. Federal Register Announcement
Comments on this data collection effort were solicited in the Federal Register on June 8 in Volume
75, Number 109. During the first comment period prior to submission to OMB, no substantial
comments were received.
8B. Consultations Outside the Agency
Consultation on the study design and data sources and needs occurred during the design phase and
will continue throughout the study. The purpose of such consultations is to ensure the technical
soundness of the study and the relevance of its findings, and to verify the importance, relevance, and
accessibility of the information sought in the study.
Consultation was conducted by the research firm, Abt Associates Inc., contracted by NSF to evaluate
the Noyce Program, its data collection subsidiary, Abt-SRBI, and consultant Frances Lawrenz,
Principal Investigator from the previous evaluation of the Noyce Program. Senior technical staff
from these organizations who are conducting the study are listed below.
Abt Associates, Inc.

Ellen Bobronnikov
Beth Gamse
Cristofer Price
Radha Roy
Carter Smith
Marc Moss
Laurie Bozzi
Amanda Parsad

617-349-2718
617-3492808
301-634-1852
301-347-5722
617-349-2543
617-349-2825
617-349-2485
301-634-1791

Abt-SRBI

Paul Schroeder
Andy Weiss

301-608-3883
301- 608-3883

University of Minnesota

Frances Lawrenz

(612) 625-2046

Abt Associates Inc.

Section A: Introduction

A-5

In addition to the above, an evaluation advisory committee has provided input on the study design and
data collection plan. The advisory panel members represent some of the nation’s leading researchers
on teacher preparation, educational policy, and evaluation design. The committee includes:


William Haver—Virginia Commonwealth University



Nancy Hopkins-Evans—School District of Philadelphia



Elizabeth Stuart—Johns Hopkins



Jonathan Supovitz—University of Pennsylvania, Consortium for Policy Research in
Education (CPRE)



Donald Wink—University of Illinois at Chicago



James Wyckoff—University of Virginia/Virginia Center for Education Policy

In addition, the proposed data collection instruments are being pilot tested with respondents drawn
from the target populations. Respondents will be asked to comment on the clarity and content of the
questions. The duration of the data collections will be recorded to help with an accurate estimation of
time burden.

A.9. Payments or Gifts to Respondents
No payment or gift will be provided to respondents.

A.10. Assurance of Confidentiality
Respondents will be advised that any information on specific individuals will be maintained in
accordance with the Privacy Act of 1974. Data collected will be available to the evaluation contractor,
the contractor hired to manage data collection, and at the aggregate level to NSF officials and staff.
Data are processed in accordance to Federal and State privacy statutes. Detailed procedures for
making information available to various categories of users are specified in the Education and
Training System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The system limits access to
personally identifiable information to authorized users. Data submitted will be used in accordance
with criteria established by NSF for monitoring research and education grants, and in response to
Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified
researchers and contractors in order to coordinate programs and to a Federal agency, court or party in
court, or Federal administrative proceeding, if the government is a party.
Individuals surveyed will be assured that the information they provide will not be released in any
form that identifies them as individuals and their responses will be kept confidential. The contractor
will be expected to maintain the confidentiality, security, and integrity of survey data and interview
data. The web-based survey data, notes, and transcripts of the interviews will be maintained on a
secure server with appropriate levels of password and other types of protection.
All assurances of confidentiality will be reviewed by the contractor’s Institutional Review Board prior
to data collection.

Abt Associates Inc.

Section A: Introduction

A-6

A.11. Questions of a Sensitive Nature
The proposed surveys ask for demographic information (gender, race, ethnicity, and marital status)
from Noyce recipients on a voluntary basis, thus respondents may choose not to provide information
that they believe is sensitive in nature. This information is being collected so that NSF can answer
questions about how effective the Noyce program has been in the recruitment and preparation of a
diverse population of students. All survey questions will be reviewed by the contractor’s Institutional
Review Board prior to fielding.

A.12. Estimates of Response Burden
A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden
The target population for this study includes individuals who participated in Noyce project funds
between 2003 and 2009. Our study will include a total of 224 awards over the 2 year base contract.
Respondents from the 2003-2008 grants (n=125) will be surveyed and interviewed in 2010-2011, and
respondents from the 2009 grants (99) will be surveyed and interviewed in 2011-2012. Additionally,
those Noyce recipients from 2003-2008 grants who have completed their obligatory teaching years2
(n≈50) will be surveyed a second time in 2011-12, to learn more about whether these teachers
continue to teach in high-need districts and schools. Finally, if the first option year is funded, the
study will survey Noyce recipients who are beyond their obligatory teaching years as well as
Teaching Fellows and Master Teaching Fellows who were first supported in 2009 grants, to learn
more about how these individuals progress through their careers. The burden for the option period is
not currently included in the calculations in this section.
Online web surveys will be administered to all Noyce project PIs and Noyce recipients, as well as
STEM Faculty identified by the PIs as active in the Noyce Program, and K-12 principals from schools
from which Noyce recipients taught in the most recent year available from monitoring data. The
survey instruments are included in Appendices D-H. The annual response burden for surveys is
estimated to be 1,356.43 hours per year for an average of 2,510 respondents.
Telephone interviews will be conducted with a sample of Noyce recipients, PIs, STEM faculty, and
K-12 principals. The interview protocols are included in Appendices I-M. The annual response
burden for interviews is expected to be 57.5 hours for 95 respondents.
Exhibit 2 indicates the number of average respondents per year to be surveyed and interviewed for
each respondent type and the time demand these surveys and/or interviews will place on each
individual respondent.

2

Noyce recipients are required to teach for two years in a high-need district for each year of support they
receive.

Abt Associates Inc.

Section A: Introduction

A-7

Exhibit 2: Estimated Annual Burden Hours

Respondent type
Surveys
PIs
STEM faculty
Noyce recipients1
Recipients in
preparation
Recipients with
teacher cert
Interns
K-12 principals
Total
Interviews
PIs
STEM faculty
Noyce recipients
Recipients in
preparation
Recipients with
teacher cert
K-12 principals
Total

Average
Number of
Respondents

Time Per
Response
(Hours)

Number of
Responses

Total Time
Burden (Hours)

124.5
450

0.75
0.5

124.5
450

93.38
225

900

0.5

900

450

500

0.67

525

351.75

110
400

0.33
0.5

110
400
2,509.5

36.3
200
1,356.43

20
20

0.75
0.75

20
20

15
15

15

0.5

15

7.5

20

0.75

20

15

20

0.5

20
95

10
57.5

1

Noyce recipients will include: a) those that are in teacher preparation; b) those that completed teacher preparation and are
teaching; c) those that completed teacher preparation but are not teaching; and d) those that dropped out of teacher
preparation. There will be separate modules for each of these types of recipients. Additionally, summer interns will be
administered a separate survey.

A.12.2. Hours Burden Estimates by Each Form and Aggregate Hours Burdens
The total response burden for surveys is estimated to be 2,712.9 hours per year for 5,019 respondents.
The total response burden for interviews is expected to be 115 hours for 190 respondents.
Exhibit 3 indicates the total number of respondents to be surveyed and interviewed during the base
study period.

Abt Associates Inc.

Section A: Introduction

A-8

Exhibit 3: Estimated Total Burden Hours

Respondent type
Surveys
PIs
STEM faculty
Noyce recipients1
Recipients in
preparation
Recipients with
teacher cert
Interns
K-12 principals
Total
Interviews
PIs
STEM faculty
Noyce recipients
Recipients in
preparation
Recipients with
teacher cert
K-12 principals
Total

Number of
Respondents

Time Per
Response
(Hours)

Number of
Responses

Total Time
Burden (Hours)

249
900

0.75
0.5

249
900

186.75
450

1,800

0.5

1,800

900

1,000

0.75

1,050

703.5

220
800

0.33
0.5

220
800
5,019

72.6
400
2,712.9

40
40

0.75
0.75

40
40

30
30

30

0.5

30

15

40

0.75

40

30

40

0.5

40
190

20
115

1

Noyce recipients will include: a) those that are in teacher preparation; b) those that completed teacher preparation and are
teaching; c) those that completed teacher preparation but are not teaching; and d) those that dropped out of teacher
preparation. There will be separate modules for each of these types of recipients. Additionally, summer interns will be
administered a separate survey.

Abt Associates Inc.

Section A: Introduction

A-9

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens
The overall annualized cost to the respondent is $37,420.65. Exhibit 4 shows the estimated total
annual costs to each group of respondents over one year for the surveys and interviews.
Exhibit 4: Estimated Cost to Respondents

Respondent type
Surveys
PIs
STEM faculty
Noyce recipients
Recipients in
preparation
Recipients with
teacher certification
Interns
K-12 principals
Total
Interviews
PIs
STEM faculty
Noyce recipients
Recipients in
preparation
Recipients with
teacher certification
K-12 principals
Total

Hourly
Salary
Estimate

Burden
Time Per
Respondent

Estimated
Cost to
Respondent

Number of
Respondents
in Category

Estimated
Annual Cost
Across All
Respondents

$44.001
$44.001

0.75
0.5

$33.00
$22.00

124.5
450

$4,108.5
$9.900.00

$15.00$19.002
$23.003

0.5

$7.50-$9.50

9006

$7,650

0.75

$17.25

525

$9,056.25

$13.004
$36.005

0.33
0.5

$4.29
$18.00

110
400

$471.90
$7,200.00
$37,420.65

$44.001
$44.001

0.75
0.75

$33.00
$33.00

20
20

$660.00
$660.00

$15.00$19.002
$23.003

0.75

15

$191.25

0.75

$11.25$14.25
$17.25

20

$345.00

$36.005

0.5

$18.00

20

$360.00
$2,216.25

1

Salary estimates for PI and STEM faculty are based on average salary estimate for full professors as reported in U.S.
Department of Education, National Center for Education Statistics. (2008). The Condition of Education 2008. (NCES 2008031). Washington, DC: U.S. Government Printing Office.
2
Salary estimates for undergraduate recipients in teacher preparation are based on Abt Associates salary for temporary staff who
are undergraduate students. Salary estimates for post-baccalaureates and career changers in teacher preparation are based on
National Science Board. 2010. Science and Engineering Indicators 2010. Arlington, VA: National Science Foundation (NSB
10-01).
3
Salary estimates for recipients with teacher certification are based on public school teacher salary estimates as reported in
NCES Projection of Education Statistics to 2015. NCES September, 2006
4
Intern salary estimates are based on Abt Associates salary for temporary staff who are undergraduate students.
5
Salary estimates for K-12 principals are based on average salary estimate for principals as reported U.S. Department of
Education, National Center for Education Statistics. (2007). The Condition of Education 2007. (NCES 2007-064).
Washington, DC: U.S. Government Printing Office.
6
Calculated based on assumption that 50 percent of Noyce recipients are undergraduate students and 50 percent are postbaccalaureates and career-changers.

Abt Associates Inc.

Section A: Introduction

A-10

A.13. Estimate of Total Capital and Startup Costs/Operation and
Maintenance Costs to Respondents or Record Keepers
There is no overall annual cost burden to respondents that results from this study other than the time
spent responding to the survey and interviews attached as appendices to this request.

A.14. Estimates of Costs to the Federal Government
The estimated cost to the Federal Government for the data collection activities included in this request
for approval is $1,681,485. This cost estimate includes instrument development and pretesting;
student recruitment; data collection; and data processing and analysis over a three-year period.
Indirect as well as direct costs are included in these estimates.

A.15. Changes in Burden
This is a new collection of information.

A.16. Plans for Publication, Analysis, and Schedule
The purpose of this research is to examine the impact and outcomes of NSF–funded Noyce awards on
participating recipients and the institutions who received the awards. This will help NSF respond to
such questions as: How has the Noyce Program helped improve the ability of K-12 teachers to teach
science and math for different types of recipients? How has Noyce Program enhanced the
involvement of STEM faculty in K-12 teacher preparation? Has the Noyce Program impact the
number and proportion of teachers STEM majors and professionals teaching in high-need schools?
An analytic evaluation report will be prepared based on study findings. The survey and interview
data will yield important details about how the Noyce Program is being implemented across the
country as well as the paths Noyce recipients are taking throughout their teacher preparation and early
teaching years. The study team will also examine variation in program implementation (types and
levels of financial support and of Noyce-supported activities), school and district characteristics, and
recipient characteristics. Data from interviews will be examined through simple frequencies as well
as descriptive summaries of emergent themes.
Analyses of survey data will include a detailed summary that utilizes appropriate descriptive
statistics. For survey items using continuous scales, the study will calculate means and standard
deviations to describe both central trends and variation across the samples of schools. Frequency
distributions and percentages will be used to summarize answers given on ordinal scales. The study
will also compare differences in responses across different types of recipients, including, for example,
patterns of leadership responsibility for those whose first participation in the Noyce Program occurred
during their junior or senior years as well as for those whose first participation occurred as a career
changer. Similarly, the study may examine the patterns of participation and types of activities for
those individuals and projects funded at the very beginning of the Noyce Program as well as those
from the most recently awarded grants, or to examine variation across types of institutions.

Abt Associates Inc.

Section A: Introduction

A-11

The analyses will examine how program characteristics, including such things as program activities
and level of financial support; and Noyce recipient characteristics, including their status when they
began the program (i.e., junior/senior, post-baccalaureate, career changer) and year when they first
received Noyce support, influence program outcomes, including plans to teach in general, plans to
teach in high-need districts, and plans to take on leadership roles. The strength of the observed
associations between program and recipient characteristics and outcomes will be estimated using
multiple regression techniques to appropriately control statistically for other factors that could
potentially affect recipients’ plans to stay in teaching other than the recipients’ respective IHE’s level
of financial support (i.e., contextual factors such as climate of K-12 school, districts’ hiring policies
due to changes in state or district economic circumstances).
The data to be collected are clustered (or “nested”) within grantee IHEs—that is, multiple recipients
will have been supported by the same IHE. It is standard analytic procedure to take nesting into
account in the statistical models that examine the strength of the relationship between various
outcomes and predictors; therefore, the study’s analyses will use multi-level models to remove any
variation in the outcome of interest that is purely associated with the unit in which individuals are
clustered, in this case, IHE. This allows a more precise estimation of the relationship of interest.
Finally, the study will use longitudinal teacher employment and certification data from state
departments of education and teacher retirement funds to conduct a rigorous quasi-experiment to
assess the impact of Noyce on teacher recruitment, teacher retention, and teacher effectiveness. In a
sample of states, certification and employment data will be collected over a 10-year period, beginning
prior to any Noyce institutions receiving funding to the current period. The study will examine
whether the Noyce Program had an impact on the number of STEM majors and professionals being
certified, the proportion that teach in high-need districts and schools, and the retention rates in those
schools.
Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and
publication formats. The contracting agency, which will conduct this third-party study of the Noyce
program on behalf of NSF, is forbidden contractually from publishing results unless NSF had made a
specific exception. All products of the collection are the property of NSF. After the products are
delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF,
i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the
quality of the information delivered by the study that NSF decides the format (raw or analytical) and
manner (in the NSF-numbered product On-line Document System or simply a page on the NSF Web
site) in which to publish.
Before the conclusion of the study the NSF grants funded under this program may use preliminary
data to improve management and performance. For example, data generated by this study may
appear as inputs to other internal and external NSF reports (e.g., the GPRA annual Performance Plan).
At this time, NSF has no set timeline for publishing interim reports from this study.

Abt Associates Inc.

Section A: Introduction

A-12

A.16.1 Project Time Schedule
Spring 2010:
Summer/Fall 2010:
Fall 2010/2011:

Winter 2011/2012:
Spring 2011/2012:
Fall 2012:

Study design
Prepare online surveys
Administer online surveys of Principal Investigators, STEM faculty, Noyce
recipients, and K-12 administrators; Collect longitudinal teacher certification
and employment data from state departments of education and teacher
retirement funds
Conduct interviews with a sample of Principal Investigators, STEM faculty,
and Noyce recipients
Analyze data
Prepare final report

A.17. Approval to Not Display Expiration Date
The data collection instruments will display the expiration date.

A.18. Exceptions to Items 19 of OMB Form 83-I
No exceptions are sought.

Abt Associates Inc.

Section A: Introduction

A-13

References
Lawrenz, F., et al., (2009) University of Minnesota Evaluation of the Robert Noyce Teacher
Scholarship Program: Final Summary Report. Minneapolis, MN, University of Minnesota,
2009.

Abt Associates Inc.

Section A: Introduction

A-14