OMB Package Section B

Section B
Collection of Information Employing Statistical
Methods
B.1. Respondent Universe and Sampling Methods
Survey and Interview Samples

The survey data collection process will include participants from all 224 Noyce grants funded
between 2002 and 2009.3 Respondents include Principal Investigators (PIs), Noyce recipients (i.e.,
Interns, Scholars, Post-Baccalaureates, Career Changers, NSF Teaching Fellows, and NSF Master
Teaching Fellows), STEM faculty, and K-12 principals.
Noyce Recipient Sample

The study will field two waves of survey data collection of Noyce recipients. (See Appendix G for
Noyce Recipient survey). The first will be conducted in the fall 2010 and will include all Scholars,
Career Changers, Post-Baccalaureates, and Interns from those IHEs that received awards in years
2003 through 2008. Teaching Fellows and Master Teaching Fellows were only introduced in the
2009 awards. The second data collection will occur in fall 2011 and will include all Noyce recipients
who received their awards in 2009, including the Teaching Fellows and Master Teaching Fellows for
the first time. Additionally, the approximately 50 Noyce recipients from 2003-2008 grants who have
completed their obligatory teaching years4 will be surveyed a second time in 2011-12, to learn
whether these teachers continue to teach in high-need districts and schools.
The study sample will include the census of recipients across each of the six types of respondents
described above. The rationale for this strategy is based on several related factors. First, there are
different numbers of recipients within each of the possible recipient categories, and NSF is interested
in learning about all types of recipients. Second, because some recipients (e.g. Scholars, PostBaccalaureates, and Career Changers) started their Noyce activities up to seven years before the first
round of data collection in 2010, the study may not be able to locate and include all of these
individuals in the study. In fact, it is quite possible that it will be easier to locate and recruit Noyce
recipients who entered the program more recently, particularly, in the last two or three years. This
will not be an issue for the second round of survey data collection in 2011 since its focus will be on
recipients who were awarded in 2009. Third, experience from the prior study of the Noyce program
indicates that obtaining high response rates may be challenging. Taken together, these factors suggest
that to obtain a sufficient number of survey responses to compute estimates with reasonable precision
(i.e. small standard errors, modest confidence intervals), the most conservative approach would be to
include the census as our potential respondent sample.
3

This number does not include supplemental or planning grants, which are excluded from the evaluation.
The total number of grants awarded to date, including supplemental and planning grants, is 249.

4

Noyce recipients are required to teach for two years in a high-need district for each year of support they
receive.

Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-1

To follow up on the information obtained though surveys, the study will conduct telephone interviews
with a sample of recipients following each wave of survey data collection to collect more in depth
information about the processes and mechanisms by which the program operates. (See Appendices L
and M for Noyce Recipient interview protocols). In winter 2011, after the first wave of data
collection, the study will conduct interviews with approximately 50 recipients, chosen from those
recipients who had completed surveys, including recipients who started as juniors/senior, postbaccalaureates, and career changers and those who vary with respect to career stage (e.g., still in
certification program, in first two years of teaching, and in subsequent years of teaching). In winter
2012, after the second wave of survey collection, the study will conduct interviews with
approximately 25 recipients from the 2009 grants, including at least 10 with the Teaching Fellows
and Master Teaching Fellows.
Noyce Principal Investigators

The Noyce program awarded 125 grants from 2003 to 2008, and all PIs of those grants will be
surveyed in 2010; in 2011, the study will survey PIs of the 99 awards newly funded in 2009. (See
Appendix D for PI survey.)
The study will also conduct telephone interviews with a sample of PIs following each wave of survey
data collection to further explore information learned in surveys. (See Appendix I for PI interview
protocol.) This sample will include all PIs from the Noyce programs of those recipients who were
selected for interviews (estimated to be approximately 25 in winter 2011 year and 15 in winter 2012),
and information from both PIs and recipients will provide more complete data on program operations
and components from multiple perspectives.
STEM Faculty

To identify the appropriate individuals to address the study’s research questions, the study must first
determine who among the STEM faculty within an IHE can provide meaningful information to the
study. This means distinguishing between STEM faculty members who are substantially involved in
the Noyce Program and those whose association with Noyce is limited to teaching classes within
which Noyce recipients were enrolled. Individual among the former group are more likely to be able
to provide the study with useful information about such topics as the implementation of the project,
recruiting and/or selecting Noyce support recipients, and mentoring and/or monitoring Noyce support
recipients, and institutional changes that have taken place within their departments. Additionally, to
maximize the likelihood that STEM faculty respondents are familiar with the Noyce recipients, the
study who will further limit this group to those STEM faculty members who are currently working
with active Noyce projects (i.e., grants are currently active). To identify the appropriate STEM
faculty respondents, the study will ask the PIs to provide lists of the STEM faculty whom they
identify as currently engaged in these types of activities. The previous evaluation found that there are
typically between three to five STEM faculty members involved at each IHE (Lawrenz et al., 2009).
This means that the target population of STEM faculty respondents is estimated to be approximately
900. The study plans to survey the full sample of active STEM faculty members in this group. (See
Appendix J for STEM Faculty interview protocol.)
The study will conduct perform telephone interviews with a sample of STEM faculty members
following each wave of survey data collection, to get additional information on the institutional
Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-2

change that has occurred in STEM departments, focusing in particular on the changes they have made
regarding their involvement in teacher preparation. During each wave of data collection, the study
will conduct interviews with 2 faculty members at each of 10 IHEs. The study plans to select a
different set of IHEs during each year so as to increase the institutional diversity along programmatic,
geographic, enrollment, and public/private status dimensions. The sample will be selected among
those faculty members who had completed surveys.
K-12 Principals

The overarching aim of the Noyce Program is to increase the number of teachers qualified in STEM
content areas in the nation’s K-12 schools. Data from school principals will provide information
about the types of schools that employ Noyce recipients, how the Noyce Program is perceived, and
the qualifications of Noyce-trained teachers. Their responses will also be used to investigate the
relationships between characteristics of the Noyce Program, types of Noyce recipients, and teacher
recruitment and retention. School-level respondents most likely to yield useful information about
schools’ participation in Noyce will be from those schools with Noyce recipients on staff;
specifically, those schools with at least one previously supported Noyce recipient teaching in 2008-09
(based on the most recent information from the Noyce monitoring data: 466 unique schools
participated in the program in 2009-10, and by the following year, given the increasing number of
grants awarded in 2008 and 2009, the number of schools with one or more Noyce teachers is expected
to increase to 800. The study plans to survey the principals of all schools in the target population.
(See Appendix F for K-12 Principal survey.) During each wave of data collection, the study will
conduct interviews with a principal or other administrator most familiar with the performance of their
Noyce teachers and the hiring process at each of 20 K-12 schools; interview topics will include
current and historic availability and quality of math and science teachers, the performance of the
Noyce and other math/science teachers, and partnerships with local IHEs. (See Appendix K for K-12
Principal interview protocol).

B.2. Procedures for the Collection of Information
The study team will field online surveys for four broad categories of respondents: Noyce recipients
(census), PIs (census), STEM faculty (purposive sample, limited to those who are identified by PIs as
currently active), and school principals (purposive sample, limited to those whose schools had had a
Noyce teacher in the most recent monitoring data available). The study team will obtain the list of
names and e-mail addresses of PIs from the Noyce Monitoring System. Additionally, the monitoring
data will provide us with the names of recipients and schools where past Noyce participants are
currently teaching. The next step will be to locate respondents and obtain current contact
information. PIs will be contacted to ascertain whether we have the complete list of recipients, and to
provide the contact information for these recipients. This approach was used in the previous Noyce
Program evaluation, and it yielded substantial, albeit in complete current contact information. They
will also be asked to provide us with the list of STEM faculty that are currently working with their
Noyce programs. The study team will then identify faculty contact information by reviewing
department websites. Finally, contact information for principals in schools where past Noyce
participants are currently teaching can be found using online and offline methods, including phone
calls to the institutions and schools in question.
When current contact information is not available through the above sources, the study will use
multiple location methods, including: telephone, e-mail, and/or mail follow-up with the respondents;
Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-3

internet web searches on respondents’ names, discipline, and institution; and telephone, e-mail, and/or
mail follow-up with respondents’ former departments and institutions. The study team has a proven
track record in locating former and current postsecondary students, teachers, and administrators
through collaboration with institutions of higher education, K-12 schools, and use of electronic
databases. Perhaps most important, the study team will have dedicated location staff to facilitate this
process.
The study team will first notify Principal Investigators about the study via electronic mailings from
NSF and the evaluation team containing the following materials:


An introductory letter signed by the cognizant NSF official(s) that explains the purpose and
the importance of the study and the expected use of study findings at the federal level.



An introductory letter signed by the study’s project director that explains the purpose of the
study and requests PIs’ assistance in providing contact information for Noyce recipients and
a list of the STEM faculty actively involved in the Noyce Program.



An attractive fact sheet introducing the Noyce program evaluation and describing the
objectives and importance of the study, study participants, data collection activities and
schedule, participants’ and evaluators’ responsibilities, and the evaluation team’s contact
information. This fact sheet will also be used to describe the study to anyone interested in the
study throughout the study period.

Other respondents will be notified of the evaluation prior to the administration of surveys via email
once contact information has been obtained. Those respondents will receive the similar materials via
e-mail as will have been sent to the PIs.
Once relevant approvals are obtained (from OMB, from Abt Associates’ IRB, and from institutional
IRBs where applicable), we will program the survey for online data collection. The study team will
test each survey system to ensure functionality and accuracy of data capture. We plan to launch
survey data collection in the early fall 2010, and again in fall 2011. Given the academic calendar, this
is an ideal time to survey respondents. To ensure effective data collection, our approach will focus on
three areas:
Developing and Implementing an Accessible and Intuitive Survey. Achieving strong response rates
begins with a well-designed, user-friendly instrument, and continues with providing a clear and
convincing rationale for the survey and the importance of respondents’ participation. The study team
will use the knowledge and lessons learned from the prior Noyce evaluation to design instruments
that will be clear and transparent to the recipients. The instruments will take into account the
changing nature of the program and will allow for additional questions to be added each year to
account for these changes.
One challenge of this study will be motivating participation and following-up with initial nonrespondents to achieve our target response rate. The web approach will allow us to easily identify
non-respondents for follow-up emails and phone contact to encourage participation, thus substantially
increasing response rates.
Supporting Respondents During Data Collection to Ensure a High Response Rate. Prior to data
collection, the study team will provide early notification to respondents, including a signed letter from
Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-4

NSF as well as from the Abt Team. This will permit respondents to allocate time to complete the
surveys, and increase respondent “buy-in,” enabling them to ask questions before beginning data
entry and allowing our team to establish rapport with any reluctant respondents. At the designated
opening date, an e-mail message will be sent to respondents with a unique user name and password,
detailed instructions, the closing date, and project staff contact information. Detailed on-screen
instructions will be included and an extensive help functionality for survey items, including a
Frequently Asked Questions section, glossary, and navigation instructions. Throughout the data
collection cycle, a toll-free number and e-mail address will be available to ensure that potential
respondents can easily and quickly obtain answers to questions or concerns. We will conduct up to
five follow-ups per respondent.
Protecting the Confidentiality of Data Collected. Abt Associates Inc. is a conscientious guardian of
the confidentiality of data. We have conducted numerous contracts involving sensitive information;
consequently, the corporation and all project staff employ both electronic and physical safeguards to
protect data from unauthorized access. Electronic project directories, files, and databases are
accessible to project staff only and are protected by discretionary access control lists (ACLs), group
memberships, passwords, and locking workstations. Access to the data processing area and database
servers is limited to authorized personnel and building security staff all sites 24 hours, 7 days per
week. To protect against data loss, Abt also uses automated, redundant backup procedures and file
management techniques to ensure that files are not inadvertently lost or damaged. The web-based
survey data will be maintained on a secure server with appropriate levels of password protection.
B.2.1. Statistical Methodology
Primary Data Collection
As described above, the study team will be surveying the census of PIs and recipients from 20032009 Noyce grants, the STEM faculty identified by PIs as actively involved in Noyce grants, and K12 principals in schools where former Noyce recipients were recently teaching. Additionally, the
study team will conduct interviews with a convenience sample of each of the recipient types.
The primary purpose of surveying the Noyce PIs, recipients, STEM faculty, and principals in schools
where former Noyce recipients are teaching is to describe the strategies Noyce grantees use to recruit,
prepare, and retain K-12 mathematics and science teachers in high-need districts and to see how
various program components affect recipient outcomes from different types of recipients. Therefore,
for this part of the study, a comparison group will not be surveyed.
Impact Analyses Using Extant Data
To determine the impact of the Noyce program on teacher recruitment and retention, the study will be
comparing outcomes for sets of institutions within a sample of states both prior to and post receipt of
the Noyce awards. Since the institutions will have received awards during different years, the periods
prior to receipt of the Noyce award for some institutions will serve as the comparison groups for those
institutions that had already received the awards. These analyses will be based on extant teacher
employment and certification data from state departments of education and teacher retirement funds.
Preliminary inquiries have been made to these agencies to ensure the necessary data are available. A
more detailed description of the impact analyses is included in Appendix B.

Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-5

B.3. Methods to Maximize Response Rates and Deal with
Nonresponse
The study team will use the knowledge and lessons learned from the prior Noyce evaluation to design
instruments that will be clear and transparent to the recipients. The instruments will take into account
the changing nature of the program and will allow for additional questions to be added each year to
account for these changes.
One challenge of this study will be motivating participation and following-up with initial nonrespondents to achieve our target response rate. The web approach will allow us to easily identify
non-respondents for follow-up emails and phone contact to encourage participation, thus substantially
increasing response rates. In an effort to increase overall survey response rate, follow-up with
respondents will be multi-modal. Telephone and email follow-up will be used for non-respondents.
Prior to data collection, early notification will be provided to respondents, including a signed letter
from NSF as well as from the study team. This will permit respondents to allocate time to complete
the surveys, and increase respondent “buy-in,” enabling them to ask questions before beginning data
entry and allowing our team to establish rapport with any reluctant respondents. At the designated
opening date, the study team will send an e-mail message to respondents with a unique user name and
password, detailed instructions, the closing date, and project staff contact information. Detailed onscreen instructions will be included with an extensive help functionality for survey items, including a
Frequently Asked Questions section, glossary, and navigation instructions. Throughout the data
collection cycle, the study will use a toll-free number and e-mail address to ensure that potential
respondents can easily and quickly obtain answers to questions or concerns.

B.4. Test Procedures or Methods
The survey instruments developed for this data collection will be pilot-tested.

B.5. Individuals Consulted on Statistical Aspects and Individuals
Collecting and/or Analyzing Data
The contractors for collection and analysis of data in this study are Abt Associates Inc., and its data
collection subsidiary Abt-SRBI. Staff from these organization have knowledge of statistical methods,
experience in evaluation of research programs, and expertise in scientific research.

Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-6

Key personnel involved in the statistical aspects and who will be involved in collecting and analyzing
data include:
Abt Associates, Inc.

Ellen Bobronnikov
Beth Gamse
Cristofer Price
Radha Roy
Carter Smith
Marc Moss
Laurie Bozzi
Amanda Parsad

617-349-2718
617-3492808
301-634-1852
301-347-5722
617-349-2543
617-349-2825
617-349-2485
301-634-1791

Abt-SRBI

Paul Schroeder
Andy Weiss

301-608-3883
301- 608-3883

Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-7

Abt Associates Inc.

Section B: Collection of Information Employing Statistical Methods

B-8