NSF_PIRE_OMB PartA_submit

Supporting Statement for Paperwork Reduction Submission: Part A

Program Evaluation of the Partnerships for International Research and Education (PIRE) Program

Introduction

The National Science Foundation (NSF) requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, a three-year clearance for original data collection to be used in the Evaluation of the Partnerships for International Research and Education (PIRE) Program. The new data collections include a Principal Investigator Survey (Appendix B); a Postdoctoral Survey (Appendix C); a Graduate Student Survey (Appendix D); an Undergraduate Survey (Appendix E); a Foreign Senior Investigator Survey (Appendix F); and an Institutional Administrator Survey (Appendix G).

The PIRE Program

The PIRE program supports U.S. researchers who wish to pursue a research agenda in collaboration with one or more international research partners. The program funds projects across a broad array of scientific and engineering disciplines, in an effort to catalyze long-term, sustainable international partnerships for collaborative research and education that will prepare a cadre of students and early-career researchers for strong leadership and engagement in global science and engineering. PIRE awards support intellectually substantive collaborations between U.S. and foreign researchers in which the international partnership is essential to the research effort. PIRE grantees must provide educational and professional development opportunities for U.S. based researchers, especially those early in their career, including postdoctoral fellows and graduate and undergraduate students. A significant aspect of the opportunity for U.S. participants is on-site research experience at an international laboratory, institution or research site, whether university-, industry- or government-based. PIRE funding may be used only to support U.S. participants in the research and educational activities, including international travel; foreign partners in PIRE are expected to secure and contribute their own funding (for example from NSF’s counterpart agencies in other countries). However, Principal Investigators who apply for PIRE awards must include letters of support from their proposed foreign partners and evidence that these partners have access to their own independent funding to support their contributions to the research agenda.¹ A logic model of the PIRE program is included in Appendix A.

PIRE awards last five years. Classes are funded in an alternating two-year, three-year cycle so that two classes are active at any given time. The first class of funded projects was 2005, so their funding ended in 2010 to 2011; the second class was funded two years later, and ended in 2012. However, there have been some no-cost extensions. The third class was funded in 2010 and the fourth in 2012.

As of June 2013, across the 2005, 2007, 2010 and 2012 cohorts, PIRE has made a total of 59 awards. These projects range in size from relatively small, bi-national consortia (e.g., two U.S. and two non-U.S. institutions in one foreign country) to large, multi-national, multi-institutional awards (e.g., a dozen U.S. institutions and eleven non-U.S. institutions representing eight foreign nations). Many are multi-disciplinary, combining, for example, the expertise of econometricians with researchers in fluid dynamics; and, notably, many feature partnerships between academic and industrial or non-profit institutions. Collectively, these 59 PIRE projects have provided research and educational opportunities for more than 100 postdoctoral fellows, more than 625 graduate students and approximately 600 undergraduates. More than 600 U.S.-based and over 400 foreign-based faculty and researchers at university and non-academic institutions have participated in one or more PIRE-funded collaborations.

The PIRE program has the following objectives: (1) to promote opportunities for U.S. scientists and engineers to engage in international collaborations that enhance research excellence; (2) to provide international research and educational experiences for U.S. students and faculty that will prepare the U.S. science and engineering workforce for global engagement; and (3) to strengthen the capacity for U.S. researchers and institutions to build and sustain international partnerships. Beginning with the 2012 cohort of PIRE awardees, the program tailored its first objective (to support international collaborations that enhance research excellence) to focus on international partnerships that would support excellence in science, engineering and education to “inform the societal actions needed for environmental and economic sustainability and sustainable human well-being” (NSF, 2010).

There have been several notable changes to the program since its inception. In the 2005 and 2007 PIRE cohorts, budgets were capped at $2.5 million, but beginning with the 2010 cohort, these budget limitations were removed (NSF, personal communication). In addition, starting with the 2010 cohort, grantees had to propose a project of sufficient scope that its effects would extend beyond an individual PI’s research group to the participating U.S. institutions by strengthening their capacity for sustained international engagement. Most recently, the 2012 PIRE competition was limited to projects that proposed international research and education partnerships to address the NSF-wide priority in Science, Engineering, and Education for Sustainability (SEES; NSF, 2011). NSF also suggested a host of potential partner agencies, both domestic and foreign, that could provide additional funding. For example, NSF’s PIRE and USAID’s PEER programs are jointly funding collaborations between U.S. investigators and their counterparts in developing countries where science and engineering capacity is emerging (National Academy of Sciences, 2014).

Overview of the Evaluation of PIRE

The evaluation of PIRE will examine the quantity and quality of research produced by the PIRE program and its participants; measure the research and career outcomes for PIRE participants; document how PIRE is perceived as changing the way U.S. institutions support, manage, or help implement international research and educational collaborations; and explore how PIRE research has made both intended and unanticipated contributions to research and education in environmental and economic sustainability. The evaluation is also designed to capture any promising practices or lessons learned about the implementation of PIRE projects. In summary, the evaluation will address the seven research questions shown in Exhibit A.1.

The study will measure outcomes at both the project (RQs 1, 2, 5, and 7) and participant levels (RQs 3, 4, and 6) and will employ project- and participant-level comparison groups as follows:

The study will measure outcomes at both the project (RQs 1, 2, 5, and 7) and participant levels (RQs 3, 4, and 6) and will employ project- and participant-level comparison groups as follows:

1. A matched comparison group of non-PIRE, NSF-funded projects that do not require an international collaboration, but which are similar to PIRE awards along other key criteria (see Section B.1 for details of how this comparison group will be constructed). From this program-level comparison group, three participant-level comparison groups will be formed (details on construction of these participant-level groups are provided Section B.1):

1. PIs and co-PIs of comparison projects, matched to corresponding PIRE participants;

2. Postdoctoral researchers in comparison projects, matched to corresponding PIRE participants; and

3. Graduate student participants in comparison projects, matched to corresponding PIRE participants.

2. A group of respondents to nationally fielded surveys of degree recipients in science, engineering and health fields (the Survey of Doctoral Recipients, SDR; and the National Survey of Recent College Graduates, NSRCG) matched to PIRE postdoctoral, graduate student, and undergraduate student participants.

The evaluation will draw on extant data as well as require new data collection efforts. This package seeks OMB approval for the new data collection efforts, which include the online surveys of project participants and U.S. institutional officials. Although approval is sought only for the new data collection, our description of the evaluation includes both the extant and original data sources that will be considered in the study. Below is a brief summary of these data sources include (see Section A.1 for more information about the circumstances requiring these data sources).

New Data Collections

New data will come from online surveys conducted with the following groups:

• PIRE Principal and co-Principal Investigators and Principal and co-Principal Investigators in a matched comparison group of projects funded under NSF programs that do not require an international collaboration;

• PIRE postdoctoral researchers and postdoctoral researchers in a matched comparison group of projects funded under NSF programs that do not require an international collaboration;

		Data Sources: Extant			Data Sources: Primary Survey Data
Exhibit A.1 Research Questions for the Evaluation of NSF’s PIRE program	Program or Participant Level	Bibliometric	NSF program data	National data (SDR, NSRCG)	PI and co-PI	Postdoc	Graduate student	Undergraduate	Foreign participant	Institutional administrator	Analyses
RQ 1. What is the quantity and quality of the publications that PIRE projects have produced (and that are projected to be produced)?	Program	X	X		X						Descriptive
RQ 2. How does the quantity and quality of publications produced by PIRE projects—that are required, by definition, to include an international collaboration—compare to the quantity and quality of publications produced by similar NSF-funded projects that do not require an international collaboration?	Program	X	X								Relational
RQ 3. What are the program experiences of PIRE Principal Investigators (PIs and co-PIs), postdoctoral researchers, and graduate and undergraduate student participants? What are the research and educational or career outcomes for these participants?	Participant	X	X	X	X	X	X	X			Descriptive Benchmarking for career outcomes
RQ 4. How do the research and educational or career outcomes for PIRE PIs, postdoctoral researchers, and graduate student participants compare to those of similar participant groups in similar NSF-funded projects that do not require an international collaboration?	Participant	X	X		X	X	X				Quasi-experimental for research outcomes; Descriptive, relational, and benchmarking for career outcomes
RQ 5. How do international affairs representatives at PIRE PIs’ institutions perceive the effects of PIRE on their institutions’ policies and practices for supporting international research and educational collaborations?	Program									X	Descriptive
RQ 6. What are the program experiences of foreign senior investigative partners in PIRE and how do they perceive the effects of PIRE on their research and educational practices and those of their institutions?	Participant		X						X		Descriptive
RQ 7. How have PIRE projects contributed to research that may inform global societal challenges?	Program				X				X		Descriptive

• PIRE graduate student participants and graduate student participants in a matched comparison group of projects funded under NSF programs that do not require an international collaboration;

• PIRE undergraduate student participants;

• PIRE foreign senior research partners (i.e., scientists and engineers at the level equivalent to a U.S. Principal or co-Principal Investigator); and

• Institutional representatives (administrators in international affairs offices) at the institutions of PIRE lead PIs (or other official who is familiar with the institution’s policies and practices for international research and education).

Extant Sources

• Extant NSF data from PIRE and comparison projects’ proposals, award letters, budgets, and annual and final reports to NSF;

• Extant data from the biennial Survey of Doctoral Recipients (SDR) and National Survey of Recent College Graduates (NSRCG); and

• Bibliometric data from the Web of Science databases maintained by Thomson Reuters.

Program-level questions about PIRE publication quantity and quality will draw on extant NSF program data and bibliometric data. Descriptive analyses (e.g., counts and citation indices of publications produced by the PIRE projects) will address RQ 1. To address RQ 2, a relational analysis will explore associations between these bibliometric indicators and the presence or absence of PIRE’s signature characteristic—namely, the program’s requirement that projects include an international collaboration is a required component. A comparison group of non-PIRE, NSF-funded projects in which an international collaboration is not required (but which may arise despite no formal program requirement) will be used in this relational analysis.

Survey data and descriptive analyses will address the program experiences of PIRE participants (RQ 3). The study will compare career outcomes of PIRE postdoctoral, graduate and undergraduate participants (as reported in these surveys) to benchmarks from nationally representative data from the SDR and the NSRCG. In addition, the study will compare the career outcomes of PIRE participants to those of similar participant types in a matched comparison group of non-PIRE, NSF-funded projects (RQ 4). This comparison will use relational analyses with controls for characteristics that could be correlated with career outcomes.

To measure research outcomes for PIRE participants (RQ 3 and RQ 4), the study will obtain extant data for publications that are indexed in bibliometric databases; primary survey data will provide information about publications that are in progress (e.g., in press, under review) and other research products such as conference presentations and software applications. In addition, the study will compare counts and citation indices for publications by PIRE participants to those of matched participants from a comparison group of projects using a quasi-experimental, comparative interrupted time series (CITS) design. For these analyses, participants in the selected comparison projects will be matched to corresponding PIRE participants using a greedy matching distance algorithm.² Matches will be based on a pre-participation measure of one of the key research outcomes for the study (e.g., the number of publications prior to project participation; the mean citation measure for publications prior to project participation). Details of the greedy matching distance algorithm and the CITS approach are provided in Part B, Section B.3, Estimation Procedures.

Finally, survey data and descriptive analyses will be used to understand: institutional engagement in implementing PIRE project activities and any changes in policies or practices as a result (RQ 5); foreign senior investigators’ role in PIRE, their perception of the benefits and challenges of participation, and any effects of PIRE on their home institutions (RQ 6); and what role, if any, PIRE research has played in advancing knowledge or technologies to address contemporary societal challenges of a global scale (RQ 7).

Rationale for the Evaluation Design

This research design was selected after a determination that neither a randomized control trial nor a regression discontinuity (RD) design was feasible. Because the PIRE program makes awards on the basis of merit, it was not feasible to randomly assign some PIRE proposals to “award” status (i.e., a treatment group) and others to a “non-award” status (i.e., a control group). Moreover, award decisions for past cohorts of the program had already been made when the evaluation was planned. Although a regression discontinuity design was contemplated, PIRE proposals do not receive a continuous score that is compared to an exogenous “cutoff” to make funding decisions, a key requirement for an RD design. Rather, each proposal receives three or more categorical scores, from one (lower merit) to five (highest merit); subsequently, panelists discuss the rated proposals and place each into one of three categories (highly competitive, competitive or not competitive). From these categories, NSF program officers select which proposals to fund, often taking into account additional factors such as geographic and disciplinary variation across the portfolio of funded projects and the institution of the lead PI (e.g., institutions in EPSCOR states).

Another potential quasi-experimental design (other than an RD design) was also deemed inappropriate, namely a design where unfunded PIRE proposals were matched to funded PIRE projects using propensity score matching (PSM) techniques to control for selection bias. For the evaluation of the PIRE program, this method was not feasible. PSM techniques require data on a large number of pre-treatment (in this case, pre-PIRE award) characteristics of the members of the treatment and comparison groups to model selection and develop propensity scores. Data on pre-award characteristics for funded and unfunded PIRE proposals are limited. Moreover, for the PIRE program, it is far from clear what measures at the project level exist for project characteristics before the award decision is made—the project does not exist as a measurable entity until the award decision has been made. For example, the PIRE award itself could affect which individuals come together to form the research group that ultimately engages in research, produces publications, and collaborates with foreign partners. In the absence of the PIRE funding, those individuals did not form an already established group for which characteristics can be measured. At the participant level also, it is unclear what group of individuals would form the counterfactual for an unfunded PIRE proposal: without PIRE funding, what individual postdoctoral fellows, graduate students, or undergraduate students could be identified as those who “would have participated” in PIRE and could therefore comprise a valid comparison group? After considering the above evaluation designs, the current approach was adopted as the most feasible design.

Matching PIRE projects to Comparison Projects

To construct the matched comparison group of NSF-funded projects, candidate comparison projects funded by non-PIRE NSF programs will be identified and matched to each PIRE project included in the evaluation on the basis of the following criteria:

• The PI of the comparison project must not also be (or have been) PI of a PIRE project;

• The comparison project duration (award period) must be within 1 year of the duration of the PIRE project (e.g., if the PIRE award is 5 years in duration, the comparison award must be 4, 5 or 6 years in duration);

• The comparison project award amount must be within 20 percent of the PIRE project award amount;

• The comparison project must be funded by an NSF program that does not require an international component (projects in which international collaboration is not required but is encouraged, optional, or not mentioned explicitly meet this criterion);

• The comparison project must have at least one graduate student participant;

• The comparison project must have research as its primary focus (PIRE awards have research as a primary focus);

• The comparison project must engage expertise in scientific or engineering disciplines that overlap at least partially with those involved of the PIRE project (i.e., the NSF directorate in which the program funding the comparison award is housed must be commensurate with at least one of the NSF directorates that corresponds to the PIRE award and the scientific or engineering disciplines in which comparison project personnel have expertise must be the same as the expertise of at least one member of the PIRE project team);³ and

• The comparison project must include at least two institutions total (U.S. or foreign, if applicable; by definition, a PIRE award includes at least one U.S. and at least one foreign institution).

If multiple matches for a PIRE project are identified, we will select one or two projects at random for inclusion in the comparison group.

Matching PIRE and Comparison Project Participants

There are many multivariate matching techniques for identifying potential matches for each PIRE projects. One frequently used technique, propensity score matching, would require a larger sample of PIRE projects to create reliable propensity scores; in addition, to create propensity scores the evaluation would require a larger number of variables from existing data than are available on participants prior to data collection. Thus, we will use another technique, greedy matching, to match participants within each PIRE-comparison project pair. Unlike PSM techniques, greedy matching is not sensitive to sample size. Greedy matching begins by (1) randomly sorting the N PIRE participants and M comparison project participants; (2) then, the first PIRE case in the list is matched to the closest comparison participant—namely, the one most similar to the PIRE participant. Next, (3) the second PIRE participant is matched to its closest comparison participant among those remaining; and (4) this process repeats until all PIRE participants have been matched.

Here, “similarity” between PIRE and comparison participants is defined as the average distance per year between the PIRE and comparison participants’ score on a pre-test measure of research outcomes.

For example, using the number of publications as the pre-test measure, the average distance per year between the i^th PIRE participant and the j^th comparison participant is defined as

;

where X¹= { } is the number of publications across N years for PIRE participants and

X⁰= { } is the number of publications across N years for Comparison participants.

A.1 Circumstances Requiring the Collection of Data

Although individual PIRE projects may include a project-level evaluation, this study is the first evaluation of the PIRE program as a whole since its inception in 2005. Since then, the PIRE program has made 59 awards across four cohorts (2005, 2007, 2010 and 2012), each for a five-year period (not including any no-cost extensions granted to individual projects). NSF has invested more than $250 million in the PIRE program. Administered by the Office of International Science and Engineering (OISE) within NSF’s Office of Integrative and International Affairs (OIIA), the PIRE program has historically received the largest share of NSF’s OISE programming budget: for example, in FY 2011, the PIRE program had a budget of approximately $18.7 million, representing 38 percent of OISE’s total budget (NSF, 2012).

The evaluation of PIRE will address timely questions about program-supported research productivity and quality, intellectual collaborations across international boundaries, and how to most effectively promote an internationally engaged and competitive workforce in science, technology, engineering and mathematics (STEM). The evaluation will combine descriptive and comparative components, including a quasi-experimental design and multiple comparison groups. The descriptive component will examine the implementation, experiences, and outcomes of the program, and the comparative component will examine program outcomes relative to an appropriate counterfactual.

The importance of international research collaborations cannot be overstated. Through international networks of scientists, resources can be shared and ideas can be developed, tested, and implemented across traditional boundaries (National Science Board, 2008). The NSB recognized the globalization of STEM research and education, in concert with the associated opportunities and challenges for the U.S., in a 2010 publication highlighting globalization trends (NSB, 2010). Other policy makers, convened by the National Research Council in a focused workshop, have also highlighted the potential of science policy and science diplomacy to meet international challenges (NRC, 2011). Participants discussed the importance of international science engagement and global science cooperation, and the value of providing opportunities and incentives for U.S. researchers to engage in science in an international arena (NRC, 2011).

Engagement in international collaboration is important for U.S. scientists’ continued prominence in the global research community and capacity to gain insight into international research advances (NSB, 2001). An active role in international collaborations means that the U.S. can expect to reap benefits not otherwise realized.

Recent research found that when U.S. and U.K. researchers engaged in collaborations, the impact of their resulting research significantly increased (as measured by citation rates) especially for U.S. corresponding authors (Luo et al., 2011). Similarly, a recent bibliometric analysis found that papers written by collaborators from multiple countries were cited more than those with authors from a single country (Adams, 2013). Benefits of international collaboration may include increased access to physical resources and funding; additional opportunities to benefit from the expertise of collaborators; and access to populations, records, historical materials, and circumstances that provide “natural experiments” (Goodnow, 2008). International collaborations can facilitate expansion of U.S. markets and promote opportunities for international economic exchange (NSB, 2001). Finally, international networks of scientists can more easily share resources and develop, test, and implement ideas across traditional boundaries (NSB, 2008).

Although policy-makers and funding agencies have emphasized the increasing need for U.S. STEM researchers to become globally engaged with their counterparts in other countries, little research has systematically examined the effectiveness of programs explicitly designed to foster international collaboration. Recent studies have, however, documented that participants in programs designed to promote international STEM collaboration report acquisition of knowledge and skills, as well as plans to engage in international collaborations in the future (e.g., Flattau et al., 2009; Institute of International Education, 2009; Spencer, 2008).

NSF has contracted with Abt Associates to conduct an evaluation of PIRE that will address seven overarching research questions (RQs), shown in Exhibit A.1. The evaluation will include the census of the 59 PIRE projects awarded between 2005 through 2012. A matched comparison group will be constructed by matching each PIRE project to one or more projects funded by other NSF programs that do not require international collaboration (although some projects may include international collaboration). The goal of the matching is to identify projects that differ from a given PIRE project only in the program’s relative emphasis on international collaboration. From this comparison group of projects, three comparison groups of project participants (PIs, postdoctoral researchers, and graduate students) will be formed. Details of the construction of these comparison groups are provided in Part B, Section B.1.

The evaluation will provide a comprehensive picture of the activities supported through the PIRE program, educational and career outcomes of participants, the extent to which participants sustain international collaborations beyond the end of the PIRE award period, and research outcomes including the number of research reports published; the quality of research publications, as measured by bibliometric citation indices; and the ratio of publications that include one or more foreign co-authors. The evaluation will employ data from extant sources and new online surveys. Below we describe these data sources in more detail.

New Data Collections

Primary data collection activities for which approval is sought entails online, web-based surveys of PIRE Principal and co-Principal Investigators (PIs and co-PIs), postdoctoral, graduate and undergraduate participants, foreign senior investigators, and administrators within international affairs or study abroad offices at the U.S. institutions of the lead PIs. The web surveys of PIs and co-PIs, postdoctoral researchers and graduate student participants will also be administered to matched groups of corresponding participants in other, non-PIRE NSF-funded projects. Supporting Statement B, Section B.1 and Exhibit B.3 provide details of the sample sizes for the respondent groups. Surveys of PIs/co-PIs, postdoctoral researchers and graduate students include items from nationally representative surveys (the Survey of Doctoral Recipients, SDR; the National Survey of Recent College Graduates, NSRCG). Surveys also include, wherever possible, items or adaptations of items from previously administered surveys that received OMB clearance. In each survey, individual items serve at least one of the following purposes:

• Outcome data: items that will allow construction of key outcome measures;

• Descriptive data: items that will permit the construction of summary descriptive statistics on program implementation;

• Covariate data: items that will measure characteristics that are likely associated with outcomes and must be included in analytic models to control for this relationship;

• Matching data: items that will allow matching of project participants to respondents to the SDR and the NSRCG for purposes of benchmarking;

• Benchmarking data: items that will allow comparison of project participants on key employment outcomes to nationally representative data from the SDR and NSRCG; and

• Administrative data: items that will allow implementation of skip patterns to minimize the number of items presented to each respondent and to permit automatic filling of appropriate text based on respondent characteristics.

Appendix H shows, for each survey item, the sources of items from these existing surveys, along with the research question(s) the item addresses and its anticipated use in analyses.

Surveys of PIs in PIRE and comparison projects. A census of PIs and a sample of co-PIs for PIRE projects will be surveyed to provide descriptive data and to permit quasi-experimental analyses of the impact of PIRE on key outcomes. Descriptive information to be collected will include PIRE PIs’ experiences collaborating with international partners, including perceived benefits and challenges; how PIs perceive the effects of PIRE on their access to expertise, data, equipment or place-based phenomena; the effect of PIRE on their colleagues’ or institutions’ support for international research opportunities; and any contributions of PIRE research for societal challenges (e.g., processes or technologies for “big data;” advances that may foster more sustainable human-environmental interactions). A matched sample of PIs and co-PIs from a matched group of comparison projects will also be asked to complete these surveys (with the exception of items specific to PIRE). Including this comparison group of PIs will permit quasi-experimental analyses of the differences in the quantity and quality of research between PIRE and its comparison group. Relational analyses will explore associations between the presence or absence of a program requirement for international collaboration and other key outcomes.

Surveys of postdoctoral participants in PIRE and comparison projects. A census of PIRE postdoctoral fellow participants will be surveyed to provide descriptive data on the nature of activities in which PIRE postdocs engaged, including the duration of any research experience abroad, how international experiences affected their career outcomes, and the persistence of international collaboration beyond the period of their participation in PIRE (e.g., if the postdoctoral appointment has ended). A sample of postdoctoral researchers from a matched comparison group of other NSF-funded projects will permit exploration of the association of PIRE with: the number of pending publications and other works that are in press, under review or in preparation (i.e., not available in bibliometric databases); the persistence of international collaboration beyond the duration of the PIRE award or the end of the postdoctoral period; and subsequent job characteristics and career trajectories. Data from the survey will also be used to benchmark employment and career outcomes against a nationally representative sample of respondents to the SDR, matched to PIRE postdoctoral recipients on year of PhD receipt and STEM discipline.

Surveys of graduate student participants in PIRE and comparison projects. For PIRE graduate students, descriptive data will be collected on the nature of activities in they were engaged, the duration of any research experience abroad, and what benefits and challenges they perceived working with foreign senior colleagues or peers. Data from a matched group of graduate students from the comparison group of NSF projects will facilitate analyses of the relationship between participation in PIRE or an NSF-funded project in a comparison group and educational and career outcomes, including degree completion, employment characteristics, persistence of existing international collaboration, and engagement in subsequent international collaborations. As with PIs and postdoctoral fellows, data will be collected on pending publications and other works that are not yet indexed in bibliometric databases.

Surveys of undergraduate participants in PIRE. A sample of undergraduate participants in PIRE will be surveyed about their research or education experiences abroad. This survey will focus on whether and how undergraduates’ education is enhanced by an international research experience, including learning gains (e.g., understanding the research process); satisfaction with components of the experience (e.g., mentoring, interactions with other students, faculty); how the experience affected their interest in and motivation to pursue additional coursework in science and plans to earn graduate degree or pursue a career involving scientific research. For former undergraduates who have completed their bachelor’s degree, educational and employment outcomes will be compared to benchmark data from a nationally representative sample of respondents to the NSRCG. These benchmark data will provide important information about how PIRE contributes to the STEM workforce, beginning at the postsecondary level.

Surveys of foreign senior investigators in PIRE. A sample of senior foreign investigators (i.e., individuals at a career level equivalent to Principal or co-Principal Investigator) will be surveyed to assess their perception of PIRE’s effects on their research, the education of their students and postdoctoral trainees, their persistence in international collaborations, particularly with U.S. scientists or engineers, and how participation in PIRE may have affected practices or policies at their institutions. As in the survey of U.S.-based PIs, this survey will ask foreign participants about the benefits and challenges of PIRE, the persistence of international collaboration beyond the duration of the PIRE award (if the award period has ended), and whether their participation in PIRE caused them (or their institutions) to make any changes in how they train their students and emerging scientists.

Surveys of Institutional administrators at PIRE PIs’ U.S. institutions. For each of the 59 PIRE projects, an administrative official in an appropriate international affairs office(s) at the lead PI’s institution in the U.S. will be asked to participate in a survey that asks how the institution supports international research and/or educational collaborations; what institutional changes in educational policies or practices for graduate or undergraduate students have occurred as a result of PIRE; what role any pre-existing institutional partnerships with educational institutions in foreign countries have played in the PIRE project; how PIRE has affected these prior partnerships or fostered the creation of new inter-institutional, international partnerships; and any challenges that have resulted from the participation of graduate or undergraduate students in sciences or engineering in education or research abroad, and how the institution has responded to those challenges.

Extant Data

Extant data sources includes NSF program data, which provide information about PIRE projects as well as identified comparison projects and bibliometric (publication and citation) data for projects and participants.

Extant NSF program data will be used for four purposes:

• To identify candidate non-PIRE NSF-funded projects for the comparison group;

• To extract full bibliographic data on publications that resulted from the project;

• To identify participants in the project for bibliometric analyses and new survey data collections; and

• To inform descriptive analyses of the PIRE program.

Extant nationally representative survey data from the Survey of Doctoral Recipients (SDR) and the National Survey of Recent College Graduates (NSRCG) will provide nationally representative benchmarks for selected employment indicators of PIRE participants. The SDR is a longitudinal survey of a nationally representative sample of science, engineering, and health (SEH) doctorate recipients. The NSRCG is a longitudinal survey that shares many of the same questions as the SDR but targets students who received a bachelor’s or master’s degree in a science, engineering, or health (SEH) field from a U.S. institution. Conducted every two years, the SDR surveys respondents until they reach 76 years of age, adding a new cohort of doctorates in each survey cycle. The most recent data available are from the 2010 wave.⁴ The NSRCG was conducted every two years through 2010, after which it was discontinued. The SDR and NSRCG include several key employment indicators of interest for this evaluation. These items from the SDR and/or NSRCG have been included in the postdoctoral, graduate student, and undergraduate student surveys (see Appendix H for a list of survey items and sources) to facilitate direct comparisons between these PIRE participants and benchmarks from these nationally representative extant data.

A sample of science and engineering doctoral recipients from the most recently available SDR wave will be matched to PIRE participants who completed a doctorate (e.g., postdoctoral PIRE participants or former PIRE graduate students or undergraduates who have since completed their PhD). This SDR sample will be matched based on the year of receipt and disciplinary field of the doctoral degree. Data from this SDR sample will provide benchmarks against which to compare PIRE participants. Likewise, a similarly matched sample of science and engineering bachelor’s and master’s degree recipients from the 2010 NSRCG wave will be selected to provide benchmarks against which to compare PIRE undergraduate and master’s-level graduate students. Although descriptive in nature, such benchmarking analyses help provide a national context in which to interpret outcomes for PIRE participants.

Extant bibliometric data from Thomson Reuters’ Web of Science will allow relational analyses of research productivity and quality associated with PIRE and NSF projects in the comparison group and a comparative interrupted time series analysis of research productivity and quality (as measured by publication and citation indices) for participants in PIRE and matched participants in projects in the comparison group. Two types of extant bibliometric data will be collected from the Web of Science:

• Citation indices for publications resulting from the PIRE and comparison group projects (to inform RQ 1 and RQ 2); and

• Publication and citation indices for PIRE and comparison participants’ publications, both before and after the start of their project participation (to inform RQ 3 and RQ 4).

A.2 Purposes and Uses of the Data

The primary purpose of the proposed information collection is program evaluation. The data collected will allow NSF to evaluate the performance of the PIRE program in terms of research productivity and quality and the extent to which participants maintain international collaborations after participation. The evaluation will: permit comparison of educational and career outcomes of postdoctoral, graduate student and undergraduate PIRE participants to nationally representative trends; explore associations between international collaboration and research and career outcomes of participants; acquire information on how U.S. institutional policies or practices may foster or hinder international collaborations like those in PIRE and whether institutional changes have occurred due to PIRE; and describe the program for purposes of program improvement. In particular, the results of the evaluation will inform NSF’s development of the next PIRE solicitation (for the 2017 cohort) to be issued in 2016. In addition, outcomes of the program would be reported to the leadership of NSF’s Office of Integrative and International Affairs, Division of Graduate Education, Division of Undergraduate Education, Directorate for Education and Human Resources, and the general public. If the study yields findings of more general interest, the results may be submitted for dissemination at conferences or publication in a peer-reviewed journal.

Although PIs of PIRE projects submit annual and final reports to NSF, these reports do not provide data about the effects of participation on subsequent educational or career outcomes, including any sustained or new collaborations with international partners, the effects of PIRE on institutions’ educational policies and practices, or foreign partners’ perception of the PIRE program’s benefits and suggestions for improvements. Consequently, the study will conduct surveys to answer those research questions that cannot be answered through existing sources alone.

A.3 Use of Information Technology to Reduce Burden

Online surveys will be used to collect information from participants. Web-based surveys have become a commonly used avenue for data collection in recent years given their facilitation of respondents’ data entry across computer platforms. Web-based surveys employ user-friendly features such as automated tabulation and data entry with custom controls such as checkboxes, standard menus, and predefined charts and graphics. In addition, survey skip patterns automatically move the respondent forward to the next appropriate section, simplifying the survey-taking experience. Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents’ attention before they are submitted. Also, given the ease by which data can be automatically uploaded into standard analysis software, there is increased efficiency for researchers seeking to analyze said data. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and thus reduce burden.

A.4 Efforts to Identify Duplication

This study does not duplicate other NSF efforts. There has been no evaluation of the PIRE program.

Data from proposals, annual reports and final reports that PIs have already submitted to NSF will be available to the evaluation contractor and the NSF program staff involved in this study. Surveys will be constructed to ask about elements of the projects that are not captured in NSF’s existing program data and that are not available in bibliometric databases. Items in the surveys will be drawn primarily from existing surveys where possible, including the Survey of Doctoral Recipients (SDR), the National Survey of Recent College Graduates (NSRCG), and surveys cleared by OMB for other NSF program evaluations⁵ (see Appendix H for sources of all survey items). The study team will analyze data from extant large-scale surveys (SDR, NSRCG) for the purposes of benchmarking postdoctoral, graduate and undergraduate participant outcomes.

In addition, the evaluation will use extant bibliometric data on publications and citations to avoid asking survey respondents to provide this information (respondents will be asked to report only works in preparation, under review or in press, as well as other works not indexed in bibliometric indices, such as conference presentations). Thomson Reuters’ Web of Science will provide publication and citation data for PIRE and comparison projects and participants.

A.5 Small Business

No information for this research will be collected from small businesses.

A.6 Consequences of Not Collecting the Information

Since its inception in 2005, the PIRE program has not been evaluated. If this information is not collected, NSF will be unable to document the specific implementation of PIRE projects and assess the impact of PIRE on participants. Without this information collection, NSF will not be able to assess the degree to which the PIRE program is meeting its goals or contributing to the mission of the Foundation.

A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

All data collection will comply with 5 CFR 1320.6.

A.8 Selection of Public Comments and Consultation with People Outside the Agency

Federal Register Announcement

Comments on this data collection effort were solicited in the Federal Register on April 10, 2014 (vol. 79, no. 69, p. 19931). No comments were received. A copy of the notice is included in Appendix N.

Consultations Outside the Agency

Consultation on the evaluation needs, study design, data sources, and analyses occurred during the design phase and will continue throughout the study. The purpose of such consultation is to ensure the technical soundness of the study and the relevance of the findings, and to verify the importance, relevance and accessibility of the information sought in the study.

Consultation was conducted by the research firm Abt Associates, contracted by NSF to design and conduct an evaluation of the PIRE program. Senior technical personnel from Abt Associates who are conducting the study are listed below:

• Alina Martinez, Principal Associate, Abt Associates (Principal Investigator)

• Carter Epstein, Scientist, Abt Associates (Project Director)

• Amanda Parsad, Senior Scientist, Abt Associates

• Laurie Bozzi, Associate, Abt Associates

In addition, an external working group of subject matter experts has provided input on the study design and data collection plan. The study’s group of subject matter experts includes:

• Jonathan Adams, Chief Scientist, Digital Science, Macmillan Publishers Ltd.

• Rajika Bhandari, Institute of International Education

• Susan Cozzens, School of Public Policy, Georgia Institute of Technology

• Irwin Feller, Emeritus, Pennsylvania State University

• Diana Hicks, School of Public Policy, Georgia Institute of Technology

In addition, the proposed data collection instruments were pilot tested with respondents drawn from the target populations. Respondents were asked to comment on the clarity and content of the survey questions, as well as the proposed duration of the data collection to help with accurate estimates of time burden.

A.9 Payments or Gifts to Respondents

No payments or gifts will be provided to respondents.

A.10 Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected will be available to the evaluation contractors and contractors hired to manage data and data collection software, and to NSF staff after the contractor has removed personally identifiable information and taken other appropriate measures to minimize disclosure risk. Data will be processed in accordance to Federal and state privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Project Participant and Project Results Information Base System of Records (63 Fed, Reg. 264, 271-272 January 5, 1998). The system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.

All assurances of confidentiality will be reviewed by the contractor’s Institutional Review Board prior to data collection.

A.11 Questions of a Sensitive Nature

The proposed surveys ask for demographic information (gender, race, ethnicity, citizenship, disability status) from PIRE and comparison project participants on a voluntary basis; respondents may choose not to provide information that they believe is sensitive in nature. This information is being collected so that NSF can understand whether the PIRE program is supporting a diverse population of participants and to control for factors that could affect the outcomes and impacts of PIRE. All instruments will be reviewed by the contractor’s Institutional Review Board prior to fielding.

A.12 Estimates of Response Burden

Exhibit A.3 presents the number of PIRE and comparison project participants that will be included in the target survey samples. For primary data collection, our calculations of sample sizes are based on the average number of personnel who participate in PIRE projects (one PI and four co-PI, two postdoctoral researchers, twelve graduate students, eight undergraduate students, and five senior personnel)⁶ and power analyses conducted in order to ensure valid estimates of program outcomes and impacts. (Details of the sampling plan are provided in Section B.1; Power analyses are provided in Appendix K.)

Exhibit A.3: Number of Survey Respondents
Respondent group	N of Respondents from PIRE projects	N of Respondents from Comparison Projects	Total N	Estimated time to complete survey (mins)
PIs and co-PIs	295	395	690	35
Postdoctoral researcher	118	158	276	25
Graduate students	484	767	1,251	25
Undergraduate students	472	NA	472	25
Foreign senior investigators	295	NA	295	30
Institutional administrators	118	NA	118	15
		Total:	3,102

PIRE and comparison group PIs and co-PIs. We will include the census of PIs (n=59) and co-PIs (n=236) of PIRE projects funded in fiscal years 2005 through 2012 for a total of 295 PIRE PIs and co-PIs. For comparison projects, we estimate that we will identifying one project match for each of 39 PIRE awards (39 comparison projects) and two project matches for each of 20 PIRE awards (40 comparison projects) for a total of 79 comparison projects. We expect each comparison project to have up to five PIs and co-PIs, for a population of 395. We will include the census of comparison PIs.

Other PIRE participants. We plan to include four independent groups of other PIRE participants, classified by their status at the time they participated in PIRE activities:

• Postdoctoral researchers (PIRE and comparison projects);

• Graduate students (PIRE and comparison projects);

• Undergraduate students (PIRE projects); and

• Senior researchers at foreign institutions (PIRE projects).

We will survey the census of PIRE and comparison group postdoctoral researchers (118 from PIRE and 158 from comparison projects). From the population of PIRE and comparison project graduate students we will select samples (484 from PIRE projects and 767 from comparison projects). We will survey the census of PIRE undergraduate students (an estimated total of 472) and the census of 295 PIRE foreign senior investigators. Census estimates are based on estimates that PIRE projects have, on average, one lead PI and four co-PIs, up to eight undergraduates, and up to five senior foreign investigators.

Institutional Administrators at PIRE lead PIs’ institutions. Up to two administrators at appropriate international affairs or study abroad offices at the U.S. institutions of the lead PI for each the 59 PIRE projects will be invited to participate in an online survey, for a total of 118 administrators.

The total number of respondents who will be invited to complete an online survey is 3,102. Details of the power calculations supporting these sample sizes are provided in Part B, Section B.1 and Appendix K.

Estimates of the time needed to complete each survey are shown in Exhibit A.4. These estimates were informed by pilot testing (n= 5 PIs/co-PIs; n=4 postdoctoral participants; n=7 graduate students; n=3 undergraduate students; and n=2 foreign senior investigators; the institutional administrator survey estimate is based on a subset of similar items from the PI survey). The PI/co-PI survey is expected to take an average of 35 minutes to complete. The postdoctoral, graduate and undergraduate student surveys are each expected to take 25 minutes to complete. The foreign senior investigator survey is expected to take 30 minutes to complete and the institutional administrator (i.e., administrator(s) of international affairs offices) survey is expected to take 15 minutes to complete.

Number of Respondents, Frequency of Responses, and Annual Hour Burden

Web-based surveys will be administered once to the 690 PIs and co-PIs (including those from the PIRE and comparison groups); 276 postdoctoral participants (including those from the PIRE and comparison groups); 1,251 graduate student participants (including those from the PIRE and comparison groups); 472 undergraduate student participants; 295 foreign investigators; and 118 institutional administrators (survey instruments are included in Appendices B through G). The total annual response burden for surveys is estimated to be 1,417 hours across the total of 3,102 survey respondents. Exhibit A.4 illustrates the number of average respondents per year to be surveyed for each respondent type and the time demands those surveys will place on respective respondents.

Burden Estimates by Each Form and Aggregate Hour Burdens

The total burden across all forms is estimated at 1,417 hours across 3,102 respondents. The hour burden by form and aggregate burden are included in Exhibit A.6. Data will be collected from individuals using only one form, in one survey wave only; thus the hours by form is the same as the hour estimates for respondent groups.

Exhibit A.4: Estimated Total Burden Hours
Respondent Type	# of Respondents1	Time Per Response (Hours) 2	Total Time Burden (Hours)
Surveys
Principal/co-Principal Investigators	690	0.58	400
Postdoctoral researchers	276	0.42	116
Graduate students	1,251	0.42	525
Undergraduate students	472	0.42	198
Foreign senior investigators	295	0.50	148
Institutional administrators	118	0.25	30
Total	3,102		1,417
1 The number of respondents for the PI/co-PI, postdoctoral and graduate student surveys include respondents from the PIRE and comparison groups. 2 The exact number of items per respondent for each survey depends on whether or not they participated in any international travel for research or educational purposes, whether or not the PIRE or comparison project period is still active, and their status as a current or former participant in the project; estimated burden takes this variation into account.

Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost is $40,914. Exhibit A.5 shows the estimated total annual costs to each group of respondents.

A.13 Estimates of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden regarding capital, operation, or maintenance costs to respondents as a result of this study. Respondent burden is confined to time spent responding to the online survey.

A.14 Estimates of Cost to the Federal Government

The estimated cost to the Federal Government for the data collection activities included in this request for approval is $402,465. This cost estimate includes instrument development and pretesting and recruitment and data collection; it does not include the cost of obtaining and processing extant data for which approval is not sought.

Exhibit A.5: Estimated Cost to Respondents
Respondent Type	Hourly Salary Estimate1	Burden Time Per Respondent	Estimated Cost to Respondent	Number of Respondents in Category	Estimated Annual Cost Across All Respondents
Surveys
Principal/co-Principal investigators	$ 39.82	0.58	$ 23.10	690	$ 15,939
Postdoctoral researchers	22.01	0.42	9.24	276	2,550
Graduate students2	21.79	0.42	9.15	1,251	11,447
Undergraduate students2	18.64	0.42	7.83	472	3,696
Foreign senior investigators	39.82	0.50	19.91	295	5,873
Institutional administrators	47.77	0.25	11.94	118	1,409
Total					$40,914
Notes: 1 Salary estimates are based on education and profession figures provided by the U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), "Salaries, Tenure, and Fringe Benefits of Full-Time Instructional Faculty Survey" (IPEDS-SA:99), and Winter 2005-06 through Winter 2011-12, Human Resources component, Salaries section. (This table was prepared July 2012); National Science Foundation, National Center for Science and Engineering Statistics, Survey of Doctorate Recipients (2010); U.S. Department of Commerce, Census Bureau, Current Population Survey (CPS), March 1996 through March 2012. (This table was prepared November 2012); and Bureau of Labor Statistics, Occupational Employment and Wages, May 2012 (http://www.bls.gov/oes/current/oes119033.htm, accessed March 29, 2014). 2 The salary for former graduate or undergraduate students who have received a Bachelor’s or post-graduate degree will be higher than those who are still seeking degrees, however, we won’t know the number of the former until analysis is underway. Thus, this amount reflects an average of the two populations: those who have ‘some college’ and those who have received their respective Bachelor’s or post-graduate degree.

A.15 Changes in Burden

This is a new collection of information.

A.16 Plans for Publication, Analysis, and Schedule

The purpose of this study is to inform NSF about the quantity and quality of research produced by the PIRE program; about the extent to which PIRE is fostering sustained international collaborations, at both the individual and institutional levels; about how educational and career outcomes for participants in PIRE compare to those of participants in other NSF-funded projects and to national trends for science and engineering degree-holders; whether or not PIRE is contributing to research needed to address transnational or global societal challenges; and about potential areas for program improvement, including how well PIRE facilitates productive research collaborations with foreign scientists and engineers.

Descriptive analyses will be conducted of data from extant NSF administrative records, bibliometric data compilation, and surveys. For survey items using continuous scales, analyses will calculate means and standard deviations to describe both central trend and variation. Frequency distributions and percentages will be used to summarize answers given on categorical or ordinal items. In addition, cross-tabulations will be used to illustrate differences in responses between groups or the distribution of responses across subgroups of interest. Selected educational and employment outcomes for postdoctoral, graduate and undergraduate student participants in PIRE will be compared to national trends using means and standard deviations; t-tests for statistical significance of any differences observed will be conducted at an alpha-level of .05. Descriptive summaries of emergent themes from responses to open-ended items will be prepared and reported, as well. These items will help NSF understand how institutional practices and policies support PIRE and contributions of PIRE to societal challenges that cannot be captured in closed-ended items.

To examine how research outcomes for PIRE at the project level differ from research outcomes observed when project funding is not explicitly directed at supporting international collaboration, we will use regression models with each outcome as the dependent variable and independent variables including a dichotomous PIRE indicator (1=PIRE project, 0=comparison project) covariates to control for differences in project level characteristics (e.g., award cohort, duration, funding amount), and variables that indicate matched pairs or matched groups (if we use many-to-one matching). Results will be presented as tables showing the coefficients on the PIRE indicator, as well as any coefficients on covariates. The coefficient on the PIRE treatment indicator represents the unique relationship of PIRE to differences in number of publications (or another outcome) controlling for other factors included in the model. We will also present descriptive tables that show means and percentages for the outcomes overall, for field of discipline, and for other major subgroups of interest. Similar methods will be used to examine how the educational and career outcomes for PIRE participants differ from educational and career outcomes when project funding is not explicitly directed at supporting international collaboration.

Comparative interrupted time series (CITS) models will be used to compare differences in differences for pre- and post-participation research outcomes for PIRE participants relative to non-PIRE participants. These models will include appropriate indicators for onset of participation and membership in the PIRE or comparison group, along with controls for variables that could affect the outcomes (e.g., time elapsed since onset of participation, field of research, demographics, etc.). These analyses will report the mean pre-post differences in number of publications, proportion of publications with a foreign co-author, and average citation index for the PIRE participant group (e.g., PIRE postdoctoral researchers), the corresponding comparison participant group, the regression-adjusted difference, the standard error of this difference, the statistical significance of any observed difference and the effect size (using Hedges’s g). Details of these analyses are included in Supporting Statement Section B and Appendix J.

An evaluation report will be prepared based on these analyses. This report will describe the research outcomes of PIRE and how these outcomes compare to those in a matched group of NSF-funded projects that do not require international collaborations; the research and career outcomes of individual participants in PIRE and how these compare either to national trends (for selected career outcomes) or to outcomes for participants in the comparison group; the duration and types of international research and educational opportunities in which PIRE participants engage; the subsequent international collaborations of former PIRE participants; the experiences of foreign senior investigators collaborating with U.S. scientists and engineers (both established and in training) in PIRE; and how PIRE contributes both to institutional practices supporting international engagement of U.S. researchers and students and to the body of knowledge needed to address global-scale challenges in information technology and energy and environmental sustainability. The full evaluation report will be preceded by an executive summary of findings and will include a background discussion of the context of the study, including program goals and relevant issues in the globalization of research in STEM fields, explanations of the data collection and analytic methods, and a discussion of the limitations of the study. Results will be discussed within the study and program context. Appendices will include technical information and data collection instrumentation.

In addition to the full evaluation report and its executive summary, a separate dissemination report will be prepared with the intent of making the key evaluation findings accessible to a broad audience. This brief report will describe the quality of research produced by PIRE using established bibliometric indices that are standardized by research field and national or world aggregated data; associations between participation in a program where international collaboration was required versus programs where it was not required, with key academic and career outcomes of participants; differences in publication quantity and quality for participants in PIRE relative to participants in the comparison group; and the extent to which researchers in PIRE sustain international collaborations (relative to researchers in the comparison group and relative the extent to which collaborations across institutions within the U.S. are sustained).

Finally, the evaluation will produce a database combining extant and survey data collected for the evaluation with all personal identifying information removed (each respondent’s name will be replaced with a unique, random study ID and each project will have a unique, random project ID). In written reports, pseudonyms will replace actual names of individuals and institutions. Disclosure risk analysis will also be performed to further mitigate the risk of inferential disclosure of individuals.

NSF will make publicly available the executive summary, final report, and other documents describing the findings of the evaluation of the PIRE program after review and clearance by NSF. The contractor conducting the program evaluation on behalf of NSF will submit the report and PIRE database for review by the program, the Contracting Officer’s Representative, and other NSF officials for clearance. NSF will review the quality of the reporting, data analyses, findings, and descriptions of the limitations of the study; ensure that there are no errors in the description of the PIRE program, the 59 awarded projects and their participants, nor in the descriptions of the comparison projects included in the evaluation and their participants; and assess compliance with privacy laws, regulations, and policies. Once the review is completed, NSF will make public the executive summary, the final report, and other documents describing the findings through a variety of means that are tailored to the information needs of the public and the program stakeholders. In addition, on a case-by-case basis, NSF could support dissemination of information through conferences and publication avenues by the contractor. Exhibit A.6 outlines the study timeline.

Exhibit A.6: Project Timeline
Activity	Timeframe
Extant data extraction, NSF Program Data	April-May, 2014
Bibliometric data extraction	July-November, 2014
Construct study samples	May-July 2014
Update contact information of respondents	August 2014
Field surveys	2 months after OMB approval
Analyze data	4-8 months after OMB approval
Prepare reports of findings	9-13 months after OMB approval
Present findings at selected conferences	13-15 months after OMB approval

A.17 Approval to Not Display Expiration Date

The data collection instruments will display the expiration date.

A.18 Exceptions to Items 19 of OMB Form 83-I

No exceptions are sought.

References

Adams, J. 2013. The fourth age of research. Nature, 497, p. 557-560.

Carney, J., Smith, W.C., Parsad, A., Johnston, K. and Millsap, M. (2008) Evaluation of the Faculty Early Career Development (CAREER) Program. Report prepared for the National Science Foundation under contract GS-10F-0086K, Order No. D050540.

Flattau, P.E., Lal, B., Laskey, A., & Ford, J. J. 2009. Portfolio Evaluation of the National Science Foundation's Grants Program on "International Research and Education in Engineering" (IREE). Washington, DC: Institute for Defense Analyses, Science & Technology Policy Institute.

Gamse, B., Smith, W.C., Parsad, A., Dreier, J., Neishi, K., Carney, J., Caswell, L., Breaux, E., McCall, T. and Spader, J. (2010). Evaluation of the National Science Foundation's GK-12 Program: Final Report, Volume I: Technical Report. Prepared under contract GS10F-0086K, Order # NSFDACS06D1412.

Goodnow, J. 2008. The Benefits of Cross-Cultural Collaboration. Testimony in International Collaborations in Behavioral and Social Sciences Research: Report of a Workshop (pp. 47-63). Washington, D.C.: National Academies. Retrieved from www.nap.edu/openbook.php?record_id=12053&page=47

Luo, J., Flynn, J. M., Solnick, R. E., Ecklund, E. H., & Matthews, K. R. W. (2011). International stem cell collaboration: How disparate policies between the United States and the United Kingdom impact research. PLoS ONE, 6(3), e17684. doi:10.1371/journal.pone.0017684

Martinez, A., Neishi, K., Parsad, A., Whittaker, K. and Epstein, C (2012). Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report. Prepared for the National Science Foundation under Contract GS-10F-0086K, Order No. NSFDAC S09T1516.

Martinez, A., Epstein, C., Parsad, A. and Whittaker, K. (2012). Evaluation of NSF’s International Research Fellowship Program: Final Report. Prepared for the National Science Foundation under Contract GS-10F-0086K, Order No. NSFDAC S09T1516

National Academy of Sciences (2014). Partnerships for Enhanced Engagement in Research (PEER) Science. See: http://sites.nationalacademies.org/pga/dsc/peerscience/index.htm

National Center for Education Statistics. Retrieved April, 30, 2008 from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007154

National Research Council, Committee on Global Science Policy and Science Diplomacy. 2011. U.S. and International Perspectives on Global Science Policy and Science Diplomacy: Report of a Workshop. Washington, DC: National Academies Press.

National Science Board. 2008. International Science and Engineering Partnerships: A Priority for U.S. Foreign Policy and Our Nation’s Innovation Enterprise. NSB-08-4. Arlington, VA: National Science Foundation.

National Science Board. 2010. Globalization of science and engineering research: A companion to science and engineering indicators 2010. NBS-10-3. Arlington, VA: National Science Foundation.

National Science Foundation (2010). Science, Engineering, and Education for Sustainability (SEES) home page. Retrieved February 27, 2014 from http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504707

National Science Foundation (2012). FY 2013 Budget Request to Congress Retrieved March 27, 2014 from http://www.nsf.gov/about/budget/fy2013/pdf/12-OISE_fy2013.pdf

National Science Foundation (2011). National Science Foundation: FY 2012 Budget Request to Congress. Retrieved 5/1/11 from http://www.nsf.gov/about/budget/fy2012/pdf/fy2012_rollup.pdf

Wine, J., Janson, N., and Wheeless, S. (2011). 2004/09 Beginning Postsecondary Students Longitudinal Study (BPS:04/09) Full-scale Methodology Report (NCES 2012-246). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC. Retrieved June 20, 2012 from http://nces.ed.gov/pubs2012/2012246_1.pdf

Appendices

1. PIRE Program Logic Model

2. Principal Investigator Survey

3. Postdoctoral Survey

4. Graduate Student Survey

5. Undergraduate Student Survey

6. Foreign Senior Investigator Survey

7. Institutional Administrator Survey

8. Survey Supplementary Materials:

1. Research Fields for surveys

2. List of Countries for surveys

3. Survey Items: Mapping to Research Questions and Sources

9. Approach to Matching

10. Estimation of Impacts

11. Power Calculations

12. Non-response bias analysis

13. Recruitment Materials

14. 60-day Federal Register Notice

1 NSF uses “Principal Investigator” and “co-Principal Investigator” to refer to U.S.-based award recipients; these terms do not apply to foreign personnel who may play a role on a particular project commensurate to that of a PI.

2 Participants will be matched from within each PIRE-comparison project matched pair.

3 NSF awards are typically made from programs within divisions that sit within directorates; for programs like PIRE that can cut across programs, divisions or directorates, NSF has, a priori, identified which directorates each of the 59 PIRE awards corresponds to – this was done partly to facilitate identification of proposal reviewers with the necessary expertise

4 If SDR 2012 data become available, the evaluation will make use of the most recently available wave.

5 These include: Recipient Survey for the Evaluation of NSF’s (S-STEM) program; the Applicant and Foreign Host Surveys for the Evaluations of NSF’s IRFP and East Asia and Pacific Summer Institutes (EAPSI) programs; the Graduate Student Survey for the Evaluation of NSF’s GK-12 Program; surveys approved for the Evaluation of NSF’s Robert Noyce Teacher Scholarship program; and surveys approved for the Evaluation of NSF’s Faculty Early Career Development (CAREER) program.

6 Estimates from NSF, June 19, 2013 (RFQ # DACS13Q2124, Response to Contractor Questions.

3