EAPSI Supporting Statement revision (part of the 10/12 response)

Supporting Statement for Paperwork Reduction Submission

Evaluation of National Science Foundation’s East Asia and
Pacific Summer Institutes and International Research
Fellowship Program
Section A
Introduction

The National Science Foundation requests that the Office of Management and Budget
(OMB) approve, under the Paperwork Reduction Act of 1995, a three year clearance to
conduct data collection efforts for an outcome evaluation of the National Science
Foundation’s East Asia and Pacific Summer Institutes (EAPSI) and International Research
Fellowship (IRFP) Program.
These two programs offer early career researchers an opportunity to forge collaborative
relationships with foreign scientists and engineers, albeit through different
interventions. Launched in 1999, EAPSI provides $5,000 of support to US graduate
students to spend the summer (two months) conducting research in seven countries in
East Asia and the Pacific region. The program is designed to immerse US scholars into
the scientific and social culture of the host location. IRFP, established in 1992, provides
support to post-graduate scientists (generally a year or two after the receipt of a
doctoral degree), for a research experience abroad lasting from 9 to 24 months, with no
restriction on geographical area. Awards range from $57,000 to $200,000, depending
on the location, cost and duration of the project, and the applicants’ family status.
To assess the program effectiveness, NSF plans to collect data that are designed to
explore the fellowship experiences and educational and career outcomes of EAPSI and
IRFP fellows as well as the influence of the programs on host scientists and their
institutions and on US scientists and their institutions. The primary methods of data
collection will include analyses of NSF program records and surveys of fellows, unfunded
applicants, US advisors of fellows, and foreign hosts. There are a bounded (or limited)
number of respondents within the general public who will be affected by this research.
A.1. Circumstances Requiring the Collection of Data

Launched in the early 1990s, the IRFP program has never been evaluated. The EAPSI
program precursor, the Japan program, was last evaluated by an outside contractor in
2002. The evaluation was focused on program participants for years 1992 to 1998. The
NSF Office of International Science and Engineering administering the program wishes
to collect data on the performance of each program, to collect evidence about the merit
of continuing/expanding current funding continued expenditures and to revise the
programs should it prove necessary.

New data collection is required for a systematic assessment of the programs. While the
extant records, such as participant progress reports and application data, provide some
information to describe the applicant and participant population, these data are
inadequate to document participants’ fellowship experiences as well as educational or
career trajectories. In addition, the evaluation study seeks to describe the effects of the
programs on the US institutions as well as on the host institutions. Extant documents
do not contain information related to these and program outcomes. To supplement and
enhance extant program data, census surveys of fellows, unfunded applicants, US
advisors, and foreign hosts will be conducted.
A.2. Purposes and Uses of the Data

The primary purpose for collection of this information is program evaluation. The
National Science Board highlighted the importance of international research and
education in its call for increased NSF involvement in promoting international research
collaboration. The IRFP and the EAPSI programs reflect NSF’s commitment to support
the active engagement of students and junior researchers in international research
experiences. Thus, given the confluence of heightened interest in and need for
international collaboration, NSF’s Office of International Science and Engineering’s
decision to commission an evaluation of these two programs is timely.
The evaluation study will focus on three programmatic areas:
1. EAPSI and IRFP fellows’ experiences applying for, and participating in, each
program;
2. Educational and career outcomes of EAPSI and IRFP fellows and how they
compare to those of unfunded applicants and to national doctoral degree
recipients; and
3. The effects of the programs on host scientists and their institutions, as well as on
US scientists and their institutions.
Data collected in the study would be used by program staff to help them manage the
programs. In addition, outcomes of the program would be reported to the Office of
International Science and Engineering leadership and possibly to other stakeholders
outside of NSF, as appropriate. If the program yields findings of more general interest,
the results may be submitted for a publication in a peer-reviewed journal.
A.3. Use of Information Technology to Reduce Burden

In order to reduce respondent burden, internet-based surveys will be used to collect
information from participants. As the populations being surveyed in this study are
highly educated scientists, engineers, and other technical professionals, they are
expected to have easy access to and be fluent in the use of web-based technologies.

The use of web-based systems facilitates accuracy, completeness, and speed of data
entry, also reduces respondent burden. Web-based surveys employ user-friendly
features, such as automated tabulation, data entry with custom controls such as
checkboxes, data verification with error messages for easy online correction, standard
menus, and predefined charts and graphics. In addition, survey skip patterns
automatically move the respondent forward into the next appropriate section, reducing
time burden on respondents and simplifying the survey-taking experience. This
approach also allows for easy identification of non-respondents and facilitates followup.
Furthermore, data entered by participants can be automatically uploaded into standard
analysis software, eliminating an additional data entry step, thus increasing the
efficiency of the researcher(s) conducting the study.
Finally, email will be used to send respondents their invitations to complete the survey
and to follow-up with the non-respondents to ensure their participation.
A.4. Efforts to Identify Duplication

This evaluation does not duplicate other NSF efforts. The Japan program evaluation
(EAPSI precursor) was limited to the program participants from 1992 to 1998. This
proposed data collection is not duplicative, as it focuses on individuals who participated
in the program starting the following year, 1999. No evaluation of the IRFP program –
established in 1992 – has been conducted.
A.5. Small Business

No small businesses will be involved in this study.
A.6. Consequences of Not Collecting the Information

Failure to collect the information proposed in this request would prevent NSF from
assessing the role IRFP and EAPSI programs play in promoting international
collaborations in STEM research and education, which the National Science Board
identified as one of the top priorities for the agency. In addition, data collected in the
evaluation would allow IRFP and EAPSI program managers to take corrective actions
should the evidence emerge that these are necessary. Finally, the evaluation would
provide systematic outcome data on the programs, which could be reported to OMB,
Congress, and other stakeholders outside NSF.
A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The project will fully comply with the guidelines of 5 CFR 1320.5. No special
circumstances apply to this data collection.
A.8. Consultation Outside the Agency

Comments on this data collection efforts were solicited in the Federal Register on March
12, 2010 (vol. 75, no. 48, p.11941). During the first comment period, one comment was
received from the Tea Party of New Jersey. The comment did not address this data

collection for which this request seeks approval, nor did the comments have bearing on
the collection, and therefore, NSF is proceeding with seeking approval from OMB.
Consultation on the study design was conducted by the research firm, Abt Associates
Inc., contracted by NSF to design and conduct the evaluations of the EAPSI and IRFP
evaluations.
To provide input on all aspects of the EAPSI and IRFP evaluation design, an Advisory
Panel of five experts was convened to guide the study. The Panel members have
provided input on the study design, and participated in an in-person meeting to discuss
the study design. Advisory Panel members include: Christopher Hill, George Mason
University; Susan Cozzens, Georgia Institute of Technology; Irwin Feller, Pennsylvania
State University; Terrence Russell, Association for Institutional Research; and Nicholas
Vonortas, George Washington University. The Advisory Panel will continue to consult
throughout the duration of the evaluation.
A.9. Payments or Gifts to Respondents

No payment or gift will be provided to respondents.
A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be
maintained in accordance with the Privacy Act of 1974. Data collected will be available
to the evaluation contractors, contractors hired to manage data and data collection
software, and at the aggregate level to NSF staff. Data will be processed in accordance
to Federal and State privacy statutes. Detailed procedures for making information
available to various categories of users are specified in the Education and Training
System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The system limits access to
personally identifiable information to authorized users. Data submitted will be used in
accordance with criteria established by NSF for monitoring research and education
grants, and in response to Public Law 99-383 and 42 USC 1885c. The information
requested may be disclosed to qualified contractors in order to coordinate programs
and to a Federal agency, court or party in court, or Federal administrative proceeding, if
the government is a party.
Individuals surveyed will be assured that the information they provide will not be
released in any form that identifies them, and that their responses will be kept
confidential to the extent provided by law. The contractor will be expected to maintain
the confidentiality, security, and integrity of the survey data. The web-based survey
data and notes will be maintained on a secure server with appropriate levels of
password and other types of protection. Proposed procedures for protecting the data
and privacy of respondents will be reviewed by the contractor’s Institutional Review
Board prior to data collection.

A.11. Questions of a Sensitive Nature

The proposed surveys ask for demographic information (gender, race/ethnicity,
citizenship status, and income) from participants on a voluntary basis, thus respondents
may choose not to provide information that they feel is sensitive in nature. All survey
questions will be reviewed by the contractor’s Institutional Review Board prior to
fielding. Copies of the surveys can be found in Appendix A.
A.12 Estimates of Response Burden

The total number of respondents targeted for this study is estimated at 6,744, which
represents the universe of applicants (fellows and unfunded applicants) to the programs
in the years 1992-2009 for the IRFP program and the years 1999-2009 for the EAPSI
program, as well as the foreign hosts of fellows in both programs, EAPSI foreign location
staff, and the US advisors of fellows in the EAPSI program. Assuming a 75% response
rate for each survey group, the total number of respondents is estimated to be 6,266,
resulting in an estimated response burden for the surveys of 1,982.5 hours over one
year. Details on these calculations are provided in A.12.1 and A.12.2.
A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

Table A.12.1 below indicates the number of respondents and expected number of
responses for each category of respondent type and the time demand these surveys will
place on each individual respondent and on all respondents in aggregate.
Table A.12.1
Respondent Type

Targeted
group

Number of
respondents#

Time per
response (hours)

Total time
burden
(hours)
EAPSI Fellows
1,300
975
0.5
487.5
EAPSI Unfunded Applicants
810
608
0.5
303.8
EAPSI US Advisors
1,047
785
0.25
196.3
EAPSI Foreign Hosts
1,300
975
0.25
243.8
EAPSI Foreign Location Staff
20
20
0.5
10.0
IRFP Fellows
581
436
0.5
217.9
IRFP Unfunded Applicants
1,105
829
0.5
414.4
IRFP Foreign Hosts
581
436
0.25
108.9
Total
6,744
5,063
N/A
1,982.5
#
The above estimates for the number of responses for each type of respondent assume a 75% response
rate which is comparable to response rates we have achieved on other studies of a similar scope
respondent type.

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

As each respondent will complete the survey once, the annual burden and the
aggregate burden will be the same as shown in Table A.12.1.

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost to respondents is $73,859.50. The following chart shows the
estimated total annual costs to each group of respondents over one year for the
surveys.
Table A.12.3
Respondent
Type

Targeted
Group

Number of
Respondents*

Time Per
Response
(hours)

Total
Time
Burden
(hours)
487.5

Hourly
salary
estimate

Estimated
cost per
respondent

Estimated
overall
cost

35
17.5
17,062.50
EAPSI Fellows
1,300
975
0.5
35
17.5
10,631.25
EAPSI
Unfunded
Applicants
810
608
0.5
303.8
43
10.75
8,441.44
EAPSI US
Advisors
1,047
785
0.25
196.3
43
10.75
10,481.25
EAPSI Foreign
Hosts
1,300
975
0.25
243.8
43
21.5
430.00
EAPSI
Location Staff
20
20
0.5
10.0
35
17.5
7,625.63
IRFP Fellows
581
436
0.5
217.9
35
17.5
14,503.13
IRFP
Unfunded
1,105
829
0.5
414.4
Applicants
43
10.75
4,684.31
IRFP Foreign
581
436
0.25
108.9
Hosts
73,859.50
Total
6,744
5,063
1982.5
#
Assumes a 75% response rate.
Figures are rounded to the nearest whole dollar. Based on an average salary estimates for PhDs in Science and
Engineering as reported in National Science Foundation's, Science and Engineering Indicators - 2006. National
Science Foundation, Division of Science Resources Statistics. Arlington, VA (NSB 06-01) [February 2006], Figure 322.

Url: http://www.nsf.gov/statistics/seind06/figures.htm
(Used estimates for faculty with 5-9 years of experience @$70,000 and 15-19 years of experience @ $85,000)

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to
Respondents or Record Keepers

There is no overall annual cost burden regarding capita, operation, or maintenance
costs to respondents that results from this study, other than the time spent responding
to the survey.
A.14. Estimates of Costs to the Federal Government

The estimated cost to the Federal Government for the data collection activities included
in this request for approval is $711,385. This cost estimate includes instrument
development and pretesting; location and recruitment; data collection; and data
processing.

A.15. Changes in Burden

This is a new collection of information.
A.16. Plans for Publication, Analysis, and Schedule

To provide the NSF with an understanding of the outcomes of the IRFP and EAPSI
programs, the contractor will prepare two reports for NSF—one for each program—that
describe the study and findings. The reports will characterize the research and
educational activities supported by the initiatives, as well as outcomes in the careers of
fellows and unfunded applicants.
This comparative evaluation will help NSF respond to questions about the impact of the
EAPSI and IRFP programs on participants’ research collaborations and activities. By
surveying the awardees of the program and their comparisons as well as the foreign
hosts, we can learn more about the perceived benefits of participation in these
programs.
Analyses of survey data will include a detailed summary that utilizes appropriate
descriptive statistics. For survey items using continuous scales, the study will calculate
means and standard deviations to describe both central trend and variation. Frequency
distributions and percentages will be used to summarize answers given on ordinal
scales. Comparative analyses between the fellows and unfunded applicants will be
conducted using propensity score matching to adjust for potential selection bias.
The project schedule is shown in Table A.16. Surveys are planned to begin in October
2010.
Table A.16
Activity
Update contact information of applicants
Field surveys
Analyze data
Prepare report of findings – EAPSI program
Prepare report of findings – IRFP program

Timeframe
1-3 months after OMB approval
4-12 months after OMB approval
12-18 months after OMB approval
19-27 months after OMB approval
19-27 months after OMB approval

A.17. Approval to Not Display Expiration Date

The data collection instruments will display the expiration date.
A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions are sought.

Section B
Statistical Methods
B.1. Respondent Universe and Sampling Methods

The universe of respondents for which the clearance is sought includes eight groups: (1)
EAPSI fellows (n=1,300); (2) IRFP fellows (n=581); (3) EAPSI foreign hosts (up to 1,300);
(4) IRFP foreign hosts (up to 581); (5) EAPSI US advisors (1,047); (6) EAPSI unfunded
applicants (n=810); (7) IRFP unfunded applicants (n=1,105); (8) key staff at EAPSI foreign
location (n=20). Note, the number of foreign hosts and advisors may be lower than the
total number of fellows if foreign hosts or advisors worked with more than one fellow.
We propose to survey the universe of each of these respondent groups.
EAPSI and IRFP Fellows. The fellows sample includes participants from cohorts that span
the program years 1999-2009 for EAPSI and 1992-2009 for IRFP.
EAPSI and IRFP hosts. The hosts sample includes participants from cohorts that span the
program years of 1999-2009 for EAPSI and 1992-2009 for IRFP.
EAPSI US advisors. The universe of US advisors includes participants from cohorts that
span the program years of 1999-2009 for EAPSI. IRFP fellows’ advisors are not included
in the survey as in contrast to EAPSI fellows, IRFP fellows have earned their degrees and
therefore do not have an advisor (doctoral degree is a condition for participation).
Unfunded applicants – comparison groups. The evaluation design incorporates the use
of unfunded applicants as a comparison group for each program. These individuals who
had applied to the EAPSI and IRFP programs, but were ultimately not awarded the
fellowships (referred to as “unfunded applicants”). The key advantage of using this
group as a comparison is in the similarity of interest and motivation to engage in
international collaboration, and intent to conduct research in foreign countries between
the applicants and the fellows.
EAPSI foreign location key staff. Interviews will be conducted with key contact staff at
each of the EAPSI foreign locations. These are representatives of the foreign partner
organizations who are familiar with, and help administer, the EAPSI fellows in their
countries.
We anticipate a response rate of at least 75 percent from the respondent group based
on previous surveys conducted with students and early career scientists of NSFsponsored programs. Response rates are projected based on similar surveys conducted
with samples of graduate students and early career researchers who participated in NSF
programs. Table B.1 illustrates the response rates for various evaluation studies of NSF

programs that surveyed graduate students and early career individuals, which were
used to estimate the expected response rates for this project.
Table B.1
Program

CAREER Fellows
IGERT Former Students
GK-12 Fellows
GK-12 Fellows

Response Rate

84%
74%
MS 45%
PhD 57%
MS 83%
PhD 92%

Length of Time Between
Participation and Data
Collection
0-10 years
0-10 years
5-10 years
0-5 years

B.2. Information Collection Procedures/Limitations of the Study

The following steps will be taken to collect survey data on various populations described
in the previous section.
Step 1: mining NSF data. NSF program data consists of e-Jackets (for years from 2001 to
2009), paper applications (for years 2000 and prior), and internal program files. Internal
program files contain information useful in locating respondents (including names,
discipline, institution at the time of application, address and phone number at time of
application).
Step 2: locating respondents. Once NSF data are organized into a central database, the
following steps for locating respondents will be taken:
1. Use names and other available information from the NSF records to conduct
Google and other web-based searches.
 Obtain contact information from individuals’ own web pages (as
academic researchers and many graduate students have research
team/lab or home page websites) and verify
 Obtain contact information from posted articles, presentations, and
other materials.
2. Contact PhD advisors to request information about their students’ whereabouts.
This approach would probably be most effective for relatively recent applicants
(within the past 5 years).
 If the name of the advisor is missing, use ProQuest to locate the fellows
dissertation abstract and document the name if the advisor (only for
doctoral thesis).
3. For difficult cases for whom email addresses could not be found based on the
searches in 1-2 above, use the following procedure:
 Use names and NSF program contact information to search AccurInt
(database linked to LexisNexis) to verify or update the addresses and
phone numbers in the NSF records.



Mail an invitation to participate in the survey (with information on how
to access the survey web site) to the latest known address. Use the US
mail Forwarding Service to obtain information on the change of address.
The invitation will be mailed up to three times using the forwarding
address provided by the US postal service. Also, dial the phone numbers
identified in AccurInt. Once individuals have been contacted, verify that
the individuals are indeed the individuals being sought by the study, and
then invite them to participate in the survey and provide instructions on
how to access the web site.

Step 3: Web survey. Once approval is obtained from OMB, we will program the surveys
for online data collection. The study team will test each survey system to ensure
functionality and accuracy of data capture; survey data collection is scheduled to begin
in fall 2010.
All subjects will be sent an invitation email by NSF, introducing the study and the
contractor conducting the study (Abt Associates). Abt will follow up with another email,
containing a link to the survey, username, and password. Three email reminders and
three telephone reminders will be used to boost response rates. The survey will be
open for two months during the academic year. Throughout the data collection cycle, a
toll-free number and e-mail address will be available to ensure that potential
respondents can easily and quickly obtain answers to questions or concerns.
Estimation Procedure

The purpose of this proposed activity is to collect data from participants and unfunded
applicants of the EAPSI and IRFP programs in an effort to measure the initial and
potential long-term impact of these programs. Analysis will include a descriptive
reporting using the measures of central tendency and frequency distributions. Data
from awardees and those who applied for but did not receive the fellowship awards of
the EAPSI and IRFP programs will be compared using propensity score matching.
Propensity score matching that will allow a comparison of the fellows (treatment group)
to unfunded applicants (comparison group) selected based on their similarity to the
awarded applicants. With this approach, the fellows would be compared to unfunded
applicants who are as similar as possible to them in terms of observable characteristics,
allowing us to determine what the fellows’ outcomes would have been had they not
received the IRFP or the EAPSI award, had other characteristics been equivalent. The
PSM models are a way of matching members of different groups based on a range of
characteristics that will allow more accurate estimates of the program effects. Appendix
B contains additional details about the PSM approach.
B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Method to maximize response rate are described in detail in section B.2. Briefly, these
will include the following procedures:
1. Extensive location techniques to identify correct email address

2.
3.
4.
5.
6.
7.

Web format of the survey
Minimization of spam filtering
Invitation from NSF to participate in the study
Skip patterns, to reduce burden on respondents
Extensive email and telephone follow-up
Availability of a toll-free number and email address for questions.

We will examine the bias in estimates because of nonresponse by following the three
steps described below. Based on the analysis we will adjust the weights of responding
students to account for student nonresponse.
1. Examination of Response Rates. The first step will be to monitor the overall response
rate, as well as by year and by relevant subgroups (e.g., by discipline, or by gender and
race/ethnicity). High response rates (over 80 percent) for the entire sample as well as
for subgroups might indicate no need for further analysis of bias due to nonresponse.
Large differences in the response rates by strata and for subgroups serve as indicators
that potential biases may exist. For example, if response rate from an important
subgroup is very low then any difference in the characteristic of interest between this
subgroup and other subgroups would result in a bias in the estimates. From the survey
results we will examine whether there are differences in the characteristics in the
subgroups, especially in a stratum where the response rate is low.
2. Comparison of estimates based on respondents to estimates from external sources.
For questions where there are data available from an external source for some
characteristic of interest (e.g disciplines, proportion n tenure track positions), we will
compare the estimates from our survey responses to those from nationally available
data. A large difference may indicate bias in the survey estimates assuming that the
external source provides an unbiased estimate.
3. Nonresponse Propensity Model. Finally, should the response rate fall below 80
percent we will construct a propensity model to estimate the probability of a student in
responding to the survey both for responding and nonresponding students; this is called
a propensity score. The estimated propensity scores come from a logistic regression
model. The model will be based on variables which are available both for nonresponding
and responding students. Students will be grouped using the estimated propensity
scores. Within each group we will compare the frame characteristics of responding and
nonresponding students. This grouping in addition to assessing the bias will also provide
a method of forming weighting classes for adjusting the weights of responding students
to reduce the bias due to nonresponse.

B.4. Tests of Procedures or Methods

Experts in the field are part of the Advisory Panel and they have reviewed the study
design and data collection instruments. The survey was pilot-tested with former
graduate students, who were asked to comment on the clarity and content on the
questions and to record the time required to complete the survey. Minor revisions,
including shortening the length of the survey, resulted from this feedback. (The median
time to completion was 32 minutes. The survey was shortened by removing individual
items and sections in order to reduce the respondent burden to 30 minutes.)
Once the survey instruments are programmed, they will be tested online by Abt
researchers familiar with the project.
B.5. Names and Telephone Numbers of Individuals Consulted

Key personnel who have been involved in the statistical aspects and who will be
involved in collecting and analyzing data are presented in the table below. The
contractor for collection and analysis of data in this study is Abt Associates Inc.,
Cambridge, MA. Staff have knowledge of statistical methods, experience in evaluation
of research programs, and expertise in scientific research were involved in the design.
Members of the Advisory Panel were also consulted in the design, and who may also be
consulted in the analysis of data. Finally, NSF program staff members familiar with the
programs have been included in the design of the evaluation.
Table B.5 Individuals Consulted
Name
Abt Associates Inc.
Alina Martinez
W. Carter Epstein
Fatih Unlu
K.P. Srinath
Luba Katz
National Science Foundation
John Tsapogas
Jong-on Hahm
Susan Parris
Edward Murdy
Advisory Panel
Irwin Feller
Susan Cozzens

Role
Project Director, Senior Associate
Associate
Economist, Scientist
Statistician, Survey Sampling and
Methodology
Associate
Program Coordinator, Office of
International Science and Engineering
EAPSI Program Manager, Office of
International Science and Engineering
IRFP Program Manager, Office of
International Science and Engineering
Senior Program Manager, Office of
International Science and Engineering
Professor Emeritus of Economics, Penn
State
Professor of Public Policy and Director of its
Technology Policy and Assessment Center,
Georgia Institute of Technology

Phone
617-349-2312
617-349-2543
301-634-1836
617-349-2313
703-292-7799
703-292-7223
703-292-7225
703-292-8711

814-865-0691
404-385-0397

Terrence Russell
Christopher Hill
Nicholas Vonortas

Principal, Terrence Russell LLC, Executive
Director Emeritus, the Association for
Institutional Research
Director, Doctoral Program in Public Policy,
George Mason University
Dept of Economics; Director, Center for
International Science and Technology
Policy, George Washington University

850-228-9273
703-993-2270
202-378-6230

Appendices
Appendix A: Survey Instruments
Appendix B: PSM Details