EAPSI_IRFP_Eval_ Supporting Statement 7.1.10 Part A

Supporting Statement for Paperwork Reduction Submission

Evaluation of National Science Foundation’s East Asia and Pacific Summer Institutes and International Research Fellowship Program

Section A

Introduction

The National Science Foundation requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, a three year clearance to conduct data collection efforts for an outcome evaluation of the National Science Foundation’s East Asia and Pacific Summer Institutes (EAPSI) and International Research Fellowship (IRFP) Program.

These two programs offer early career researchers an opportunity to forge collaborative relationships with foreign scientists and engineers, albeit through different interventions. Launched in 1999, EAPSI provides $5,000 of support to US graduate students to spend the summer (two months) conducting research in seven countries in East Asia and the Pacific region. The program is designed to immerse US scholars into the scientific and social culture of the host location. IRFP, established in 1992, provides support to post-graduate scientists (generally a year or two after the receipt of a doctoral degree), for a research experience abroad lasting from 9 to 24 months, with no restriction on geographical area. Awards range from $57,000 to $200,000, depending on the location, cost and duration of the project, and the applicants’ family status.

To assess the program effectiveness, NSF plans to collect data that are designed to explore the fellowship experiences and educational and career outcomes of EAPSI and IRFP fellows as well as the influence of the programs on host scientists and their institutions and on US scientists and their institutions. The primary methods of data collection will include analyses of NSF program records and surveys of fellows, unfunded applicants, US advisors of fellows, and foreign hosts. There are a bounded (or limited) number of respondents within the general public who will be affected by this research.

A.1. Circumstances Requiring the Collection of Data

Launched in the early 1990s, the IRFP program has never been evaluated. The EAPSI program precursor, the Japan program, was last evaluated by an outside contractor in 2002. The evaluation was focused on program participants for years 1992 to 1998. The NSF Office of International Science and Engineering administering the program wishes to collect data on the performance of each program, to collect evidence about the merit of continuing/expanding current funding continued expenditures and to revise the programs should it prove necessary.

New data collection is required for a systematic assessment of the programs. While the extant records, such as participant progress reports and application data, provide some information to describe the applicant and participant population, these data are inadequate to document participants’ fellowship experiences as well as educational or career trajectories. In addition, the evaluation study seeks to describe the effects of the programs on the US institutions as well as on the host institutions. Extant documents do not contain information related to these and program outcomes. To supplement and enhance extant program data, census surveys of fellows, unfunded applicants, US advisors, and foreign hosts will be conducted.

A.2. Purposes and Uses of the Data

The primary purpose for collection of this information is program evaluation. The National Science Board highlighted the importance of international research and education in its call for increased NSF involvement in promoting international research collaboration. The IRFP and the EAPSI programs reflect NSF’s commitment to support the active engagement of students and junior researchers in international research experiences. Thus, given the confluence of heightened interest in and need for international collaboration, NSF’s Office of International Science and Engineering’s decision to commission an evaluation of these two programs is timely.

The evaluation study will focus on three programmatic areas:

1. EAPSI and IRFP fellows’ experiences applying for, and participating in, each program;

2. Educational and career outcomes of EAPSI and IRFP fellows and how they compare to those of unfunded applicants and to national doctoral degree recipients; and

3. The effects of the programs on host scientists and their institutions, as well as on US scientists and their institutions.

Data collected in the study would be used by program staff to help them manage the programs. In addition, outcomes of the program would be reported to the Office of International Science and Engineering leadership and possibly to other stakeholders outside of NSF, as appropriate. If the program yields findings of more general interest, the results may be submitted for a publication in a peer-reviewed journal.

A.3. Use of Information Technology to Reduce Burden

In order to reduce respondent burden, internet-based surveys will be used to collect information from participants. As the populations being surveyed in this study are highly educated scientists, engineers, and other technical professionals, they are expected to have easy access to and be fluent in the use of web-based technologies.

The use of web-based systems facilitates accuracy, completeness, and speed of data entry, also reduces respondent burden. Web-based surveys employ user-friendly features, such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. In addition, survey skip patterns automatically move the respondent forward into the next appropriate section, reducing time burden on respondents and simplifying the survey-taking experience. This approach also allows for easy identification of non-respondents and facilitates follow-up.

Furthermore, data entered by participants can be automatically uploaded into standard analysis software, eliminating an additional data entry step, thus increasing the efficiency of the researcher(s) conducting the study.

Finally, email will be used to send respondents their invitations to complete the survey and to follow-up with the non-respondents to ensure their participation.

A.4. Efforts to Identify Duplication

This evaluation does not duplicate other NSF efforts. The Japan program evaluation (EAPSI precursor) was limited to the program participants from 1992 to 1998. This proposed data collection is not duplicative, as it focuses on individuals who participated in the program starting the following year, 1999. No evaluation of the IRFP program – established in 1992 – has been conducted.

A.5. Small Business

No small businesses will be involved in this study.

A.6. Consequences of Not Collecting the Information

Failure to collect the information proposed in this request would prevent NSF from assessing the role IRFP and EAPSI programs play in promoting international collaborations in STEM research and education, which the National Science Board identified as one of the top priorities for the agency. In addition, data collected in the evaluation would allow IRFP and EAPSI program managers to take corrective actions should the evidence emerge that these are necessary. Finally, the evaluation would provide systematic outcome data on the programs, which could be reported to OMB, Congress, and other stakeholders outside NSF.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The project will fully comply with the guidelines of 5 CFR 1320.5. No special circumstances apply to this data collection.

A.8. Consultation Outside the Agency

Comments on this data collection efforts were solicited in the Federal Register on March 12, 2010 (vol. 75, no. 48, p.11941). During the first comment period, one comment was received from the Tea Party of New Jersey. The comment did not address this data collection for which this request seeks approval, nor did the comments have bearing on the collection, and therefore, NSF is proceeding with seeking approval from OMB.

Consultation on the study design was conducted by the research firm, Abt Associates Inc., contracted by NSF to design and conduct the evaluations of the EAPSI and IRFP evaluations.

To provide input on all aspects of the EAPSI and IRFP evaluation design, an Advisory Panel of five experts was convened to guide the study. The Panel members have provided input on the study design, and participated in an in-person meeting to discuss the study design. Advisory Panel members include: Christopher Hill, George Mason University; Susan Cozzens, Georgia Institute of Technology; Irwin Feller, Pennsylvania State University; Terrence Russell, Association for Institutional Research; and Nicholas Vonortas, George Washington University. The Advisory Panel will continue to consult throughout the duration of the evaluation.

A.9. Payments or Gifts to Respondents

No payment or gift will be provided to respondents.

A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected will be available to the evaluation contractors, contractors hired to manage data and data collection software, and at the aggregate level to NSF staff. Data will be processed in accordance to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.

Individuals surveyed will be assured that the information they provide will not be released in any form that identifies them, and that their responses will be kept confidential to the extent provided by law. The contractor will be expected to maintain the confidentiality, security, and integrity of the survey data. The web-based survey data and notes will be maintained on a secure server with appropriate levels of password and other types of protection. Proposed procedures for protecting the data and privacy of respondents will be reviewed by the contractor’s Institutional Review Board prior to data collection.

A.11. Questions of a Sensitive Nature

The proposed surveys ask for demographic information (gender, race/ethnicity, citizenship status, and income) from participants on a voluntary basis, thus respondents may choose not to provide information that they feel is sensitive in nature. All survey questions will be reviewed by the contractor’s Institutional Review Board prior to fielding. Copies of the surveys can be found in Appendix A.

A.12 Estimates of Response Burden

The total number of respondents targeted for this study is estimated at 8,359, which represents the universe of applicants (fellows and unfunded applicants) to the programs in the years 1992-2009 for the IRFP program and the years 1999-2009 for the EAPSI program, as well as the foreign hosts of fellows in both programs, EAPSI foreign location staff, and the US advisors of fellows in the EAPSI program. Assuming a 75% response rate for each survey group, the total number of respondents is estimated to be 6,266, resulting in an estimated response burden for the surveys of 2,489.25 hours over one year. Details on these calculations are provided in A.12.1 and A.12.2.

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

Table A.12.1 below indicates the number of respondents and expected number of responses for each category of respondent type and the time demand these surveys will place on each individual respondent and on all respondents in aggregate.

Table A.12.1
Respondent Type	Targeted group	Number of respondents#	Time per response (hours)	Total time burden (hours)
EAPSI Fellows	1,434	1,075	0.5	537.5
EAPSI Unfunded Applicants	1,401	1,050	0.5	525
EAPSI US Advisors	1,434	1,075	0.25	268.75
EAPSI Foreign Hosts	1,434	1,075	0.25	268.75
EAPSI Foreign Location Staff	20	15	0.5	10
IRFP Fellows	567	425	0.5	212.5
IRFP Unfunded Applicants	1,502	1,126	0.5	563
IRFP Foreign Hosts	567	425	0.25	106.25
Total	8,359	6,266*	N/A	2,489.25
# The above estimates for the number of responses for each type of respondent assume a 75% response rate which is comparable to response rates we have achieved on other studies of a similar scope respondent type.

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

As each respondent will complete the survey once, the annual burden and the aggregate burden will be the same as shown in Table A.12.1.

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost to respondents is $92,333.75. The following chart shows the estimated total annual costs to each group of respondents over one year for the surveys.

Table A.12.3
Respondent Type	Targeted Group	Number of Respondents*	Time Per Response (hours)	Total Time Burden (hours)	Hourly salary estimate	Estimated cost per respondent	Estimated overall cost
EAPSI Fellows	1,434	1,075	0.5	537.5	35	17.5	18,812.50
EAPSI Unfunded Applicants	1,401	1,050	0.5	525	35	17.5	18,375.00
EAPSI US Advisors	1,434	1,075	0.25	268.75	43	10.75	11,556.25
EAPSI Foreign Hosts	1,434	1,075	0.25	268.75	43	10.75	11,556.25
EAPSI Location Staff	20	15	0.5	7.5	43	21.5	322.50
IRFP Fellows	567	425	0.5	212.5	35	17.5	7,437.50
IRFP Unfunded Applicants	1,502	1,126	0.5	563	35	17.5	19,705.00
IRFP Foreign Hosts	567	425	0.25	106.25	43	10.75	4,568.75
Total	8,359	6,266		2,489.25			92,333.75
# Assumes a 75% response rate. Figures are rounded to the nearest whole dollar. Based on an average salary estimates for PhDs in Science and Engineering as reported in National Science Foundation's, Science and Engineering Indicators - 2006. National Science Foundation, Division of Science Resources Statistics. Arlington, VA (NSB 06-01) [February 2006], Figure 3-22. Url: http://www.nsf.gov/statistics/seind06/figures.htm (Used estimates for faculty with 5-9 years of experience @$70,000 and 15-19 years of experience @ $85,000)

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden regarding capita, operation, or maintenance costs to respondents that results from this study, other than the time spent responding to the survey.

A.14. Estimates of Costs to the Federal Government

The estimated cost to the Federal Government for the data collection activities included in this request for approval is $711,385. This cost estimate includes instrument development and pretesting; location and recruitment; data collection; and data processing.

A.15. Changes in Burden

This is a new collection of information.

A.16. Plans for Publication, Analysis, and Schedule

To provide the NSF with an understanding of the outcomes of the IRFP and EAPSI programs, the contractor will prepare two reports for NSF—one for each program—that describe the study and findings. The reports will characterize the research and educational activities supported by the initiatives, as well as outcomes in the careers of fellows and unfunded applicants.

This comparative evaluation will help NSF respond to questions about the impact of the EAPSI and IRFP programs on participants’ research collaborations and activities. By surveying the awardees of the program and their comparisons as well as the foreign hosts, we can learn more about the perceived benefits of participation in these programs.

Analyses of survey data will include a detailed summary that utilizes appropriate descriptive statistics. For survey items using continuous scales, the study will calculate means and standard deviations to describe both central trend and variation. Frequency distributions and percentages will be used to summarize answers given on ordinal scales. Comparative analyses between the fellows and unfunded applicants will be conducted using propensity score matching to adjust for potential selection bias.

The project schedule is shown in Table A.16. Surveys are planned to begin in October 2010.

Table A.16
Activity	Timeframe
Update contact information of applicants	1-3 months after OMB approval
Field surveys	4-12 months after OMB approval
Analyze data	12-18 months after OMB approval
Prepare report of findings – EAPSI program	19-27 months after OMB approval
Prepare report of findings – IRFP program	19-27 months after OMB approval

A.17. Approval to Not Display Expiration Date

The data collection instruments will display the expiration date.

A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions are sought.