Overview of Proposed GRFP Study

An Evaluation of the National Science Foundation’s Graduate Research Fellowship Program: Overview of proposed study

Program Overview

As part of the National Science Foundation’s (NSF) continued commitment to graduate student education in the U.S., the Graduate Research Fellowship Program (GRFP), which began in 1952, seeks to promote and maintain advanced training in STEM fields by annually awarding roughly 2,000 fellowships¹ to U.S. citizens, nationals, and permanent residents for graduate study in research-based programs. The goals of the program are:

• To select, recognize, and financially support individuals early in their careers with the demonstrated potential to be high achieving scientists and engineers, and

• To broaden participation in science and engineering of underrepresented groups, including women, minorities, and persons with disabilities.

Underpinning the program goals are NSF’s broader strategic organization goals, including that of performing as a model organization. To achieve this goal and to become a model Federal steward, representing excellence in management and fiscal responsibility, NSF seeks to “learn through assessment and evaluation of NSF programs, processes, and outcomes; continually improve them; and employ outcomes to inform NSF planning, policies, and procedures” (http://www.nsf.gov/news/strategicplan/nsfstrategicplan_2011_2016.pdf, pp. 16-17, italics in original). Thus, excellence in management is an underlying goal of each NSF program, including the GRFP.

Purpose and Need for Study

NSF is seeking to conduct a study that has three purposes:

• Provide descriptive information related to the GRFP program goals on the demographics, educational decisions, career preparation, aspirations and progress, as well as professional productivity, of GRFP Fellows and comparable non-recipient applicants and national populations of graduate students and doctorate recipients.

• Provide rigorous evidence of the impact of the GRFP on individuals’ educational decisions, career preparations, aspirations and progress, as well as professional productivity

• Provide an understanding of how the program is implemented by universities and whether and how specific program policies could be adjusted to make the program more effective in meeting its goals.

Previous studies of the GRFP were largely completed in the mid-1970s to the mid-1990s. The most recent study, published in 2002, examined GRFP Fellow cohorts through 1993, and is now dated. The NSF GRF program collects data on an ongoing basis through multiple sources that is used for program management and accountability purposes. These sources include reports from the GRFP Committee of Visitors, annual surveys of the review panelists, comments from Fellows and university GRFP coordinating officials, and data compiled from the applications. In addition, GRFP Fellows submit annual activity reports, the format for which was revised in 2010 to include activities that contribute to career preparation such as acquisition of research skills and other professional skills, data on career plans, internships, and other sources of financial support. The data are tracked over multiple years to examine trends and identify gaps that need to be addressed in subsequent competitions. However, the data, while useful, offered limited information for prior years. In addition, they did not address program impact or implementation.

Thus, NSF needs current information on several fronts to inform future decisions about program structure and design that cannot be addressed either with NSF data or existing national databases. These include: (a) how GRFP Fellows differ from their peers in terms of demographics, educational trajectories, and career outcomes; (b) the impact of the GRF program on Fellows in terms of educational trajectories, career outcomes, and professional productivity; (c) the impact of the GRF program on institutions in terms of student diversity and quality in STEM graduate programs; and (d) program implementation. The current study, being conducted for NSF by NORC at the University of Chicago, is intended to address each of these areas.

Research Questions and Study Approach

Research Questions

The study focuses on the following research questions:

RQ1. What is the impact of the GRFP fellowship on the graduate school experience? 

RQ2. What is the impact of the GRFP fellowship on career outcomes?

RQ3. What are the effects of the GRFP on institutions?

RQ4. Is the program design effective in meeting program goals?

While RQ1 and RQ2 are framed in terms of impact, a necessary component of the research is examining how the Fellows compare with peers in terms of demographics, aspirations, educational trajectories, career outcomes, and professional productivity, to help address the program goals. RQ3 and RQ4 are designed to address both the GRF program goals as well as the underlying NSF strategic goal of excellence in management.

Data Sources

To address the research questions, the study will use both primary and secondary data sources. In terms of primary data collection, the study will:

• Collect data from Fellows and carefully-matched counterparts (QG2 Honorable Mentions)² through a survey (GRFP Follow-Up Survey) that asks about graduate school experiences, educational attainment, career outcomes, employment characteristics, and professional productivity (RQ1 and RQ2). The survey also asks Fellows about the influence of program elements (choice, flexibility, and monetary value) on their decision to enroll in and successfully complete STEM graduate programs (RQ4).

• Collect data from two samples of institutions:

  1. In-depth data from six institutions (“institutional site visit sample”) gathered from site visits which will encompass in-person interviews with administrators, faculty, and staff to understand: (1) the current climate, (2) perceived impact of the program on Fellows, institutions, and programs, (3) program implementation, and (4) GRFP policies. The in-depth data will be used to address RQ3 and RQ4.

  2. Targeted data from a larger sample of 20 institutions (“institutional phone sample”) gathered from shorter phone interviews more narrowly focused on implementation and specific GRFP policies (RQ4 and, to a more limited extent, RQ3).

These two institutional samples serve two different purposes, each suited to the type of data to be collected. By limiting the site visit sample to six institutions and broadening the institutional phone sample to include 20 institutions, we balance the need for in-depth data collection with the goals of minimizing respondent burden and collecting data from a broader pool of institutions.

• Review and analyze similar federal fellowship programs using data collected from websites, program materials, and interviews with program officers managing these programs. This part of the study will help inform GRFP policies and best practices (RQ4). The findings will be valuable in understanding how best to support Fellows and help develop a more diverse STEM workforce.

Secondary data sources include the Doctorate Records File to provide a national context and national comparison group.

Analysis

Descriptive analysis will be used to examine the composition, experiences, and outcomes of Fellows, non-recipients, and national peers and will provide evidence of GRFP participation of underrepresented groups and trends in program selection, recognition, and financial support of early career scientists and engineers. To measure impact, the study will model outcomes using quasi-experimental methods to compare outcomes of the treatment group (Fellows) with outcomes of plausibly similar control groups (QG2 Honorable Mentions). These methods are widely accepted as the best methods on which to base causal inferences in the absence of a randomized experiment (i.e., when it is not feasible to randomly assign participants to treatment and control groups). To examine implementation, the study will use qualitative methods to code and analyze the institutional interviews and to draw out lessons learned from the interviews with program officers.

Findings and Dissemination

The data collected in this study and the analytic reports will provide a comprehensive look at the GRFP, its impact on Fellows, institutions, and the science and engineering workforce, and the extent to which the program is meeting its goals. In particular, the findings will provide information on:

• The influence the GRFP has had on the decisions, experiences, academic attainment, and career outcomes of Fellows compared with carefully-matched peers ;

• The extent to which the program has broadened the participation of underrepresented groups in STEM at the graduate level;

• The perceived effects on institutions in terms of financial impact, enrollment, student diversity and quality (among others);

• Whether specific design elements—choice, flexibility, and monetary value—are working as intended and the extent to which they are valued; and

• The need (if any) for changes in the way the program is structured to make it more effective.

Thus, overall, the study findings will provide valuable insights to NSF on the impact of its investments in the GRFP and inform its program management. In conjunction with findings from the review of similar federal fellowship programs, the findings may prove valuable to the larger community of program officers administering these programs as well as to the graduate education policy community in understanding how best to support graduate education and help develop a more diverse STEM workforce.

Study results will be reported to the Division of Graduate Education, EHR/NSF, distributed within the community of universities who participate in GRFP, and published on the NSF website. Limited print copies of the full report will be made available to NSF as well as 500 copies of a printed executive summary that can be disseminated more widely and that will be useful to a variety of public audiences. A policy brief reporting on the review and analysis of federal fellowship programs will be made available to all federal fellowship program managers and may be of interest to the larger foundation community as well. Findings of more general policy or methodological interest will be distributed more broadly, through conference presentations and submissions for publication in peer-reviewed journals.

1 The annual number of fellowships awarded increased from approximately 1,000 to 2,000 in 2010.

2 The quality groupings (QG) refer to the categories assigned to each GRFP participant upon applying to the fellowship program. QG1 is the highest ranking an applicant can receive. The study sample includes the highest two categories: QG1 and QG2 applicants.