Crosswalk between research questions and data sources

Appendix C. Crosswalk between Research Questions and Data Sources

Primary data sources:

1. GRFP Follow-Up Survey of current and former graduate students [termed “Survey (Current)” and “Survey (Former) in the tables below], Fellows and Honorable Mentions

2. In-Person interviews with institutional administrators, faculty and staff during site visits [termed “Interviews (Site visits) in the tables below]

3. Phone interviews with institutional administrators, faculty, and staff [termed “Interviews (phone) in the tables below]

4. Phone interviews with program officers of federal fellowship programs similar to the GRFP and review of program materials [exempt from OMB review—see page 2]

Secondary data sources: Survey of Doctorate Recipients (SDR), Survey of Earned Doctorates (SED), Integrated Postsecondary Education Data System (IPEDS), Barron’s Profiles of American Colleges. These data sources will be used to:

• Define a comparison group of national peers

• Obtain characteristics of institutions, including reputation and ranking, to be used in the modeling or to look at differences in selectivity of institutions hosting Fellows, Honorable Mentions, and the national peers

• Calculate outcomes in terms of degree attainment and time to degree

• Examine career trajectories and characteristics of academic and non-academic employment

• Examine future professional productivity

Tables C.1-C.4 provide a crosswalk of data sources and analysis for each of the research questions.

Table C.1. Research Question 1: Data Sources and Analysis

RQ 1: What is the impact of the GRFP fellowship on the graduate school experience?
Data Sources	Analysis
Survey (Current); Survey (Former) Section I (A): GRFP Award Status Decision to attend graduate school or to study a STEM field at the graduate level Decision to attend a particular institution or to change institutions Ability to/effect on: opportunities for research or to work with faculty or variety of faculty Get additional financial support Perceptions of being a good student or an asset to faculty projects Job search Cover living expenses Change advisors or departments during graduate study Section II (C): Experiences with Program Program climate, quality, and offerings: Ratings of program attended along various dimensions (such as reputation, research experience, guidance received, curriculum, quality of instruction, environment for minority students and women, professional development opportunities, etc.) Whether the program had a scholarly climate and offered opportunities for enrichment, professional growth, collaboration, presenting own research, developing career skills, travel to non-U.S. institutions for research and training, learn about ethical practices Opinions about accessibility and collegiality (or not) of faculty Participation in various research and professional activities: Frequency with which student engaged in a set of activities measuring intellectual engagement and climate (participated in professional organizations, contacted other scholars, offered or asked for critiques on scholarly work, performed own research etc.) Extent to which student worked with people other than advisor or worked with international collaborators, undertook interdisciplinary research, learned organizational/managerial skills etc. Importance of different dimensions of intended career path (working for social change, high income potential, creativity and initiative, availability of jobs etc.) Section II (D): Professional Productivity and Financial Support During Graduate School Number of papers presented while in graduate school at national/international meetings or conferences Type and number of publications by whether student was primary author Number of patents for which student applied while in graduate school Types of grants/contracts for which student applied as PI or co-PI while in graduate school Sources of financial support during graduate school Whether student worked for pay during graduate school, number of hours, type of work, and reason for working Participation in an internship (paid/unpaid, type) Section IV (F). Educational Background Community college attendance Participation in NSF-sponsored programs Section IV (Demographics) Gender Race/ethnicity Marital status and number of dependents Highest educational attainment of parents Citizenship Disability status and type of disability 2. Survey (Former) Section B: Graduate School Background Information Degrees earned Start date and completion date of degree (to calculate time to degree) Leaves of absence: duration and reasons Change in primary field of study and why 3. Interviews (Site Visits) Faculty: How do Fellows benefit from their GRFP Fellowship? What are the career goals of your GRFP Fellows, and do they differ from those of the other students in your department? Compared to other students, to what extent are the Fellows developing the personal and professional skills necessary for success in their chosen field after graduating? How do the experiences of Fellows differ from those of other students in the program? Probe for: whether Fellows are fully integrated into the program or if their source of funding isolates them; whether the GRFP funding provides greater autonomy/flexibility since it is not tied to an advisor or lab; whether program guidelines affect Fellows’ service to the department in terms of TAing/RAing? To what extent do Fellows contribute to the research activity of the department? Are the educational and research experiences of Fellows similar to those of other students? Do Fellows have different opportunities or make different choices compared to other students? If there are differences, what are they? How has this changed over the past few years? Compared to the other students in your department, do Fellows differ in the length of time they need to finish? Senior University Administrators: Are there supports or activities provided by the university to the Fellows that are separate from those provided to other graduate students? Compared to other students, to what extent are Fellows contributing to the research endeavors of the university while they are in graduate school? To what extent are they supporting the department through service and teaching? How has this changed over the past few years? (Probe specifically for changes in Fellows’ participation in teaching and research.) [Compared to other students] To what extent are [GRFP Fellows] succeeding in STEM fields upon graduation? Departmental/Graduate Studies Staff: How do Fellows benefit from their GRFP Fellowship? What kinds of supports are offered to Fellows that are different than those offered other graduate students? In your opinion, are these helpful to Fellows in terms of timely progress towards degree or better integration into the department? How does your department financially support its graduate students, for example, how many students receive full support to the completion of their degree, and how is aid awarded? How would the department be affected if GRFP funding were to disappear? Does the GRFP figure into the financial planning of the department? Now let’s talk about how the Fellows in your department actually use their Fellowships. When do most Fellows use the three years of the Fellowship? How common is it for Fellows to place their Fellowship on reserve for one or two years? How do most GRFP students secure funding when they are not receiving GRFP support? What supplemental funding, if any, is provided to Fellows by the department? How do the guidelines on when Fellows may use their funding affect the experiences of the Fellows and the department? How has this changed over the past few years? Do Fellows participate in [teaching assistantship and research assistantship] opportunities to the same degree as their peers? How has this changed over the past few years? 4. Interviews (Phone) Let’s talk about how the Fellows in your department actually use their Fellowships. When do most Fellows use the three years of the Fellowship? How common is it for Fellows to place their Fellowship on reserve for one or two years? How are most Fellows funded when they are not receiving GRFP support? What supplemental funding, if any, is provided to Fellows by the department? How do the GRFP policies on when Fellows may utilize their funding affect the experiences of the Fellows and the department? How do the policies affect the Fellows’ progress to degree completion? What are the requirements and opportunities for TAing and RAing in the department? Do Fellows participate in these opportunities to the same degree as their peers? How do the program guidelines about the amount of service Fellows may provide to the institution while funded by the GRFP affect the experiences of Fellows and the department? Could this policy be improved for the Fellows? How has the service provided by Fellows changed over the past few years? How do the experiences of Fellows differ from those of other students in the program? Probe for: whether Fellows are fully integrated into the program or if their source of funding isolates them; whether the GRFP funding provides greater autonomy/flexibility since it is not tied to an advisor or lab; What kinds of supports are offered to Fellows that are different than those offered to other graduate students? In your opinion, are these helpful to Fellows in terms of timely progress towards degree or better integration into the department?	This question is addressed by comparing graduate student experiences (such as participation in STEM graduate study, selection of institution, professional productivity, career aspirations, graduate degree attainment and time-to-degree) of Fellows with those of a matched comparison group of similar but non-awarded GRFP applicants. In addition, Fellows’ experiences will be compared with those of a matched comparison group of doctoral students nationally. Compare responses of Fellows and Honorable Mentions: Descriptive analysis Statistical modeling, adjusting for covariates Compare outcomes of Fellows and matched counterparts in national data: Descriptive analysis Statistical modeling, adjusting for covariates Selected qualitative data from the interviews will be used to provide context and background for the quantitative analyses and to examine how faculty and staff view the Fellows, their integration into and contribution to the department ; the extent to which different supports are offered to Fellows that are not offered to other graduate students; and the amount of graduate funding available to Fellows in addition to their Fellowship. In addition, selected data from faculty and senior administrators will be analyzed to see how well perceptions about student outcomes align with the quantitative data

Table C.2. Research Question 2: Data Sources and Analysis

RQ2. What is the impact of the GRFP fellowship on career outcomes?
Data Sources	Analysis
1. Survey (Former) Section III (E): Job History Current employment status If not working, reason for not working Most recent year of employment Number of jobs since leaving program Whether first or current/most recent employment was in an academic position and if so, type of academic position held Characteristics of current/most recent employment: type of employer, number of employees, type of job, annual salary, primary and secondary work activities, whether related to field of graduate study, whether first job after leaving program If not first job after leaving program, characteristics of first employer/job (see above) Section III (E): Professional Productivity Since leaving graduate school: Number of papers presented at national and international meetings Number and types of publications produced Patent-related activity: Number of applications Number granted Number granted that have been commercialized or licensed Number, types, and amount of grants/contracts awarded as PI Types of teaching activities Types of professional services undertaken	This question is addressed by comparing the career outcomes (for example, in terms of academic and non-academic career choices, science and engineering careers versus careers in other fields, job characteristics, and professional productivity) of Fellows with those of a matched comparison group of similar but non-awarded GRFP applicants, and other national populations of doctoral students. Compare outcomes of Fellows and Honorable Mentions: Descriptive analysis Statistical modeling, adjusting for covariates Compare a subset of outcomes of Fellows and matched counterparts in national data: Descriptive analysis Statistical modeling, adjusting for covariates

Table C.3. Research Question 3: Data Sources and Analysis

RQ3. What are the effects of the GRFP on institutions?
Data Sources	Analysis
1. Interviews (Site Visits) Senior University Administrators: What is your overall impression of the NSF Graduate Research Fellowship Program (GRFP)? How does it compare, in reputation, with other fellowship programs? What trends, if any, have you noticed in the granting of GRFP Fellowships? Has the recent increase in the number of Fellowships awarded contributed to these trends? [If needed: for example, in terms of quality of students, racial, ethnic, and gender diversity, field of study, etc.] How has this increase impacted your graduate program, if at all? We are interested in the impact of GRFP on the university. To what extent does the program help: Recruit students to STEM programs at your university? Offset the costs necessary to fund students? Diversify the student body of STEM programs? How would your university be affected if GRFP funding were to disappear? Does the GRFP figure into the financial planning of the graduate studies office or any of your graduate programs? Does the current amount of funding provided by the GRFP adequately meet the needs of graduate students at your university? How is the cost-of-education allowance provided by the Fellowship used by the university? Are Fellows provided any kind of supplemental funding if the allowance cannot cover their financial needs? Are there supports or activities provided by the university to the Fellows that are separate from those provided to other graduate students? Compared to other students, to what extent are Fellows contributing to the research endeavors of the university while they are in graduate school? To what extent are they supporting the department through service and teaching? How has this changed over the past few years? (Probe specifically for changes in Fellows’ participation in research and teaching.) To what extent are they succeeding in STEM fields upon graduation? Faculty: What is your overall impression of the NSF Graduate Research Fellowship Program (GRFP)? How does it compare, in reputation, with other fellowship programs? What does it mean to faculty members that a student is a GRFP Fellow? How does a GRFP Fellowship influence the admissions decisions of your department? How does receiving a GRFP Fellowship influence faculty members’ willingness to work with a prospective student? How does your department benefit from hosting GRFP Fellows? To what extent do Fellows contribute to the research activity of the department? Are the educational and research experiences of Fellows similar to those of other students? Do Fellows have different opportunities or make different choices compared to other students? If there are differences, what are they? How has this changed over the past few years? How would your department be affected if GRFP funding were to disappear? [Probes to be used as necessary:] Does the GRFP figure into the financial planning of the department? Department/Graduate Studies Staff: What is your overall impression of the NSF Graduate Research Fellowship Program (GRFP)? How does it compare, in reputation, with other fellowship programs? How does your department benefit from hosting GRFP Fellows? Let’s talk about the program’s enrollment patterns in terms of gender, ethnicity, and Master’s/Ph.D. student ratios. Do Fellows differ from other graduate students in terms of these characteristics? To what extent does the GRFP promote diversity among graduate students enrolled in your department? What kinds of supports are offered to Fellows that are different than those offered other graduate students? In your opinion, are these helpful to Fellows in terms of timely progress towards degree or better integration into the department? How does your department financially support its graduate students, for example, how many students receive full support to the completion of their degree, and how is aid awarded? How would the department be affected if GRFP funding were to disappear? Does the GRFP figure into the financial planning of the department? 2. Survey (current); Survey (former) Section I (A): GRFP Award Status Decision to attend graduate school or to study a STEM field at the graduate level Decision to attend a particular institution or to change institutions Section II (D): Professional Productivity and Financial Support During Graduate School Number of papers presented while in graduate school at national/international meetings or conferences Type and number of publications by whether student was primary author Number of patents for which student applied while in graduate school Types of grants/contracts for which student applied as PI or co-PI while in graduate school	Possible effects of the GRFP on graduate institutions are assessed through a series of interviews focusing on financial impact including adequacy of the cost-of-education allowance and ability to free up resources to provide funding to other students, the extent to which Fellows participate in departmental teaching and research (“service to the department”), effects on student diversity and (to the extent feasible) student quality, and effects, if any, on scholarly productivity and research. Data from the interviews will be used to draw out broad themes regarding perceived effects on the institution and perceived benefits to the department of hosting GRFP Fellows A limited set of questions on the surveys ask Fellowship Recipients about the influence of the award on their ability to attend graduate school or to study a STEM field on graduate school, and professional productivity while in graduate school. Analyzed at the institutional level, this provide information will provide evidence of impact on graduate programs and institutions.

Table C.4. Research Question 4: Data Sources and Analysis

RQ4. Is the program design effective in meeting program goals?
Data Sources	Analysis
1. Survey (current); Survey (former) Section I (A): GRFP Award Status Decision to attend graduate school or to study a STEM field at the graduate level Decision to attend a particular institution or to change institutions Ability to change advisors Those who did not accept the Fellowship are asked a limited set of questions whether particular GRFP requirements influenced their decision to refuse the award alternative sources of funding Those who accepted the Fellowship are asked whether the Fellowship would be better if certain requirements were either eased or changed (for example, five years instead of three years of funding; no service requirements etc.) Section II (D): Professional Productivity and Financial Support During Graduate School Whether student worked during the program and if so, the reason why (for example, to cover living expenses, to pay for school, to support family) 2. Survey (Former) Section III (E): Job History Current employment status If not working, reason for not working Most recent year of employment Number of jobs since leaving program Whether first or current/most recent employment was in an academic position and if so, type of academic position held Characteristics of current/most recent employment: type of employer, number of employees, type of job, annual salary, primary and secondary work activities, whether related to field of graduate study, whether first job after leaving program If not first job after leaving program, characteristics of first employer/job (see above) Section III (E): Professional Productivity Since leaving graduate school: Number of papers presented at national and international meetings Number and types of publications produced Patent-related activity: Number of applications Number granted Number granted that have been commercialized or licensed Number, types, and amount of grants/contracts awarded as PI Types of teaching activities Types of professional services undertaken 3. Interviews (Phone) NSF is interested in learning how some particular policies of the GRFP are working and the extent to which they could be improved. We are interested in both your experiences with these policies as well as your opinions, suggestions for improvements, and ideas. Let’s talk about how the Fellows in your department actually use their Fellowships. When do most Fellows use the three years of the Fellowship? How common is it for Fellows to place their Fellowship on reserve for one or two years? Has this pattern changed over the past few years? How are most Fellows funded when they are not receiving GRFP support? What supplemental funding, if any, is provided to Fellows by the department? How do the GRFP policies on when Fellows may use their funding affect the experiences of the Fellows and the department? How do the policies affect the Fellows’ progress to degree completion? Does the current amount of funding provided by the GRFP adequately meet the needs of graduate students at your university? How is the cost-of-education allowance provided by the Fellowship used by the university? How does the institution cover tuition if the cost-of-education allowance of $10,500 is insufficient? How do the experiences of Fellows differ from those of other students in the program? Probe for: whether Fellows are fully integrated into the program or if their source of funding isolates them; whether the GRFP funding provides greater autonomy/flexibility since it is not tied to an advisor or lab; What kinds of supports are offered to Fellows that are different than those offered to other graduate students? In your opinion, are these helpful to Fellows in terms of timely progress towards degree or better integration into the department? What are the requirements and opportunities for TAing and RAing in the department? Do Fellows participate in these opportunities to the same degree as their peers? How do the program guidelines about the amount of service Fellows may provide to the institution while funded by the GRFP affect the experiences of Fellows and the department? Could this policy be improved for the Fellows? How has the service provided by Fellows changed over the past few years? The program requires that the status of Fellows is decided on an annual basis—i.e. whether they are in a “Tenure” or “Reserve” status for the following Fellowship year. How do you think this policy works? Is there any need to change it? The program also requires that Fellows are affiliated with a U.S. institution. Are there instances (for example, in particular fields) where you would suggest revisiting this policy? Is there anything about the program policies [refer to the Administrative Guide if needed] that, if changed, would improve the program or be beneficial for your institution, the graduate programs, or Fellows? 4. Interviews (Site Visits) Senior University Administrators: Does the current amount of funding provided by the GRFP adequately meet the needs of graduate students at your university? How is the cost-of-education allowance provided by the Fellowship used by the university? How could the GRFP be improved? What changes to the program would most benefit your university? Faculty: [How do] program guidelines affect Fellows’ service to the department in terms of TAing/RAing? Department/Graduate Studies Staff: Now let’s talk about how the Fellows in your department actually use their Fellowships. When do most Fellows use the three years of the Fellowship? How common is it for Fellows to place their Fellowship on reserve for one or two years? How do most GRFP students secure funding when they are not receiving GRFP support? What supplemental funding, if any, is provided to Fellows by the department? How do the guidelines on when Fellows may use their funding affect the experiences of the Fellows and the department? How has this changed over the past few years? What are the expectations and opportunities for TAing and RAing in the department? Do Fellows participate in these opportunities to the same degree as their peers? How could the GRFP be improved? What ideas would you like to communicate to NSF? [If perceived problems are reported:] What solutions would you propose?	This question is addressed by asking (a) Fellows about the impact of GRFP on decision to go to graduate school in a STEM field, the impact of particular program elements on choice, flexibility, and ability to fund and complete their graduate programs, and future employment and professional productivity in STEM fields; (b) students who refused the Fellowship the role that particular requirements played in the decision; (c) institutional administrators and faculty about whether the program could be improved; and (d) program officers managing similar federal fellowship programs about what they have learned from their programs regarding implementation and promising practices. In terms of broadening participation in STEM fields, the survey will provide data on the influence of the GRFP on decisions to go to graduate school in a STEM field among underrepresented minorities and women. It will also shed light on whether these groups then go on to work in STEM fields and the extent of their professional productivity compared with other fellowship recipients. The survey will also provide an indication about the extent to which the design elements are valued by recipients and whether recipients see the need for change. The data will also show whether some of the fellowship requirements are barriers to acceptance of the award. Data from the interviews (primarily the phone interviews but also, to a smaller extent, the site visit interviews) will be used to draw together a portrait of how the program is working, how the different program elements are being implemented, and respondents’ recommendations on how the program can be improved These data may be supplemented by information obtained from program officers of similar federal fellowship programs about what appears to be working well in their programs

1