NASA ESD NESSF Impact Assessment (MT Generic Clearance) SHORT FORM (5-12-22)

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 09/30/2024

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA Earth and Space Science Fellowship (NESSF) Impact Assessment

2. TYPE OF COLLECTION:

	Attitude/Behavior Scale
	Baseline Survey
	Cognitive Interview Protocol
	Consent Form
	Focus Group Protocol
	Follow-up Survey
	Instructions
	Satisfaction Survey
	Usability Protocol

GENERAL OVERVIEW:

To foster the development of graduate students’ research skills and scientific knowledge and further the research interests and goals of NASA, NASA funds graduate student-designed research through a series of fellowship and grant-funded programs such as the NASA Earth and Space Science Fellowship (NESSF). Over the years, the program has grown and in 2021, the NASA Earth Science Division received 351 applications and funded 58 proposals.

Program managers and other NASA stakeholders are interested in understanding the impact of receiving grant funds on participants’ educational and career outcomes. Programs that serve similar purposes and populations of students, such as, the National Science Foundation’s (NSF) Graduate Research Fellowship Program (GRFP) have been rigorously evaluated to provide descriptive information about participants and estimate program impact. Such evaluations are critical to ensuring that grant and fellowship programs are meeting intended goals and providing student researchers with high-quality experiences. These formative evaluations can also provide important feedback for program improvement efforts.

Survey data from this collection will provide descriptive information about current and past program applicants and participants and an estimate of program impact. Additionally, data will be used to inform possible areas for program improvement. Data of this nature has never been collected from NESSF program applicants and participants, however doing so will provide evidence of program impact and support recommendations for potential program improvements. This information collection is being filed under methodological testing to ensure that the survey, which has been adapted from the survey used in the evaluation of the NSF GRFP, displays strong measurement properties with the current population. Additionally, the survey has been scaled back to reduce the burden on respondents, making it a new measure that needs to undergo methodological testing.

3. INTRODUCTION AND PURPOSE:

NESSF proposers, including awardees, who submitted proposals to NASA Earth Science Division from 2011 to 2014 will be asked to fill out a web-based (Survey Monkey) survey. This survey contains four categories of questions: career outcomes, graduate school experience, integration into scientific community, and prior connections to NASA. This NESSF survey is adapted from the survey used in the evaluation of the NSF GRFP. We are interested in measuring and comparing career and educational outcome of NESSF awardees and proposers who received high scores but not selected for funding. The purpose for this methodological testing is to validate our adapted instrument and its reliability to explain the way in which NESSF awardees’ career and educational outcome are impacted by NESSF.

1. RESEARCH DESIGN OVERVIEW:

Evaluation results from fellowship programs similar to NESSF have demonstrated that such merit-based funding positively impacts the career and educational outcome of awardees (Evaluation of the National Science Foundation’s Graduate Research Fellowship Program – Final Report, 2014; The Career Achievements of National Research Service Award Postdoctoral Trainees and Fellows: 1975 – 2004, 2006). As such, fellowship programs such as NESSF have been viewed as a desirable evidence-based way to increase doctoral degree attainment, flexibility in choosing research projects, and professional productivity. While there is literature documenting the generalized outcome from fellowship programs similar to NESSF, we are not aware of any evaluations specific to NASA fellowship awardees. We are specifically interested in if and how NESSF promotes the integration of awardees into the NASA scientific community and contributes to NASA workforce needs. From an equity standpoint, we are also interested in if and how NESSF attracts applicants from a diverse background.

This NESSF survey will be conducted on a web-based platform (Survey Monkey). We will use a quasi-experimental design to understand and estimate the impact of NESFF. Propensity score matching will be used to match awardees and non-awardees that received similar scores on their proposals, as well as being similar in other measured characteristics. We are targeting proposers from the 2011 to 2014 timeframe as they would have had an estimated 5 to 8 years of post-Ph.D experiences to allow us to adequately assess NESSF’s impact on their career outcome.

4. TIMELINE: Testing of the NESSF survey will take place in June – July 2022.

5. SAMPLING STRATEGY:

The number of Earth Science Division NESSF proposers who receive excellent, very good, or good adjectival rating is 350 or below per year, which translates to 1400 eligible survey respondents for the four years of interest (2011 – 2014).

Table 1. Calculation chart to determine statistically relevant number of respondents

Data Collection Source	(N) Population Estimate	(A) Sampling Error +/- 5% (.05)	(Z) Confidence Level 95%/ Alpha 0.05	(P) *Variability (based on consistency of intervention administration) 50%	Base Sample Size	Response Rate	(n) Number of Respondents
Earth Science Division NESSF Proposers	1400	N/A	N/A	N/A	1400	25%	350
TOTAL							350

6. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
Earth Science Division NESSF Proposers	350	1	25	146
TOTAL				146

7. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

8. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? Yes  No

2. If yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? Yes  No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

Yes  No

Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

9. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? Yes  No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? Not applicable.

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? We will target Earth Science Division NESSF proposers between 2011 – 2014 who received excellent, very good, or good adjectival rating (1400 respondents or below)

10. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent:  Active  Passive

6. How will the information be collected:

 Web-based or other forms of Social Media

 Telephone

 In-person

 Mail

 Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used?  Yes  No

11. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST: 

 Consent form

 Instrument (attitude & behavior scales, and surveys) NOTE: Instructions are included in the instrument

 Protocol script (Specify type: Script)

 Instructions

 Other (Specify ________________)

12. GIFTS OR PAYMENT:  Yes  No  If you answer yes to this question, please describe and provide a justification for amount.

13. ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $6875. The cost is based on stipend and administrative fee for a graduate student intern for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

14. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Amy P. Chen

Title: Senior Principal Research Scientist, Early Career Research & GLOBE

NASA Earth Science Division

Email address or Phone number: [email protected]; (202) 860-6424

Date: 12/14/2023

References

Bartolone, J.B., Halverson, M.L., Hoffer, T.B., Wolniak, G.C., Setlak, L., Hedberg, E.C., Nielsen, E., Nhuan-Le, V. and Yisak, M. (2014) Evaluation of the National Science Foundation’s Graduate Research Fellowship Program. NORC.

Mantovani, R., Look, M.V., Wuerker, E. (2006) The Career Achievements of National Research Service Award Postdoctoral Trainees and Fellows: 1975–2004. National Institutes of Health.

NASA Office of STEM Engagement 6