Supporting STatement PART A
A.1. Circumstances that Make Data Collection Necessary
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The National Science Foundation (NSF) seeks approval for the collection of research and development data through the project Computer and Information Science and Engineering (CISE) Research Experiences for Undergraduates (REU) Past Participant Survey – 2021 Impact of REU Participation on Career Pathways.
Every year the National Science Foundation (NSF) funds hundreds of Research Experience for Undergraduates (REU) activities through its REU program, which provides undergraduate students at US higher education institutions with opportunities to work with faculty on a research project. REUs are offered in two forms: REU Sites or REU Supplements. REU Sites are based on independent proposals to initiate and conduct projects that engage a number of students in research, and REU Supplements are included as a component of proposals for new or renewal NSF grants or cooperative agreements or may be requested for ongoing NSF-funded research projects.
Through these experiences, the REU program seeks to expand student participation in all kinds of research – both disciplinary and interdisciplinary—encompassing efforts by individual investigators, groups, centers, national facilities, and others. REU participation is also a means by which NSF seeks to recruit and retain a diverse population of future computing researchers.
The Directorate of Computer and Information Science and Engineering (CISE) is seeking to evaluate the effectiveness of these CISE REU program goals through a one-time online survey of past REU participants and mentors, along with a comparison group of former participants who engaged in other research experiences and activities.
A.2. Purpose and Use of the Information
Indicate how, by whom, how frequently, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
Data will be collected via two versions of a one-time online survey through Qualtrics. One version will be for former participants in undergraduate research experiences. The other version will be for former mentors in the NSF REU program. The Center for Evaluating the Research Pipeline (CERP) at the Computing Research Association (CRA) – on behalf of NSF – is developing, managing the administration of, and conducting data analysis for the surveys. The specific research and evaluation objectives are to:
Identify the career trajectory of the REU participants since their participation in the REU program including degrees they received, institutions they attended, and their current status (e.g., employed, graduate students).
Document the structure of the REU experience that the respondents participated in. These may include the type of REU (e.g., Site, Supplement), location of REU, and timing of REU.
Describe the REU mentors’ perceptions of the REU program on the student participants and the mentors’ career development.
Examine the skills the participants gained and experiences they had during their REU participation. These may include technical skills, information on graduate school application process, and research training.
Analyze the predictors of computing career choices, specifically focusing on the experiences and participant characteristics that are associated with the participants’ interest in and ultimate selection of research careers in computing.
Compare the backgrounds, skills, education, and career pathways of NSF REU participants to a comparison group of participants who have engaged in other, non-NSF REU research activities and experiences.
Ultimately, findings from the analysis of this data collection will be used by NSF to improve the CISE NSF REU Program in order to better reach its goals of providing meaningful research opportunities to undergraduate students and, in doing so, attract a broad range of students to computing research careers. A single report will be created and delivered to NSF in late 2021. This report will become a resource for ongoing future decision-making related to the CISE REU program.
A.3. Use of Information Technology and Burden Reduction
Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.
The REU Past Participant Survey will be administered online via a unique Qualtrics survey link and is designed to be completed in 20 minutes. Potential respondents will be identified as follows: (1) NSF will provide to CERP a list of names and contact information for its former CISE REU participants and mentors; and (2) Comparison group participants with prior (but non-NSF REU) CISE research experiences will be identified from a pool of respondents who previously participated in a CERP-administered survey and gave consent to be contacted for future research. In this way, only those who are already eligible to complete the survey will be recruited.
The REU Mentor Survey will also be administered online via a unique Qualtrics survey link and is designed to be completed in 20 minutes. NSF will provide a list of names and contact information for current and former CISE REU award recipients (i.e., mentors). As such, for all three types of survey respondents, only those who are already eligible for the study will be recruited for participation. All contacts for initial recruitment and reminders will be made electronically. (See Section B.2 for additional details on recruitment procedures.)
Both of the online surveys will include programmed skip patterns so that respondents are only asked questions relevant to them and their experiences. In order to enhance the convenience of the survey completion, the survey will allow respondents to skip items, pause, and continue completing the survey, as needed.
A.4. Efforts to Identify Duplication and Use of Similar Information
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purpose described in item 2 above.
To date, although there have been numerous evaluations of individual NSF CISE REU programs and groups of programs operating within the same program year, there has been no comprehensive evaluation of the full NSF CISE REU program and its participants that: (1) includes measurement of participants’ longer term outcomes, such as degrees received and career choices; (2) examines how NSF CISE REU features that vary across programs are associated with program outcomes; and (3) includes a comparison group with exposure to other non-NSF CISE REU experiences that allows the CISE NSF REU program to “benchmark” its outcomes against other programs and experiences. Moreover, in smaller evaluations of NSF REUs, small sample sizes have meant that it has not been possible to examine whether experiences and impacts of NSF REUs are the same for different subgroups of students, such as those who have traditionally been underrepresented in computing.
The current study is the first to be able to address all of these needs through its study design, comprehensive survey assessment, and targeted recruitment of participants that includes NSF CISE REU participants and mentors from a variety of sites and years, as well as comparison participants who will have had diverse alternate research experiences.
A.5. Impacts on Small Businesses or Other Small Entities
If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.
The data collection involved does not impact small businesses or other small entities.
A.6. Consequences of Collecting the Information Less Frequently
Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
If the data are not collected, NSF will not have information that may help it to: (1) understand how NSF CISE REUs impact participating students; (2) identify what features of NSF CISE REUs are associated with better student outcomes, such as skill development or retention in computing research; (3) pinpoint how the benefits of NSF CISE REU program experiences compare to those of other non-NSF research experiences; and (4) understand whether experiences in NSF REUs are similar for students from a variety of backgrounds.
The survey will only be administered once; thus, the question about conducting the survey less frequently is not applicable.
A.7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5
Explain any special circumstances that would cause an information collection to be conducted in a manner:
Requiring respondents to report information to the agency more often than quarterly
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it
Requiring respondents to submit more than an original and two copies of any document
Requiring respondents to retain records other than health, medical, government contract, grant-in-aid, or tax records for more than 3 years
In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use
Requiring respondents to submit proprietary trade secret or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
Of the items noted above, there are no special circumstances for the collection of data for the proposed study. Data collection will be conducted according to the guidelines in 5.CFR 1320.5.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency
If applicable, provide a copy and identify the date and page number of publications in the Federal Register of the agency’s notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.
Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on April 12, 2021 (Vol. 86, No. 68, page 19031). No public comments were received.
Consultations Outside of the Agency
There will be no consultations outside of the Agency. Acting on behalf of NSF as Agency contractors, the Computing Research Association’s Center for Evaluating the Research Pipeline (CERP) is designing, administering, and analyzing data from the two surveys. CERP will also be identifying an appropriate list of comparison participants from a previous CERP survey administration to be included in the data collection for the REU Past Participants Survey.
A.9. Explanation of Any Payment or Gift to Respondents
Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
To enhance survey response rates, all survey respondents will be offered the opportunity to enter into a raffle to win one of 200 $10 electronic gift cards.
A.10. Assurance of Confidentiality Provided to Respondents
Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
Before agreeing to participate, all respondents will have an opportunity to review the purpose, procedures, and confidentiality measures that will be implemented in the study via an IRB-approved consent form and process. Respondents will be given information about CERP, its contact information, and its role in conducting the survey on behalf of NSF. In the consent process, potential respondents will be informed of the following:
That their responses will be kept confidential.
That they can stop participating at any time.
All reports generated from the study will not identify individual respondents.
The data collected will include email addresses; however, this information will be removed from survey responses at the completion of data collection, and CERP will create a unique, anonymous identifier for their survey records.
Respondents can request that any data collected as part of their participation be removed completely from CERP records.
The unique identifier will be stored with their identifying information, separately from the rest of their data, using CERP’s encrypted cloud storage servers.
Access to the data in the encrypted cloud storage is limited to the senior research team.
The survey will be administered online using the Qualtrics platform, which describes its security credentials as being the “gold-standard” of US government security compliance, with ISO 27001, 27012, and 27018 certifications as well as FedRAMP authorization. An additional reminder that respondents’ data will remain confidential will also be included in the introduction of the survey.
A.11. Justification for Sensitive Questions
Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
Primary topics in the REU Past Participant Survey and the REU Mentor Survey include non-sensitive questions related to academic background, research experiences, employment history and current employment, and respondents’ attitudes and beliefs about their own or their mentees’ experiences in research. Both surveys also invite respondents to provide demographic and background information. There are no survey questions that are anticipated to be of a sensitive nature.
All respondents in both surveys will be informed in IRB-approved consent forms and at the beginning of the survey that they can skip any questions they do not wish to answer. As part of the consent process, they will be informed that all of their responses will be kept private and that personally identifiable information will be removed from survey data when data collection has been completed. They will also be told that they can request permanent removal of all or parts of the data they provide, without penalty.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
Provide estimates of the hour burden of the collection of information. The statement should:
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour-burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.
Each one of the one-time, online surveys (REU Past Participant Survey and REU Mentor Survey) are designed to be completed in 20 minutes or less. The estimated number of individuals (past participants, mentors, and comparison group participants) who will be receiving one of these two surveys is 27,300. Based on an approximate response rate of 30%, there will be an estimated 8,190 respondents when the data collection is completed. Based on 8,190 estimated responses and 20 minutes per respondent, the estimate for this data collection is 2,730 burden hours.
Because there is just one survey administration, the annual cost to respondents is the same as the total cost. Although we do not precisely know the occupations (and thus the wage rates) of the respondents we will recruit, we have calculated the cost for the collection of information as follows:
All potential survey respondents have engaged in or led research experiences during their undergraduate careers. A central goal of such programs is to provide participants who are considering research careers with an experience that can help them decide whether they enjoy and are well-suited for such a career. As such, the most common profession among the potential survey respondents who are past participants is likely to be a computing research professional. Respondents to the REU Mentor Survey are also likely to be computing research professionals.
The Bureau of Labor Statistics publishes wage information for workers in the category of “Computer and Information Research Scientists.” The median annual wage of this group was $126,830 in May 2020.1 Assuming a 40-hour workweek over the course of 52 weeks annually, the hourly wage for this occupation is approximately $61.00 per hour.
By multiplying the hourly wage by the number of survey hours, the overall cost to survey respondents would be approximately $166,530, or $20.33 per respondent.
A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14). The cost estimates should be split into two components: a) a total capital and start-up cost component annualized over its expected useful life, and b) a total operation and maintenance and purchase of services component.
The collection of this information does not add any additional costs to respondents or record keepers other than the hour burden costs described in Sections A.12 and A.14.
A.14. Annualized Cost to Federal Government
Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.
The estimated annualized (total) cost to the Federal Government for conducting the study is $19,160. Since this data collection will occur only once in the life of this contract and all one-off costs are associated with this single data collection point, the annualized cost is equal to the total cost of this data collection and reporting instance. This total includes costs associated with the study design; survey development, programming, and administration (including incentives and cost of survey software), analysis, reporting, and presentation of the results.
A.15. Explanation for Program Changes or Adjustments
Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.
Not applicable. This is a new submission.
A.16. Plans for Tabulation and Publication and Project Time Schedule
For collections of information whose results are planned to be published, outline plans for tabulation and publication.
Data from the REU Past Participant Survey and the REU Mentor Survey are expected to be collected over a one- to two-month period in Fall 2021. After data collection closes, CERP will spend two months preparing, analyzing, and summarizing the data, including: data cleaning and new variable construction, computing survey response rates, running descriptive statistics of the responding participants, providing a summary of survey item responses, and answering each the research and evaluation objectives identified in Section A.2 (“Purpose and Use of the Information”) and reproduced below:
Identify the career trajectory of the REU participants since their participation in the REU program including degrees they received, institutions they attended, and their current status (e.g., employed, graduate students).
Document the structure of the REU experience that the respondents participated in. These may include the type of REU (e.g., Site, Supplement), location of REU, and timing of REU.
Describe the REU mentors’ perceptions of the REU program on the student participants and the mentors’ career development.
Examine the skills the participants gained and experiences they had during their REU participation. These may include technical skills, information on graduate school application process, and research training.
Analyze the predictors of computing career choices, specifically focusing on the experiences and participant characteristics that are associated with the participants’ interest in and ultimate selection of research careers in computing.
Compare the backgrounds, skills, education, and career pathways of NSF REU participants to a comparison group of participants who have engaged in other, non-NSF REU research activities and experiences.
Survey data will be analyzed using SPSS and R statistical software packages. Analyses will be summarized with a combination of text, tables, and graphical representations. Objectives 1 – 4 will be answered through summarized responses from CISE REU participants and mentors. Objective 5 will be answered by comparing key outcomes for different subsets of participants, based on background/demographic characteristics (e.g., men versus women), or by REU features (e.g., Site versus Supplement REU types). Objective 6 will be answered by comparing outcomes for past NSF CISE REU participants and comparison participants who have had other non-NSF REU experiences.
A draft report of the results will be submitted to NSF for feedback in late Fall, with approximately two weeks allotted for submitting feedback and questions. CERP will respond to the feedback and revise the report for a period of approximately two weeks. CERP will submit the final report to NSF near the end of 2021.
REU Past Participant Survey Project Timeline
Activity |
Expected Activity Period |
Conduct survey |
September – October 2021 |
Cleaning, analysis of survey data |
October – November 2021 |
Draft report to NSF |
November 2021 |
Final report to NSF |
December 2021 |
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
Not applicable. The expiration date for OMB approval will be displayed on all forms/questionnaires associated with this information collection.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act.”
Not applicable. There are no exceptions to the certification statement.
1 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Computer and Information Research Scientists,
at https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm (visited June 10, 2021).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Marietta Bowman |
| File Modified | 0000-00-00 |
| File Created | 2021-08-05 |