2. CISE_REU_SitesSupplements_Evaluation_OMB Statement A_FINAL

Supporting STatement PART A

A.1. Circumstances that Make Data Collection Necessary

Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The National Science Foundation (NSF) seeks approval for the collection of research and evaluation data through the project Computer and Information Science and Engineering (CISE) Research Experiences for Undergraduates (REU) Sites and Supplements Evaluation.

Every year the National Science Foundation (NSF) funds hundreds of Research Experience for Undergraduates (REU) activities through its REU program. The Directorate of Computer and Information Science and Engineering (CISE) is seeking to evaluate the effectiveness of the CISE REU program.

The REU program provides undergraduate students at US higher education institutions with opportunities to work with faculty on a research project. They can take the form of REU Sites or REU Supplements. REU Sites are based on independent proposals to initiate and conduct projects that engage a number of students in research. REU Supplements are included as a component of proposals for new or renewal NSF grants or cooperative agreements, or they may be requested for ongoing NSF-funded research projects.

By offering this opportunity to undergraduate students, the REU program seeks to expand student participation in all kinds of research — both disciplinary and interdisciplinary — encompassing efforts by individual investigators, groups, centers, national facilities, and others. The REU experience integrates research and education to attract a diverse pool of talented students into careers in science and engineering, including teaching and education research related to science and engineering.

To date, CISE REU Principal Investigators (PIs) who want to understand the impact of their individual REU programs have conducted their own evaluations or have used the participant application and measurement materials provided by the CISE REU Toolkit, an online resource maintained by Dr. Audrey Rorrer of the University of North Carolina Charlotte. (The Toolkit is being phased out after 2021.) However, there have been limited opportunities to gather data that provide a more comprehensive understanding of the impact of NSF CISE REUs as a whole, across a variety of Site and Supplement implementations. There is also little available data to compare NSF CISE REU participant outcomes to those of students who have other undergraduate research experiences or no undergraduate research experience. Additionally, at the individual REU level, PIs have been unable to see how the experiences and outcomes of participants in their own REU implementation compare to those of participants in similar REUs nationwide. The current project will address these information needs.

A.2. Purpose and Use of the Information

Indicate how, by whom, how frequently, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The Center for Evaluating the Research Pipeline (CERP) at the Computing Research Association (CRA) – on behalf of NSF – will collect, analyze, and provide reports on data to inform the project’s research and evaluation questions, which include the following:

1. Who are the students reached through the NSF REU Program, and how do they compare to students participating in other types of research experiences and to students in the broader CISE community?

2. How do CISE REU Sites and REU Supplements differ from other research experiences (e.g., other REUs, internships, and independent research projects)?

3. To what extent are the goals of the NSF REU Program being met by the individual projects within the program, including recruitment and retention of students in science and engineering fields and increasing diversity in these fields?

4. In what ways does participation in REU Sites, REU Supplements, internships, and/or other independent research experiences impact student attitudes and pathways to CISE careers and other research experiences?

5. In what ways does participation in the REU Sites and REU Supplements impact recruitment and retention of students who are underrepresented in computing?

There will be three primary users for the information collected in the project, including (1) NSF stakeholders; (2) NSF REU CISE Program PIs; and (3) the broader CISE community. For each audience, results will be shared via written reports. The information provided to each user type will differ somewhat in focus and is further described below.

NSF Stakeholders

Yearly reporting and a final project report to NSF will include information pertaining to the full set of research and evaluation questions described above. Reports will provide NSF with the following information:

1. A full profile of the demographic and background characteristics of participants in the NSF CISE REU program, including both REU Sites and REU Supplements. These students’ profiles will be compared to those of students with other, non-NSF REU experiences and students without research experiences.

2. A detailed description of the features and components that different REU projects offer to participants, as compared with non-NSF research experiences.

3. An analysis of how successful each individual REU is in achieving key outcomes of the REU program, overall and relative to other NSF CISE REU programs. This will include both an analysis of the immediate impact of the program as well as a longer-term (one year post-program) analysis of program outcomes.

4. An analysis of what types of undergraduate research experiences (and what components of undergraduate research programs more generally) are most strongly correlated with sustained (one year post-program) changes in student attitudes, skills and knowledge, and intentions to pursue additional CISE research and/or career pathways.

5. An analysis of the features of NSF REU programs that are most strongly associated with the recruitment and retention of students who are underrepresented in computing.

The findings from this data collection will be used by NSF to understand and improve the impact of the CISE REU program, including increasing recruitment and retention in science and engineering and promoting a diverse group of computing/STEM careers.

NSF CISE REU Program PIs

Reports will also be provided annually to PIs running individual REU projects who are participating in the evaluation. Reports will provide PIs with the following information:

1. A profile of the demographic and background characteristics of students participating in their REU.

2. A summary of how their REU participants changed over the course of their REU, comparing pre-program and post-program attitudes, self-assessed skills and knowledge, and intentions to pursue additional CISE research and/or career pathways.

3. A summary of how their REU participants perceived the REU.

4. A comparison of their own REU participants versus participants in other NSF CISE REUs at similar institutions, in each of the areas described above.

The results of this report will be used by the individual REU PIs to understand their own participants and outcomes, overall and relative to other NSF CISE REUs. The PIs can use this information to identify and address any needed changes in their recruitment of participants and/or how they run their REU program.

The CISE Community

Information will also be shared via a project report for the broader CISE community. It will include the following:

1. An aggregated summary of changes from pre-program to post-program, across all NSF CISE REU program participants.

2. An aggregated summary of how CISE REU participants compare to students with other research experiences, both initially (prior to participation) and one year after their research experience, in terms of their demographic and background characteristics, their intentions to pursue additional CISE research and/or career pathways, and their recruitment of students from underrepresented groups in computing.

3. A set of key findings, recommendations, and lessons learned from the project.

The purpose of sharing this information with the broader CISE community is to publicize any key findings that may promote the community’s adoption of research-supported best practices for engaging undergraduate students – and underrepresented student populations specifically – in research, regardless of whether those experiences are NSF-sponsored.

A.3. Use of Information Technology and Burden Reduction

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.

All project surveys will be administered online via a unique Qualtrics survey link. All surveys are designed to be completed in 20 minutes or less, and they include programmed skip patterns so that respondents are only asked questions relevant to them and their experiences. In order to enhance the convenience of the survey completion, the survey will allow respondents to skip items, pause, and continue completing the survey at a later time, as needed.

A.4. Efforts to Identify Duplication and Use of Similar Information

Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purpose described in item 2 above.

Related projects that collect data relating to NSF REU program evaluations are described below. A summary of these projects and their features is shown in Table 1.

CISE REU Toolkit:

Historically, one resource for PIs seeking evaluation data for their site has been the NSF CISE REU Toolkit resources described in Section A.1. Past users of the Toolkit received a report describing their site applicants and their participants’ pre and post program survey responses, but the reports had some methodical and informational limitations. For example, individual participants’ pre- and post-program surveys were never matched, PIs did not receive any data on their performance relative to similar REUs or students with other research experience or no research experience, and there was no measurement of longer-term changes. Additionally, the evaluation services offered through this resource sunsetted in 2021 and will no longer be maintained by Dr. Rorrer; it is intended to be replaced by the current data collection.

Other individually conducted REU evaluations:

Individual NSF REU PIs can conduct their own evaluations of their programs, which can take many forms, depending on their goals and information needs. The most significant limitation of these evaluations is that because they are designed and conducted at the individual REU level, they do not use metrics or protocols that would be commonly defined and used across a broad set of REUs. Thus, their ability to evaluate their performance against other NSF and non-NSF research experiences is lacking. Moreover, individually created and administered program evaluations do not allow NSF to examine overall program effectiveness and conduct comparative analysis of different REU programs.

CISE REU Past Participant Survey:

The CISE REU Past Participant Survey 2021 (OMB Clearance Number 3145-0265), another current NSF REU data collection project, will gather retrospective data on the experiences of REU participants and mentors who took part in REUs since 2013, along with data from past participants who had other non-NSF research experiences. This is a valuable complement to the current project, in that it collects long-term data from former REU participants, with a comparison group of participants. However, it does not duplicate the data collected in the current study. The Past Participant Survey will not be collecting real-time data on current REU programs, nor will it be collecting pre- and post-program data to measure changes in participants after REU participation. It also will not be collecting data at the individual REU project level. The current data collection will be filling in those gaps.

ETAP Application and Exit Survey:

Currently, NSF uses the ETAP (Education and Training Application) for collecting some data on NSF REU applicants, PIs, and participants. Use of the ETAP system will be required for all NSF Sites applying for new or renewal awards, starting in 2021. As a concurrently operating data collection effort, the ETAP has some overlap with the current project in its collection of NSF applicant/participant demographic and background information and REU PI and site information, although it will only be collecting the data from NSF REU Sites (and not Supplements), and only those awards that are new or renewal awards as of 2021. In addition, the ETAP collects some post-only REU outcome data from participants (again, within the subset of REUs described previously). To minimize duplication of data collection, project staff has met with ETAP staff to discuss opportunities to access ETAP data, and we have aligned our survey measures with ETAP’s where possible.

Table 1. Features of Data Collection Projects Relating to REU Evaluation

Data Collection	Matched pre/post survey methodology	Longer term follow-up	Comparison to similar REUs	Comparison to non-NSF REUs and no research
CISE REU Toolkit (sunsetted)	No matching
Individually conducted REU evaluations	Varies	Varies
CISE REU Past Participants Survey		X	No individual project analysis	X
ETAP Application and Exit Survey	Post only		Applicants/ participants
CISE REU Sites and Supplements Evaluation	X	X	X	X

No single existing project – or combination of projects – can answer the research and evaluation questions that are being addressed with the current data collection. The NSF CISE REU Sites and Supplements Evaluation will simultaneously do the following: (1) collect pre, post, and follow-up data across dozens of currently operating REU implementations and institutions; (2) examine how NSF CISE REU features that vary across currently operating programs are associated with program outcomes; (3) provide individual REUs with data that compare their program outcomes with those of REUs in similar institutions; and (4) include measurement of comparison groups of current students without research experience and current students with non-NSF REU experiences, which will allow the CISE NSF REU program to “benchmark” the outcomes of its currently funded REUs against other programs and experiences via a real-time, non-retrospective measurement design. As allowed, the project staff will collaborate with the ETAP project staff to get access to existing REU data and align measures to ensure that no duplicate data are collected.

A.5. Impacts on Small Businesses or Other Small Entities

If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

The data collection may impact small businesses that have previously been conducting evaluations for REUs that choose to participate in the current collection. Although such impacts are a potential consequence of this information collection, we encourage PIs of those projects to continue working with other evaluators for any of their REU information needs that fall outside of the scope of work for this effort.

A.6. Consequences of Collecting the Information Less Frequently

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

Need for this collection

NSF needs to collect the data proposed in this project to (1) better understand the extent to which the REU program is achieving its goals; and (2) have comparative participation and outcome data on REU programs that are currently funded. If the data are not collected, NSF will not have information that may help it to: (1) understand the demographic and background characteristics of the students who are currently being funded by NSF CISE REUs, including how they compare to students participating in other types of research experiences and to students in the broader CISE community; (2) understand how NSF CISE REU Sites and REU Supplements differ from other research experiences, both in terms of their structure and content as well as the outcomes they achieve; (3) determine whether and to what extent individual REU projects are meetings NSF’s goals for the program; (4) learn how participation in REU Sites, REU Supplements, internships, and/or other independent research experiences impact student attitudes and pathways to CISE careers and other research experiences; and (5) learn whether the NSF REU program is helping to diversify the research pipeline by increasing recruitment and retention of students who are underrepresented in computing.

Need for proposed frequency of collection

The project design includes measurement of NSF REU participants at three timepoints: prior to REU participation, immediately after participation, and approximately one year after completing participation. Each of these measurements is essential for answering the research and evaluation questions. Pre-program surveys will gather participant information and establish participant baseline attitudes, skills, knowledge, and intentions to continue in a CISE career pathway. Immediate post-program surveys will capture changes in key metrics during the course of the REU program. One-year follow-ups are required to measure the extent to which any gains seen post-program are maintained over time.

The project includes measurement of a comparison group of students who are not NSF REU participants at two timepoints: prior to any research participation, and at a follow-up timepoint that corresponds to the follow-up measurement of the REU participant respondents. Initial surveys to gather baseline data will be conducted with the comparison group during the same period when pre-program surveys are being completed by the REU Site participant subset (late Spring 2022/23), and follow-up surveys will be conducted during the same period when follow-up surveys are being conducted with the REU participants (late Summer 2023/24). These measurements are required to mirror the data collection schedule of the REU participants, in order to track the extent to which non-NSF REU students also change on key measures over the same time period, relative to the REU participants. This quasi-experimental design will help to determine the unique impact of REUs relative to other undergraduate research experiences and no undergraduate research experience.

Finally, the project includes forms completed by NSF CISE REU Principal Investigators at two timepoints. The Time 1 form will gather basic information about the REU prior to the commencement of the evaluation, including PI names and contact information and REU project information. The Time 2 form will be distributed at the end of the REU term, at the same time participants are completing their post-surveys. It will include questions confirming the number of student participants and measuring PI reports of the REU’s content and perceived impact. Each form is required to accurately track the REUs being evaluated and to document the participants in and features of each REU. Each form will take approximately 10 minutes to complete.

There are no know technical or legal obstacles to reducing data collection burden.

A.7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5

Explain any special circumstances that would cause an information collection to be conducted in a manner:

• Requiring respondents to report information to the agency more often than quarterly

• Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it

• Requiring respondents to submit more than an original and two copies of any document

• Requiring respondents to retain records other than health, medical, government contract, grant-in-aid, or tax records for more than 3 years

• In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study

• Requiring the use of a statistical data classification that has not been reviewed and approved by OMB

• That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use

• Requiring respondents to submit proprietary trade secret or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

Of the items noted above, REU students participating in this study will be asked to complete a pre-test survey and then a post-test survey approximately 10 weeks later when their REU ends, which may fall within the same quarter. REU PIs who choose to have your REUs evaluated will also be asked to complete a Time 1 and Time 2 PI REU Information Form approximately 10 weeks apart, which may fall within the same quarter. However, participation is completely voluntary and is not required of any survey respondents.

There are no other special circumstances for the collection of data for the study. Data collection will be conducted according to the guidelines in 5.CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

If applicable, provide a copy and identify the date and page number of publications in the Federal Register of the agency’s notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on September 8, 2021 (Vol. 86, No. 171, page 50378). No public comments were received.

Consultations Outside of the Agency

There will be no consultations outside of the Agency. Acting on behalf of NSF as Agency contractors, the Computing Research Association’s Center for Evaluating the Research Pipeline (CERP) is designing, administering, and analyzing data from the project surveys. CERP will also be identifying an appropriate list of comparison participants from a previous CERP survey administration to be included in the data collection.

CERP will be collaborating with another Agency contractor acting on behalf of NSF, Mathematica, to learn more about the ETAP administration and data. Mathematica has designed and implemented the ETAP system being used by NSF for REU applications and an Exit Survey.

A.9. Explanation of Any Payment or Gift to Respondents

Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Comparison group respondents will be offered a chance to win one of 150 $10 Amazon gift cards for their participation in the initial survey and again in the one-year follow-up survey. NSF CISE REU participants will not be offered incentives for their initial (pre-program) and Time 2 (post-program) surveys, since their project PIs will be encouraging their participation in their individual program evaluation. The NSF REU participants will be offered a chance to win one of 150 $10 Amazon gift cards for their participation in the one-year follow-up survey.

A.10. Assurance of Confidentiality Provided to Respondents

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

Before agreeing to participate, all respondents will have an opportunity to review the purpose, procedures, and confidentiality measures that will be implemented in the study via an IRB-approved consent form and process. Respondents will be given information about CERP, its contact information, and its role in conducting the survey on behalf of NSF. In the consent process, potential respondents will be informed of the following:

• That their responses will be kept confidential.

• That they can stop participating at any time.

• All reports generated from the study will not identify individual respondents.

• Any identifying information collected in the study will be removed from survey data files and stored separately from the survey data. CERP will create a unique, anonymous identifier for each respondent and will use the identifier to distribute and/or link the surveys they complete over the course of the study.

• Respondents can request that any data collected as part of their participation be removed completely from CERP records.

• The unique identifier will be stored with their identifying information, separately from the rest of their data, using CERP’s encrypted cloud storage servers.

• Access to the data in the encrypted cloud storage is limited to the senior research team.

The surveys will be administered online using the Qualtrics platform, which describes its security credentials as being the “gold-standard” of US government security compliance, with ISO 27001, 27012, and 27018 certifications as well as FedRAMP authorization. An additional reminder that respondents’ data will remain confidential will also be included in the introduction of the survey.

A.11. Justification for Sensitive Questions

Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

There are no survey questions that are anticipated to be of a sensitive nature. Primary topics in the student surveys include non-sensitive questions related to academic background, research experiences, employment history and current employment, and respondents’ attitudes and beliefs about their experiences in research. Student surveys will also invite respondents to provide demographic and background information.

Primary topics in the PI REU Information Forms at Time 1 and Time 2 focus on gathering information about the REUs being evaluated, including PI contact information, REU project details, and perceptions of student learnings and impacts after participating in the REU.

All respondents will be informed in IRB-approved consent forms and at the beginning of their first survey that they can skip any questions they do not wish to answer. As part of the consent process, they will be informed that all of their responses will be kept private and that personally identifiable information will be removed from survey data when data collection has been completed. They will also be told that they can request permanent removal of all or parts of the data they provide, without penalty.

A.12. Estimates of Hour Burden Including Annualized Hourly Costs

Provide estimates of the hour burden of the collection of information. The statement should:

• Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour-burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

• Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.

Hour Burden of Data Collection, by Respondent Type

Information will be collected from three respondent types: (1) REU participants, (2) comparison group participants, and (3) REU Principal Investigators (PIs).

1. REU Participants

Each NSF CISE REU participant will be asked to complete three surveys: (1) a pre-test before they begin their REU project; (2) a post-test, after their REU ends; and (3) a follow-up survey, approximately one year after their participation ends. As such, each REU participant will complete their pre- and post-survey in their first year of participation. They will complete their follow-up survey one year after their participation in the REU. Within the data collection timeline for this project, there will be two cohorts of REU participants completing a full data collection cycle (pre-test, post-test, and follow-up), plus a third cohort of Year 3 summer REU participants who will only complete a pre-test and a post-test, but no follow-up survey.

Based on current estimates of NSF-funded CISE REU Site and Supplement projects and current expressed interest from REU PI applicants in using CERP evaluation, we expect approximately 1,000 REU participants per year to complete pre-surveys. (See Statement B, Item B1 for more detail on these estimates.) Thus, across the two full-year and one partial (one-third) year cohorts, it is estimated that a total of 2,333 REU respondents will complete a 20-minute pre-survey in the project. Of these 2,333 REU participant respondents, we expect that approximately 70% of the original REU respondents will complete a 20-minute post-survey. For the follow-up survey (occurring in each participant’s second year of project participation), only the REU participants from the first two years of the data collection would be able to complete the survey within the data collection time range of the study. It is expected that approximately 50% of the original REU respondents will complete a 20-minute one-year follow-up survey. This would result in a total of 4,966 20-minute surveys completed by REU respondents, for a total of 1,655.33 burden hours for this subset of respondents.

The table below provides the number of expected respondents, frequency of response, and annual hour burden for each of the three cohorts of REU respondents in the project.¹

Table 2. Estimated Survey Burden – REU Participants

REU Participant Survey Type	Timing and frequency	COHORT 1 Expected number of responses	COHORT 2 Expected number of responses	COHORT 3 Expected number of responses (Partial year)	Time required	Annual participant burden
Pre-survey	R completes in first year, prior to REU start	1,000	1,000	333	20 mins each	778 hrs (pre) + 544 hrs (post) 1,322.00 Total
Post-survey (70% of original N)	R completes in first year, after REU ends	700	700	233	20 mins each
Follow-up survey (50% of original N)	R completed in second year, one year after REU ends	500	500	Not conducted	20 mins each	333 hrs
Total	--	2,200	2,200	566	20 mins	1655 hrs

2. Comparison Group Participants

Each comparison group participant will be asked to complete two surveys: (1) a pre-test; and (2) a follow-up survey. Each comparison group participant will complete their pre- survey in their first year of participation, and they will complete their follow-up survey near the beginning of their second year of participation. Within the data collection timeline for this project, there will be two cohorts of comparison group participants completing a full data collection cycle (pre-test and follow-up).

Based on CERP data from previous recruitment efforts with students who are similar to the targeted comparison group, we expect to recruit approximately 1,000 comparison group respondents to complete a pre-survey in each of the two years. (See Statement B, Item B1 for more detail on these estimates.) Thus, across the two cohorts, it is expected that a total of 2,000 REU respondents will complete a 20-minute pre-survey in the project. Of these respondents, we expect that approximately 50% of the original comparison group respondents (N = 1,000) will also complete a 20-minute follow-up survey. This would result in a total of 3,000 20-minute surveys completed by comparison group respondents, for a total of 1,000 burden hours for this subset of respondents

The table below provides the number of expected respondents, frequency of response, and annual hour burden for the two cohorts of comparison group respondents in the project.

Table 3. Estimated Survey Burden – Comparison Group Participants

Comparison Group Participant Survey Type	Timing and frequency	COHORT 1 Expected number of responses	COHORT 2 Expected number of responses	Time required	Annual participant burden
Pre-survey	R completes in first year of study participation	1,000	1,000	20 mins each	667 hrs
Follow-up survey (50% of original N)	R completed in second year of study participation	500	500	20 mins each	333 hrs
Total	--	1,500	1,500	20 mins	1,000 hrs

3. NSF CISE REU Principal Investigators (PIs)

To gather basic information about each REU being evaluated, each project PI will be asked to complete two short information forms. The first information form will gather basic information about the REU project and will be administered prior to pre-testing the REU participants. The second information form will be administered at the end of the REU period (approximately 10 weeks later) and will gather information about the status of REU participants and PI reports of the REU’s content and perceptions of student experiences. Similar to the cohorts described for the REU participants, within the data collection timeline for this project, there will be two full-year cohorts of REU PIs completing forms, plus a third cohort of Year 3 REU PIs running REU projects that occur through August 2024.

Based on current estimates of NSF-funded CISE REU Site and Supplement projects and current expressed interest from REU PI applicants in using CERP evaluation, we expect full sets of PI REU Information Forms (Time 1 and Time 2) from approximately 100 REU PIs per year for each of the first two years. In the third partial year, we expect Time 1 and Time 2 PI REU Information Forms from approximately 33 PIs. (See Statement B, Item B1 for more detail on these estimates.) Thus, across the full project period, it is estimated that a total of 233 NSF CISE REU PIs will complete both a 10-minute Time 1 PI REU Information Form and a 10-minute Time 2 PI REU Information Form. This would result in a total of 466 10-minute PI REU Information Forms completed by REU respondents, for a total of 77.67 burden hours for this subset of respondents.

The table below provides the number of expected PI respondents, frequency of response, and annual hour burden for each of the cohorts of REU PIs in the project.

Table 4. Estimated Survey Burden – NSF CISE REU Principal Investigators

REU PI Form	Timing and frequency	COHORT 1 Expected number of responses	COHORT 2 Expected number of responses	COHORT 3 Expected number of responses (Partial year)	Time required	Annual participant burden
T1 Information Form	PI completes prior to REU start	100	100	33	10 mins each	39 hrs (T1) 39 hrs (T2) 78 hrs total
T2 Information Form	PI completes after REU ends	100	100	33	10 mins each

Annualized Cost to Respondents, by Respondent Type

Annualized costs are presented below for each respondent type and overall.

1. REU Participants

REU participants will be students while participating in pre-test and post-test surveys. Most will also be students at the time they complete their follow-up surveys. According to data from the Economic Policy Institute², the average 2020 hourly wage of workers with “some college” was $22.51. The estimated number of burden hours for these participants in their first year of survey participation is 1,322. At the average hourly rate for this population, the annual cost for Year 1 and Year 2 survey participation is $12,248.

2. Comparison Group Participants

Comparison group participants will be students while participating in their pre-test. Most will also be students at the time they complete their follow-up surveys. The estimated number of burden hours for these participants survey participation is 667 in Year 1. At the average hourly rate of $22.51 for this population, the annual cost for their Year 1 and Year 2 participation is $17,498.

3. NSF CISE REU Principal Investigators

The Bureau of Labor Statistics publishes wage information for workers in the category of “Computer and Information Research Scientists,” which would include REU Principal Investigators. The median annual wage of this group was $126,830 in May 2020.³ Assuming a 40-hour workweek over the course of 52 weeks annually, the hourly wage for this occupation is approximately $61.00 per hour. The estimated number of burden hours for these PIs is 78. Completion of both their Time 1 and Time 2 PI REU Information Forms would occur in the PIs first year of project participation; thus, all costs for this survey burden would be accrued in their Year 1 participation and would total $4,706.

4. Total Annualized Cost, by Year

Summing all REU participants, comparison group participants, and REU PIs over the whole study, the cost for their Year 1 and Year 2 survey burden hours, and the follow-up surveys, is $49,452.

A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14). The cost estimates should be split into two components: a) a total capital and start-up cost component annualized over its expected useful life, and b) a total operation and maintenance and purchase of services component.

The collection of this information does not add any additional costs to respondents or record keepers other than the hour burden costs described in Sections A.12 and A.14.

A.14. Annualized Cost to Federal Government

Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.

The estimated annualized (total) cost to the Federal Government for conducting the study is $208,428. This cost includes the staff salaries based on the expected number of hours project staff will be spending on the project, the cost of survey software, and survey incentives over the course of the project ($729,497; over 42 months). The average annualized amount is calculated by taking a 12-month sum assuming an equal distribution of cost across all months.

A.15. Explanation for Program Changes or Adjustments

Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.

Not applicable. This is a new submission.

A.16. Plans for Tabulation and Publication and Project Time Schedule

For collections of information whose results are planned to be published, outline plans for tabulation and publication.

Year 1 Plans

An overview of plans for Year 1 data collection, data tabulation, and reporting are shown in Table 5 at the end of this section.

Data Collection: The start of the first year of data collection for the study will coincide with the Summer 2022 REU programs; as such, data collection for Year 1 will begin in late Spring 2022 and will include Time 1 and Time 2 PI REU Information Form completion and pre- and post-program survey completion by participants in REU programs during Summer 2022, Fall 2022, and Spring 2023. (The table shows continuous data collection during Year 1 because start and end dates for REU programs will vary.) Comparison group pre-test surveys will also be administered at the same time that REU pre-program surveys are collected, in Spring 2022.

Data tabulation: Cleaning, analysis, and tabulation of PI data and REU participants’ pre- and post-program survey data will be conducted using SPSS and R on a yearly basis after the completion of the summer REU term. In the first year, cleaning, analysis, and tabulation of participant and comparison group data will be conducted during September, October, and November of 2022.

Reporting: The first set of individual, award-level reports to PIs summarizing program participants and outcomes will be provided to PIs who ran REUs during the Summer 2022 term. These will be delivered in December 2022 and January 2023. A report to NSF that includes results across all REUs completed during Summer 2022 and comparisons to non-NSF REU students on demographics and baseline characteristics will be delivered in February or March 2023. (See Section A.2 for additional details on report contents.)

Table 5. REU Evaluation Project Timeline – First Year of Data Collection

Year 1 Data Collection Timeline (May 2022 – April 2023)

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

Yr 1 REU PI REU Information Forms and participant data collection (pre and post)

Yr 1 Comparison group pre data collection

Cleaning and analysis of Yr 1 (Summer) REU pre/ post data & comparison group pre data

Yr 1 Reporting for PIs

Yr 1 Reporting for NSF

Year 2 Plans

An overview of plans for Year 2 data collection, data tabulation, and reporting are shown in Table 6 at the end of this section.

Data Collection: Data collection for Year 2 will begin in late Spring 2023 and will include Time 1 and Time 2 PI REU Information Form completion and pre- and post-program survey completion by participants in REU programs during Summer 2023, Fall 2023, and Spring 2024. (Again, the table shows continuous data collection during Year 2 because start and end dates for REU programs will vary.) Year 2 comparison group pre-test surveys will also be administered at the same time that REU pre-program surveys are collected, in Spring 2023. In addition, follow-up surveys from the first year REU and comparison group participants will be collected in late Summer 2023.

Data tabulation: Cleaning, analysis, and tabulation of PI data and REU participants’ pre- and post-program survey data will be conducted after the completion of the summer REU term. In this second year, cleaning, analysis, and tabulation of PI and participant data will be conducted during September, October, and November of 2023 for the following data: REUs occurring in Fall 2022, Spring 2023, and Summer 2023, and Year 2 comparison group pre-tests. Cleaning, analysis and tabulation of the longer-term follow-up surveys from first year respondents will take place in February through April of 2024.

Reporting: The individual, award-level reports to PIs summarizing program participants and outcomes will be provided to PIs who ran REUs during the terms noted in the Data Tabulation section above. These will be delivered in December 2023 and January 2024. A report to NSF that includes results across all REUs during Fall 2022 through Summer 2023, along with pre-survey comparisons to non-NSF REU students on demographics and baseline characteristics, will be delivered in February or March 2024.

Table 6. REU Evaluation Project Timeline – Second Year of Data Collection

Year 2 Data Collection Timeline (May 2023 – April 2024)

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

Yr 2 REU PI REU Information Forms and participant data collection (pre and post)

Yr 2 Comparison group pre data collection

Yr 1 REU participant & comp group follow-up data collection

Cleaning and analysis of Yr 2 REU pre/post data and comparison group pre data

Cleaning and analysis of Yr 1 participant & comp group follow-up data

Yr 2 Reporting for PIs

Yr 2 Reporting for NSF

Year 3 Plans

Year 3 of the study is a partial year, running from May to August 2024, when the current contract period ends. An overview of plans for Year 3 data collection and reporting for this period are shown in Table 7 at the end of this section.

Data Collection: Data collection for Year 3 will begin in late Spring 2024 and will include Time1 and Time 2 PI REU Information Form completion and pre- and post-program survey completion only by PIs and participants in REU programs during Summer 2024 only. In addition, follow-up surveys from the second year REU and comparison group participants will be collected in late summer 2024. Data will be delivered to NSF in August 2024.

Reporting: A final, summary report to NSF that compiles all data across the entire evaluation period for both REU participants and comparison group participants will be delivered in prepared and submitted, as will a report for the broader CISE community.

Table 7. REU Evaluation Project Timeline – Third (Partial) Year of Data Collection

	Year 3 Data Collection Timeline (May 2024 –August 2024)
	May	Jun	Jul	Aug
Yr 3 REU PI REU Information Forms and participant data collection (pre and post)
Yr 2 REU participant & comp group follow-up data collection
Final NSF report development/submission
Final CISE community report development

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

Not applicable. The expiration date for OMB approval will be displayed on all forms/questionnaires associated with this information collection.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act.”

Not applicable. There are no exceptions to the certification statement.

1 In these estimates, annual hour burdens are defined from the perspective of the respondent (i.e., the survey time required of each respondent per year of participation), rather than the project calendar year.

2 EPI analysis of Current Population Survey Outgoing Rotation Group microdata. Retrieved from https://www.epi.org/data/#cpsorg on October 13,2021.

3 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Computer and Information Research Scientists,

at https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm (visited June 10, 2021).