Supporting statement PART b: Collection of Information Employing Statistical Methods
B.1. Respondent Universe and Selection Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
Overview:
On behalf of the Directorate of Computer and Information Science and Engineering (CISE) of the National Science Foundation (NSF), the Computing Research Association’s (CRA) Center for Evaluating the Research Pipeline (CERP) will be conducting the NSF CISE REU Sites and Supplements Evaluation project. The project is designed to evaluate the impact of the NSF CISE Research Experiences for Undergraduates (REU) program. The project will collect data from three groups of student survey respondents: (1) NSF CISE Site or Supplement REU participants, (2) undergraduate students who participate in other, non-NSF research experiences, and (3) undergraduate students who do not participate in research. The project will also collect information from NSF CISE REU PIs who sign up to participate in the NSF-sponsored REU program evaluation.
Description and Numerical Estimates:
NSF CISE REU Site and Supplement participants
NSF CISE REU participants will include undergraduate students who participate in REU projects in which the project’s Principal Investigator (PI) chooses to use NSF-sponsored program evaluation services. The evaluation data collection period will occur over a span of 2 1/3 years; it will begin with projects running in Summer 2022 and will continue through the Summer 2024 term. NSF records indicate that between 2018 and 2020, the number of REU participants has had a yearly average of almost 2,900 Site and Supplement CISE REU participants1. Estimating that slightly more than one third of these participants will be in REUs in which the PIs will choose to be included in the NSF-funded evaluation would result in a yearly total of approximately 1,000 REU participants for each of the first two years of data collection2, and 333 participants in the third (partial) year. In total, there would be approximately 2,333 REU participants who will be recruited for the evaluation. Samples will not be used; all of these participants will be invited to participate.
Comparison group participants who have had no research experience or who have non-NSF REU undergraduate research experiences
The NSF REU Sites and Supplements Evaluation Project will employ two comparison groups. One will be students who do not participate in undergraduate research. The other will be students who have a non-NSF undergraduate research experience.
For both groups, CERP will prepare a single list of potential comparison students using respondents to a previously administered CERP survey. This survey, called the Data Buddies Survey (DBS), is administered annually to students in computing, and it collects a range of information about respondents, including their backgrounds, research experience, and intentions to engage in research as an undergraduate. Potential comparison respondents will be identified if they meet the following criteria: (1) they gave CERP permission to contact them for future studies; (2) they are current undergraduates; and (3) they had not engaged in any formal REUs at the time that they completed the DBS.
Using data from the most recently available DBS collection, approximately 1,200 undergraduate students who were recruited from this ongoing cohort completed surveys in 2020. Of this number, approximately 20% had already engaged in formal research. Thus, there were slightly more than 1,000 undergraduates students in the cohort who had not already engaged in formal research and would thus qualify to be comparison group respondents for the current study. We expect similar numbers for the 2021 and (future) 2022 DBS cohorts, which will be used to pull comparison group participants for the current study.
Thus, across the two cohorts of comparison group data collection, the number of undergraduate DBS respondents who are expected to agree to participate in the new study is estimated to be approximately 2,000 students (1,000 in each of two years). The respondents will be assigned to the appropriate comparison group (no research experience or non-NSF REU research experience) based on their responses on their follow-up surveys, which will be administered slightly more than one year after their pre-surveys. (Any respondents who have gone on to participate in an NSF REU will not be included in a comparison group.)
Previous years of DBS data suggest that 50 – 60% of undergraduates who have not had an undergraduate research experience plan to do so during their college career; thus, with a recruitment target of 2,000 comparison participants for the pre-survey and an expected follow-up survey completion rate of 50%, we expect that there will be 1,000 complete sets of comparison participant pre- and follow-up surveys. If 50-60% of these respondents go on to have a formal research experience, this would result in 500 – 600 comparison participants with a (non-NSF) REU research experience and 400 – 500 comparison participants without any research experience.
NSF CISE REU Site and Supplement Principal Investigators (PIs)
NSF CISE REU PIs who choose to use NSF-sponsored program evaluation services will be asked to complete two short forms about their REU and participating students. Based on past NSF records and PI interest expressed so far in using NSF/CERP for evaluation, we expect that approximately 100 PI’s will participate in evaluations of their REUs for each of the first two years of data collection, and 33 PIs in the third (partial) year. In total, there would be approximately 233 REU PIs who will be sign up to participate in the evaluation, and all will be asked to provide information about their REU(s).
Expected Response Rates
REU participant respondents
Initial Survey and REU Post-Program Survey
CERP believes that REU participant respondents’ response rates for their initial surveys, will, on average, be very high. This is because the request to participate will come from their REU advisor at the beginning of their REU experience, when motivation to participate in all REU activities is especially strong. REU participant post-program response rates are also expected to be high (about 70% of those completing an initial survey), again due to the personalized appeals from their project PI to take part in the surveys.
Follow-Up Survey
As follow-up surveys will be conducted after about one year, we expect some drop-off in response rates from the initial survey respondents. Specifically, we expect about 50% of those who completed an initial survey to also complete a one-year follow-up.
For all survey administrations, CERP’s recruitment process will include personalized appeals to participate, an emphasis on the respondents’ ability to contribute to understanding and improving undergraduate research experiences, and multiple attempts to elicit participation. Efforts to maximize participation rates are further described in Section B.3. Analyses of whether there is any systematic difference in those respondents who do versus do not complete a follow-up survey will also be conducted.
Comparison group respondents
Initial Survey
Pre-survey estimates of 1,000 respondents each year for comparison group participants have been estimated based on response rates that CERP has achieved in previous survey research projects, and it reflects a response rate of approximately 30% of those eligible. For this study, we expect to see response rates that are at least as high as that, and perhaps higher, given that both recruitment efforts and incentives for this collection will be greater than for previous studies.
Follow-Up Survey
Similar to expected response rates for REU participants, as follow-up surveys will be conducted after about one year, we expect some drop-off in response rates from the initial survey respondents. Specifically, we expect about 50% of those who completed an initial survey to also complete a one-year follow-up.
As with REU participant respondents, CERP’s recruitment process will include personalized appeals to participate, an emphasis on the respondents’ ability to contribute to understanding and improving undergraduate research experiences, and multiple attempts to elicit participation. Efforts to maximize participation rates are further described in Section B.3. Analyses of whether there is any systematic difference in those respondents who do versus do not complete a follow-up survey will also be conducted.
NSF CISE REU Site and Supplement PI respondents
All PIs in this study will be choosing to participate in the REU evaluation and will be motivated to provide basic REU information to allow for accurate documentation of their REU and its participating students. As such, we expect all of them to complete the basic REU information included in the brief Time 1 PI REU Information Form. Given the same factors – that PIs are selecting into the evaluation voluntarily and they will be motivated to provide accurate data for their REU evaluations – along with the very small time burden of the Time 2 form, we also expect PI participation that is close to 100% for the Time 2 PI REU Information Form.
B.2. Procedures for the Collection of Information
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
Overview
Data collection for this project intends to assess the immediate and short-term (one-year) impacts of NSF CISE REU participation. The specific research and evaluation questions include the following:
Who are the students reached through the NSF REU Program, and how do they compare to students participating in other types of research experiences and to students in the broader CISE community?
How do CISE REU Sites and REU Supplements differ from other research experiences (e.g., other REUs, internships, and independent research projects)?
To what extent are the goals of the NSF REU Program being met by the individual projects within the program, including recruitment and retention of students in science and engineering fields and increasing diversity in these fields?
In what ways does participation in REU Sites, REU Supplements, internships, and/or other independent research experiences impact student attitudes and pathways to CISE careers and other research experiences?
In what ways does participation in the REU Sites and REU Supplements impact recruitment and retention of students who are underrepresented in computing?
Procedure
To answer the project’s core research and evaluation questions, three types of student respondents will be recruited for the survey: (1) NSF CISE REU participants; (2) a comparison group of undergraduates with other, non-NSF REU experiences; (3) a comparison group of undergraduates who do not engage in research experiences. Participating REU PIs – all of whom have chosen to receive evaluation services – will also complete two PI REU Information Forms just before and right after their REU program runs to document basic aspects of their program. Procedures for data collection are described below. In addition, an overview of the data collection instruments and timeline is included in the attached document, “NSF CISE REU Sites and Supplements Evaluation: Data Collection Overview – Tools and Timing.”
NSF CISE REU participants and PIs
All NSF-funded CISE REU PIs will be invited to use NSF-sponsored program evaluation services that are the basis for the current study. Although not mandatory, PI participation in the study will eliminate the burden on PIs to design, conduct, and fund their own evaluation of their REU. Data collection will take place among REUs in which the PI has signed up to participate in the evaluation.
Once interested PIs have officially signed up for the evaluation, CERP will coordinate with each project’s designated REU contact to complete the Time 1 and Time 2 PI REU Information Forms and administer the participant pre- and post-program surveys at the appropriate times. CERP will provide instructions and timelines to the contact, along with all recruitment scripts and online survey links for their own PI forms and the participant surveys. The process for data collection is as follows:
Approximately two weeks prior to the REU start date, CERP will send the REU PI: (1) a Qualtrics link to the Time 1 PI REU Information Form, which they will complete; (2) a standard pre-survey recruitment email for the PI’s to send to all of the students who will be participating in their REU; and (3) an anonymous Qualtrics survey link (provided by CERP) for the students to complete their pre-survey. (Samples will not be drawn). The recruitment email will include: (1) a description of the survey project and its purpose, (2) an emphasis on respondents’ ability to contribute to understanding how research experiences impact undergraduates, and (3) a request to complete three online surveys – the current one, a post-program survey, and a one-year (approximately) follow-up survey. The email will contain contact information for CERP that respondents can use to ask additional questions about the survey or their participation in it.
Approximately one week prior to the REU start date, the REU PI will send a reminder email to their REU participants that once again describes the project and its purpose and invites them to participate by completing the online Qualtrics survey. The reminder will again include contact information for CERP for any questions about the survey.
Immediately after the REU program end date, the REU PI will complete a Time 2 PI REU Information Form and will distribute a CERP-provided Qualtrics link to a post-program participant survey to their REU participants. Again, the email with the survey link will contain all of the relevant study information and will remind respondents about the importance of their participation. Two reminder emails will be sent at subsequent one-week intervals to ensure high response rates.
Approximately one year later, in late summer, all former REU participants who completed a pre-program survey will receive an email request from CERP to complete a follow-up survey. The email will remind students of their participation in the study, describe the study goals and the importance of their participation, and provide a link to the Qualtrics survey, along with contact information for CERP for any questions about the study or the survey. Once again, two reminder emails at one-week intervals will be sent to those who have not yet completed a survey.
Comparison participants (non-NSF REUs and no research experience)
As described previously, there will be two comparison groups. One will be students who do not participate in undergraduate research. The other will be students who have a non-NSF undergraduate research experience. For both groups, CERP will generate a single list of potential comparison respondents from the Data Buddies Survey (DBS), as described in Section B.1. Samples will not be drawn.
To coincide with the timing of the pre-test for the first set of REU participants in Summer 2022, the identified comparison respondents will be sent an initial recruitment email in late Spring 20223. The email will introduce CERP as the research and evaluation arm of the Computing Research Association (CRA), which has a strong reputation within the computing community. The email will include a description of CERP as the organization conducting a survey on behalf of NSF, a description of the survey project and its purpose, an emphasis on respondents’ ability to contribute to understanding how research experiences impact undergraduates, and a request to complete the current online survey and a one-year (approximately) follow-up survey. The email will contain a link to the survey in Qualtrics as well as contact information for CERP that respondents can use to ask additional questions about the survey or their participation in it.
Approximately one week later, all respondents who have not yet completed a survey will be sent a reminder email that once again describes the project and its purpose and invites them to participate by completing the online Qualtrics survey. The reminder will again include contact information for CERP for any questions about the survey.
After another week has passed, respondents who still have not completed a survey will be sent a final email with a reminder about the project and a request, with a link, to complete the online Qualtrics survey. CERP contact information will again be included.
Approximately one year later, in late summer, all comparison group participants who completed a pre-survey will receive an email request from CERP to complete a follow-up survey. The email will remind students of their participation in the study, describe the study goals and the importance of their participation, and provide a link to the Qualtrics survey, along with contact information for CERP for any questions about the study or the survey. Once again, two reminder emails at one-week intervals will be sent to those who have not yet completed a survey. Participant responses to this follow-up survey will define their inclusion in the comparison group with no research experience versus the comparison group with other, non-NSF REU research experience.
Statistical methodology for stratification and sample selection
Not applicable. There will be no stratification or sample selection for the NSF REU participants. Every REU participant that takes part in an REU in which the PI has elected to use NSF-sponsored REU evaluation services will be recruited for data collection. Similarly, every eligible comparison group participant will be recruited for data collection.
Estimation procedure
For each stage of data collection in the study, analyses will be conducted to determine whether there are systematic differences in the characteristics of those who respond to survey instruments or items, as follows below.
Respondent-level non-response analyses
To understand whether there are systematic differences in who responds to the project surveys, CERP will conduct and report on respondent-level non-response bias analysis. This analysis will vary depending on the type of respondent.
NSF CISE REU participants: For REU participant pre-test response rates, CERP expects survey completion rates to be close to 100%. This estimate is based on data from past studies, in which participants who are just beginning an REU program are highly motivated to comply with requests from project personnel. CERP will be able to calculate pre-test response rates – at the individual project level and for REUs overall -- with data provided by the REU PIs in their Time 1 PI REU Information Form, which will include the number of students participating in the REU. CERP will also examine whether there are different rates of survey participation as a function of other project-level variables that are provided by PIs, including CISE division, REU term (Summer, Fall, Spring), and REU type (Site or Supplement).
For a subset of REU Site projects that use NSF’s ETAP system to have students apply for their REU, CERP will also have access to participant-level data gathered at the REU application phase, along with a record of whether the applicant was accepted into an REU program. CERP will use this opportunity to (1) match ETAP data for students who go on to be participants to the appropriate REU pre-program survey; (2) identify REU participants in ETAP who do not complete an REU pre-program survey; and (3) compare ETAP data on the demographics and background characteristics for REU participants who did versus did not complete an REU pre-test survey, to test for non-response bias.
Analyses of respondent-level non-response bias will also be conducted with REU participant post-program surveys. After matching REU participants’ pre-program surveys to their post-program surveys, respondents who complete a full set of forms will be compared to those who complete only a pre-program survey. The two groups will be compared on participant-level and project-level characteristics. The same procedure will be used to examine respondent-level non-response bias in the follow-up survey.
Comparison group participants: Because the comparison group participants will be recruited from a set of respondents to a previous CERP survey, there will be a large amount of pre-existing data on these respondents that will allow for comparisons of those who do versus do not agree to complete the pre-survey for this study. Basic respondent-level non-response analyses on the pre-test survey will be completed by comparing gender, race/ethnicity, and other background characteristics for the recruited respondents who do versus do not participate. Analysis of the follow-up survey non-response bias will use comparison group pre-survey data to compare the demographic and background profiles of those that do versus do not complete a follow-up survey.
Item-level non-response analyses
In addition to respondent-level non-response analyses, concerns have also been raised about non-response bias at the levels of individual survey items.4 Although non-response bias analyses of all survey items are not feasible, CERP will monitor and report on survey items that are unanswered by a large percentage of respondents overall and within each of the respondent groups.
Reporting of non-response findings
A description and summary of all non-response analyses will be included in project reports to NSF.
Degree of accuracy needed for the purpose described in the justification
As noted in Section A.1 and A.2, the current study’s purpose and goals incorporate multiple levels of analysis and reporting. One purpose of the study is to provide individual REU PIs with data about the participants and outcomes of their own REU. A second, broader purpose and set of goals is to provide NSF and the CISE community with information about REU participants and outcomes as a whole, using an aggregated data set that allows for multiple types of comparisons and investigations of the correlates of desired program outcomes.
Degree of accuracy needed for individual PI reports
Each participating REU will receive a yearly report with: (1) a profile of the demographic and background characteristics of students participating in their REU; (2) a summary of pre- to post-program changes among their REU participants; (3) a summary of how their REU participants perceived the REU; and (4) a comparison of their REU own participants versus participants in REUs at similar institutions.
We expect high response rates for both the pre- and post-program surveys from all REU participants, given that there will be interpersonal and professional incentives for their participation (the PI or a senior staff member on their REU will be recruiting the participants to complete the surveys), as well as recruitment methods that will enhance participation (compelling participation messages and multiple reminders). For any response rates that dip below 70%, the reports will include descriptive information on any differences in which participants responded; for individual REUs, statistical comparisons based on response status may not be able to be conducted due to small numbers of students in individual REUs. Thus, we expect that this reporting will reflect a high degree of accuracy regarding pre- to post-program changes in REU participants.
Degree of accuracy needed for NSF reports
NSF will receive multiple reports providing the following: (1) profiles of the participants in the NSF CISE REU program, along with comparison group students with other, non-NSF REU experiences and students without research experiences; (2) descriptions of the features and components that different REU projects offer to participants, as compared with non-NSF research experiences; (3) analyses of how successful each individual REU is in achieving key outcomes of the REU program, overall and relative to other NSF CISE REU programs; (4) analyses of the correlates of undergraduate research experiences that are most strongly correlated with sustained (one-year post-program) changes in students; and (5) analyses of the features of NSF REU programs that are most strongly associated with the recruitment and retention of students who are underrepresented in computing.
As noted in the accuracy discussion for individual REU PI reports, the pre- and post-program REU participant surveys – which contribute data to answer all of the above information needs – are expected to have high response rates, and when aggregated for NSF reporting, should allow for a high degree of accuracy in answering questions about REU participants, their experiences, and their immediate post-program changes.
For the NSF reports, having a large number of comparison participants will be needed at both the pre-survey and the follow-up survey for accurate reporting of comparisons between REU participants and other undergraduates. The initial single group of comparison participants recruited for the study will eventually be sorted into two groups, based on whether these individuals go on to have research experiences between their pre-survey and their follow-up survey. Previous Data Buddies Survey data suggest that 40 – 50% of undergraduates who have not had an undergraduate research experience plan to do so during their college career; thus, with a recruitment target of 2,000 comparison participants for the pre-survey, we expect that there will be more than enough baseline comparison participants in both the “no research” or “non-NSF REU research” groups to provide an accurate data representation of how these groups compare to REU participants in the areas described above.
Completion rates for follow-ups surveys are estimated to be approximately 50% of the original respondents, for both the REU participants and the comparison groups. For both groups, despite the fact that the respondents had already agreed to be in the study, a fairly long period of time will have elapsed since their previous survey completion; thus, we expect some drop-off in interest to occur. However, respondents from both groups will be recruited with similar recruitment messages and reminders, and they will be offered the same incentives for participation. In short, there should be similar response rates across respondent types, and efforts will be made to maximize these rates. Analyses will be conducted and reported to describe any patterns of non-response at the follow-up.
In terms of overall statistical power, with a 50% response rate at follow-up, there will be adequate numbers of respondents in each group to conduct comparative follow-up analyses. It is expected that these numbers will also allow for aggregate analyses of whether recruitment and retention of underrepresented groups as a whole differ as a function of research experience type; however, the extent to which analyses are possible for some individual categories of underrepresented populations will need to be determined when data are collected. Even with large overall Ns in each group, there may be small numbers of respondents for some subgroups of interest (such as certain racial/ethnic groups) that may limit CERP’s ability to include those variables in any predictive analyses. The research team will conduct power analyses during the reporting stage and provide interpretation of the findings in that context. Furthermore, when the Ns for subgroups are respondents are small enough that the identities of those respondents is at risk, those analyses will not be included in any research reports.
Unusual problems requiring specialized sampling procedures
Not applicable. There are no specialized sampling procedures being used.
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
The study will not be implementing data collection that is less frequent than an annual cycle. REU projects run in fall, spring, and summer academic terms, and data collection timing for REU participants and PIs is tied to the REU project start and end dates. The timing of data collection for comparison groups will closely track data collection schedule for REU participants. This schedule has been carefully selected to track both short-term and longer-term outcomes among REU participants and their peers in the comparison groups.
B.3. Methods to Maximize Response Rates and the Issue of NonResponse
Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.
CERP will take several measures to maximize response rates for the surveys.
Methods to maximize response rates
Information, motivation, and resources for respondents
All communications with potential respondents will promote participation in several ways. Emails sent to potential respondents will include information about the study and its purposes, thereby ensuring that they are fully informed about the project. The emails will also underscore the importance of the respondents’ role in contributing to improving the knowledge base about undergraduate research experiences and their impacts. Furthermore, as part of the Computing Research Association (CRA), CERP will be able to leverage CRA’s strong reputation within the computing community to promote high levels of survey participation. All emails will include an introduction to CERP as the research and evaluation arm of the CRA. The emails will explain CRA’s role in administering the survey on behalf of NSF. The emails will include a link to more information about CRA and CERP, as well as contact information for CERP staff for those seeking more information about any aspects of the study.
Appeals from known REU personnel
For the subset of REU participants being asked to complete the pre- and post-program surveys, the recruitment and reminder emails described above will be sent from a member of their own REU staff (either the PI or a designated project staff member), which is likely to enhance REU participants’ interpersonal and professional incentives to take part in the study.
Access to high-quality contact information
All of the potential survey respondents will be recruited using email addresses that they themselves have provided; thus, the percentage of potential participants who will be lost due to incorrect contact information will be close to zero.
Multiple recruitment prompts
As described in Section B.2, there will be multiple recruitment invitations sent to respondents to encourage high participation rates, spaced out by one-week intervals. For comparison group pre-surveys and all follow-up surveys, which are not bound by REU program start and end dates, CERP will evaluate response rates after the final scheduled reminder to determine whether to extend the data collection window further; although surveys that are in the field longer have higher response rates, this must be balanced against the need for timely completion of data collection.5
Survey Convenience
Respondents will be able to easily link to the online Qualtrics survey – via computer, tablet, or mobile device – through a personalized survey link provided in the recruitment email sent by CERP. All surveys have been designed to minimize respondent burden wherever possible. For example, the overall survey completion time should be 20 minutes at most and may take less time for many respondents. Skip patterns and survey flow have been carefully considered in the survey design to ensure that respondents are presented with only the items that are relevant to them. Respondents will be able to start and pause the survey so that they can complete it at their convenience.
Non-response bias analysis
CERP will use available data to determine whether subgroups of respondents are participating in the survey at different rates. CERP will also examine whether individual survey items have high rates of missing responses. Both of these non-response analyses are described further in Section B.2, and both will be included in the reporting of the study results.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Background
Procedures
CERP has used this study’s planned recruitment procedures in dozens of previously survey research projects. These procedures have been tested and refined repeatedly to enhance efficiency and improve response rates.
Instrument development
Many of the items in the project surveys have been pulled directly from survey projects conducted by CERP over many years. These projects include CERP’s annual Data Buddies Survey and previous non-NSF REU evaluations, such as the Distributed Research Experiences for Undergraduates (DREU), a program run by CRA’s Widening Participation Committee and evaluated by CERP since 2013. Consequently, many items in the current instruments have been tested and refined over a period of many years. Of course, new items have also been added to these surveys, and some previously developed items have been updated to enhance their applicability to the current study purposes. For both the previously-developed and new survey items, community stakeholders and subject matter experts in REUs have been recruited to contribute to item development and refinement. CERP will also engage in significant instrument testing prior to commencing data collection, as described below.
Testing plans:
All surveys will be programmed in Qualtrics. CERP will conduct comprehensive testing of the survey, including the following:
Proofing survey question text
Confirming appropriate question types have been programmed (e.g., multiple versus single response options)
Ensuring that the survey skip logic has been appropriately programmed
Checking that the survey functions adequately in multiple browsers
Checking that the order of survey questions facilitates easy comprehension and “flow” for respondents; i.e., that the order and placement of questions feels natural and flows well
Pilot testing the survey will be conducted in two ways. First, internal CRA staff who are not familiar with the study will be asked to complete the survey and provide feedback on the clarity of items and the overall survey administration experience, including the amount of time it takes them to complete it. After that process is complete, if additional testing is warranted, pilot testing will be conducted with fewer than 10 individuals outside of the CRA organization.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
NSF has contracted with Computing Research Association’s (CRA) Center for Evaluating the Research Pipeline (CERP) to design and conduct this survey project. Names, titles, contact information, and roles of the CERP staff who will be involved in the study are described below.
Dr. Burcin Tamer, Director of CERP (202-266-2935)
Tamer will have general oversight over the project and manage the contract overall. She has led the study design (including statistical aspects of the design) and instrument development.
Heather Wright, Associate Director of CERP (202-266-2945)
Wright will be responsible for project management and staff oversight. She contributed to the study design (including statistical aspects of the design) and led instrument development.
Kristi Kelly, Research Associate (202-266-2935)
Kelly will work on data management, analysis, and reporting activities. She has contributed to the study design (including statistical aspects of the design) and instrument development.
Taniya Ross-Dunmore, Research Assistant (202-266-2935)
Ross-Dunmore will be engaged in administrative aspects of the project, including communication with REU personnel and study participants and programming and testing the online surveys.
1 This number is a non-duplicated count; i.e., if a participant takes part in an REU during two consecutive terms within a year, they would be counted twice. This is the most relevant way of counting potential respondents, because a participant would complete pre- and post-program surveys for each research experience they participate in.
2 This approximation is supported by the numbers of PIs who have expressed interest so far in being included in the evaluation. To date, 65 REU Site PI applicants have expressed interest in using CERP for their evaluations for 2022. Assuming 10 participants per site, this would be 650 REU Site participants. We expect Site PIs to continue to enroll for the first year through early May 2022. Supplement PIs (who generally work with one or two students) will also be able to enroll throughout the year.
3 This process will be repeated in 2023 for the second year of comparison group participants.
4 Baker, R., Brick, J. M., Keeter, S., Biemer, P., Kennedy, C., Kreuter, F., ... & Terhanian, G. (2016). Evaluating survey quality in today's complex environment. American Association for Public Opinion Research (AAPOR).
5 Baker, R., Brick, J. M., Keeter, S., Biemer, P., Kennedy, C., Kreuter, F., ... & Terhanian, G. (2016). Evaluating survey quality in today's complex environment. American Association for Public Opinion Research (AAPOR).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Marietta Bowman |
| File Modified | 0000-00-00 |
| File Created | 2022-01-20 |