Statement B NSF ADVANCE Evaluation

Attachment B1: Invitation and Reminder Emails for ADVANCE Surveys
Attachment B2: Consent for ADVANCE Surveys
Attachment B3: Invitation, Reminder, and Schedule Emails for Teleconference Interviews
Attachment B4: Consent for Teleconference Interviews

B. Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Below is a description of the respondents for each data collection method: (a) online surveys for representatives of ADVANCE grantee institutions/organizations; (b) an online survey for representatives of institutions/organizations that applied for the ADVANCE Institutional Transformation (IT) grant but did not receive NSF ADVANCE funding; and (c) teleconference interviews with representatives of ADVANCE awards.

(a) Online Surveys for Representatives of ADVANCE Grantee Institutions/Organizations

The universe of respondents includes 228 representatives from institutions/organizations that received NSF ADVANCE funding. (These representatives will be referred to as survey invitees throughout this document). The start dates for their ADVANCE projects range from 10/01/2001 to 9/30/2018. Seven surveys were developed with questions tailored to the focus of the ADVANCE project and whether the ADVANCE grant has ended or will still be ongoing as of 8/31/2020. Table B.1 shows the distribution of invitees by survey type. Of the 228 survey invitees, 224 will complete one survey and four will complete two.

Table B.1: Number of Survey Invitees from ADVANCE grantee institutions/organizations by Survey Type

Survey Type	Number of Survey Invitees from ADVANCE grantee institutions/organizations
Single Institution Organizational Change Ended Survey	75
Single Institution Organizational Change Ongoing Survey	17
Partnership Ended Survey	29
Single Institution Self-Assessment Ended Survey (version A)	17
Single Institution Self-Assessment Ended Survey (version B)	20
General ADVANCE Ended Survey	53
General ADVANCE Ongoing Survey	13
Single Institution Organizational Change Ended Survey and Partnership Ended Survey	2
General ADVANCE Ended Survey and Single Institution Self-Assessment Ended Survey (version A)	1
Single Institution Organizational Change Ended Survey and Single Institution Self-Assessment Ended Survey (version A)	1
*Total*	228

The 228 survey invitees represent 178 ADVANCE grantee institutions/organizations. The number of survey invitees exceeds the number of institutions/organizations because some ADVANCE grantee institutions/organizations have multiple ADVANCE awards.

The 228 survey invitees will be identified with the assistance of the Principal Investigator (PI) for the ADVANCE award. For awards that are still ongoing, this will be the current PI, and for awards that have ended it will be the last one if there was a change in PIs during the ADVANCE grant. Fast Track clearance (OMB # 3145-0215; Expiration Date: 6/30/2020) has been obtained for Windrose Vision to ask the PIs to identify the best suited member in their ADVANCE project team to complete the survey. If the PI does not respond to the request, then the survey will be sent to the PI.

(b) Online Survey for Representatives of ADVANCE Institutional Transformation (IT) Applicants

The ADVANCE IT applicants survey will be administered to . The pool of respondents for this survey is 120 individuals that applied for the ADVANCE IT award but did not receive NSF ADVANCE funding. These individuals will also be referred to as survey invitees throughout this document. The timeframe for ADVANCE IT proposals submitted ranges from 10/01/2001 through 9/30/2018.

(c) Teleconference Interviews with Representatives from Six ADVANCE Awards

Six case studies will be developed. Four will focus on awards that have faculty equity strategies that continued after ADVANCE funding ended; and two will focus on awards with strategies that did not continue. The number of case studies and interviewees is limited by the budget allocated for the evaluation.

To provide data for the case studies, a total of 12 representatives from six ADVANCE awards (two from each) will be invited to participate in a one-hour teleconference interview with Windrose Vision, the NSF evaluation contractor. One interviewee will be the representative who completed the online ADVANCE survey. Windrose Vision will ask that representative to recommend another member of their ADVANCE project team to accompany them on the interview. Then, Windrose Vision will contact both to schedule the interview at the day and time that is convenient for them.

The interviews will:

Examine the factors that enhanced or impeded the continuation of faculty equity ADVANCE strategies after NSF funding ended;
Probe further into strategies reported in the ADVANCE survey and described in the annual reports, journal articles, and other documents; and
Address any missing information and/or clarify any information from annual reports.

In selecting the ADVANCE awards for case studies, consideration will be given to factors that enhanced or impeded the continuation of faculty equity ADVANCE strategies related to:

Policies and/or procedures (e.g., parental leave or modified duties, stop the tenure clock);
Institutional infrastructures mechanisms (e.g., office space, operating budget, collection of faculty equity data, data-informed process to monitor equity in tenure and promotion decisions);
Accountability structures for administrators (e.g., a strategic plan with faculty equity goals, a committee to advise leadership on faculty equity issues, documented expectations for leadership to act on faculty climate survey results, etc.); and
Diffusion of ADVANCE strategies and products (e.g., toolkits, materials, and resources).

B.2. Procedures for the Collection of Information

B.2.1. Data Collection Procedures

The data collection procedures for the online surveys and teleconference interviews are described below.

Procedures for Online Surveys

Introductory email – The NSF Assistant Director of the Education and Human Resources (EHR) Directorate or the NSF ADVANCE Program Director will send an introductory email to survey invitees. The purpose is to provide an overview of the evaluation and encourage participation in the survey.
Invitation email (Attachment B1) – Within 24 hours of the introductory email being sent, Windrose Vision will send the invitation email to complete the online survey to invitees. The invitation explains the purpose of the evaluation and the survey; emphasizes that participation in the survey is voluntary; lists the closing date of the survey; identifies who will have access to the data; and includes a URL to access the electronic consent form. The survey will be open for six weeks but will be extended to seven weeks for survey invitees from institutions/organizations categorized as small businesses.
Reminder emails to non-respondents (Attachment B1) – Windrose Vision will send a maximum of five reminder emails at weekly intervals to encourage those who have not accessed the hyperlink to participate in the survey. The first reminder email will be sent one week after the email invitation.
Electronic Consent Form (Attachment B2) – Survey invitees will be directed to the consent form upon clicking on the hyperlink included in the invitation email. The consent form includes the following sections: purpose of the evaluation, purpose of the survey, procedures, benefits, voluntary participation, risks, confidentiality, and contact information for NSF and Windrose Vision. At the end of the form, survey invitees are asked if they consent to participate in the survey. If they consent, they will be directed to the survey.

Procedures for Teleconference Interviews

Introductory Email – The NSF Assistant Director of the EHR Directorate or the NSF ADVANCE Program Director will send an introductory email to each representative of the six selected ADVANCE awards who completed the ADVANCE survey. The introductory email explains that the ADVANCE award has been selected to be highlighted as one of six case studies and encourages representatives to participate in the teleconference interview.

Emails to interviewees (Attachment B3). Windrose Vison will do the following:

(1) Send invitation email to each representative of the six selected awards who completed an ADVANCE survey. The purpose is twofold: first to invite the representative to participate in the teleconference interview, and second to ask the representative to recommend another member of their ADVANCE project team to join the call (first choice and alternate choice in case their first person is unavailable). The invitation has the following information: purpose of the interview, length of time of the interview (one-hour), period when all interviews should be conducted, voluntary participation, and contact information for NSF and Windrose Vision.

(2) Send two invitation reminders (at weekly intervals) to the six representatives of ADVANCE awards if they do not respond.

(3) Upon receipt of the recommendations for the additional interviewee, Windrose Vision will send an invitation email to the 1^st choice.

(4) Send one reminder email to the 1^st choice. If the individual is unable to participate or does not respond, then Windrose Vision will email the alternate choice.

(5) Upon agreement to participate, send an email to schedule the date and time for the teleconference interview. Windrose Vision will ask the two representatives of each ADVANCE award to provide their 1^st, 2^nd, and 3^rd choices for the date and time.

(6) Send one reminder email to schedule the interview if representatives do not respond. Windrose Vision will compile and identify the date and time that work best for the representatives.

(7) Send a confirmation email. After the date and time of interview is confirmed, Windrose Vision will send an email with the following information: date, time, URL to join GoToMeeting, interview questions, and URL for electronic consent form.

Electronic Consent Form (Attachment B4) –The electronic consent form consists of the following sections: purpose of the evaluation, purpose of the teleconference interview, procedures, benefits, voluntary participation, risks, audio recording, confidentiality, contact information for additional questions about the interviews, consent, and audio recording. The consent section asks interviewees if they consent to participate in the teleconference interview. The audio recording section asks interviewees if they agree for the interview to be audio recorded.

B.2.2. Analysis Procedures

Quantitative Analysis of Survey Data from ADVANCE Grantee Institutions/Organizations

Quantitative and qualitative data will be analyzed to address the evaluation questions and gain a better understanding of the sustainability and diffusion of ADVANCE strategies.

Descriptive statistics –Survey responses that address the evaluation questions will be analyzed using descriptive statistics such as frequency distributions and measures of central tendency (i.e., mean, median, and mode). As appropriate, responses to open-ended questions will be grouped into categories and summarized with descriptive statistics as well.

As discussed in section B1, the survey data will enable NSF to capture the depth of the ADVANCE portfolio, consisting of projects that have ended and those that will still be ongoing as of 8/31/2020.

The surveys contain:

(a) Common questions in areas such as policies and/or procedures, infrastructure mechanisms, accountability structures, factors that affect sustainability, sources of funding, development of products (toolkits, materials, and resources), and dissemination venues. Data collected from these common questions will be aggregated across all projects.

(b) Questions aimed at the specific focus of the ADVANCE project (e.g., Single Institution Organizational Change, Partnership, Single Institution Self-Assessment). Data collected will be aggregated by survey and focus.

Inferential Statistics - A preliminary assessment of award annual reports was conducted to determine the feasibility of using a quasi-experimental design. Given the unique characteristics of each ADVANCE project and the context in which the ADVANCE strategies were implemented, no suitable comparison group could be determined. Each ADVANCE award will essentially provide their own data for comparative purposes, as data will be collected for different time points. For example, the Single Institution Organizational Change Ended survey will gather information about faculty equity policies and/or procedures, infrastructure mechanisms, and accountability structures that existed before the ADVANCE award, as well as those that were implemented during the NSF ADVANCE grant that continued after NSF funding ended.

Inferential statistics will be used to identify relationships between two or more groups of categorical data. For example, a Chi-square test of independence will be used to determine if there is a relationship between time (i.e., before, during, and after funding) and changes in policies and/or procedures. Another analysis will be to determine if there is a relationship between institutional characteristics (e.g., size, geographic area, ADVANCE funding level) and type of ADVANCE strategy that continued after funding. Sub-analysis (e.g., by policy type) will be performed where possible depending on sample size.

Qualitative Analysis – Content analysis will be used to analyze data collected from teleconference interviews and open-ended questions from the surveys. The content analysis involves: coding responses; creating categories to summarize coded responses; and identifying themes across categories to explore relationships among all responses. To ensure reliability, two Windrose Vision analysts will independently code the qualitative data. Discrepancies between the analysts’ coding will be identified and resolved. Inter-rater reliability will be estimated, with 80% agreement as the minimum acceptable inter-rater agreement.

Quantitative and Qualitative Analysis of Survey Data from ADVANCE IT Applicants

Data will be used to examine if and how developing the proposal for the ADVANCE IT grant influenced unfunded institutions/organizations in addressing STEM faculty equity issues. Quantitative data will be summarized with descriptive statistics such as frequency distributions. Responses to open-ended questions will be reviewed and grouped into categories, then the categories will be collapsed into themes, and reported using descriptive statistics, as appropriate.

B.3. Methods to Maximize Response Rates

Online surveys – Windrose Vision, in consultation with NSF ADVANCE team, will use strategies prior to and during survey administration to maximize the response rate on the online surveys. The strategies implemented prior to survey administration aim to raise awareness about the purpose of the evaluation and alert invitees that they will receive an email invitation to participate in the online surveys. The strategies implemented during survey administration will encourage participation in the online surveys.

Prior to survey administration:

NSF ADVANCE team and Windrose Vision will hold four informational webinars with past ADVANCE awardees and four with current ADVANCE awardees. To increase attendance, each information webinar will be scheduled for 15 minutes and on different days so awardees can find a time that fits in their schedule. During the webinars, attendees will be provided with information to help answer the survey questions. For example, Windrose Vision will: a) recommend that respondents have their Project Outcomes Report available; b) provide the URL for awardees to access this report since the report is public; and c) recommend that awardees with grant(s) that have ended use the final annual report to assist in answering which, if any, ADVANCE strategies still exist.
Windrose Vision will develop and disseminate an ADVANCE evaluation Frequently Asked Questions (FAQ) flyer for awardees and another one for ADVANCE IT applicants.
The NSF Assistant Director of the EHR Directorate or the NSF ADVANCE Program Director will send an introductory email to the respondents
NSF and Windrose Vision will encourage well-known researchers in the ADVANCE community to send an email to survey invitees encouraging them to participate in the surveys. Some of these researchers are former ADVANCE awardees and are members of Windrose Vision’s Technical Working Group that provided feedback on the data collection activities for this evaluation.

During survey administration:
- Windrose Vision will keep the survey open for six weeks and send five follow-up reminder emails to non-respondents weekly after the email invitation. This email will include the URL for the electronic consent form. If respondents consent to participate in the survey, they will be directed to the survey. As discussed in section A.5 of Statement A, small businesses will have seven weeks to respond.
- The NSF ADVANCE Program Director will send an email to non-respondents encouraging them to participate in the survey. The email will be sent soon after Windrose Vision sends the third reminder email.

Teleconference Interviews – Following is a summary of the approach that will be employed to maximize participation.

The NSF Assistant Director of the EHR Directorate or the NSF ADVANCE Director will send an introductory email to the representatives of awards selected to participate in teleconference interviews. In the email, the Director will explain that the interview is an opportunity to provide contextual factors that influenced the sustainability and diffusion of faculty equity strategies.
Windrose Vision will send reminder emails as discussed in Section B.2.1. Data Collection Procedures.
The NSF ADVANCE Program Director will send a reminder email to representatives who have agreed to participate but have not yet scheduled their date and time for the interview. The Director will send the email after Windrose Vision sends the second reminder email to schedule the interview.

B.4. Test of Procedures or Methods to be Undertaken

Fast Track Clearance (OMB # 3145-0215. Expiration Date: 06/30/2020) was granted to conduct cognitive testing interviews to improve the content of the surveys. The cognitive testing took place from February the 18^th to the 26^th, 2020. Representatives from 10 institutions/organizations (eight awardees and two ADVANCE IT applicants) provided feedback during a 30-minute interview. Results from the interviews and NSF feedback were used to improve survey instructions, phrasing of questions, wording of response categories, and question sequence. Following are examples of edits made to the surveys.

(1) Survey respondents will have the option to skip questions about policies and/or procedures, infrastructure mechanisms, and accountability structures in place “before” the NSF ADVANCE award if they indicate they do not have knowledge or documentation about them; (2) The labels and definitions for two policies (Parental Leave or Modified Duties and Other Accommodations for Pregnant or Lactating Women) were revised; (3) The question about the funding source for each faculty equity strategy that continued after NSF ADVANCE funding ended was replaced by one general question for all strategies; (4) The request for URLs of products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding was removed; though a question about the description of products remains. (5) The definition of “Partnership Project” was simplified; and (6) Some questions in a table format with more than five columns were restructured into multiple questions.

Windrose Vision staff also tested the online surveys to assess ease of navigation.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Windrose Vision, the NSF contractor, worked with NSF, and a Technical Working Group (TWG) to develop the evaluation design and data collection instruments. Windrose Vision staff have extensive experience collecting and using a variety of statistical techniques to analyze data from grant applicants, reviewers, and awardees. In addition, a TWG made up of awardees and experts in the field of faculty equity were consulted on the design and analysis of this evaluation. Please email Madeleine F. Wallace, Ph.D. at [email protected] for any questions related to the design and data analysis of the evaluation.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Labibah Tehreem
File Modified	0000-00-00
File Created	2021-01-13

Statement B NSF ADVANCE Evaluation