REVISED_Statement A NSF ADVANCE Evaluation

The National Science Foundation (NSF) requests approval from the Office of Management and Budget (OMB), under the Paperwork Reduction Act of 1995, to collect data for the evaluation of the ADVANCE Program. The evaluation focuses on the sustainability and diffusion of STEM faculty equity strategies generated by ADVANCE grantee institutions/organizations; and examines if and how developing the proposal for the ADVANCE Institutional Transformation (IT) grant influenced unfunded institutions/organizations to address inequities for STEM faculty. This request seeks approval for nine distinct data collections: seven self-completion online surveys for ADVANCE grantee institutions/organizations (each survey tailored to the specific project focus); one self-completion online survey for ADVANCE IT applicants who were not funded; and one instrument to conduct teleconference interviews with two representatives each from of six ADVANCE awards.

SECTION A

Overview of the NSF ADVANCE Program

The NSF ADVANCE program, established in 2001, supports projects that identify and address structural and/or policy barriers to equity for STEM faculty with gender as the primary focus. The ADVANCE program is managed in the Division of Human Resource Development (HRD) within the Directorate for Education and Human Resources (EHR). An ADVANCE Implementation Committee (AIC) advises on the management and implementation of the program and ensures that the program remains informed of the gender issues that exist within the STEM disciplines. The ADVANCE awards are funded through the following tracks .

Adaptation: These awards support the adaptation and implementation of evidence-based organizational change strategies.
Institutional Transformation (IT): These awards seek to make an impact through organizational change strategies designed to reduce or eliminate organizational barriers to the full participation of women in STEM academic careers, resulting in an academic environment that attracts women to science and engineering careers.
Institutional Transformation Catalyst (IT Catalyst) and Institutional Transformation Start (IT Start): These awards focus on institutional self-assessments to prepare institutions/organizations for transformation through the implementation of ADVANCE strategies. IT Catalyst awards also support implementation of ADVANCE strategies.
Leadership: These awards support the efforts of individuals, small groups, or organizations in the development of new strategies and programs that enable women to establish or re-establish successful academic careers; and programs for faculty and leadership skill development.
Partnership for Adaptation, Implementation, and Dissemination (PAID): These awards support analysis, adaptation, dissemination, and use of existing innovative materials and practices known to be effective in increasing the representation and participation of women in science and engineering careers.
Partnerships for Learning and Adaptation Networks: STEM Disciplines (PLAN-D) and Partnerships for Learning and Adaptation Networks: Institutions of Higher Education (PLAN-IHE): These awards promote knowledge-building about ADVANCE activities and include a protocol to identify and document the effectiveness of the activity for achieving desired outcomes in particular institutional and disciplinary contexts.

Overview of Information Collection Request

The evaluation examines both the sustainability and diffusion of faculty equity strategies, activities, and/or interventions developed by ADVANCE grantee institutions/organizations. NSF has not previously conducted a program-level evaluation of these two components for the ADVANCE program. The NSF has also not collected information on faculty equity strategies implemented by institutions/organizations that applied for an ADVANCE IT grant but did not receive funding (also referred to as ADVANCE IT applicants in this document).

This request seeks approval to administer online surveys to representatives from ADVANCE grantee institutions/organizations, as well as representatives from institutions/organizations that applied for the ADVANCE IT grant but did not receive funding. It also requests approval to conduct teleconference interviews with representatives from six ADVANCE awards. The interview data will inform the development of six case studies to illustrate the factors that contribute to or inhibit the continuation of faculty equity strategies after NSF ADVANCE funding ends, as well as the diffusion of these strategies. Section A.2.3 Information Collection Methods describes how representatives will be identified to participate in data collection activities (i.e., online surveys and teleconference interviews)

The evaluation includes ADVANCE projects with start dates from 10/01/2001 through 9/30/2018 and proposals submitted for the ADVANCE IT grant during the same time period. The status of these ADVANCE projects varies as some have ended while others will still be ongoing at the time of survey administration. ADVANCE projects that have not ended as of 08/31/2020 are referred to as ongoing.

The ADVANCE projects, funded through different NSF ADVANCE tracks, were grouped into categories based on common characteristics (e.g., single institution organizational change, partnership) and status (i.e., ended or still ongoing). Then surveys were developed for each of these categories. Table A lists the eight survey: seven for ADVANCE grantee institutions/organizations, and one for ADVANCE IT applicants. The description of each survey is in Section A.2.3 Information Collection Methods.

Table A. Mapping of Survey Types with NSF ADVANCE Funding Tracks

Survey Type	NSF ADVANCE Funding Tracks
Single Institution Organizational Change Ended Survey	IT PAID
Single Institution Organizational Change Ongoing Survey	IT Adaptation
Partnership Ended Survey	PAID PLAN-IHE PLAN-D
4. Single Institution Self-Assessment Ended Survey - version A¹	ITS ITC
5. Single Institution Self-Assessment Ended Survey - version B²	ITS ITC
General ADVANCE Ended Projects Survey	Leadership PAID
General ADVANCE Ongoing Survey	PLAN-D PLAN-IHE Adaptation Partnership
ADVANCE IT Applicant Survey	IT

¹ Version A is for ADVANCE grantee institutions/organizations that applied for another ADVANCE grant.

² Version B is for ADVANCE grantee institutions/organizations that did not apply for another ADVANCE grant.

A.1. Circumstances Making the Collection of Information Necessary

Program-level evaluation, a regular part of NSF’s scientific program administration and management, is used to improve programs and communicate impacts to stakeholders. These stakeholders include Congress, NSF leadership, NSF program staff, current grantees, past and prospective applicants, the science and engineering community, and the public. The results from the ADVANCE evaluation are critical to realizing the commitments to building and using evidence that are central to the provisions of the Evidence-Based Policymaking Act of 2018 (Pub.L.115-435).

NSF EHR has contracted Windrose Vision, LLC, a company specializing in evaluating federally funded STEM and biomedical programs, to conduct an evaluation of the NSF ADVANCE program. The evaluation will focus on the sustainability and diffusion of STEM faculty equity strategies, with gender as a primary focus, generated by ADVANCE grantee institutions/organizations. In addition, the evaluation will examine if and how developing the proposal for the ADVANCE IT grant influenced institutions/organizations to address STEM faculty equity issues even if they were not funded.

Data from surveys and teleconference interviews are necessary to:

Identify, categorize, and document grantees’ ADVANCE strategies that continued after NSF ADVANCE funding ended.
Identify the factors that contributed to or inhibited the continuation of ADVANCE strategies after NSF ADVANCE funding ended.
Aggregate and compare data across three time periods (before, during, and after funding) within the ADVANCE portfolio of grantees.
Identify, categorize, and document the characteristics of ADVANCE strategies diffused within and outside the institution/organization and the venues used.
Identify and document the unexpected outcomes and perceived most prominent impacts of the ADVANCE project from the grantee’s perspective.
Validate, cross check, and as necessary supplement information about ADVANCE strategies that grantees described in proposals and reports.
Document any changes in faculty equity that occurred at institutions/organizations that applied for the ADVANCE IT applicants but did not receive funding.

NSF EHR places a strong emphasis on evidence-based decision making. To this aim, the evaluation findings, including factors that enabled or prevented strategies from continuing after funding, will provide evidence to inform changes in ADVANCE Request for Applications (RFAs), Frequently Asked Questions (FAQs), and allocation of funds for ADVANCE tracks. The findings will help to describe how ADVANCE strategies have been diffused and identify those more likely to be diffused. In addition, evaluation findings may contribute to the design of other NSF programs aiming to continue activities after NSF funding ends.

A.2. Purpose and Use of the Information

A.2.1. Purpose of the Information

The purpose of the evaluation is to:

Examine the sustainability of different ADVANCE strategies developed by grantee institutions/organizations (six questions); and
Characterize the diffusion of ADVANCE strategies developed by grantee institutions/organizations (five questions)

ADVANCE strategy refers to any activity and/or intervention focusing on STEM faculty to address inequities in academic workplaces and/or the academic profession. These strategies include:

Reviewing, revising, updating, and/or creating policies/procedures (e.g., parental leave or modified duties, stop the tenure clock);
Putting in place infrastructure mechanisms to support faculty equity efforts (e.g., office space, operating budget, collection of faculty equity data, data-informed process to monitor equity in tenure and promotion decisions);
Establishing accountability structures for administrators (e.g., a strategic plan with faculty equity goals, a committee to advise leadership, documented expectations for leadership to act on faculty equity issues); and
Implementing other activities (e.g., mentoring, leadership and career development workshops, networking).

The following are the evaluation questions about sustainability:

What ADVANCE strategies were sustained after the NSF funding ceased?
What are the characteristics of these ADVANCE strategies?
How were sustained ADVANCE strategies funded, staffed, evaluated, and/or changed over time?
Were any non-institutional or non-organizational operating funds or resources sought and/or acquired to continue ADVANCE strategies after the NSF funding ended?
What, if any, contextual factors enhanced or limited the sustainability of ADVANCE strategies over time?
To what extent do sustained ADVANCE projects impact populations beyond STEM faculty?

The following are the evaluation questions about diffusion:

What mechanisms, if any, have been used to disseminate ADVANCE strategies?
What are the ADVANCE strategies that have been diffused through all mechanisms (NSF ADVANCE grants and other mechanisms)?
What are the characteristics of ADVANCE strategies that are most and least frequently diffused?
What ADVANCE developed strategies, if any, are likely to continue to be diffused among institutions of higher education and non-academic organizations without on-going investment by NSF?
What are the characteristics of these ADVANCE developed strategies?

In addition, the evaluation gathers information from institutions/organizations that applied for the ADVANCE IT grant but did not receive NSF ADVANCE funding. The evaluation question of interest is:

Were there any changes in faculty equity at unfunded institutions/organizations as a result of the proposal development process? If so, what were those changes?

A.2.2. Use of Information Collection

The evaluation findings will allow NSF EHR to make evidence-based decisions about ADVANCE and similar programs. These decisions will promote improvements in Request for Applications (RFAs) and Frequently Asked Questions (FAQs), as well as refinements in the allocation of funds for ADVANCE tracks. By understanding the factors that affect sustainability, NSF will be better positioned to support effective strategies that are widely diffused and continue even after funding ends. This means the NSF reaps a greater longer-term return on the investment.

A.2.3. Information Collection Methods

This section describes the data collection methods for the NSF ADVANCE evaluation. These methods include: (a) Online surveys for representatives from ADVANCE grantee institutions/organizations; (b) One online survey for ADVANCE IT applicants; and (c) Teleconference interviews with representatives from six ADVANCE awards.

A.2.3.1. Online Surveys for Representatives from ADVANCE Grantee Institutions/Organizations

Identification of Representatives to Participate in Online Surveys

The Principal Investigator (PI) from each ADVANCE award will be asked to identify the best representative to respond to the ADVANCE survey. OMB clearance has been obtained to contact the PI of each ADVANCE award (OMB # 3145-0215; Expiration Date: 6/30/2020). The representative may be the PI, or it may be another knowledgeable member of the ADVANCE project team. The number of representatives in the potential pool of survey invitees exceeds the number of institutions/organizations because some ADVANCE grantee institutions/organizations have multiple ADVANCE awards (e.g., IT, PAID, PLAN-D) and in some cases there is a different PI for each award.

Description of Online Surveys

Seven surveys were developed with questions tailored to the focus of the ADVANCE projects and the status of the ADVANCE grant (ended or still ongoing). Below is the list of the seven surveys. Please note that if the title includes the word “ended,” the survey has questions for ADVANCE projects that have ended or are expected to end by August 31, 2020. If the title includes the word “ongoing,” the survey has questions for ADVANCE projects that are expected to continue after August 31, 2020. Below is the list of ADVANCE online surveys:

Single Institution Organizational Change Ended Survey
Single Institution Organizational Change Ongoing Survey
Partnership Ended Survey
Single Institution Self-Assessment Ended Survey - version A
Single Institution Self-Assessment Ended Survey - version B
General ADVANCE Ended Survey
General ADVANCE Ongoing Survey

The description of each survey is below.

Single Institution Organizational Change Ended (Attachment A1) – This survey covers IT, and PAID projects that aimed to make systemic changes within an institution of higher education to address gender-based inequities for STEM faculty. This survey consists of seven sections covering the time periods before, during, and after NSF ADVANCE funding:

Section 1 focuses on formal policies and/or procedures implemented such as dual career, parental leave or modified duties, stop the tenure clock, etc.
Section 2 focuses on infrastructure such as having a physical space; hiring full or part-time employees; allocating operating budget; and putting into place systems for collecting, reporting, and using faculty equity data.
Section 3 focuses on accountability structures that ensure leadership responsibility for making progress on institutional goals regarding equity.
Section 4 focuses on the sources of funding for ADVANCE strategies that continued after NSF ADVANCE funding ended.
Section 5 focuses on factors that may have facilitated the continuation of ADVANCE strategies after NSF ADVANCE funding ended, as well as challenges and barriers encountered.
Section 6 focuses on products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding and their dissemination within and outside the institution/organization.
Section 7 focuses on the perceived impacts of the ADVANCE project on the culture and climate of the institution/organization.

Single Institution Organizational Change Ongoing (Attachment A2) – This survey covers IT, PAID, and Adaptation projects that aim to make systemic changes within an institution of higher education to address gender-based inequities for STEM faculty. This survey includes the sections from the Single Institution Organizational Change Ended Survey, except for those related to the strategies that continued after funding ended. The survey covers time periods before and during NSF ADVANCE funding, and request information about the steps being taken to ensure sustainability after funding ends.

Partnership Ended Survey (Attachment A3) – This survey covers PAID, PLAN-D, and PLAN-IHE projects that aimed to scale-up or disseminate equity strategies for STEM faculty. These ADVANCE projects involved more than one institution/organization working together. Only the lead institution in a partnership project will be surveyed. This survey consists of six sections covering the time periods during and after NSF ADVANCE funding:

Section 1 focuses on ADVANCE strategies (e.g., skill development efforts, policies and/or procedures, and other activities/interventions) aimed to transform the culture and climate of STEM disciplines and/or institutions.
Section 2 focuses on the sources of funding for ADVANCE strategies that continued after NSF ADVANCE funding ended
Section 3 focuses on factors that may have facilitated the continuation of ADVANCE strategies after NSF ADVANCE funding ended, as well as challenges and barriers to the continuation.
Section 4 focuses on products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding and their dissemination to others outside the partnership project.
Section 5 focuses on the most important impact of the Partnership project.

Single Institution Self-Assessment Ended Survey - version A (Attachment A4) – This survey covers the IT Start and IT Catalyst projects focused on a wide range of self-assessment activities to identify systemic inequities impacting STEM faculty. Based on the results of the self-assessments, some grantees implemented ADVANCE strategies. Version A of the survey is for ADVANCE grantee institutions that later submitted an NSF ADVANCE proposal for a different track. The survey includes question about the factors that facilitate the submission of the proposal. This survey consists of 6 sections covering the time periods during and after NSF ADVANCE funding.

Section 1 focuses on the impacts of the ADVANCE project on the institutions and populations.
Section 2 focuses on the challenges of implementing the ADVANCE project.
Section 3 focuses on social science research proposals that resulted from the ADVANCE project’s institutional assessment activities, as well as the factors that facilitated the submission of another ADVANCE proposal.
Section 4 focuses on changes at the institution as a result of the ADVANCE award.
Section 5 focuses on the sources of funding for ADVANCE strategies that continued after NSF ADVANCE funding ended.
Section 6 focuses on products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding and their dissemination within and outside the institution/organization.

Single Institution Self-Assessment Ended Survey - version B (Attachment A5) – This version B of the survey is for ADVANCE grantee institutions that did not submit a proposal for a track different than the IT Start or IT Catalyst. It includes questions from version A, except for the one about factors that facilitated applying for other ADVANCE grants. Instead it asks why the institution did not apply for other ADVANCE grants.

General ADVANCE Ended Survey (Attachment A6) – This survey covers Leadership and PAID projects. Some of these projects aimed to enable women to establish or re-establish successful academic careers (e.g., faculty and leadership skill development); while other projects aimed to scale-up or disseminate STEM faculty equity strategies. This survey consists of two sections covering the time periods during and after NSF ADVANCE funding:

Section 1 focuses on the activities and impacts of the ADVANCE Award
Section 2 focuses on products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding and their dissemination within and outside the institution/organization.

General ADVANCE Ongoing Survey (Attachment A7) – This survey covers PLAN-D, PLAN-IHE, Adaptation, and Partnership projects that aimed to scale-up or disseminate STEM faculty equity strategies. Only the lead institution will be surveyed. This survey consists of four sections covering the time period during NSF ADVANCE funding.

Section 1 focuses on ADVANCE strategies such as skill development efforts, work on policies and/or procedures, and other activities/interventions aimed to transform the institutional culture and climate as part of the ADVANCE project.
Section 2 focuses on factors that may facilitate or inhibit the continuation of ADVANCE strategies.
Section 3 focuses on products (e.g., toolkits, materials, and resources) developed with NSF ADVANCE funding and their dissemination to others outside the ADVANCE project.
Section 4 focuses on the most important impacts of the ADVANCE project.

A.2.3.2. Online Survey for ADVANCE IT Applicants

Identification of Representatives to Participate in Online Survey

The representative is the individual who submitted the proposal for the ADVANCE IT grant but did not receive NSF ADVANCE funding. The purpose of the survey is to identify any benefits at the institution/organization as a result of developing the proposal.

Description of Online Survey

ADVANCE IT Applicant Survey (Attachment A8) This survey consists of three sections covering the time period after submitting the IT proposal.

Section 1 focuses on the perceived benefits to the institutions and populations from developing the ADVANCE IT proposal. This section queries if the institution/organization submitted social science research proposals on faculty equity to NSF programs (other than ADVANCE) or other funders.
Section 2 focuses on changes made at the institution if the areas of policies and/or procedures, infrastructure mechanisms, accountability structures, or other area.
Section 3 focuses on products (e.g., toolkits, materials, and resources) developed.

A.2.3.3. Teleconference Interviews with Representatives from ADVANCE Awards

Identification of Representatives to Participate in Teleconference Interviews

A total of 12 representatives from six ADVANCE awards (two from each award) will be invited to participate in a one-hour teleconference interview with Windrose Vision, the NSF evaluation contractor. One interviewee will be the representative who completed the online ADVANCE survey. Windrose Vision will ask that representative to recommend another member of their ADVANCE project team to accompany them on the interview. Then, Windrose Vision will contact both to schedule the interview at the day and time that is convenient for them.

Description of Teleconference Interviews

Teleconference interviews (Attachment A9) will inform the development of six case studies. Four will focus on awards that have faculty equity strategies that continued after ADVANCE funding ended; and two will focus on awards with strategies that did not continue. The number of case studies and interviewees is limited by the budget allocated for the evaluation.

The teleconference interviews will:

Examine the factors that enhanced or impeded the continuation of faculty equity ADVANCE strategies after NSF funding ended;
Probe further into strategies reported in the ADVANCE survey and described in the annual reports, journal articles, and other documents; and
Address any missing information and/or clarify any information from annual reports.

In selecting the ADVANCE awards for case studies, consideration will be given to factors that enhanced or impeded the continuation of faculty equity ADVANCE strategies related to:

Policies and/or procedures (e.g., parental leave or modified duties, stop the tenure clock);
Institutional infrastructures mechanisms (e.g., office space, operating budget, collection of faculty equity data, data-informed process to monitor equity in tenure and promotion
decisions);
Accountability structures for administrators (e.g., a strategic plan with faculty equity goals, a committee to advise leadership on faculty equity issues, documented expectations for leadership to act on faculty climate survey results); and
Diffusion of ADVANCE strategies and products (e.g., toolkits, materials, and resources).

A.3. Use of Information Technology and Burden Reduction

Information technology will be employed in both the self-completion surveys administered online and in the conduct of the teleconference interviews.

A.3.1. Online Surveys

To reduce burden, the online surveys will use skip-patterns so that respondents are only presented with questions relevant to their ADVANCE project. Information will be pre-populated if possible, such as the institution/organization name and award ID for ADVANCE grantees and the institution/organization name and proposal ID for ADVANCE IT applicants. The use of open-ended questions will be kept to a minimum, as most questions include single select or multiple select questions, where the respondent is asked to choose from a predetermined set of response options. The surveys include response options in drop-down menus and multi-item matrix tables.

A.3.2. Teleconference Interviews

To reduce the burden on respondents, one-hour teleconference interviews will be conducted using GoToMeeting, an online meeting tool, and scheduled at a day and time that is convenient for the representatives of the six ADVANCE awards.

A.4. Efforts to Identify Duplication and Use of Similar Information

The evaluation of the NSF ADVANCE Program does not duplicate other NSF efforts. The following data have not been and are not currently being systematically collected by NSF:

Information on faculty equity strategies that have continued after NSF ADVANCE funding;
Data on the diffusion of ADVANCE strategies from ADVANCE grantees;
Data on the diffusion of ADVANCE strategies from ADVANCE IT applicants; and
Information on faculty equity strategies that ADVANCE IT applicants implemented as a result of the proposal development process.

A.5. Impact on Small Businesses or Other Small Entities

There are 11 ADVANCE grantees considered small entities (i.e., professional societies and professional associations). They will be given more time to complete the survey. Windrose Vision will extend the time to complete the online survey from six weeks to seven weeks for them. The extension will allow these ADVANCE grantee organizations more time to gather all related information and materials necessary to complete the online survey.

A.6. Consequences of Collecting the Information Less Frequently

This is not a request to collect longitudinal data; the online surveys and teleconference interviews will only be administered once. If data from this evaluation are not collected, NSF will be unable to determine if faculty equity strategies implemented during ADVANCE grants have continued after funding ended. The NSF will also be unable to understand the reasons strategies did not continue. Also, the NSF leadership and the ADVANCE program staff will not have necessary information to understand the diffusion of faculty equity strategies generated by ADVANCE grantees. Findings drawn from this evaluation will inform NSF of the efforts to broaden the participation of underrepresented groups, with gender as a primary focus, in the STEM workforce.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This project fully complies with all guidelines of 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A.8.1. Comments in Response to the 60-day Federal Register Notice

The 60-day Federal Register Notice was published in I Vol. 85, No. 20, on Thursday, January 30, 2020. On February 14, 2020, Ms. Lisa Schnall, a Senior Attorney Advisor from the Office of Legal Counsel, U.S. Equal Employment Opportunity Commission, requested a copy of the surveys and supporting statement. NSF provided the requested information. No other request for information or feedback on the surveys has been received. We will be moving forward with our request for clearance.

A.8.2. Comments in Response to Consultations Outside Agency

The proposed design is grounded in extensive background research and discussions with the NSF ADVANCE program team and NSF EHR Evaluation and Monitoring Group members. The evaluation design was developed in consultation with Windrose Vision, an independent contractor with experience evaluating NSF programs, and members of an external Technical Working Group (TWG) organized by Windrose Vision. The members of the TWG provided feedback on the evaluation design, survey questions, teleconference interview questions, and strategies to increase participation of ADVANCE grantees. The criteria for selecting TWG members included expertise and knowledge of NSF’s ADVANCE program, faculty equity in STEM disciplines, and diffusion. Most of the TWG members who provided feedback on the evaluation worked on ADVANCE projects at grantee institutions/organizations. The TWG members’ email, institution, and expertise are listed in Attachment A10.

A.9. Explanation of Any Payment or Gift to Respondents

No payment or gifts will be provided to respondents.

A.10. Assurance of Confidentiality Provided to Respondents

Survey data will be kept confidential and maintained in accordance with the requirements of the Privacy Act of 1974. The personally identifiable information (PII) will be removed from survey responses prior to analysis and will not be shared with NSF and the public. Survey data files will be password-protected to prevent unauthorized access or improper disclosure. Only designated staff from Windrose Vision, the evaluation contractor, who have a need to know and are bound by confidentiality will have access to the data for analysis and reporting purposes. Any publication of survey findings will not include the respondents’ PII. The survey results will be reported in aggregate form only, for example by category of institution/organization or by geographic region.

Since 12 individuals will participate in teleconference interviews for the six case studies, a statement will be added in the consent form indicating that the name and affiliation of interviewees may be disclosed in any publication. Potential interviewees will have the option to consent or not to the release of their names and affiliations. Individuals who consent will be interviewed and those who do not consent will not be interviewed.

A.11. Justification for Sensitive Questions

Data collection instruments do not include any sensitive questions.

A.12. Estimates of Hour Burden Including Annualized Hourly Costs

A.12.1. Estimates of Hour Burden

Every effort will be made to minimize the burden on respondents, including as described in section A.3 above. Table A.12.1 shows the estimate of hour burden for online surveys and teleconference interviews. The expected burden level for this evaluation is 207.50 hours.

A.12.1: Estimate of Burden

Data Collection Type	Number of Individuals	Participation Time (in minutes)	Burden (in annual hours)
Single Institution Organizational Change Ended Survey	75	45	56.25
Single Institution Organizational Change Ongoing Survey	17	30	8.5
Partnership Ended Survey	29	30	14.5
Single Institution Self-Assessment Ended Survey (version A)	17	30	8.5
Single Institution Self-Assessment Ended Survey (version B)	20	30	10
General ADVANCE Ended Survey	53	30	26.5
General ADVANCE Ongoing Survey	13	30	6.5
Single Institution Organizational Change Ended Survey and Partnership Ended Survey	2	75	2.5
General ADVANCE Ended Survey and Single Institution Self-Assessment Ended Survey (version A)	1	60	1
Single Institution Organizational Change Ended Survey and Single Institution Self-Assessment Ended Survey (version A)	1	75	1.25
Institutional Transformation Applicant Survey	120	30	60
Teleconference interviews with representatives of ADVANCE awards	12	60	12
Total	360	-	207.50

A.12.2. Annualized Cost to Respondents

Data are only collected once from survey respondents and interviewees. The overall cost is estimated to be $14,090, $13,275 for survey respondents and $815 for interviewees. The cost is derived by multiplying the average hourly wage of a full professor at a public university by the total burden hours (see Tables A.12.2.1a and A.12.2.1b).

A.12.2.1a: Estimate Costs for Survey Respondents

Data Collection		Respondent Type		Hourly Salary Estimate^*		Burden Hours per Respondent	Total Number of Respondents	Total Burden Hours	Estimated Annualized Costs
Single Institution Organizational Change Ended Survey		Full Professor		$67.91		45	75	56	$3,820
Single Institution Organizational Change Ongoing Survey		Full Professor		$67.91		30	17	9	$577
Partnership Ended Survey		Full Professor		$67.91		30	29	15	$985
Single Institution Self-Assessment Ended Survey (Version A)		Full Professor		$67.91		30	17	9	$577
Single Institution Self-Assessment Ended Survey (Version B)		Full Professor		$67.91		30	20	10	$679
General ADVANCE Ended Survey		Full Professor		$67.91		30	53	27	$1,799
General ADVANCE Ongoing Survey		Full Professor		$67.91		30	13	7	$441
Single Institution Organizational Change Ended Survey and Partnership Ended Survey		Full Professor		$67.91		75	2	3	$170
General ADVANCE Ended Survey and Single Institution Self-Assessment Ended Survey (Version A)		Full Professor		$67.91		60	1	1	$68
Single Institution Organizational Change Ended Survey and Single Institution Self-Assessment Ended Survey (Version A)		Full Professor		$67.91		75	1	1.25	$85
Institutional Transformation Applicant Survey		Full Professor		$67.91		30	120	60	$4,074
		Total Cost for Survey Respondents:							$13,275
A.12.2.1b: Estimated Costs for Interviewees
Data Collection	Respondent Type		Hourly Salary Estimate^*		Burden Hours per Respondent		Total Number of Respondents	Total Burden Hours	Estimated Annualized Costs
Teleconference Interviews	Full Professor		$67.91		1		12	12	$815
Total Cost for Interviewees:									$815
Total Annualized Cost to Respondents:									$14,090

^*The hourly rate was estimated by taking the national average 2018/19 salaries of $141,314 for full professors at public universities, dividing them by 2080 hours. For salary data see: https://www.aaup.org/sites/default/files/2018-19_ARES_Final_0.pdf

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers

There are no capital, maintenance, or operating costs to respondents.

A.14. Annualized Cost to the Federal Government

The annual total cost to the government is estimated to be $241,620 ($240,000 for the online surveys, $1,620 for the teleconference interviews). The following costs are assumed for the eight surveys: (1) Single Institution Organizational Change Ended Survey, (2) Single Institution Organizational Change Ongoing Survey, (3) Partnership Ended Survey, (4) Single Institution Self-Assessment Survey (version A), (5) Single Institution Self-Assessment Survey (version B), (6) General ADVANCE Ended Survey, (7) General ADVANCE Ongoing Survey, and (8) ADVANCE IT Applicants Survey.

The cost for each of the eight online surveys is estimated to vary between $25,000 and $30,000, depending on the complexity of the survey. Using the upper bound, the total is estimated to be $240,000 (eight surveys X $30,000). The cost of six one-hour teleconference interviews is estimated to be $1,620 [six interviews X $175 (hourly rate of contractor’s senior staff to serve as interviewer) X $95 (hourly rate of contractor’s staff to serve as a note taker)]. The teleconference interviews will be conducted with two representatives each from six ADVANCE projects. The estimates provided do not include the development of the survey questions, interview questions, scheduling the teleconference interviews, data cleaning and analysis, or preparation of reports.

A.15. Explanation for Program Changes or Adjustments

This is a new collection of information.

A.16. Plans for Tabulation and Publication and Project Time Schedule

The data collection, analysis, and reporting will be conducted by Windrose Vision, a company that specializes in evaluating STEM and biomedical programs. Windrose Vision will report the responses to each of the evaluation questions listed in Section A.2.1 Purpose of the Evaluation. NSF will determine if the findings will be published and disseminated.

A.16. Estimated Project Time Schedule

Activity	Time Schedule (Assume clearance in September 2020)
Complete administration of online surveys	2 months after OMB clearance
Complete tele-conference interviews for case studies	6 months after OMB clearance
Complete analysis of teleconference interview data and integrate results with survey data	8 months after OMB clearance
Develop final report and case studies	11 months after OMB clearance

A.17. Reasons Why Display of OMB Expiration Date is Inappropriate

No exceptions are sought. Data collection instruments will display the OMB Expiration Date.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are sought from the Paperwork Reduction Act.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Labibah Tehreem
File Modified	0000-00-00
File Created	2021-01-13

REVISED_Statement A NSF ADVANCE Evaluation