Request for OMB Clearance
Quantitative
Evaluation of the ADVANCE Program (ADVANCE)
Supporting
Statement for Paperwork Reduction Act Submission
This request for Office of Management and Budget (OMB) review asks for clearance for two surveys, an institutional outcomes indicator data form (indicator data form), and two accompanying glossary and instructions sheets to be used in the Quantitative Evaluation of the ADVANCE Program (ADVANCE). Directed at increasing the participation and advancement of women in academic science and engineering careers, ADVANCE is an agency-wide National Science Foundation (NSF) activity managed by the Directorate for Education and Human Resources (EHR). The NSF funds research and education in mathematics, science, and engineering through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research and/or educational institutions in all parts of the United States. The Foundation accounts for about 20 percent of Federal support to academic institutions for basic research. EHR is responsible for the health and continued vitality of the Nation’s science, technology, engineering, and mathematics (STEM) education and for providing leadership in the effort to improve education in these areas.
The goal of ADVANCE is to increase the representation and advancement of women in academic science and engineering careers, thereby developing a more diverse science and engineering workforce. ADVANCE encourages institutions of higher education and the broader science, technology, engineering and mathematics (STEM) community, including professional societies and other STEM-related, not-for-profit organizations, to address various aspects of STEM academic culture and institutional structure that may differentially affect women faculty and academic administrators. As such, ADVANCE is an integral part of NSF’s multifaceted strategy to broaden participation in the STEM workforce, and it supports the critical role of the Foundation in advancing the status of women in STEM academic careers.
For many decades, an increasing number of women have obtained STEM doctoral degrees, yet women continue to be significantly underrepresented in almost all STEM academic positions. The degree of underrepresentation varies among STEM disciplines, although women’s advancement to senior ranks and leadership is an issue in all fields. The ADVANCE program seeks to combat the cumulative effect of a variety of external factors that impact the number of women entering and advancing in academic STEM careers but are unrelated to women’s ability, interest, and technical skills, such as:
Organizational constraints of academic institutions;
Differential effects of work and family demands;
Implicit and explicit bias; and
Underrepresentation of women in academic leadership and decision-making positions.
The Quantitative Evaluation focuses on two components of ADVANCE: the Fellows and Institutional Transformation (IT) components for Cohorts 1 and 2. Fellows awards, funded in 2002 and 2004, went to individuals with doctorates in STEM fields showing strong potential for and interest in pursuing academic careers. These individuals held postdoctoral positions or were disadvantaged in their academic careers because they had been out of the workforce due to family responsibilities or relocation of a spouse. Fellows were provided with three years of salary and research support at host institutions to establish strong sustainable research and education careers in academe. This clearance request applies to the 59 individuals in both cohorts of Fellows awards.
The five-year IT awards seek to make an impact through organizational change strategies designed to reduce or eliminate organizational barriers to the full participation of women in STEM academic careers, resulting in an academic environment that fosters women in and attracts women to science and engineering careers. Through ADVANCE IT grants, NSF supports comprehensive, institution-wide projects at institutions of higher education (IHEs) to transform institutional practices and climate. Recognition of the importance of taking an organizational approach is based on research indicating that the lack of women’s full participation in science and engineering academic careers is often a systemic consequence of the academic culture and organizational structure of institutions of higher education. This clearance request pertains to the 19 IHEs participating in the first two (2001 and 2003) cohorts of IT project awards.
Three broad research questions guide the evaluation:
How do selected gender equity outcomes for STEM women faculty, at both the individual and institutional levels, compare between the 19 Cohorts 1 and 2 grantees and other similar U.S. four-year colleges and universities that have not subsequently received ADVANCE IT awards? How have the IT activities been implemented and how successful have the grantees been at achieving medium- and long-term outcomes and longer term impacts?
What innovative institutional level measures of changes in gender equity climate and practices can be developed, and how can these be applied to evaluating the outcomes of the IT awards?
How do the career trajectories of Fellows compare to those of similar individuals who were not awarded these grants?
Westat will collect information about the outcomes and impacts associated with IT awards by surveying the first two cohorts of IT awardees. Both cohorts of Fellows will also be surveyed to obtain information on their experiences as ADVANCE Fellows as well as their subsequent career trajectories. The evaluation will apply a quasi-experimental design using secondary data from the Survey of Doctorate Recipients (SDR) to compare institutional STEM faculty outcomes for the 19 Cohorts 1 and 2 IT grantee institutions with those for all similar U.S. colleges and universities that have not received ADVANCE IT awards. Data from the SDR will also be used to compare career outcomes for Fellows and individuals with similar characteristics who did not receive these awards.
Primary data for the outcome evaluation of the ADVANCE program will be requested in three forms. For the IT component, each of the 19 IT Principal Investigators (PIs) will be asked to designate one or more individuals (who may include themselves) at their institution to 1) respond to a survey during a two-way videoconference discussion using the WebEx platform, and 2) complete a data form that requests quantitative institutional-level outcome data on faculty gender equity in STEM fields. For the Fellows component, 59 recipients of the ADVANCE Fellows awards will be asked to respond to a mail survey.
The IT survey and outcome indicator data form will be emailed to the most current project PI listed in NSF files. The forms will be introduced by a letter from Westat (Appendix A) describing the evaluation’s purpose and assuring the respondents that their answers will not be reported in a way that can link to individual responses or reveal the institutions associated with particular comments. The IT survey, which requests information about each project’s IT-funded activities and institutional-level outcomes at different points in time, will be administered once, through a WebEx-supported process facilitated by the contractor. The PIs of each of the 19 IT projects will be asked to designate the most appropriate respondents for the survey at their institution. The introductory page of the survey instrument contains a section requesting each respondent’s name and contact information and verifying the start and end dates on record for the IT grant period.
The IT survey has three main sections:
Section A: Institutional Context and Culture (i.e., efforts outside ADVANCE targeted at faculty equity, accessibility of institutional faculty data, and the general impact of the institution’s leaders and resource and policy environment on faculty gender equity-focused activities).
Section B: ADVANCE IT Activities (i.e., broad approaches, implementation of policies and practices, structural changes within the institution, and effective and challenging activities).
Section C: ADVANCE IT Outcomes and Examples (changes in faculty hiring and promotion, changes in faculty satisfaction and collegiality, changes in institutional culture, and examination of social science research publications and other scholarship based on project activities).
Respondents will receive the IT survey several weeks in advance and will be expected to have reviewed the questions prior to the facilitated WebEx session. However, we anticipate that respondents will clarify and refine the answers—both for themselves and for the data collectors—as part of the associated discussion. The data collection process will be interactive, affording respondents the opportunity to add or elaborate on information they deem important for understanding the outcomes and impact of ADVANCE IT activities at their institution. By the same token, data collectors will be able to probe, as needed, to clarify the responses. (A copy of the IT survey and a one-page glossary of terms and phrases are provided in Appendix B.)
The IT quantitative data form will collect institutional-level outcome data for two time periods: 2001 and 2008, which for most Cohorts 1 and 2 awardees roughly correspond to the periods just before the beginning, and shortly after the end, of their IT grants. (However, 13 institutions received no-cost time extensions into 2009.) Selection of the 2001 and 2008 time periods enables comparison with data from 2001 and 2008 administrations of the SDR. The form requests data for the STEM departments/units targeted by IT-funded activities in the following areas: faculty recruitment, retention, tenure and promotion; leadership and administrative positions; and salary and professional benefits for tenure-track and tenured faculty. NSF strongly encouraged IT project personnel to collect these data items as part of a toolkit of quantitative indicators designed both to assist projects in monitoring their activities and to provide analytical and policy support to EHR. However, our review of project documents shows considerable inconsistency across the Cohorts 1 and 2 IT awardees with respect to which items were collected and how these were reported. Consequently, our evaluation represents the first comprehensive effort to collect these important outcome data in a consistent format to facilitate systematic analysis. (A copy of the IT outcome indicator data form and accompanying instructions is provided in Appendix C.)
The Fellows survey is designed to obtain information about each individual’s Fellowship-funded activities and outcomes. It will be administered once, through a paper-based survey sent by the U.S. mail to the last known address of the 59 ADVANCE Fellows. The survey will be accompanied by a letter of introduction that explains the purpose of the evaluation, describes the nature of the information being requested, assures the respondents that their answers will be kept completely confidential, and requests that the survey be returned to Westat in a self-addressed stamped envelope within two weeks of receipt.
The Fellows survey contains six sections:
Section A: Personal Information (i.e., start and end dates and institution associated with the grant, year Ph.D. was awarded, and Fellow’s racial/ethnic background, birth date, and current marital status).
Section B: At the Time You Applied for the ADVANCE Fellowship (i.e., employment status, marital status, job search status, career constraints, and career goals).
Section C: During the ADVANCE Fellowship Period (i.e., employment positions, work time allocation, ancillary grant fund allocation, professional activities, satisfaction with resources and other support, contextual professional influences, professional goals, changes in personal circumstance, activities, and accomplishments).
Section D: From the End of the ADVANCE Period Until Today (i.e., employment status currently and since the Fellowship, career constraints, career contributions from the Fellowship, and other relevant information respondents choose to add).
Section E: Questions for Comparison Purposes (as of October 1, 2008, based on Survey of Doctoral Recipients) (i.e., employment status, details, and salary; total earned income in 2007; marital status; and children living in the home).
Section F: Closing (i.e., request to attach current curriculum vitae and for current email and phone number in the event clarification is needed).
(Copies of the cover letter and the paper-based Fellows survey are provided in Appendix D.)
In order to inform program improvement, NSF is requesting evaluation of the first two cohorts of institutions in the IT component and both cohorts of Fellows awardees.
The primary purpose of this data collection is to provide information about outcomes for the 59 recipients of ADVANCE Fellowships and the 19 Cohorts 1 and 2 IT grantee institutions. NSF will use the data to understand changes in the participation and advancement of women in academic science and engineering careers. All information collected will be used to provide analytical and policy support to EHR, assisting NSF in making decisions about current ADVANCE programming, future funding, and other initiatives to improve STEM education. It may also provide information for NSF’s Government Performance and Results Act (GPRA) report.
Both the IT survey and quantitative indicator data collection form will use computer technology to minimize the response burden. For the IT survey, the person(s) whom the PI designates as the most appropriate survey respondents will participate in a two-way videoconference meeting facilitated by contractor staff using WebEx technology. When a contractor staff member schedules a two-way videoconference meeting with respondents at each institution, the WebEx system will generate an email to the PI with basic instructions on how to join the meeting. Respondents will connect by PC to the scheduled meeting using the email link and meeting password provided. They will also connect by phone by dialing a toll free number and entering the phone access code provided. Alternatively, when they join the meeting by PC, respondents can provide a phone number to have the system phone them. Contractor staff will also use video feed via webcam and respondents will be invited, but not required, to connect a webcam to their PC to incorporate a video of their participation in the conference.
For the indicator data form, IT project personnel will use a Microsoft Excel file composed of formatted one-page worksheets to enter the requested data on STEM faculty for each unit targeted by the ADVANCE project at their institution. This computer application, likely to be familiar to all respondents, will simplify the process of organizing and conveying a variety of detailed quantitative data elements. Questionable or incomplete entries will be evident to contractor staff at the time of the WebEx-enabled videoconference and can be called to respondents’ attention before data are analyzed and the results submitted to NSF.
Employing computer technology would not meaningfully reduce the response burden for the small number of individuals asked to respond to the Fellows survey.
Selected respondents to the IT survey may be interviewed as part of a companion formative evaluation of the IT component of the ADVANCE program being conducted for NSF by another contractor. However, the other evaluation focuses on program processes rather than outcomes and is primarily qualitative in nature, so there is minimal if any overlap. As noted earlier in Section A, the quantitative indicator data collection form is the only means of collecting comprehensive institutional level outcome data across the 19 Cohorts 1 and 2 IT grantees. Moreover, these data are not available in any national dataset, including the Integrated Postsecondary Education Data System (IPEDS). Wherever possible, we will prefill data items using information from past project reports and ask that the respondents confirm the accuracy of these data as a way of minimizing response burden and unnecessary duplication of effort on the part of the IT grantees.
The data that will be collected from the 59 Fellows refers to their experiences with the Fellowship awards as well as their subsequent career trajectories. There is no other source for this information.1 Whenever possible, we will minimize respondent burden and unnecessary duplication of effort by prefilling individual items (e.g., demographic data) based on information collected in prior project reports.
No information is to be collected from small businesses.
If the data for this evaluation are not collected, NSF will be unable to document outcomes for the first two cohorts of ADVANCE IT grantees or for recipients of ADVANCE Fellows awards. It will not be able to assess the degree to which the early ADVANCE program met its goals or to comply fully with the mandate that the Foundation evaluate its STEM education programs.
The data collection is in compliance with 5 CFR 1320.6.
To ascertain and so minimize the burden associated with completing the outcome indicator data form, over a period of several months we solicited feedback on the form from seven individuals representing Cohorts 1 and 2 IT projects that exhibit a diversity of geographic, programmatic, and institutional characteristics. Specifically, project representatives were asked to comment on the availability and feasibility of obtaining the requested data items, as well as the understandability and ease of use of the form. Based on their comments, we added introductory descriptions, shortened the form by eliminating data elements deemed too difficult to collect, clarified definitions, and revised selected items.
AT NSF’s Joint Annual Meeting in June 2009, Westat project staff had the opportunity to test a few possible questions for the IT survey with current ADVANCE grantees. In addition, we utilized feedback from one representative of a Cohort 1 IT grantee institution, along with comments from three members of the evaluation’s Advisory Panel (AP), in making final revisions to the IT survey instrument. The same three AP members also reviewed and offered suggestions on the Fellows survey. Finally, survey experts outside the evaluation team were consulted to assist in estimating the response burden for the Fellows survey and accompanying materials.
The Federal Register notice was published at 74 FR 60300, on November 20, 2009, and one comment was received from Roger Clegg, President and General Counsel, Center for Equal Opportunity: “The ADVANCE Program is described as “address[ing] the underrepresentation and inadequate advancement of women,” and “gender equity outcomes,” for STEM faculty. We hope that the Program does not contemplate the use of quotas, numerical goals, or other discrimination or preferences as ways to address underrepresentation (a dubious term) or gender equity (likewise dubious). Such discrimination and preference is presumptively unconstitutional when engaged in by government, including federal government, agencies (see Mississippi University for Women v. Hogan, 458 U.S. 718 (1982) (gender discrimination requires an “exceedingly persuasive justification”), and faculty discrimination on the basis of sex is illegal under Title VII of the 1964 Civil Rights Act, 42 U.S.C. 2000e et seq.”
We responded via email on November 30, 2009: Thank you for your comment in response to the Federal Register notice published November 20, 2009 “NATIONAL SCIENCE FOUNDATION Agency Information Collection Activities: Proposed Collection, Comment Request.” The NSF ADVANCE program does not use quotas, numerical goals, or other discrimination or preferences. Further information on the ADVANCE program is available at: http://www.nsf.gov/advance.
An institutional grant of at least $4,000 will be made to all IT projects to assist with the costs of collaborative participation in the IT survey as well as completion of the outcome indicator data form. Those institutions that prove willing and able to supply more detailed outcome information at the departmental level will receive grants of $10,000. Respondents to the Fellows survey will receive no payments or gifts.
The primary data collected in both surveys will be kept strictly confidential and maintained in accordance with the requirements of the Privacy Act of 1974. Data will be available only to eight designated project staff members who have signed a pledge of confidentiality and possess up-to-date certificates attesting to their having completed Westat’s Human Subjects and Data Security trainings. Respondents to both the IT Survey and Fellows Survey will be informed about procedures to maintain the confidentiality and security of the data in the cover letters that will accompany both surveys (see Appendices A and D) and will also be reminded of data confidentiality and security in the introductions to both survey instruments (see Appendices B and D).
The respondents’ names and contact information are the only individually identifiable information collected by the IT survey; this information is necessary to allow Westat staff to follow up for clarification of responses, if needed. The IT outcome indicator data form will collect information on salary and employment aggregated at the department/unit level, thereby protecting individual identities.
The Fellows survey does request personally identifying information, including name, contact information, and birth date, as well as information on grants and publications. It also asks respondents to provide copies of their most recent curriculum vitae. Hard copies of the returned Fellows surveys will be kept in a locked file cabinet; an alphanumeric code will be associated with each name to protect the respondent’s identity. Other hard copy materials, including curriculum vitae, will also be kept in a locked file cabinet.
For both the IT and Fellows surveys, data maintained in electronic form, including any WebEx recordings generated automatically by the system, will be stored on a password-protected server accessible only to designated project staff. Evaluation results will be reported in aggregated form and will include no information that will enable identification of specific individuals. For the IT survey, we will also make every effort to conceal the specific institutional affiliations associated with all comments.
The IT survey does not contain personal questions of a sensitive nature. The Fellows survey includes questions requesting data that may be construed as sensitive, including salary and income data as well as data on relationship status. This information is crucial to effectively evaluate the outcomes of the ADVANCE Fellows component. As noted in Section A.10, Fellows’ names will be associated with alternate alphanumeric identifiers to eliminate the danger that any unintentional disclosure may enable individual identifiers to be connected with sensitive information. In addition, all the safeguards discussed in Section A.10 will apply to these data.
In keeping with NSF’s program goals, the instrument will be administered using methods designed to collect essential evaluation data with the least possible burden to respondents.
The estimated one-time annual response burden is 1,859 person-hours. There are two classes of respondents: IT respondents and Fellows. IT respondents are of two types: PIs on ADVANCE IT-supported projects and PI’s designees, who for purposes of the IT survey are assumed to be at the associate professor level, but for purposes of the outcome indicator data collection form are assumed to be instructor-level researchers. Burden hours per response are estimated on the basis of discussions with NSF program officers, discussions with a sample of IT project personnel, and Westat’s experience in administrating similar surveys.
The calculations used to determine overall response burden and estimates by type of module are presented in Exhibit 1.
Respondent type |
Number of respondents |
Hours perrespondent type |
Annual person-hour total |
PIs and designees – Survey |
60 |
1.5 |
90 |
PI designees – Indicator Data Form |
19 |
90.0 |
1,710 |
Fellows |
59 |
1.0 |
59 |
Total |
138 |
92.5 |
1,859 |
The overall one-time cost to the respondents is estimated to be $39,501. The hourly wage rates are based on information found in Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2008-09 Edition, Teachers—Postsecondary, retrieved from http://www.bls.gov/oco/ocos066.htm. Calculations are shown in Exhibit 2.
Respondent type |
Calculation(# of respondents x hourly wage x hours to complete) |
Cost burden |
PIs and designees – Survey |
60 x $33.61 ($69,911/yr associate professor) x 1.5 hours |
$3,025 |
PI designees- – Indicator Data Form |
19 x $20.39 ($42,406/yr instructor) x 90 hours |
$34,867 |
Fellows |
59 x $28.20 ($58,662/yr assistant professor) x 1 hour |
$1,609 |
Total |
$39,501 |
|
The one-time cost burden to Fellows respondents is minimal.
In light of the one-time cost burden to IT respondents or record-keepers in responding to the survey instrument and outcome indicator data form, an institutional grant of at least $4,000 will be made to all IT projects. Larger grants of up to $10,000 will go to those IT institutions that need considerable additional resources to support more in-depth data collection. As described previously, both the IT survey and the data form were prescreened by sources outside the NSF and the evaluation team and were modified to minimize the overall burden to study respondents.
The total estimated cost to the government for data collection, analysis, and reporting activities for this study is approximately $417,327, as shown in Exhibit 3.
Personnel |
$311,527 |
Incentives |
$100,000 |
Computing |
$4,113 |
Copying, postage, telephone |
$1,687 |
Total Costs |
$417,327 |
No changes in burden are anticipated.
The data are being collected for evaluation purposes. We anticipate that data collection for both surveys will occur in April, May and June of 2010 and that data collection for the indicator data form will extend through July of 2010. The final report, incorporating analyses of these data as well as comparisons with the 2001 and 2008 SDR data, will be completed in the late summer of 2011.
Westat is conducting this third-party study of ADVANCE on behalf of NSF. After the products are delivered, NSF determines whether the quality of the products merits publication by NSF (i.e., NSF is the exclusive publisher of the information being gathered). Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF website) in which to publish.
Not applicable.
No exceptions apply.
1 We have no way of knowing how many, if any of the Fellows may be included in the sample for the Survey of Doctorate Recipients (SDR). Moreover, while the SDR does contain a limited number of the same items as the Fellows Survey, it does not ask questions pertinent to the Fellowship experience nor does it ask about all aspects of the respondents’ subsequent career trajectories of interest in this evaluation.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | splimpto |
| File Modified | 0000-00-00 |
| File Created | 2021-02-03 |