Request for OMB Clearance
Quantitative
Evaluation of the ADVANCE Program (ADVANCE)
Supporting
Statement for Paperwork Reduction Act Submission
This request for Office of Management and Budget (OMB) review asks for clearance for two surveys and one accompanying glossary and instruction sheet to be used in the Quantitative Evaluation of the ADVANCE Program (ADVANCE). Directed at increasing the participation and advancement of women in academic science and engineering careers, ADVANCE is an agency-wide National Science Foundation (NSF) activity managed by the Directorate for Education and Human Resources (EHR). The NSF funds research and education in mathematics, science, and engineering through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research and/or educational institutions in all parts of the United States. The Foundation accounts for about 20 percent of Federal support to academic institutions for basic research. EHR is responsible for the health and continued vitality of the Nation’s science, technology, engineering, and mathematics (STEM) education and for providing leadership in the effort to improve education in these areas.
The goal of ADVANCE is to increase the representation and advancement of women in academic science and engineering careers, thereby developing a more diverse science and engineering workforce. ADVANCE encourages institutions of higher education and the broader science, technology, engineering and mathematics (STEM) community, including professional societies and other STEM-related, not-for-profit organizations, to address various aspects of STEM academic culture and institutional structure that may differentially affect women faculty and academic administrators. As such, ADVANCE is an integral part of NSF’s multifaceted strategy to broaden participation in the STEM workforce, and it supports the critical role of the Foundation in advancing the status of women in STEM academic careers.
For many decades, an increasing number of women have obtained STEM doctoral degrees, yet women continue to be significantly underrepresented in almost all STEM academic positions. The degree of underrepresentation varies among STEM disciplines, although women’s advancement to senior ranks and leadership is an issue in all fields. The ADVANCE program seeks to combat the cumulative effect of a variety of external factors that impact the number of women entering and advancing in academic STEM careers but are unrelated to women’s ability, interest, and technical skills, such as:
Organizational constraints of academic institutions;
Differential effects of work and family demands;
Implicit and explicit bias; and
Underrepresentation of women in academic leadership and decision-making positions.
The Quantitative Evaluation focuses on two components of ADVANCE: the Fellows component and Institutional Transformation (IT) component for Cohorts 1 and 2. Fellows awards, funded in 2002 and 2004, went to individuals with doctorates in STEM fields showing strong potential for and interest in pursuing academic careers. These individuals held postdoctoral positions or were disadvantaged in their academic careers because they had been out of the workforce due to family responsibilities or relocation of a spouse. Fellows were provided with three years of salary and research support at host institutions to establish strong sustainable research and education careers in academe. This clearance request applies to the 59 individuals in both cohorts of Fellows awards.
The five-year IT awards seek to make an impact through organizational change strategies designed to reduce or eliminate organizational barriers to the full participation of women in STEM academic careers, resulting in an academic environment that fosters women in and attracts women to science and engineering careers. Through ADVANCE IT grants, NSF supports comprehensive, institution-wide projects at institutions of higher education (IHEs) to transform institutional practices and climate. Recognition of the importance of taking an organizational approach is based on research indicating that the lack of women’s full participation in science and engineering academic careers is often a systemic consequence of the academic culture and organizational structure of institutions of higher education. This clearance request pertains to the 19 IHEs participating in the first two (2001 and 2003) cohorts of IT project awards.
Three broad research questions guide the evaluation:
How do selected gender equity outcomes for STEM women faculty, at both the individual and institutional levels, compare between the 19 Cohorts 1 and 2 grantees and other similar U.S. four-year colleges and universities that have not subsequently received ADVANCE IT awards? How have the IT activities been implemented and how successful have the grantees been at achieving medium- and long-term outcomes and longer term impacts?
What innovative institutional level measures of changes in gender equity climate and practices can be developed, and how can these be applied to evaluating the outcomes of the IT awards?
How do the career trajectories of Fellows compare to those of similar individuals who were not awarded these grants?
We will employ a quasi-experimental design (QED) using secondary datasets from the Survey of Doctorate Recipients (SDR) for comparison purposes to address a subset of the evaluation questions for both the IT and Fellows components. Both components will also include design elements not associated with the QED to address other descriptive evaluation questions. QEDs are generally considered one rigorous alternative to a randomized control trial (RCT) design when an RCT is not feasible or possible.1 As described below, at both the design and analysis stages, we have added features to enhance the comparability of the comparison groups and treatment groups for both the IT and Fellows components of the evaluation. The RFP for this evaluation required that comparison data be drawn from existing secondary datasets.
For the IT component QED, we plan to use a design comparing selected IT grantee outcomes to the same outcomes for other similar 4-year U. S. colleges and universities that have not received ADVANCE IT awards. Grantee outcomes will be collected from all 19 Cohorts 1 and 2 IT grantees with the help of contractor-hired data collectors working in conjunction with each of the 19 institutions. Comparison group outcomes will be extracted from the 2001 (pretest) and 2008 (posttest) administrations of the SDR. The design will allow us to compare changes in outcomes for the 19 IT grantees with changes in outcomes for the same period for other, similar U.S. 4-year colleges and universities based on SDR responses aggregated at the national level. 2
This design represents the best compromise between the requirements of a rigorous QED and the constraints imposed both by the relatively small number of grantees and the evaluation requirement of using available secondary national data sources for comparison purposes. Further, we have built the following additional features into the QED for the IT component to account for the fact that the IT grantee institutions may represent a special case of all 4-year US colleges and universities (for example, small liberal arts colleges are not well-represented among grantees):
In constructing the comparison group, we will select responses from 4-year colleges and universities in the SDR with an overall profile similar to that of the 19 ADVANCE IT grantees. Of the 19 IT grantees, according to the Carnegie classification, 9 are classified as research universities with very high research activity, 8 as research universities with high research activity, 1 as a Masters-level college and university and 1 as a Baccalaureate college. In order to construct the comparison group in this way, we will first need to manually recode the institutional identification of all respondents from the SDR according to the Carnegie classification.
We will exclude from the comparison group responses the 19 Cohorts 1 and 2 IT grantees as well as institutions that have received IT ADVANCE awards in subsequent cohorts, because including them might dilute the potential impacts of the program.
We will examine pre-post changes in identified outcomes rather than focusing on post-program outcomes per se, which will correct for some of the inherent incomparability between the treatment and comparison groups.
However, the types of outcome data collected by the SDR, which will be used for the QED comparison, are limited to academic position, rank, tenure, salary and job satisfaction. Consequently, to understand how IT grantees implemented their projects and measure changes in other outcomes, such as university policies, practices, structures and culture as well as faculty productivity, we will collect a much broader range of data from IT grantees (treatment group only) through the IT Survey.
The Fellows component will utilize a QED with equating 3 that compares outcomes for Fellows with those for SDR respondents with similar demographic and academic characteristics. Primary data on 2002 and 2004 awardees will be collected using the Fellows Survey. A comparison group will be identified from the SDR database using a propensity scoring matching (PSM) approach to control for bias when randomization is not possible. PSM uses a predicted probability of membership in the treatment vs. comparison group to create a counterfactual group, based on observed predictors obtained from logistic regression. In order to do this, we need individual-level data on demographic characteristics such as education, discipline, year of receipt of the doctorate, and race/ethnicity.
Primary data for the outcome evaluation of the ADVANCE program will be requested in two main forms. For the IT component, each of the 19 IT Principal Investigators (PIs) will be asked to designate several individuals (who may include themselves) at their institution to respond to a survey during a two-way videoconference discussion using the WebEx platform. In addition, contractor-hired data collectors will assist the 19 grantee institutions in extracting institutional outcome data for the evaluation. For the Fellows component, 59 recipients of the ADVANCE Fellows awards will be asked to respond to a mail survey.
The IT survey will be emailed to the most current project PI listed in NSF files. The forms will be introduced by a letter from Westat (Appendix A) describing the evaluation’s purpose and assuring the respondents that their answers will be reported in a way that cannot easily be linked to individual responses or reveal the institutions associated with particular comments. The IT survey, which requests information about each project’s IT-funded activities and institutional-level outcomes at different points in time, will be administered once, through a WebEx-supported process facilitated by the contractor. The PIs of each of the 19 IT projects will be asked to designate the most appropriate respondents for the survey at their institution. The introductory page of the survey instrument contains a section requesting each respondent’s name and contact information and verifying the start and end dates on record for the IT grant period.
The IT survey has three main sections:
Section A: Institutional Context and Culture (i.e., efforts outside ADVANCE targeted at faculty equity, accessibility of institutional faculty data, and the general impact of the institution’s leaders and resource and policy environment on faculty gender equity-focused activities).
Section B: ADVANCE IT Activities (i.e., broad approaches, implementation of policies and practices, structural changes within the institution, and effective and challenging activities).
Section C: ADVANCE IT Outcomes and Examples (changes in faculty hiring and promotion, changes in faculty satisfaction and collegiality, changes in institutional culture, and examination of social science research publications and other scholarship based on project activities).
Respondents will receive the IT survey several weeks in advance and will be expected to have reviewed and competed as many of the questions as possible prior to the facilitated WebEx session. However, we anticipate that respondents will also clarify and refine the answers—both for themselves and for the data collectors—as part of the associated discussion. The data collection process will be interactive, affording respondents the opportunity to add or elaborate on information they deem important for understanding the outcomes and impact of ADVANCE IT activities at their institution. By the same token, data collectors will be able to probe, as needed, to clarify the responses. (A copy of the IT survey and a one-page glossary of terms and phrases are provided in Appendix B.)
Contractor-hired data collectors working in conjunction with each of the grantee institutions will collect institutional-level outcome data for two time periods: 2001 and 2008, which for most Cohorts 1 and 2 awardees roughly correspond to the periods just before the beginning, and shortly after the end, of their IT grants. Selection of the 2001 and 2008 time periods enables comparison with data from 2001 and 2008 administrations of the SDR. We will request institutional level data for STEM fields targeted by IT-funded activities in the following areas: faculty recruitment, retention, tenure and promotion; leadership and administrative positions; and salary and professional benefits for tenure-track and tenured faculty.
The Fellows survey is designed to obtain information about each individual’s Fellowship-funded activities and outcomes. It will be administered once, through a paper-based survey sent by the U.S. mail to the most recent verified address of the 59 ADVANCE Fellows. The survey will be accompanied by a letter of introduction that explains the purpose of the evaluation, describes the nature of the information being requested, assures the respondents that every effort will be made to ensure that their answers will be kept confidential, and requests that the survey be returned to Westat in a self-addressed stamped envelope within two weeks of receipt.
The Fellows survey contains six sections:
Section A: Personal Information (i.e., start and end dates and institution associated with the grant, year Ph.D. was awarded, and Fellow’s racial/ethnic background, birth date, and current marital status).
Section B: At the Time You Applied for the ADVANCE Fellowship (i.e., employment status, marital status, job search status, career constraints, and career goals).
Section C: During the ADVANCE Fellowship Period (i.e., employment positions, work time allocation, ancillary grant fund allocation, professional activities, satisfaction with resources and other support, contextual professional influences, professional goals, changes in personal circumstance, activities, and accomplishments).
Section D: From the End of the ADVANCE Period Until Today (i.e., employment status currently and since the Fellowship, career constraints, career contributions from the Fellowship, and other relevant information respondents choose to add).
Section E: Questions for Comparison Purposes (as of October 1, 2008, based on Survey of Doctoral Recipients) (i.e., employment status, details, and salary; total earned income in 2007; marital status; and children living in the home).
Section F: Closing (i.e., request to attach current curriculum vitae and for current email and phone number in the event clarification is needed).
(Copies of the cover letter and the paper-based Fellows survey are provided in Appendix C.)
In order to inform program improvement, NSF is requesting evaluation of the first two cohorts of institutions in the IT component and both cohorts of Fellows awardees.
The primary purpose of this data collection is to provide information about outcomes for the 59 recipients of ADVANCE Fellowships and the 19 Cohorts 1 and 2 IT grantee institutions. NSF will use the data to understand changes in the participation and advancement of women in academic science and engineering careers. All information collected will be used to provide analytical and policy support to EHR, assisting NSF in making decisions about current ADVANCE programming, future funding, and other initiatives to improve STEM education. It may also provide information for NSF’s Government Performance and Results Act (GPRA) report.
The IT survey will use computer technology to minimize the response burden. For the IT survey, the person(s) whom the PI designates as the most appropriate survey respondents will participate in a two-way videoconference meeting facilitated by contractor staff using WebEx technology. When a contractor staff member schedules a two-way videoconference meeting with respondents at each institution, the WebEx system will generate an email to the PI with basic instructions on how to join the meeting. Respondents will connect by PC to the scheduled meeting using the email link and meeting password provided. They will also connect by phone by dialing a toll free number and entering the phone access code provided. Alternatively, when they join the meeting by PC, respondents can provide a phone number to have the system phone them. Contractor staff will also use video feed via webcam and respondents will be invited, but not required, to connect a webcam to their PC to incorporate a video of their participation in the conference.
Employing computer technology would not meaningfully reduce the response burden for the small number of individuals asked to respond to the Fellows survey.
Selected respondents to the IT survey may be interviewed as part of a companion formative evaluation of the IT component of the ADVANCE program being conducted for NSF by another contractor. However, the other evaluation focuses on program processes rather than outcomes and is primarily qualitative in nature, so there is minimal if any overlap.
The data that will be collected from the 59 Fellows refers to their experiences with the Fellowship awards as well as their subsequent career trajectories. There is no other source for this information.4 Whenever possible, we will minimize respondent burden and unnecessary duplication of effort by prefilling individual items (e.g., demographic data) based on information collected in prior project reports.
No information is to be collected from small businesses.
If the data for this evaluation are not collected, NSF will be unable to document outcomes for the first two cohorts of ADVANCE IT grantees or for recipients of ADVANCE Fellows awards. It will not be able to assess the degree to which the early ADVANCE program met its goals or to comply fully with the mandate that the Foundation evaluate its STEM education programs.
The data collection is in compliance with 5 CFR 1320.6.
To ascertain and so minimize the burden associated with collecting institutional level outcome data, over a period of several months we solicited feedback from seven individuals representing Cohorts 1 and 2 IT projects that exhibit a diversity of geographic, programmatic, and institutional characteristics. Specifically, project representatives were asked to comment on the availability and feasibility of obtaining specific data items.
AT NSF’s Joint Annual Meeting in June 2009, Westat project staff had the opportunity to test a few possible questions for the IT survey with current ADVANCE grantees. In addition, we utilized feedback from one representative of a Cohort 1 IT grantee institution, along with comments from three members of the evaluation’s Advisory Panel (AP), in making final revisions to the IT survey instrument. The same three AP members also reviewed and offered suggestions on the Fellows survey. Finally, survey experts outside the evaluation team were consulted to assist in estimating the response burden for the Fellows survey and accompanying materials.
The Federal Register notice was published at 74 FR 60300, on November 20, 2009, and one comment was received from Roger Clegg, President and General Counsel, Center for Equal Opportunity: “The ADVANCE Program is described as “address[ing] the underrepresentation and inadequate advancement of women,” and “gender equity outcomes,” for STEM faculty. We hope that the Program does not contemplate the use of quotas, numerical goals, or other discrimination or preferences as ways to address underrepresentation (a dubious term) or gender equity (likewise dubious). Such discrimination and preference is presumptively unconstitutional when engaged in by government, including federal government, agencies (see Mississippi University for Women v. Hogan, 458 U.S. 718 (1982) (gender discrimination requires an “exceedingly persuasive justification”), and faculty discrimination on the basis of sex is illegal under Title VII of the 1964 Civil Rights Act, 42 U.S.C. 2000e et seq.”
We responded via email on November 30, 2009: Thank you for your comment in response to the Federal Register notice published November 20, 2009 “NATIONAL SCIENCE FOUNDATION Agency Information Collection Activities: Proposed Collection, Comment Request.” The NSF ADVANCE program does not use quotas, numerical goals, or other discrimination or preferences. Further information on the ADVANCE program is available at: http://www.nsf.gov/advance.
No payments or gifts will be made to either the IT Survey or the Fellows Survey respondents.
The primary data collected in the Fellows Survey will be kept confidential and maintained in accordance with the requirements of the Privacy Act of 1974. For the IT Survey, even though the respondents are institutions, all efforts will also be made to report data in a manner that does not link specific comments to identifiable persons or institutions. Data for both surveys will be available only to eight designated project staff members who have signed a pledge of confidentiality and possess up-to-date certificates attesting to their having completed Westat’s Human Subjects and Data Security trainings. In the cover letter that will accompany the IT Survey respondents will be informed about efforts to minimize the ability to link comments to specific institutions as well as procedures to maintain the security of the data (see Appendix A). Respondents to the Fellows Survey will be similarly informed about procedures to maintain the confidentiality and security of the data in the cover letter that will accompany the survey (see Appendix C). Respondents will also be reminded of these considerations in the introductions to the respective survey instruments (see Appendices B and C).
The respondents’ names and contact information are the only individually identifiable information collected by the IT survey; this information is necessary to allow Westat staff to follow up for clarification of responses, if needed. The salary and employment outcome data will be aggregated at the institutional level, thereby protecting individual identities.
The Fellows survey does request personally identifying information, including name, contact information, and birth date, as well as information on grants and publications. It also asks respondents to provide copies of their most recent curriculum vitae. Hard copies of the returned Fellows surveys will be kept in a locked file cabinet; an alphanumeric code will be associated with each name to protect the respondent’s identity. Other hard copy materials, including curriculum vitae, will also be kept in a locked file cabinet.
For both the IT and Fellows surveys, data maintained in electronic form, including any WebEx recordings generated automatically by the system, will be stored on a password-protected server accessible only to designated project staff. Evaluation results will be reported in aggregated form and will include no information that will enable identification of specific individuals. For the IT survey, we will also make every effort to conceal the specific institutional affiliations associated with all comments.
The SDR data to be used is confidential data under a license with the Science Resources Statistics Division of the National Science Foundation. We have obtained access to the data and are committed to adhering to all the conditions set forth by the license.
The IT survey does not contain personal questions of a sensitive nature. The Fellows survey includes questions requesting data that may be construed as sensitive, including salary and income data as well as data on relationship status. This information is crucial to effectively evaluate the outcomes of the ADVANCE Fellows component. As noted in Section A.10, Fellows’ names will be associated with alternate alphanumeric identifiers to eliminate the danger that any unintentional disclosure may enable individual identifiers to be connected with sensitive information. In addition, all the safeguards discussed in Section A.10 will apply to these data.
In keeping with NSF’s program goals, the instrument will be administered using methods designed to collect essential evaluation data with the least possible burden to respondents.
The estimated one-time annual response burden is 149 person-hours. There are two classes of respondents: IT respondents and Fellows. IT respondents are of two types: PIs on ADVANCE IT-supported projects and PI’s designees, who for purposes of the IT survey are assumed to be at the associate professor level. Burden hours per response are estimated on the basis of discussions with NSF program officers, discussions with a sample of IT project personnel, and Westat’s experience in administering similar surveys.
The calculations used to determine overall response burden and estimates by type of module are presented in Exhibit 1.
Respondent type |
Number of respondents |
Hours perrespondent type |
Annual person-hour total |
PIs and designees – Survey |
60 |
1.5 |
90 |
|
|
|
|
Fellows |
59 |
1.0 |
59 |
Total |
119 |
2.5 |
149 |
The overall one-time cost to the respondents is estimated to be $4,634. The hourly wage rates are based on information found in Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2008-09 Edition, Teachers—Postsecondary, retrieved from http://www.bls.gov/oco/ocos066.htm. Calculations are shown in Exhibit 2.
Respondent type |
Calculation(# of respondents x hourly wage x hours to complete) |
Cost burden |
PIs and designees – Survey |
60 x $33.61 ($69,911/yr associate professor) x 1.5 hours |
$3,025 |
|
|
|
Fellows |
59 x $28.20 ($58,662/yr assistant professor) x 1 hour |
$1,609 |
Total |
$4,634 |
|
As shown in Exhibit 2, the one-time cost burden to Fellows respondents is minimal. The one-time cost burden to IT survey respondents is also quite moderate, averaging about $160 per grantee institution.
The total estimated cost to the government for data collection, analysis, and reporting activities for this study is approximately $417,327, as shown in Exhibit 3.
Personnel |
$411,527 |
|
|
Computing |
$4,113 |
Copying, postage, telephone |
$1,687 |
Total Costs |
$417,327 |
No changes in burden are anticipated.
The data are being collected for evaluation purposes. We anticipate that data collection for the Fellows Survey will occur in June, July and August of 2010 and that data collection for the IT Survey and associated efforts will occur in fall 2010. The final report, incorporating analyses of these data as well as comparisons with the 2001 and 2008 SDR data, will be completed in the late summer of 2011.
Westat is conducting this third-party study of ADVANCE on behalf of NSF. After the products are delivered, NSF determines whether the quality of the products merits publication by NSF (i.e., NSF is the exclusive publisher of the information being gathered). Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF website) in which to publish.
NSF has every intention of making the findings from this evaluation available to interested audiences, while still respecting the relationship of trust established with the survey respondents. We will use various methods to communicate the results, as appropriate, including presentations at STEM conferences and creation of a user-friendly report for dissemination.
Not applicable.
No exceptions apply.
1 C.f., US Government Accountability Office, PROGRAM EVALUATION: A Variety of Rigorous Methods Can Help Identify Effective Interventions, November, 2009, GAO -10-30 www.gao.gov/new.items/d1030.pdf; What Works Clearinghouse: http://ies.ed.gov/ncee/wwc/references/iDocViewer/Doc.aspx?docId=20&tocId=2#quasi
2 Frehill and colleagues compared outcomes for ADVANCE grantees with those for all 4-year U.S. colleges and universities in measuring intermediate indicators of institutional transformation, (C.f. Frehill, L. M., Jeser-Cannavale, C., & Malley, J. E. (2007). Measuring outcomes: Intermediate indicators of institutional transformation. In A. J. Stewart, J. E. Malley, and D. LaVaque-Manty (Eds.), Transforming science and engineering: Advancing academic women (pp. 298–322). Ann Arbor, MI: University of Michigan Press.
3 See What Works Clearinghouse http://ies.ed.gov/ncee/wwc/references/iDocViewer/Doc.aspx?docId=20&tocId=2#quasi
4 We have no way of knowing how many, if any of the Fellows may be included in the sample for the Survey of Doctorate Recipients (SDR). Moreover, while the SDR does contain a limited number of the same items as the Fellows Survey, it does not ask questions pertinent to the Fellowship experience nor does it ask about all aspects of the respondents’ subsequent career trajectories of interest in this evaluation.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | splimpto |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |