Supporting Statement (3145-0136)

REQUEST FOR CLEARANCE

 

National Science Foundation

Directorate of Education and Human Resources

Division of Undergraduate Education

Science, Technology, Engineering, and Mathematics Talent Expansion Program

Monitoring System (STEP)

Attachment I

Section A

Introduction

This request for Office of Management and Budget (OMB) review asks for a renewal of clearance for the monitoring data collection for the  Science, Technology, Engineering, and Mathematics Talent Expansion Program (STEP) that is part of the National Science Foundation's (NSF) Directorate for Education and Human Resources (EHR) Generic Clearance (OMB 3145-0136), which expires on March 31, 2011. The STEP program is administered by the EHR Division of Undergraduate Education (DUE).

A.1. Circumstances Requiring the Collection of Data

STEP responds to the critical need for a highly trained and educated workforce in established or emerging science, technology, engineering, and mathematics (STEM) disciplines (Type 1) and for educational research on retention and degree attainment in STEM (Type 2). For further information about the NSF STEP program, please go here.
 

STEP seeks to increase the number of students (U.S. citizens or permanent residents) receiving associate's or baccalaureate degrees in established or emerging STEM fields. Increasing the number of undergraduate students obtaining degrees in STEM fields will provide a workforce that is prepared to ensure a healthy economy, respond to demands for national security, and maintain and elevate the quality of life and standard of living in the United States through technological and scientific advancements.

 

The program awards grants to institutions of higher education in the United States and its territories, to consortia thereof, or to nonprofit entities that have established consortia among such institutions of higher education. The institutions of higher education must offer either associate's degrees or baccalaureate degrees in STEM. Associate's degree-granting institutions with a demonstrated record of articulation to STEM baccalaureate programs need not necessarily grant associate's degrees in STEM fields in order to be eligible for this program.

The STEP monitoring system will be the primary source of data on the extent to which project goals are being achieved. The monitoring system consists of a Web-based data collection instrument that will be completed by the principal investigator (PI) of the lead institution for each STEP award and by the institution leader for each partner institution associated with the award.

A.2. Purposes and Uses of the Data

The information collected in this task is required for effective administration, communication, and program and project monitoring; for meeting reporting requirements; for measuring attainment of NSF's program, project, and strategic goals as laid out in NSF’s Strategic Plan; and as a baseline for future program evaluations.

The primary purpose of this collection is to provide data and information for effective program management and monitoring of program activities. This data collection activity is designed to track the extent to which STEP projects meet the objectives of the program. Within the DUE division, this information is used to administer and monitor the progress of the program. The findings are used to recommend, among other things, administrative changes in program functions, level of award support, individual program focus and emphasis, and recruiting efforts.

In recent guidance from the Director of OMB, M-10-32, the need for rigorous evaluations and the objectives of evaluations of programs were clearly outlined, including the use of evaluation resources. Because the collection of data contained in this monitoring effort contributes to the formal evaluation of the program and provides regular measures of program performance by accumulating operating information from each project in the program, this guidance is particularly pertinent to this request.

“Improving and coordinating the use of existing evaluation resources. In addition to the voluntary evaluation initiative, agencies should continue to carefully assess, report on, and allocate the base funds and resources that the agencies have for conducting evaluation. Agencies are encouraged to share information beyond what is requested in guidance and consult with OMB’s Resource Management Offices (RMOs) to coordinate and improve the design, implementation, and utilization of evaluations.”

 

These directives establish an ongoing need for NSF to engage in an interactive process of collecting information and using it to improve program services and processes.

The STEP program also uses the data to fulfill reporting requirements. As a part of its performance assessment activities, NSF relies on the judgment of external experts to maintain high standards of program management. Directorate and Office advisory committees (ACs) meet twice a year, while Committees of Visitors (COVs) for divisions or programs meet once every three years. Data collected in the STEP monitoring system may be used to report to these committees on program activities. In addition, NSF is required to measure the attainment of its program, project, and strategic goals by the President’s Accountable Government Initiative, the Government Performance and Results Act (GPRA) Modernization Act of 2010, and the NSF’s Strategic Plan. Data collected in the system help NSF management examine their progress towards the Foundation’s goals and respond to these reporting requirements.

Finally, the data can also be used as a preliminary step in more detailed future evaluation efforts. EHR makes these data available to NSF staff, EHR contractors with responsibility for the collection, and DUE program managers and their staff and contractors. Information collected may also be disseminated, in aggregate form, to the broad undergraduate science, technology, engineering, and mathematics education community upon request to and approval from the NSF.

Each PI/institution leader of a STEP award is required to provide annual data using the Web-based data collection system (see appendices A1 and A2). The following is an overview of the types of information collected:

·         Contact information for each PI/institution leader

·         Total undergraduate enrollment for each institution, including the institution's cutoff date for determining total enrollment

·         Total undergraduate full-time equivalent (FTE) enrollment for each institution, including the institution's definition of an FTE  undergraduate student and method for calculating undergraduate FTE enrollment

·         Each institution's requirements for selecting a major [4-year institutions]/considering a student a STEM major [2-year institutions]

·         Aggregated data by STEM discipline (as well as non-STEM and undeclared), including the number of majors, degrees granted, and transfers to 4-year institutions [2-year institutions only], broken out by gender and race/ethnicity

·         Information on strategies/activities supported under STEP grants to increase the number of STEM graduates or STEM majors transferring to 4-year institutions [2-year institutions only] (this includes status, general categories, targeted student groups, brief description, number of undergraduate/high school students participating, indications of success, and implementation challenges)

·         Information on challenges involved in increasing the number of STEM graduates or STEM majors transferring to 4-year institutions [2-year institutions only] not related to the previously mentioned strategies/activities

·         Information on influences beyond the scope of STEP grants and not included in the previously mentioned strategies/activities or challenges that may affect the number of STEM graduates or STEM majors transferring to 4-year institutions [2-year institutions only]

·         Estimates of the number of distinct undergraduate and high school students participating in the strategies/activities carried out under STEP grants

·         Number of undergraduate students receiving direct funding under STEP grants

·         Types of efforts for which undergraduate students were funded

A crosswalk of data elements can be found in appendix B.

A.3. Use of Information Technology To Reduce Burden

Web-based systems can facilitate respondents' data entry across computer platforms. One innovative feature of many of the individual Web systems is the thorough editing of all submitted data for completeness, validity, and consistency. Programmed data validations prevent most invalid data from entering the system, and questionable or incomplete entries are called to respondents' attention before they are submitted to NSF. Web-based data collection instruments employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus and predefined charts and graphics. All of these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden. Other efforts to reduce burden include pre-filling previously entered respondent contact information and allowing users to carry forward previously entered  textual data that are relevant to the current collection cycle (e.g., explanations regarding the cutoff date used to determine total undergraduate enrollment, definitions for a full-time equivalent undergraduate student).

The data for this monitoring effort are collected by 508-compliant Web-based data collection instruments. The instrument structure allows respondents to move between a menu screen and screens addressing individual topics. The question format is composed of text boxes, radio buttons, and checkboxes. Respondents may enter and leave the instrument as often as they desire and continue to change their responses until they complete their authorized final submission. Respondents have access to screen-specific help material as well as a Help & Instructions feature, which includes a discipline crosswalk, frequently asked questions, and a glossary with definitions for each race/ethnicity category, to assist them in completing their responses. Additionally, since the collection is Web-based, minor changes in wording and displays can be easily made in response to user feedback.

A.4. Efforts To Identify Duplication

This system does not duplicate other NSF efforts. Comparable data are not currently being collected on an annual basis for the STEP program. In addition, the collection is coordinated with the NSF FastLane Project Reports system (OMB 3145-0058) to ensure that the two collections do not collect similar data. Aggregate data will be shared with NSF-funded researchers as appropriate, thereby minimizing the possibility that other researchers will duplicate these efforts in their own future collections.

A.5. Small Business

No information is to be collected from small businesses.

A.6. Consequences of Not Collecting the Information

Without this information NSF would be unable to document the effectiveness or output of the STEP program. The Foundation would be unable to disseminate information to other projects and institutions about successful approaches to increasing the number of STEM graduates or STEM majors transferring to 4-year institutions. Additionally, without this feedback NSF would have no way of making systematic modifications to the STEP program (e.g., adequacy of funding amount, duration of award, institutional supports needed). Moreover, NSF would be unable to comply fully with the congressional mandate that the Foundation evaluate its science, technology, engineering, and mathematics education programs. 

 

In addition, the enabling legislation for the STEP program (Public Law 107–368, National Science Foundation Authorization Act of 2002) stipulates the following: 

(D)(i) In order to receive a grant under this paragraph, an eligible applicant shall establish targets to increase the number of students studying toward and completing associate’s or bachelor’s degrees in science, mathematics, engineering, or technology.

(ii) A grant under this paragraph shall be awarded for a period of 5 years, with the final 2 years of funding contingent on the Director’s determination that satisfactory progress has been made by the grantee toward meeting the targets established under clause (i).

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collection will comply with 5 CFR 1320.6.

A.8. Consultation Outside the Agency

A.9. Payments or Gifts to Respondents

No payments or gifts will be provided to respondents.

A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data are processed according to Federal and State privacy statues. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.

 

The opening screen of the STEP system states the following:

 

 

The Federal Government has a continuing commitment to monitor its awards to identify and address any inequities based on gender, race, ethnicity, or disability of the PIs/co-PIs, trainees, or other participants.

 

Information from this data collection system will be retained by the NSF, a Federal agency, and will be an integral part of its Privacy Act System of Records in accordance with the Privacy Act of 1974 and maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998). All individually identifiable information supplied by individuals or institutions to a Federal agency may be used only for the purposes outlined in the system of records notice and may not be disclosed or used in identifiable form for any other purpose, unless otherwise compelled by law. These are confidential files accessible only to appropriate NSF officials, their staffs, and their contractors responsible for monitoring, assessing, and evaluating NSF programs. Only data in highly aggregated form, or data explicitly requested as “for general use,” will be made available to anyone outside of the NSF for research purposes. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c.

A.11. Questions of a Sensitive Nature

The STEP monitoring system collects the name, telephone and fax numbers, and e-mail address of each PI/institution leader. The instrument does not collect information on individual students, only aggregated counts of majors, degrees granted, and transfers to 4-year institutions, broken out by gender and race/ethnicity; counts of students participating in strategies and activities supported by STEP grants; and counts of students receiving direct funding under STEP grants. This information is collected in order to monitor the award sites, to provide continuity, and to assess the success of the award programs. The respondents will be PIs (for lead institutions) and institution leaders (for partner institutions) who will presumably obtain the student demographic data from their institution's Institutional Research office. Individualized (institution-specific) data are provided only to STEP program staff and to contractors authorized by NSF. Any public reporting of the data will be in aggregate form.

A.12 Estimates of Response Burden

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The total number of average annual respondents is 213 and the estimated average annual response burden is 5,538 hours. The Web-based data collection is an annual activity of the STEP program. Data will be collected from each lead institution and its corresponding partner institution(s); the respondents will be award PIs (for lead institutions) and institution leaders (for partner institutions). The population for the first year of the clearance will consist of 121 STEP awards, including awards granted in FY 2004, FY 2005, FY 2006, FY 2007, FY 2008, and FY 2009. There will be 121 PIs and 92 institution leaders, or 213 total respondents. For the second year of the clearance, the 19 FY 2004 awards will be dropped from the population and we anticipate approximately 20 FY 2010 awards will be added to the population. There will be approximately 122 PIs and 95 institution leaders, or 217 total respondents. For the third year of the clearance, the 22 FY 2005 will be dropped from the population and we anticipate approximately 20 FY 2011 awards will be added to the population. There will be approximately 120 PIs and 89 institution leaders, or 209 total respondents. The total number of annual respondents was calculated as follows:

No. of Respondents in Year One:

FY 2004 19 awards/PIs & 9 partner institutions/institution leaders

FY 2005 22 awards/PIs & 18 partner institutions/institution leaders

FY 2006 22 awards/PIs & 21 partner institutions/institution leaders

FY 2007 22 awards/PIs & 9 partner institutions/institution leaders

FY 2008 20 awards/PIs & 33 partner institutions/institution leaders

FY 2009 16 awards/PIs & 2 partner institutions/institution leaders

Year One Totals: 121+ awards/PIs & 92+ partner institutions/institution leaders = 213 respondents

No. of Respondents in Year Two:

FY 2005 22 awards/PIs & 18 partner institutions/institution leaders

FY 2006 22 awards/PIs & 21 partner institutions/institution leaders

FY 2007 22 awards/PIs & 9 partner institutions/institution leaders

FY 2008 20 awards/PIs & 33 partner institutions/institution leaders

FY 2009 16 awards/PIs & 2 partner institutions/institution leaders

FY 2010 20 awards/PIs & 12 partner institutions/institution leaders

Year Two Totals: 122 awards/PIs & 95 partner institutions/institution leaders = 217 respondents

No. of Respondents in Year Three:

FY 2006 22 awards/PIs & 21 partner institutions/institution leaders

FY 2007 22 awards/PIs & 9 partner institutions/institution leaders

FY 2008 20 awards/PIs & 33 partner institutions/institution leaders

FY 2009 16 awards/PIs & 2 partner institutions/institution leaders

FY 2010 20 awards/PIs & 12 partner institutions/institution leaders

FY 2011 20 awards/PIs & 12 partner institutions/institution leaders

Year Three Totals: 120 awards/PIs & 89 partner institutions/institution leaders = 209 respondents

213 respondents in clearance year one + 217 respondents in clearance year two + 209 respondents in clearance year three = 639 respondents over three years / three years = 213 respondents per year

Based on the average response burden per respondent for the last three collection cycles, we estimate the average response burden per respondent to be 26 hours.

The burden estimate is outlined below:

Clearance Year	Type of Respondent	No. of Respondents	Burden Hours per Respondent	Total Annual Burden Hours
One	PI/Institution Leader	213	26	5,538
Two	PI/Institution Leader	217	26	5,642
Three	PI/Institution Leader	209	26	5,434
Three-Year Total		639		16,614
Average Annual		213		5,538

 

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

As mentioned above, respondents will be award PIs and institution leaders. The estimated total average annual response burden is 5,538 hours. There is only one form with slight wording differences for 2-year institutions and for 4-year institutions. This form accounts for the entire average annual response burden of 5,538 hours. The average annual response burden was calculated as follows:

 

Clearance Year	Type of Respondent	No. of Respondents	Burden Hours per Respondent	Total Annual  Burden Hours
One	PI/institution leader	213	26	5,538
Two	PI/institution leader	217	26	5,642
Three	PI/institution leader	209	26	5,434
Three-Year Total		639		16,614
Average Annual		213		5,538

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost to the respondents for the STEP data collection is estimated to be $121,915. The following table shows the annualized estimates of costs to PI/institution leader respondents, who are generally university professors. The estimated hourly rate is based on a report in the April 16, 2010, edition of The Chronicle of Higher Education (2010). (“What Professors Earn.” The Chronicle of Higher Education, 56(31), A10, Washington, D.C.: The Chronicle of Higher Education, Inc.). According to the report, the average salary of an associate professor across all types of doctoral-granting institutions (public, private, church-related) was $83,511. When divided by the number of standard annual work hours (2,080), this calculates to approximately $40 per hour.

 

Clearance Year	Type of Respondent	No. of Respondents	Burden Hours per Respondent	Average Hourly Rate	Estimated Annualized Cost
One	PI/institution leader	213	26	$40	$221,520
Two	PI/institution leader	217	26	$40	$225,680
Three	PI/institution leader	209	26	$40	$217,360
Three-Year Total		639			$664,560
Average Annual		213			$221,520

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents or record-keepers that results from the distance monitoring of the STEP program other than the time spent responding to the data collection instrument.

 

It is usual and customary for individuals involved in education and training activities in the United States to keep descriptive records. The information being requested is from records that are maintained as part of normal educational or training practice. Furthermore, the majority of PIs/institution leaders are active or former grantees or participants in programs or projects once funded by NSF. In order to be funded by NSF, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0058. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane, an Internet-based forms system. Thus, PIs/institution leaders who are the respondents to the STEP data collection make use of standard office equipment (e.g., computers), Internet connectivity that is already required as a startup cost and maintenance costs under OMB 3145-0058, and free software (e.g., Microsoft Explorer or Mozilla Firefox) to respond. The information requested is typical of educational and research portfolios and would be maintained as part of normal practice. Thus, there are no capital and startup costs or operation and maintenance costs to respondents or record-keepers.

A.14. Estimates of Costs to the Federal Government

Computing the annualized cost to NSF for the STEP data collection was done by taking the budgets for three years and calculating the costs for each of the following operational activities involved in producing, maintaining, and conducting the STEP data collection:

 

 

The annualized cost was computed as one-third of the total three-year costs; thus, the annualized cost to NSF for the STEP data collection is $387,892.

A.15. Changes in Burden

The previously reported total annual response burden for this collection was 3,295 hours for 177 respondents; the current request for 5,538 hours for 213 respondents is an increase of 2,243 hours. The number of respondents has increased, and the estimates of burden hours per respondent are based on actual burden data collected during the last three data collection cycles.

 

There have been no major changes in the instrument that would affect the burden. Minor changes have been made to the wording of questions to clarify the desired responses. In addition, some text boxes have been replaced with check boxes to help standardize responses.

A.16. Plans for Publication, Analysis, and Schedule

This is a recurring study. Data collection is conducted annually, beginning in mid-February and ending in mid-April. Extensions are granted by NSF program officers as necessary. Agency staff can access the data through the online system as needed.

 

Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. ICF Macro, the contractor that conducts this data collection on behalf of NSF, is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF and NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information collected that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.

 

The data from this collection will be used for internal review purposes and to monitor the STEP projects, as well as for reporting to Congress and OMB. Reports to NSF management, PIs, and Congress dealing with characteristics and performance of the STEP program will include statistical tables and charts generated from the database. At this time NSF has no set timeline for publishing interim reports from this study.

A.17. Approval to Not Display Expiration Date

Not applicable

A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions apply.

Section B

Introduction

B.1. Respondent Universe and Sampling Methods

B.2. Information Collection Procedures/Limitations of the Study

This data collection uses a Web-based instrument. The PIs and institution leaders associated with each STEP award will be required to provide project information each year during the duration of their NSF funding. The latest program solicitation for the STEP program can be found here.

NSF understands the limitations of this data collection, particularly in terms of using the data to determine program effectiveness. Data collected through the STEP system are not used to determine the ultimate effectiveness of its STEM educational interventions, but are used in program planning and management, to report on agency activities and goals, and to lay the groundwork for future evaluations.

B.2.1. Statistical Methodology for Stratification and Sample Selection

This data collection is a census, so no sampling is required.

B.2.2. Estimation Procedure

Not applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not applicable

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Not applicable

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Past collection have had 100 percent response rates, and NSF expects that the rate will remain the same. Considerable effort is made for follow up for non-response and incomplete responses. Biweekly during the collection, the contractor sends e-mail messages and calls institutions that have not logged into the system. The contractor also notifies all institutions that are still entering data when the collection reaches its midpoint and when the system closing date is one week away. Examples of e-mail messages sent to the STEP respondents are included in appendix C. The collection is part of reporting required of awardees.

B.4. Tests of Procedures or Methods

The questions in this collection were reviewed by STEP PIs during the development of the system. In addition, many of the items and response categories follow formats that are already in use within other tasks, such as the Noyce monitoring system, also cleared as part of the EHR Generic Clearance (OMB 3145-0136), including demographic questions. User feedback is also taken into consideration during system updates. The system has been in operation since 2007 and has been tested extensively.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency

Susan H. Hixson, National Science Foundation, (703) 292-4623

William P. Neufeld, National Science Foundation, (703) 292-5148

Connie Kubo Della-Piana, National Science Foundation, (703) 292-5309

Contractors

ICF Macro of Bethesda, MD will be responsible for data collection and analysis under the direction of Lea Mesner, (301) 657-3070.