REQUEST FOR CLEARANCE
National Science Foundation
Directorate for Education and Human Resources
Division of Graduate Education
Graduate STEM Fellows in K-12 Education
Monitoring System (GK-12)
Attachment B
This request for Office of Management and Budget (OMB) review asks for a renewal of clearance for the monitoring data collection for the National Science Foundation (NSF) Division of Graduate Education (DGE) Graduate STEM Fellows in K-12 Education (GK-12) program under the Directorate for Education and Human Resources (EHR) Generic Clearance (OMB 3145-0136), which expires on March 31, 2011. The EHR Generic Clearance includes collections of information about NSF's education and training (E&T) activities.
The GK-12 program was initiated in 1999. The GK-12 program provides funding for graduate students in NSF-supported science, technology, engineering, and mathematics (STEM) disciplines to acquire additional skills that will broadly prepare them for professional and scientific careers in the 21st century. Through interactions with teachers and students in K-12 schools and with other graduate fellows and faculty from STEM disciplines, graduate students can improve communication, teaching, collaboration, and team building skills while enriching STEM learning and instruction in K-12 schools. Through this experience, graduate students can gain a deeper understanding of their own STEM research. In addition, the GK-12 program provides institutions of higher education with an opportunity to make a permanent change in their graduate programs by incorporating GK-12 like activities in the training of their STEM graduate students. Expected outcomes include improved communication, teaching, collaboration, and team building skills for the fellows; professional development opportunities for K-12 teachers; enriched learning for K-12 students; and strengthened and sustained partnerships in STEM between institutions of higher education and local school districts.
The GK-12 monitoring system consists of a Web-based data collection instrument that will be completed by the principal investigators (PIs) of the awards, with additional data submitted by NSF-funded graduate fellows and by the lead cooperating teachers and NSF-supported teachers who work with the graduate fellows.
The information collected in this task is required for effective administration, communication, and program and project monitoring; for meeting reporting requirements; for measuring attainment of NSF's program, project and strategic goals as laid out in NSF’s Strategic Plan; and as a baseline for future program evaluations.
The primary purpose of this data collection is to provide data and information for effective program management and monitoring of program activities. This data collection activity is designed to track the extent to which GK-12 projects meet the objectives of the program. Within the DGE division, this information is used to administer and monitor the progress of the program. The findings are used to recommend, among other things, administrative changes in program functions, level of award support, individual program focus and emphasis, and recruiting efforts.
In recent guidance from the Director of OMB, M-10-32, the need for rigorous evaluations and the objectives of evaluations of programs were clearly outlined, including the use of evaluation resources. Because the collection of data contained in this monitoring effort contributes to the formal evaluation of the program and provides regular measures of program performance by accumulating operating information from each project in the program, this guidance is particularly pertinent to this request.
“Improving and coordinating the use of existing evaluation resources. In addition to the voluntary evaluation initiative, agencies should continue to carefully assess, report on, and allocate the base funds and resources that the agencies have for conducting evaluation. Agencies are encouraged to share information beyond what is requested in guidance and consult with OMB’s Resource Management Offices (RMOs) to coordinate and improve the design, implementation, and utilization of evaluations.”
These directives establish an ongoing need for NSF to engage in an interactive process of collecting information and using it to improve program services and processes.
The GK-12 program also uses the data to fulfill reporting requirements. As a part of its performance assessment activities, NSF relies on the judgment of external experts to maintain high standards of program management, to provide advice for continuous improvement of NSF performance. Directorate and Office advisory committees (ACs) meet twice a year, while Committees of Visitors (COVs) for divisions or programs meet once every three years. Data collected in the GK-12 monitoring system can be used to report to these committees on program activities. In addition, NSF is required to measure the attainment of its program, project and strategic goals, as identified by the President’s Accountable Government Initiative, the Government Performance and Results Act (GPRA) Modernization Act of 2010, and NSF’s Strategic Plan. These data help NSF management examine their progress towards these goals and respond to these reporting requirements. In addition, some of the GK-12 PIs also use the data collected to report information at the school district and State level.
Finally, the data can also be used as a preliminary step in more detailed future evaluation efforts. EHR makes these data available to NSF staff, EHR contractors with responsibility for the collection, and DGE program managers and their staff and contractors. Information collected may also be disseminated, in aggregate form, to current and prospective applicants to the GK-12 program and to the broad STEM education community upon request to and approval from the NSF. DGE is using monitoring system data as a baseline for an evaluative research study to be conducted by Abt Associates.
Data are collected using the GK-12 Data Collection System, an online system developed to facilitate electronic data collection. Respondents are GK-12 awardees who provide project and school information, NSF-funded graduate fellows, and the lead cooperating teachers and NSF-supported teachers, who enter their own participant data, as well as some additional information about their experiences in the GK-12 program. An overview of the data collected is summarized below by form type and major sections:
Annual Award Form:
1. Award Information
2. Partnerships and Collaborations
3. Participating Schools in Your GK-12 Project
4. Principal Investigator and Co-Principal Investigator Information
5. Project Coordinator Information
6. Graduate Fellow Information
7. Cooperating Teacher Information
8. Award Evaluator Information
9. Other Participant Information
10. Academic Achievements Related to GK-12 Activities and Themes
11. Additional Funding Sources
12. Program Outreach Activities
13. International Experience
14. Program Impact
15. Verify Fellows’ Interactions with K-12 Students
Annual NSF-Funded Graduate Fellow Form:
1. Contact Information
2. Permanent Contact Information
3. Demographic Information
4. Academic History
5. Previous Teaching Experience
6. Current Graduate Work
7. Research Advisor
8. Interactions with K-12 Students
9. Honors and Awards During GK-12 Project Participation
10. Development Activities
11. International Experience
12. Research and Professional Experience
Annual Lead/NSF-Supported Teacher Form:
1. Contact Information
2. Demographic Information
3. Degrees
4. Teaching Experience and Credentials
5. Experience Working with GK-12 Fellows
6. Honors and Awards During GK-12 Project Participation
7. Development Activities
8. International Experience
9. Professional Experience
Changes and modifications to data elements requested and approved for the 2010-2011 data collection, are still in software development and revised system screens are not yet available. Screenshots of the previous version of the system are provided as appendices A1, A2, and A3; however, please see the crosswalks of data elements (appendices B1, B2, and B3) for details on the most current questions, wording of the system's data elements, and their categorization by Staff and Project Participant Characteristics, Project Implementation Characteristics, and Project Outputs, as required by OMB.
EHR tends to favor Web-based systems because they can facilitate respondents’ data entry across computer platforms. One innovative feature of many of the individual Web systems is the thorough editing of all submitted data for completeness, validity, and consistency. Editing is performed as data are entered. Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents’ attention before they are submitted to NSF. Web-based systems employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
The GK-12 system follows the success of other Web-based data collection systems at NSF. The Web-based software facilitates respondents' data entry by ensuring more complete and correct data submissions and thus reducing the need for follow up. Unlike earlier systems, which provided software on disks, newer Web-based systems like GK-12 do not allow respondents to revise the field sizes, thus ensuring comparability of data across sites. Fields are also marked with out-of-range indicators, and respondents are warned to check their data if they appear to be out-of-range. GK-12 sites can view data submitted in previous collection cycles. Since most program participation is on a multiyear basis, this feature makes updating the previous year's data, particularly those on participants, far easier and less burdensome. The GK-12 system is compliant with Section 508 of the Rehabilitation Act.
Data collected under the GK-12 program are not available anywhere else. The GK-12 data collection does not duplicate other NSF efforts.
Whenever possible, data are drawn from existing NSF databases. Questions are asked only if the information requested is not available elsewhere. NSF has examined its data collection requirements to ensure that the requested data are not available from other Federal sources.
Changes to NSF’s FastLane system are monitored on a regular basis, and as new data elements are added to (or deleted from) the FastLane application and/or project reporting system, the GK-12 instrument is modified accordingly. The instrument does not duplicate items available through FastLane.
No information is to be collected from small businesses.
The information requested here is not available elsewhere. Without this information, NSF would be restricted in describing the activities of the GK-12 program. Without this feedback, NSF would have no way of making systematic modifications to the GK-12 program (e.g., adequacy of funding amount, duration of award, and institutional supports needed). These data will ensure that NSF makes informed decisions about future directions of the GK-12 program. Finally, without this information NSF would find it difficult to meet GPRA and other requirements and would be unable to comply fully with the congressional mandate that the Foundation monitor its STEM education programs.
The data collections will comply with 5 CFR 1320.6.
The notice inviting comments on the EHR Generic Clearance (OMB 3145-0136) was published in the Federal Register November 9, 2010, Volume 75, Number 216, pages 68829-68830. No comments were received.
The PIs who use the GK-12 data collection system were consulted in its design and planning. A pilot test was conducted with two awards in 2002. Feedback from system users is solicited annually at various GK-12 meetings (e.g., the annual orientation meeting for new awards and the annual projects meeting). In addition, user comments submitted during the collection period are taken into consideration for system improvements.
No payments or gifts will be provided to respondents.
Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data are processed according to Federal and State privacy statues. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.
The opening screen on the GK-12 system states the following:
The Federal Government has a continuing commitment to monitor its awards to identify and address any inequities based on gender, race, ethnicity, or disability of the principal investigators, co-principal investigators, graduate fellows, cooperating teachers or other participants.
Information from this data collection system will be retained by the NSF, a Federal agency, and will be an integral part of its Privacy Act System of Records in accordance with the Privacy Act of 1974 and maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998). All individually identifiable information supplied by individuals or institutions to a Federal agency may be used only for the purposes outlined in the system of records notice and may not be disclosed or used in identifiable form for any other purpose, unless otherwise compelled by law. These are confidential files accessible only to appropriate NSF officials, their staffs, and their contractors responsible for monitoring, assessing, and evaluating NSF programs. Only data in highly aggregated form, or data explicitly requested "for general use," will be made available to anyone outside of the NSF for research purposes. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grant and in response to Public Law 99-383 and 42 USC 1885c. The Social Security number (SSN) and date of birth will be maintained in accordance with the requirements of the Privacy Act of 1974. Submission of the SSN is voluntary. It is used for quality control, program evaluation, and matching with other data sets maintained in the Education and Training System of Records 63 Fed. Reg. 264, 272 (January 5, 1998).
GK-12 collects data that are considered of a private nature, including the name, phone number, e-mail address, and disability status of the PIs, award evaluators, cooperating teachers, graduate fellows, and other participants. In addition, street address and social security numbers are requested from project fellows. These data are collected in order to monitor the site's participant populations, and to follow up with the participants over time. Respondents have the option of not providing information that they consider privileged and may mark their gender, race, ethnicity, and/or disability as "not reported." For SSN, fellows may opt to report the whole number, the last 4 digits, another unique identifier of their choice, or select “not reported.” Because the program requires that all GK-12 Fellows be U.S. citizens or permanent residents, the question of GK-12 fellow citizenship is directly addressed both in the Award form (by asking the PIs to verify the GK-12-required citizenship/residency requirement for each fellow) and in the Fellow form (by not providing a "not reported" option for citizenship and also checking the genesis of fellows by requesting place of birth). Individuals' data are provided only to GK-12 program staff and consultants conducting studies using the data as authorized by NSF. Any public reporting of the data is in aggregate form.
The estimated total number of annual respondents is 2,040, with an average annual response burden of 4,560 hours. The Web-based collection is an annual activity of each award site of the GK-12 program. We estimate an annual average of 120 awards, with an average of 7 NSF-funded graduate fellows and 9 lead/NSF-supported cooperating teachers per award, resulting in an average 120 awardees, 840 NSF-funded fellows, and 1,080 lead/NSF-supported teachers responding each year. The number of respondents was estimated using data from the current portfolio of GK-12 awards, and the average annual hour burden for all respondents was determined using the burden reported by respondents during the last three collection cycles.
The estimated annual burden is calculated below.
|
Respondent Type |
Estimated Average Annual Number of Respondents |
Estimated Average Annual Burden Hours Per Respondent |
Estimated Annual Burden Hour Total |
|
Awardees |
120 |
14 |
1,680 |
|
NSF-Funded Graduate Fellows |
840 |
1.5 |
1,260 |
|
Lead/NSF-Supported Teachers |
1,080 |
1.5 |
1,620 |
|
Total |
2,040 |
|
4,560 |
As mentioned above respondents will be Awardees (as reported by the project’s lead PI with help from the project coordinator), NSF-funded graduate fellows, and lead/NSF-supported cooperating teachers. The estimated total annual response burden is 4,560 hours. There is a different Web-based form for each respondent type. The annual burden by form was calculated as follows:
|
Form Type |
Respondent Type |
Number of Respondents |
Burden Hours Per Respondent |
Total Burden Hours |
|
Award Form |
Awardees |
120 |
14 |
1,680 |
|
Fellow Form |
NSF-Funded Graduate Fellows |
840 |
1.5 |
1,260 |
|
Teacher Form |
Lead/NSF Supported Teachers |
1,080 |
1.5 |
1,620 |
|
Total |
|
2,040 |
|
4,560 |
The overall annualized cost to the respondents is estimated to be $123,720. The following table shows the annualized estimates of costs to respondents. The estimated hourly rate for PIs is based on a report in the April 16, 2010, edition of The Chronicle of Higher Education (2010). (“What Professors Earn.” The Chronicle of Higher Education, 56(31), A10, Washington, D.C.: The Chronicle of Higher Education, Inc.). According to the report, the average salary of an associate professor across all types of doctoral-granting institutions (public, private, church-related) was $83,511. The hourly rate for fellows is based on their annual stipend of $30,000. The rate for cooperating teachers was established by using the Bureau of Labor Statistic's May 2009 National Occupational Employment and Wage Estimates, which estimates the mean annual wage for those in Education, Training, and Library Occupations to be $49,530. Each of these average annual wages was then divided by the number of standard annual work hours (2,080) to determine an average hourly rate for each respondent type. Those rates and the total costs are indicated in the table below:
|
Respondent Type |
Number, Rate, and Burden |
Cost |
|
Awardees |
(120 x $40/hour x 14 hours) |
$67,200 |
|
NSF-Funded Graduate Fellows |
(840 x $14/hour x 1.5 hours) |
$17,640 |
|
Lead/NSF-Supported Teachers |
(1,080 x $24/hour x 1.5 hours) |
$38,880 |
|
Total |
|
$123,720 |
There is no overall annual cost burden to respondents or record keepers that results from GK-12 other than the time spent responding to the online questionnaire. It is usual and customary for individuals involved in implementing a GK-12 award to keep descriptive records. The information being requested is from records that are maintained as part of normal practices of GK-12 projects, including graduate fellow training and collaboration with K-12 teachers. Furthermore, the majority of respondents are active or former grantees or participants in programs or projects once funded by NSF. In order to be funded by NSF, institutions must follow the instructions in the NSF Grant Proposal Guide (GPG) that is cleared under OMB 3145-0058. The GPG requires that all applicants submit requests for NSF funding and that all active NSF awardees do administrative reporting via FastLane, an Internet-based forms system. Thus, PIs, fellows, and cooperating teachers and/or other administrators who are the respondents to the GK-12 data collection task make use of standard office equipment (e.g., computers), Internet connectivity that is already required as a startup cost and maintenance costs under OMB 3145-0058, and free software (e.g., Microsoft Explorer or Mozilla Firefox) to respond. Thus, there are no capital and startup costs or operation and maintenance costs to respondents or record-keepers.
Computing the annualized cost to NSF for the GK-12 data collection was done by taking the budgets for three years and calculating the costs for each of the following operational activities involved in producing, maintaining, and conducting the GK-12 data collection:
|
Operational Activities |
Cost Over Three Years |
|
System Development (includes initial development of the database and Web-based application, and later changes requested by the program-e.g., increased reporting tools, additional validations) |
$538,107 |
|
System Maintenance, Updates, and Tech Support (system requires updates each year before opening the collection; maintenance is required to keep the system current with technology, e.g., database servers, operating systems) |
$236,511 |
|
Data Collection Opening and Support (e.g., online and telephone support to respondents and contacting respondents to encourage completion of the questions), Reporting (as defined by DGE), and Followup activities (e.g., providing data to other consultants) |
$403,383 |
|
Three-Year Total for All Operational Activities |
$1,178,001 |
The annualized cost was computed as one-third of the total three-year costs; thus, the annualized cost to NSF for the GK-12 collection is $392,667.
The burden for this collection has decreased from 6,120 hours to an annual average burden of 4,560 hours, a decrease of 1,560 hours resulting in a savings of $44,640 annualized cost to respondents for the hour burden compared to the annualized cost ($168,360) submitted for the prior clearance, even though current respondent rates per hour are higher than the previous rates. The new, lower burden is based on the burden reported by respondents and reflects the change that fewer fellows and corresponding teachers are now expected to respond. NSF now requires only NSF-funded fellows to respond and invites only lead/NSF-supported teachers to respond. Furthermore, it takes these experienced teachers an average of 1.5 hours, not the previously projected two hours, to complete the forms. Since it was previously cleared, there have been no changes in the questions that directly affect burden hours; however improvements to the user interface significantly reduced the burden to awardees.
This is a recurring study. Data collection is conducted annually, beginning in March and ending in May. The data are collected for internal review purposes, for monitoring the NSF-funded fellows, as well as for use in reporting to Congress. Reports to NSF management and Congress dealing with the characteristics and performance of the GK-12 program include tables and charts generated from the database. In addition, respondents are able to access tables that display summary information for data entered in the current and previous collections.
Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. ICF Macro, the contractor conducting this data collection on behalf of NSF, is forbidden contractually from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.
Before the conclusion of the study, both NSF and the funded GK-12 projects may use preliminary data to improve management and performance. For example, data generated by this study may appear as inputs to other internal and external NSF reports (e.g., the GPRA Annual Performance Plan). At this time, NSF has no set timeline for publishing interim reports from this study.
Not applicable
No exceptions apply.
The sample size is the entire universe of respondents. There are currently an average of 2,040 awardees, NSF-funded graduate fellows, and lead/NSF-supported cooperating teachers working on GK-12 awards and this number is expected to remain stable throughout the clearance period.
|
Population |
Estimated Universe Size |
Sample Size |
|
GK-12 Awardees, NSF-Funded Graduate Fellows, and Lead/NSF-Supported Cooperating Teachers |
2,040 |
2,040 |
This data collection uses a Web-based instrument. Each GK-12 project will provide project information each year during the duration of their NSF funding.
NSF understands the limitations of this data collection, particularly in terms of using the data to determine program effectiveness. Data collected through the GK-12 system are not used to determine the ultimate effectiveness of its STEM educational interventions, but are used in program planning and management, to report on agency activities and goals, and to lay the groundwork for future evaluations.
This data collection is a census, so no sampling is required.
Not applicable
Not applicable
Not applicable
Not applicable
The collection is part of reporting required of awardees; lead principal investigators are responsible for reporting awardee data and ensuring that data are collected from graduate fellows and cooperating teachers, and have access to status information on the Web site indicating which participants have responded. Past collections have had 100 percent response rates from awardees and graduate fellows, and NSF anticipates that the rate will remain the same. In 2010 NSF started to collected data only from the lead and NSF-supported teachers. The response rate from this group was 96 percent against NSF’s projected 85 perent response rate. NSF anticipates that the rate will remain within the 85-95 percent range. ICF Macro will continue to provide outstanding technical support to all system users during each collection and continue to encourage PIs to follow up with their projects’ fellows and teachers to maintain the noted response rates. Appendix C includes examples of the e-mail messages that announce the opening of the system and remind awardees to log in and enter data.
This system has been operational since 2002 and is tested extensively before every data collection. In addition, many of the items and response categories follow formats that are already in place at NSF. User feedback is also taken into consideration during system updates, to ensure that the system is continually improved.
Agency
Sonia Ortega, National Science Foundation, (703) 292-8697
William Neufeld, National Science Foundation, (703) 292-5148
Contractors
ICF Macro will be responsible for data collection and analysis under the direction of Lea Mesner, (301) 657-3070.
| File Type | text/html |
| File Title | Supporting Statement |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |