31450187 Part A

Supporting Statement (3145-0187)

Evaluation of the Graduate Teaching Fellows in K-12 Education (GK-12) Program: OMB 3145-0187

Section A

Introduction

Introduction

This request for Office of Management and Budget (OMB) review asks for reinstatement of clearance to evaluate the NSF’s Graduate Teaching Fellows in K-12 Education (GK-12) Program.  Previously cleared under 3145-0187 was an evaluation of the GK-12 program consisting of a qualitative analysis of case studies of 21 sites and a quantitative analysis of survey data collected from all of the project sites.  These evaluation activities resulted in evaluation reports submitted to the NSF by the contractor in 2005.  These findings informed program management and dissemination regarding the outcomes of GK-12 funding for participating Fellows, teachers, universities, and schools.

 

This reinstatement with change is to collect data from GK-12 participants (i.e., Fellows, teachers, PIs, and faculty advisors) in order to examine the long-term impacts of participation in the program.  There are no other studies at this point that examine the initial and long-term impacts of the GK-12 program on participating Fellows, teachers, K-12 students, PIs, institutions of higher education, and local school districts. 

 

The proposed study has the following objectives:

1.    To evaluate the short term career outcomes of recently graduated Fellows;

2.    To assess the program’s outcomes for participating Individuals (Fellows, Teachers, Students) and Organizations (Institutions of Higher Education, K-12 Schools); and

3.    To inform and assist program management, including providing information for internal reporting to NSF constituents, external reporting to Congress and other interested parties, which will inform future program solicitations. 

This study will utilize a quasi-experimental design to assess the outcomes of the GK-12 program for participants.  Program outcomes for participating Fellows will be measured through comparison of data from Fellows and a matched comparison group of non-Fellows; perceptions of faculty advisors on the outcomes of GK-12 participation for Fellows; and comparison with national benchmark data.  Outcomes for K-12 teachers and other participants will be assessed through participant self-report and triangulation among multiple data sources.  

Data for this research will be provided by samples of the following categories of individuals:

1.  GK-12 Fellows

2.  Non-GK-12 graduate students (serving as a comparison group for the Fellows)

3.  K-12 teachers who participate in GK-12

4.  GK-12 principal investigators

5.  Faculty members who administer GK-12 awards

6.  Faculty members who advise GK-12 Fellows

This evaluative research will include the development of web-based surveys and interviews to collect data from the individuals listed above.  The information gathered from this study will be used along with the data from the existing monitoring system (separately cleared as OMB 3145-0136) and a review of extant data on GK-12 projects to examine the impacts of the GK-12 program on its participants.

The GK-12 Program:  Initiated in 1999, the GK-12 program has funded approximately 169 projects to date.  It is a fellowship program that offers graduate students in science, technology, engineering, and mathematics (STEM) disciplines to acquire additional skills that will broadly prepare them for professional and scientific careers in the 21st century.  Through interactions with teachers and students in K-12 schools and with other graduate fellows and faculty from STEM disciplines, Fellows can improve communication, teaching, collaboration, and team building skills while simultaneously enriching STEM learning and instruction in K-12 schools. Through this experience, graduate students can gain a deeper understanding of their own STEM research.  In addition, the GK-12 program provides institutions of higher education with an opportunity to make a permanent change in their graduate programs by incorporating GK-12 like activities in the training of their STEM graduate students.

The program has four major goals:

1.  To improve communication, teaching, collaboration, and team building skills for the fellows;

2.  To provide professional development opportunities for K-12 teachers;

3.  To enrich learning for K-12 students, and

4.  To strengthen and sustain partnerships in STEM between institutions of higher education and local school districts.

A.1. Circumstances Requiring the Collection of Data

Data are required to conduct an evaluation of the GK-12 program for program assessment and impact and to fulfill requirements of OMB’s Program Assessment Rating Tool (PART) process and the Government Performance and Results Act (GPRA). While descriptive statistics and in-depth information on project structures and activities are available from the already approved Distance Monitoring System (OMB # 3145-0136), there is a need to evaluate the impacts of the program on its participants, especially Fellows who have graduated and are now in the early stages of their career. Data collected through this research will allow us to examine the impacts of the training that GK-12 Fellows receive on their career trajectories, the impact of GK-12 participation for K-12 teachers and students, and, finally, the program's impact on STEM graduate education at individual institutions of higher education.

A.2. Purposes and Uses of the Data

The primary purpose for collection of this information is program evaluation. NSF will use the analysis of responses to prepare and publish reports. The data collected will allow NSF to respond to questions of program impact from within the Foundation and from Congress. Documenting the short and long-term impacts of the GK-12 program will inform future GK-12 program policy decisions and contribute to the wider NSF discussion on the future of science, technology, engineering, and mathematics (STEM) graduate education. Specifically, this research is designed to answer the following research questions:

Fellow Outcomes:

1. What is the impact of participating in GK-12, both while enrolled and after graduation, for Fellows’

   1. Professional preparation? (teamwork, communication, teaching, and research)

   2. Academic progression and retention towards their degrees?

   3. Career choices and advancement?

   4. Understanding of the responsibilities of STEM professionals for outreach and social awareness?

Teacher Outcomes:

2. What are the outcomes of participating in GK-12 for K-12 teachers’

   1. STEM content knowledge?

   2. Use of STEM and GK-12 related pedagogical practices and tools?

   3. Participation in STEM education activities?

   4. Access to STEM educational curricula and resources? (e.g., classroom resources, collaborative opportunities etc.)

K-12 Student Outcomes:

3. What are the outcomes of participating in GK-12 for K-12 students’ knowledge of and interest in STEM fields and STEM-related careers?

Outcomes for Participating K-12 Schools and IHEs:

4. To what extent have GK-12 inspired ideas and practices been incorporated into the professional preparation of STEM graduate students at participating institutions of higher education?

5. To what extent have participating STEM faculty members and university staff developed, strengthened, and sustained partnerships with local school districts?

Appendix A maps the research topics to data sources.

A.3. Use of Information Technology To Reduce Burden

In order to reduce respondent burden, internet-based surveys and telephone interviews will be used to collect information from participants. We describe both below.

 

Samples of GK-12 Fellows, non-GK-12 graduate students, PIs, K-12 teachers, and faculty advisors will be asked to complete a web-based survey to learn about the impacts of their GK-12 participation.  The survey questions for respondents in each category will be standardized, and primarily close-ended.  These questions will collect descriptive data about project experiences and impacts.  Web based surveys have become more and more commonly used in recent years. NSF tends to favor Web-based systems because they can facilitate respondents' data entry across computer platforms. One innovative feature of many of the individual Web systems is that information once entered into the system can be presented to the respondent for verification, thus reducing the respondent burden. Another valuable feature is that there can be a thorough editing of all submitted data for completeness, validity, and consistency. Editing is performed as data are entered. Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents' attention before they are submitted to NSF. Web-based surveys employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. In addition, survey skip patterns automatically move the respondent forward into the next appropriate section, creating less confusion and simplifying the survey-taking experience. This approach also allows for easy identification of non-respondents and facilitates follow-up. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.

Data from the existing program monitoring databases, maintained under the EHR Generic Clearance (OMB # 3145-0136), will be used to identify GK-12 Fellows, teachers, and PIs. In case the databases do not have the contact information for a GK-12 Fellow, or the information is not current, and the information is not available through their home department, Internet searches will be used to update contact information for the graduates. Email will be used to send respondents their invitations to complete the survey and follow-up with the non-respondents to ensure their participation.

 

After data from the survey are analyzed, we will select a sub-sample of respondents to participate in brief telephone interviews. These interviews are necessary to shed light on questions that arise during analysis, as well as examine more deeply into program effects and impacts. For example, while the survey may ask PIs whether they have developed any new courses as a result of the GK-12 program, the interviews can probe into the how and why of creating those courses - what was the need? how were they developed? How well are they operating now? The interviews will be guided by an open-ended protocol that can be tailored to each respondent. Email will be used to invite GK-12 participants, including graduates, GK-12 project PIs, advisors, and K-12 teachers to participate in interviews and to schedule the interviews.

A.4. Efforts To Identify Duplication

This evaluation does not duplicate other NSF efforts. There are no other studies at this point that examine the initial and long-term impacts of the GK-12 program on participating Fellows, teachers, students, PIs, institutions of higher education, and local school districts. Project monitoring data for GK-12 gathered via OMB 3145-0136 will be available to the contractor and the NSF staff working on this research. These data will be used to pre-fill surveys as appropriate. Future data collection tasks likewise will also draw on whatever prior program data exists, thus preventing duplication.

A.5. Small Business

No information for this research will be collected from small businesses.

A.6. Consequences of Not Collecting the Information

If this information is not collected, NSF will be unable to document the initial and longer-term impacts of the GK-12 program as required by GPRA, nor will it be able to adequately assess program performance as required by PART, and the Deficit Reduction Act of 2005.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collections will comply with 5 CFR 1320.6.

A.8. Consultation Outside the Agency

Comments on this data collection effort were solicited in the Federal Register on August 31, 2007 (Volume 72, Number 169). A copy of the notice is included in Appendix B. During the first comment period prior to submission to OMB, no substantial comments were received.

The GK-12 Program Announcement and Guidelines, coupled with consultation with GK-12 Program Staff, were major sources of information in designing the data collection instruments for this research. Discussions with NSF staff in the Division of Graduate Education informed the development of the surveys and interview protocols, as did discussions with staff members from ORC Macro, the company responsible for the GK-12 Distance Monitoring system.

A.9. Payments or Gifts to Respondents

We have not budgeted for nor will we offer any incentives to encourage survey or interview completion. Incentives should be unnecessary in most cases, particularly with GK-12 Fellows who are currently enrolled and those who are recent graduates, as NSF has been in recent and regular contact with these respondents. All of the GK-12 participants should be tech-savvy and motivated to assist the NSF.

A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to evaluation contractors, contractors hired to manage data and data collection software, and at the aggregate level to NSF officials and staff. Data are processed in accordance to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.

Individuals surveyed will be assured that the information they provide will not be released in any form that identifies them as individuals and their responses will be kept confidential. The contractor will be expected to maintain the confidentiality, security, and integrity of survey data and interview data. The web-based survey data and notes and transcripts of the interviews will be maintained on a secure server with appropriate levels of password and other types of protection.

All assurances of confidentially will be reviewed by the contractor’s Institutional Review Board prior to data collection.

A.11. Questions of a Sensitive Nature

The proposed surveys ask for demographic information (gender, race/ethnicity, and citizenship status) from Fellows, comparison graduate students, and teachers. This information is being collected so that NSF can answer questions about whether GK-12 has increased participation from individuals traditionally underrepresented in scientific disciplines.

The GK-12 Fellows/comparison survey will also ask for the name of graduated respondents’ employer. This information is being collected so that characteristics of graduates’ employer can be coded and calculated (such as employer size, whether graduates are employed in “Fortune 500” companies, etc.). We are asking just for the employer name, instead of multitude specific questions about employers’ characteristics (a) to reduce burden and (b) because respondents may not always know the specific characteristics of their employers. The study will never report the actual names of employers.

All of these questions will be reviewed by the contractor’s Institutional Review Board prior to fielding. These items will be voluntary. Respondents may choose not to provide information that they feel is privileged. Any individualized data that are collected are provided only to program staff and consultants conducting studies using the data as authorized by NSF. Any public reporting of data is in aggregate form.

A.12 Estimates of Response Burden

The requested burden for this evaluation is 3,289 hours for 8,081 respondents.

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The target population for this study includes individuals who participated in GK-12 projects funded between 2000 and 2004. There are 138 such projects; all will be included in the study.

Online web surveys will be administered to every GK-12 project PI as well as samples of Fellows, comparison graduate students, teachers, and faculty advisors from these projects. Survey sample sizes have been targeted to be large enough to be able to detect differences between groups relating to important characteristics with 80% power when we do a two-sided statistical test at 5% level of significance. The total number of respondents for surveys is estimated to be 7,343.

Telephone interviews will be conducted with the PI and a sample of Fellows, teachers, and faculty participants from each GK-12 project. The total number of respondents that estimate for interviews is estimated 738 over one year.

The estimated overall response burden for both interviews and surveys is estimated to be 3,289 hours over one year. The estimated average time per survey and interview is based on similar surveys and interviews conducted in earlier studies and will be confirmed in pilot tests in Spring 2008.

The chart below indicates the number of respondents to be surveyed/interviewed for each category of respondent type and the time demand these surveys or interviews will place on each individual respondent, and then aggregated across all respondents.

Estimated Burden Hours
Respondent Type	Frequency of response	Time per response (in hours)	Number of responses	Total time burden (in hours)
GK-12 Fellows (Survey)	1	.5	1738	869
GK-12 Fellows (Interview)	1	.5	200	100
Comparison graduate students (Survey)	1	.5	1862	931
K-12 teachers (Survey)	1	.33	1867	616
K-12 teachers (Interview)	1	.5	200	100
GK-12 principal investigators (Survey)	1	.5	138	69
GK-12 principal investigators (Interview)	1	.5	138	69
GK-12 Faculty members (Interview)	1	.5	200	100
Faculty advisors (Survey)	1	.25	1738	435
Total			8081	3289

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

Each respondent will receive the survey one time. Interviews will also be conducted one time per respondent. The table below contains the same values as the table in section A.12.1 and shows the total burden hours. Because the survey is given to each respondent only one time and the interview is only conducted one time per respondent, the annual burden and the aggregate burden are the same.

 

 

Estimated Burden Hours
Respondent Type	Frequency of response	Time per response (in hours)	Number of responses	Total time burden (in hours)
GK-12 Fellows (Survey)	1	.5	1738	869
GK-12 Fellows (Interview)	1	.5	200	100
Comparison graduate students (Survey)	1	.5	1862	931
K-12 teachers (Survey)	1	.33	1867	616
K-12 teachers (Interview)	1	.5	200	100
GK-12 principal investigators (Survey)	1	.5	138	69
GK-12 principal investigators (Interview)	1	.5	138	69
GK-12 Faculty members (Interview)	1	.5	200	100
Faculty advisors (Survey)	1	.25	1738	435
Total			8081	3289

 Internet surveys will be administered to GK-12 Fellows, comparison graduate students, K-12 teachers, GK-12 PIs, and faculty advisors.

6. Survey questions for Fellows and comparison graduate students pertain to their graduate experiences, perceptions of the importance of various training elements and adequacy of training, the specific activities they engaged in as part of GK-12, the skills they developed or enhanced as a result of their participation in the GK-12 program, their location in the labor market, their current job and the impact of their GK-12/graduate training on their careers.

7. Survey questions for K-12 teachers pertain to their GK-12 experiences and the impact of those experiences on their own professional development and the learning of their students.

8. Survey questions for GK-12 project PIs and faculty advisors pertain to the perceived impacts of the GK-12 program for participating individuals and institutions.

The four survey instruments are included in Appendix C ¹[1]. The estimated average time per survey is based on similar surveys conducted in earlier studies and will be confirmed in pilot tests in Spring 2008.

Interviews will be conducted with samples of GK-12 Fellows, K-12 teachers, GK-12 PIs, and participating faculty members. The interview protocols are included in Appendix D.

[1] We have developed four slightly different versions of the teacher survey. The only difference with each version is that some questions have been tailored to reference math vs. science teachers and elementary vs. high school teachers. We have only included the high school math version of the teacher survey in Appendix C.

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The overall annualized cost to respondents is $86,313 following chart shows the estimated total annual costs to each group of respondents over one year for the surveys and interviews that assess the impacts of GK-12.

Estimated Costs to Respondents
Survey Type	Hourly Salary Estimate	Burden Time per Respondent	Estimated Cost to Respondent	Number of Respondents in Category	Estimated Annual Cost Across all Respondents
Survey of GK-12 Fellows	$20.00a	.5	$10.00	1738	$17,380
Survey of comparison graduate students	$20.00a	.5	$10.00	1862	$18,620
Interviews with GK-12 Fellows - Enrolled	$14.00b	.5	$7.00	200	$1,400
Survey of K-12 teachers	$28.00c	.33	$9.24	1867	$17,251
Interviews with K-12 teachers	$28.00c	.5	$14.00	200	$1,400
Survey of GK-12 principal investigators	$45.00d	.5	$22.50	138	$3,105
Interviews with GK-12 principal investigators	$45.00d	.5	$22.50	138	$3,105
Interviews with GK-12 Faculty members	$45.00d	.5	$22.50	200	$4,500
Survey of Faculty advisors	$45.00d	.25	$11.25	1738	$19,552
aThis salary estimate is based on an average of two salary rates because the GK-12 Fellows sample and the comparison sample include both graduated and enrolled respondents.  There are roughly equal numbers of graduated and enrolled students in both the Fellows and comparison sample so we use the average salary for the two groups which is $20.00.  The salary rate we used for the graduated respondents is $26.00 which is based on the estimated salary for PhDs in science and engineering occupations who had their Ph.D.s for less than five years as reported in National Science Board, Science and Engineering Indicators 2006. Arlington, VA: National Science Foundation, 2006 (NSB-06-01A), Appendix table 3-5, page 4. The second rate is for enrolled respondents.  For the enrolled respondents, we used a salary rate of $14.00 which is based on NSF's contribute to GK-12 student stipends as reported in the GK-12 program solicitation NSF 07-555.  Thus we're using a mixed salary rate of $20.00 = ($14.00 + $26.00)/2 for calculating estimated burden.   b Current GK-12 trainee salaries are based on NSF contribution to GK-12 student stipends as reported in the GK-12 program solicitation, NSF 07-555   cPublic school teacher salary estimate based on NCES Projection of Education Statistics to 2015. NCES September, 2006 d Current GK-12 faculty salaries are based on an average salary estimate for full professors as reported in U.S. Department of Education, National Center for Education Statistics. (2007). The Condition of Education 2007 (NCES 2007-064). Washington, DC: U.S. Government Printing Office.

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents that results from this study other than the time spent responding to the survey and interviews attached as appendices to this request.

A.14. Estimates of Costs to the Federal Government

The cost burden to the government currently approved by OMB for this collection of information is $306,041. The proposed changes in the number and type of respondents and adjustments in the data requests result in an estimated cost to the government of all data collection, analysis and reporting activities of $938,182, an increase of $632,141 over the previous cost burden. The costs of information collection are based on and extrapolated from the budget developed in performance of NSF Contract Number GS10F0086K.  Indirect as well as direct costs are included in these estimates. The data are being collected over a three-year period. 

A.15. Changes in Burden

 

The inventory numbers for the now expired 3145-0187 covered 16 individual collection tasks or instruments The OMB inventory records show a total number of respondents of 3,645 and total hours of 1,253.  For this renewal, 16 of the previous instruments are discontinued, 0 are revised, and 9 are new.  This renewal anticipates an annual average burden of 2,529 annual hours, for 8,081 responses.  The change in burden results in an increase in total hour burden of 1,276 hours and 4,436 respondents. This change in burden is due to shifts in the number and type of respondents and adjustments in the data requested.  The charts below summarize the changes in burden in the individual instruments.

 

  

Hour Changes in Task Burden for GK-12 (3145-0187)
Respondent Type	Survey/Interview	Previously cleared burden (in hours)	Currently requested burden (in hours)	Change in burden (in hours)
Case Studies
Fellows	Interview	72.00	0	-72.00
Fellows	Focus Group	180.00	0	-180.00
Non-Fellow	Interview	29.00	0	-29.00
Teachers	Interview	65.00	0	-65.00
Teachers	Focus Group	180.00	0	-180.00
Principal Investigator	Interview	12.00	0	-12.00
Principal	Interview	32.00	0	-32.00
Faculty advisor	Interview	12.00	0	-12.00
Evaluator	Interview	12.00	0	-12.00
Graduate School Dean	Interview	4.50	0	-4.50
Trainer for GK-12 Projects	Interview	6.00	0	-6.00
Subtotal		604.50	0	-604.50
Census Study
Fellows	Survey	269.00	0	-269.00
Non-GK-12 Fellows	Survey	122.00	0	-122.00
Teachers	Survey	215.00	0	-215.00
Principal Investigator	Survey	20.00	0	-20.00
District Staff	Survey	22.00	0	-22.00
Subtotal		648	0	-648.00
New Instruments
GK-12 Fellows	Survey	0	869.00	869.00
GK-12 Fellows	Interview	0	100.00	100.00
Comparison graduate students	Survey	0	931.00	931.00
K-12 teachers	Survey	0	616.00	616.00
K-12 teachers (Interview)	Interview	0	100.00	100.00
GK-12 principal investigators	Survey	0	69.00	69.00
GK-12 principal investigators	Interview	0	69.00	69.00
GK-12 Faculty members	Interview	0	100.00	100.00
Faculty advisors	Survey	0	435.00	435.00
Subtotal		0	2529.00	2529.00
TOTAL		1252.5	3289.00	2036.5

A.16. Plans for Publication, Analysis, and Schedule

The purpose of this research is to examine the impact and outcomes of NSF–funded GK-12 awards on participating Fellows, K-12 teachers, K-12 students, and the institutions who received the awards. This will help NSF respond to such questions as: How has NSF helped enhance the graduate experiences of masters’ and Ph.D. graduates in the science and engineering workforce? How has NSF enhanced the communication and teaching skills of graduate students in a way that will benefit both the science community and the larger public in the future? How has NSF helped improve the ability of K-12 teachers to teach science and math?

An analytic evaluation report will be prepared based on study findings. The analysis of the Fellows will consist both of simple frequencies as well as comparison of Fellows’ data with data from a matched comparison group of non-Fellows. Cross-tabs with Chi square, analysis of variance and various regression techniques will be used to assess the similarities and differences between GK-12 Fellows and comparison non-GK-12 respondents. Data from the teacher survey will be examined through frequencies to assess the outcomes of their GK-12 participation, looking at outcomes overall as well as comparing outcomes for teachers within specific grade levels (Elementary, Middle, High school). Data from interviews will be examined through simple frequencies as well as descriptive summaries of emergent themes.

Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats. The contracting agency, which will conduct this third-party study of the GK-12 program on behalf of NSF, is forbidden contractually from publishing results unless NSF had made a specific exception. All products of the collection are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product On-line Document System or simply a page on the NSF Web site) in which to publish.

Before the conclusion of the study the NSF grants funded under this program may use preliminary data to improve management and performance. For example, data generated by this study may appear as inputs to other internal and external NSF reports (e.g., the GPRA annual Performance Plan). At this time, NSF has no set timeline for publishing interim reports from this study.

A.17. Approval to Not Display Expiration Date

Not Applicable

A.18 Exceptions to Item 19 of OMB Form 83-I

Not Applicable

1

14