Supporting Statement (3145-0182)

Generic Clearance of National Science Foundation-sponsored Graduate Education Impacts or Legacy (GEIL).  (Previously entitled the "Evaluation of the Initial Impacts of the Integrative Graduate Education Research and Traineeship (IGERT) Program) (OMB 3145-0182)

 Section A

Introduction

 

For over fifty years NSF has funded directly and indirectly (e.g. via institutions), tens of thousands of individuals who pursue post-undergraduate education or research training. The graduate traineeship and fellowship programs coordinated through NSF’s Division of Graduate Education (DGE) include programs such as: 

• The Integrative Graduate Education Research and Traineeship (IGERT) program: IGERT provides grants to institutions to recruit and support doctoral students in interdisciplinary Science, Technology, Engineering, and Mathematics programs (STEM).
• The Graduate Teaching Fellows in K-12 Education (GK-12) program: GK-12 provides grants to institutions to support STEM graduate students’ acquisition of skills that will prepare them for careers in the 21st century.
The Graduate Research Fellowship (GRF) program: GRF provides three years of funding to eligible individuals for graduate study leading to research-based masters or doctoral degrees at an IHE of their choice.

Other NSF programs that support graduate education include:

To expand this clearance, the potential data collection activities and studies proposed under this generic clearance include:

A.1. Circumstances Requiring the Collection of Data

The graduate programs relevant to the Graduate Education Impacts or Legacy (GEIL) clearance support NSF’s “Learning” goal as outlined in its 2006 Strategic Plan (NSF 06-48), under which NSF aims to “Cultivate a world-class, broadly inclusive science and engineering workforce.” These programs are subject to evaluation by OMB’s Program Assessment Rating Tool (PART) process, and the Deficit Reduction Act of 2005 (P.L. 109-171), which established the Academic Competitiveness Council and requires it to “Identify the effectiveness of [federal] programs,” “Determine the extent to which the programs have undergone independent, external evaluation based on sound, scientific principles,” and “Ascertain the extent to which the programs have quantitative evidence of achieving their stated goals.” NSF needs data on its graduate programs for program monitoring and to support program analysis, impact assessment, and evaluation activities. While descriptive statistics and in-depth information on project structures and activities are available from ongoing evaluation activities, e.g. the annual monitoring data collection, there is a need to evaluate the long-term impacts of programs on participants.  The proposed research activity will examine progress made towards initial and long-term program goals, those changes we can measure on the way to the ultimate achievement of long-term program goals for all of NSF’s programs.  The first sub-activity will focus on the IGERT traineeship program. 

The IGERT program: The IGERT program was developed to meet the challenges of educating Ph.D. scientists and engineers with the multidisciplinary backgrounds and the technical, professional, and personal skills needed for the career demands of the future.  The program is intended to catalyze a cultural change in graduate education for students, faculty, and universities by establishing new innovative models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries.  It is also intended to facilitate greater diversity in student participation and preparation and to contribute to the development of a diverse, globally aware science and engineering workforce. The IGERT program has three major goals:

• To broaden the demographic base of the Science, Technology, Engineering, and Math (STEM) workforce through the inclusion of greater numbers of students from groups underrepresented in mathematics and science in STEM graduate education;
• To bring about departmental, institutional, and national change in the quality and culture of STEM graduate education, and influence STEM faculty and researchers; and
• To educate STEM graduate students in such a way that they will be capable not only of taking on professorial roles, but also of working with confidence and creativity in situations that demand greater cross-disciplinary and cross-sector interaction:  in industry, in national laboratories, and in the public sector.

A.2. Purposes and Uses of the Data

 

• What do IGERT and other NSF-funded graduate students do subsequent to their participation in the program? 
• Do IGERT and other NSF-funded graduate programs provide graduates with the professional and/or research skills needed for a career in science and engineering?
• Are IGERT and other NSF-funded graduates satisfied that their NSF-funded graduate education has advanced their careers in science or engineering?  
• To what extent do IGERT and other NSF-funded graduates engaged in the science and engineering workforce conduct inter- or multi-disciplinary scientific research? 
• To what extent have NSF-funded graduate projects or programs broadened the participation of diverse individuals, particularly individuals traditionally underemployed in science or engineering, including but not limited to women, minorities, and persons-with-disabilities?

A.3. Use of Information Technology To Reduce Burden

Technology will be used to reduce respondent burden in three ways, through the use of:  (1) internet-based surveys to collect data; (2) the Internet to locate program and non-program graduates and degree noncompleters; and (3) email to follow-up with the non-responders. 

All IGERT graduates and a carefully selected sample of comparison individuals will be asked to complete a web-based survey to learn about the impacts of their training on their careers. This approach has become more and more commonly used in recent years.  NSF tends to favor Web-based systems because they can facilitate respondents' data entry across computer platforms.  One innovative feature of many of the individual Web systems is that information once entered into the system can be presented to the respondent for verification, thus reducing the respondent burden.  Another valuable feature is that there can be a thorough editing of all submitted data for completeness, validity, and consistency.  Editing is performed as data are entered.  Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents' attention before they are submitted to NSF.  Web-based surveys employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics.  In addition, survey skip patterns automatically move the respondent forward into the next appropriate section, creating less confusion and simplifying the survey-taking experience.  This approach also allows for easy identification of non-respondents and facilitates follow-up.  All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden. 

Email will be used to send respondents their invitations to complete the survey and follow-up with the non-respondents to ensure their participation.  Email will also be used to invite degree noncompleters to participate in the telephone interview and to schedule the interview.

Data from the existing program monitoring databases, maintained under the EHR Generic Clearance (OMB # 3145-0136, Expires January 31, 2008), will be used to track graduates and degree noncompleters of the IGERT program and pre-fill surveys as appropriate. In case the databases do not have the contact information for a graduate or the information is not current, and the information is not available through their home department, Internet searches will be used to locate the graduates. 

A.4. Efforts To Identify Duplication

The Recurring Study of National Science Foundation-sponsored Graduate Education Impacts or Legacy (GEIL) does not duplicate other NSF efforts.  There are no studies at this point that track graduates of NSF’s graduate education programs to study the long-term impacts of their training. Project monitoring data for IGERT gathered via OMB 3145-0136 and program evaluation data gathered via OMB 3145-0182 will be available to the contractor and the NSF staff working on this research. These data will be used to pre-fill surveys as appropriate.  Future data collection tasks likewise will also draw on whatever prior program data exists, thus preventing duplication. 

A.5. Small Business

For the initial data collection instruments requested under GEIL, no information would be collected from small businesses.  It is possible that a future instrument requested under GEIL shall survey small businesses, for example, some graduates funded by NSF may have started their own businesses or work in a small business.

A.6. Consequences of Not Collecting the Information

If this information is not collected, NSF will be unable to document the initial and longer-term impacts of its graduate traineeship and fellowship programs as required by GPRA, nor will it be able to adequately assess program performance as required by PART, and the Deficit Reduction Act of 2005.  There are questions of the legacy and effectiveness of these programs and their impact on beneficiaries’ career trajectories that cannot be answered except through follow-up surveys with graduates.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collections will comply with 5 CFR 1320.6. 

A.8. Consultation Outside the Agency

Comments on this data collection effort were solicited in the Federal Register on August 31, 2007 (72 FR 50410).  A copy of the notice is included in appendix A.  During the first comment period prior to submission to OMB, no substantial comments were received.

One of the primary objectives behind the expansion of data collection under OMB 3145-0182 is to involve direct beneficiaries of NSF-funded programs (e.g., graduates with advanced degrees in science and engineering) in the NSF program development and management process, and to solicit their reactions to current and past investments NSF has made in graduate education. 

The survey and interview developed for this collection is for graduates of the IGERT program as well as for program degree noncompleters. The IGERT Program Announcement and Guidelines, coupled with consultation with the IGERT Program Committee, were major sources of information in designing this survey and interview protocol.  Discussions with NSF staff in the Directorate of Education and Human Resources, Division of Research, Evaluation and Communication and the Division of Graduate Education informed the development of the survey and interview protocol, as did discussions with staff members from ORC Macro, the company responsible for the IGERT Monitoring system, and Abt Associates, which is responsible for the study of the initial impacts of the IGERT program. PIs of the IGERT program were also solicited for their input. 

A.9. Payments or Gifts to Respondents

NSF requests the possibility of using e-gift certificates as incentives only if response rates require them.  Cash or checks would have to be sent through the mail, which would be a costly process and would require obtaining mailing addresses. 

NSF's contractors would offer respondents who are selected for the survey an e-gift certificate from an appropriate “.com” organization not in excess of $25 to encourage their participation in the study.  The incentive is proposed on the basis of our experiences in the Under-graduate Research Opportunities (URO) study OMB 3145-0121.

For the first task that will collect data from graduates and degree noncompleters of NSF-funded IGERT projects, we have not budgeted for nor will we offer such an incentive.  Incentives should be unnecessary in most cases, particularly with recent graduates, as NSF has been in recent and regular contact with IGERT respondents when they were students, and both IGERT and comparison graduates should be tech-savvy and motivated to assist the NSF.

A.10. Assurance of Confidentiality

Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974.  Data collected are available to evaluation contractors, contractors hired to manage data and data collection software, and at the aggregate level to NSF officials and staff.  Data are processed in accordance to Federal and State privacy statutes.  Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed, Reg.  264, 272 January 5, 1998).  The system limits access to personally identifiable information to authorized users.  Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c.  The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.

Individuals surveyed will be assured that the information they provide will not be released in any form that identifies them as individuals and their responses will be kept confidential.  The contractor will be expected to maintain the confidentiality, security, and integrity of survey data and interview data.  The web-based survey data and notes and transcripts of the interviews will be maintained on a secure server with appropriate levels of password and other types of protection.

All assurances of confidentially will be reviewed by the contractor’s Institutional Review Board prior to data collection.

 

A.11. Questions of a Sensitive Nature

In some cases, collections in GEIL will request information from respondents including name, address, Social Security number (SSN), date of birth, employment status, etc. These data are collected in order to measure the success of the NSF programs by combining data from this evaluation with previously approved evaluations for these programs. Information of this nature is also used to track recipients of funding and training.   OMB has previously approved the collection of SSN numbers under the EHR Generic Clearance (OMB 3145-0136) to be used as a tracking mechanism to permit follow-up studies that examine the long-term effect of the IGERT and GK-12 Program on individuals' success.

As is NSF standard survey business practice, the request for an individual's SSN will be a voluntary field.  Indeed all items of a sensitive nature are voluntary.  Respondents may choose not to provide information that they feel is privileged, such as SSN, address, or date of birth. Any individualized data that are collected are provided only to program staff and consultants conducting studies using the data as authorized by NSF. Any public reporting of data is in aggregate form.

Furthermore, sensitive questions will confirm data already available from the EHR Generic Clearance.  These data are necessary to determine whether the IGERT program has impacted the demographic diversity of its graduates.  The graduates will also be asked for their social security number and the contact information of someone who will know their whereabouts over the next few years.  These data are necessary to help locate the graduates of the IGERT program for follow-up surveys.  All of these questions will be clearly identified as voluntary and the respondents will have the option of not responding to them.  All of these questions will be reviewed by the contractor’s Institutional Review Board prior to fielding.

The proposed survey under subtask 1 asks questions for the names of their employer. This information is being collected so that characteristics of graduates’ employer can be coded and calculated (such as employer size, whether graduates are employed in “Fortune 500” companies, etc.). We are asking just for the employer name, instead of multitude specific questions about employers’ characteristics (a) to reduce burden and (b) because respondents may not always know the specific characteristics of their employers. The study will never report the actual names of employers.

A.12 Estimates of Response Burden

 

The overall estimated burden for the new generic clearance is 15,000 hours on the public (Federal Register, June 9, 2005, Volume 70, Number 110), for 4,000 respondents.  The first sub-task includes surveys of IGERT program graduates and comparison non-IGERT Ph.D.s, and interviews with IGERT degree noncompleters. The burden for this sub-task is 405 hours for 1,600 respondents.  The IGERT study was previously cleared for 1180 hours for 2,555 responses (OMB 3145-0182).

   

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The total number of respondents in the first sub-task is estimated to be 1,600 over one year.  The estimated response burden is estimated to be 405 hours over one year.

The chart below indicates the number of respondents to be surveyed/interviewed for each category of respondent type and the time demand these surveys or interviews will place on each individual respondent, and then aggregated across all respondents.

 

Estimated Interview Burden Hours
Respondent Type	Survey/Interview	Time Per Response (in hours)	Number of Responses	Total Time Burden (in hours)
IGERT graduates	Survey	0.25	800	200
Comparison non-IGERT graduates	Survey	0.25	750	188
IGERT degree noncompleters	Interview	0.33	50	17
Total			1,600	405

The estimated number of IGERT and non-IGERT respondents is based on the number of IGERT graduates (masters’ and Ph.D.) and degree noncompleters listed in the Monitoring Database who were in the 1998 – 2003 cohorts of NSF-funded IGERT programs and who received a degree (masters’ or Ph.D) or left their program without a degree in or before the 2005-06 academic year. The total number of IGERT survey respondents for this study is estimated to be 800 (225 masters’ graduates and 575 Ph.D. graduates) over one year.  The total number of comparison non-IGERT Ph.D. recipient respondents is estimated to be 750 over one year. 

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

Internet surveys will be administered to graduates of the IGERT program and a carefully selected comparison group of non-IGERT Ph.D. recipients. Survey questions pertain to their graduate experiences, perceptions of the importance of various training elements and adequacy of training, their location in the labor market, their current job and the impact of their IGERT/graduate training on their careers. The instrument is included in Appendix C.  The estimated average time per survey is based on similar surveys conducted in earlier studies and will be confirmed in pilot tests this summer.

The telephone interviews will be administered to a sample of IGERT program degree noncompleters.  Interview questions pertain to their reasons for enrolling in the program, their experiences in the program, their reasons for leaving their program, their current job and the impact of their IGERT training on their careers. The instrument is included in Appendix D.

Estimated Burden Hours for Survey and Interview Forms
Survey/Interview	Time per Response (in hours)	Number of Responses	Total Time Burden (in hours)
IGERT graduates	0.25	800	200
Comparison non-IGERT graduates	0.25	750	188
IGERT degree noncompleters	0.33	50	17
Total		1,600	405

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

A.13. Estimate of Total Capital and Startup Costs/Operation and Maintenance Costs to Respondents or Record Keepers

There is no overall annual cost burden to respondents that results from the Recurring Study of National Science Foundation-sponsored Graduate Education Impacts or Legacy other than the time spent responding to the survey attached as appendices to this request. 

A.14. Estimates of Costs to the Federal Government

Approximate Annual Total Costs
Annual Costs	Total
Personnel	$             283,236
Subcontracts	$               15,787
Other Direct Costs	$                 5,232
Sub Handling Fee	$                   568
G&A and Fee	$                 1,218
TOTAL	$             306,041

A.15. Changes in Burden

The inventory numbers at OMB for the now expired 3145-0182 covered 21 individual collection tasks or instruments. The OMB inventory records show a total number of respondents of 2,555 and total hours of 1,180 (rounded to the nearest hour).

 

For this renewal, 19 of the previous instruments are discontinued, 2 are revised, and 1 is new. This renewal anticipates an estimated annual average of 4000 total respondents and 2000 total hours; as explained in the Federal Register on June 9, 2005 (Volume 70, Number 110) a copy of the notice is found in Appendix A1. The total burden requested, however, for this initial revision (sub-task 1) following on the IGERT initial impact surveys is 405 annual hours, for 1600  responses.  The change in burden is results in a reduction in total hour burden of 775 hours and 955 respondents.

 

This change in burden is due to shifts in the number and type of respondents and adjustments in the data requested. The charts below summarize the changes in burden in the individual instruments.

 

Hour Changes in Task Burden for IGERT (Sub-task 1)
Respondent Type	Survey/Interview	Previously cleared burden (in hours)	Currently requested burden (in hours)	Change in burden (in hours)
Site Visits
Principal Investigator	Interview	60.00	0	-60.00
Co-Principal Investigators	Interview	30.00	0	-30.00
Faculty with multiple roles	Interview	100.00	0	-100.00
Faculty with single roles	Interview	167.00	0	-167.00
Funded trainees	Interview	200.00	0	-200.00
Non-funded associates	Interview	100.00	0	-100.00
University Administrators	Interview	33.00	0	33.00
Subtotal		690.00	0	-690.00
Initial Impact Evaluation
Current IGERT PIs	Survey	19.00	0	-19.00
Current IGERT department chairs	Survey	25.33	0	-25.33
Comparison non-IGERT department chairs	Survey	25.33	0	-25.33
Current IGERT faculty	Survey	76.00	0	-76.00
Comparison non-IGERT faculty	Survey	76.00	0	-76.00
Current IGERT students	Survey	101.33	0	-101.33
Comparison non-IGERT students	Survey	101.33	0	-101.33
IGERT graduates	Survey	12.50	200:00	187.50
Comparison non-IGERT graduates	Survey	12.50	188:00	175.50
Administrators at IGERT institutions	Interview	9.50	0	-9.50
Administrators at non-IGERT institutions	Interview	19.00	0	-19.00
IGERT student internship supervisors	Survey	10.00	0	-10.00
National STEM leaders	Survey	1.66	0	-1.660
International Stem Leaders	Survey	0.83	0	-0.83
Subtotal		490.33	388.00	102.33
New Instruments
IGERT degree noncompleters	Interview	0	17.00	17.00
Subtotal		0	17.00	17.00
TOTAL		1180.33	405.00	775.33

A.16. Plans for Publication, Analysis, and Schedule

The purpose of these studies is to examine the impact of NSF funding on the career trajectories of the graduate degree recipients who directly or indirectly benefited from NSF funds, particularly as a graduate trainee.  This will help NSF respond to such questions as:  How has NSF helped advance the careers of masters’ and Ph.D. graduates in the science and engineering workforce?  By following the graduates of the programs and asking them about where they are with their careers we can learn more about the perceived benefits of participation in these NSF-funded programs.

The follow-up study of the graduates of the IGERT program, for example, looks at the long-term impacts of the IGERT program and is concerned not only with how well the program is accomplishing its goals (to broaden the demographic base of the STEM workforce, to bring about change in the quality and culture of STEM graduate education, to educate graduate students to take on professorial roles and to work in a multidisciplinary and global environment), but is also concerned with the more proximal and distal impacts of the program on the program’s graduates in terms of the adequacy of their training, the level of their preparation for the work world, and finally its impact on their careers.  It is important to understand these effects based on the experiences and perspectives of those who participate in the IGERT program.  As the number of graduates coming out of the various IGERT projects is not very large, all graduates of the program for whom data is available from the Monitoring Database will be surveyed to learn about their experiences and perspectives. 

An analytic evaluation report will be prepared based on findings from the web-based survey and telephone interviews.  It is anticipated that the analyses of the survey will include simple frequencies taking into account the impact of their training on the time to degree, their entry into the labor market, their employment sector and the disciplinary fields of work, their job responsibilities and professional advancement, and their productivity. Cross-tabs with Chi square and analysis of variance will be used to assess the similarities and differences between IGERT graduates and comparison non-IGERT respondents. It is anticipated that the analyses of the interviews will include simple frequencies as well as descriptive summaries of emergent themes.

Like many agencies, NSF is reducing its reliance on formal (i.e., traditional) publication methods and publication formats.  The contracting agency, which will conduct this third-party study of the IGERT program on behalf of NSF, is forbidden contractually from publishing results unless NSF had made a specific exception.  All products of the collection are the property of NSF.  After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product On-line Document System or simply a page on the NSF Web site) in which to publish. 

Before the conclusion of the study both NSF grants funded under these programs may use preliminary data to improve management and performance.  For example, data generated by this study may appear as inputs to other internal and external NSF reports (e.g., the GPRA annual Performance Plan).  At this time, NSF has no set timeline for publishing interim reports from this study.

A.17. Approval to Not Display Expiration Date

Not Applicable

A.18 Exceptions to Item 19 of OMB Form 83-I

Not applicable.