Program Review of the Science and Technology Centers (STC): Integrative Partnerships Program (3145-new)
The National Science Foundation (NSF), requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, a three year clearance for NSF to conduct data collection efforts related to the evaluation of the NSF’s Science and Technology Centers: Integrative Partnership (STC) Program. A related data collection effort that consists of general grantee reporting describing all activities of the Center is approved for program monitoring under OMB 3145-0194.
The STC program provides multiyear (up to ten years) support to STCs as continuing awards that are among the largest (up to $4 million a year) awarded by the National Science Foundation (NSF). This support fuels innovation and builds intellectual and physical infrastructure within and among disciplines in the integrative conduct of research, education, and knowledge transfer. The STC program currently funds a total of 17 Centers—five beginning in 2000, six beginning in 2002, two beginning in 2005, and four beginning in 2006. STCs conduct world-class research through partnerships among academic institutions, national laboratories, industrial organizations, and/or other public/private entities, and via international collaborations, as appropriate. STCs enable and foster excellence in education, the integration of research and education, and the creation of bonds between learning and inquiry so that discovery and creativity more fully support the learning process. In addition, STCs capitalize on diversity through participation in Center activities and demonstrate leadership in the involvement of groups underrepresented in science and engineering.
To help fulfill the evaluation needs of the program, NSF has planned to collect data that is designed to explore the structures and processes the STCs and their participating universities have in place for developing the human capital of program participants and for fostering a variety of career paths. The primary methods of data collection will include data gathering from open sources and from records at NSF and grantee centers and from surveys of program participants. There are a bounded (or limited) number of respondents within the general public who will be affected by this research, including former graduate student and postdoctoral fellow participants of the centers. NSF will use the STC program evaluation data and analyses to provide members of an expert peer review panel with information about the program’s role in the talent development and on the career paths taken by students who participated in STCs and were involved in particular STC activities.
While descriptive statistics and in-depth information on project structures and activities are available from the already approved Grantee Reporting Requirements (OMB # 3145-0194), there is a need to evaluate the outcomes of the program for its participants, specifically former STC student participants or trainees (i.e., students who participated in at least 160 hours of STC program activities). This new information will inform the program review that will be conducted by an expert panel.
The primary purpose for collection of this information is program evaluation. Students participate in educational and research activities of the STCs, often spanning multiple years. For example, the majority of graduate students have participated in center activities for multiple years; 62% and 52% respectively in the class of 2000 and 2002 centers, while 38% and 48% have participated for only one year. Data collected will focus on the short-term career outcomes of former STC student participants and the educational and research environments that engendered them.
The data collected through this evaluation will provide information on the program’s role in the education and research training and on the career paths taken by students who participated in STCs. The evaluation will focus on answering three primary research questions; two that deal directly with education and research training of students and one that focuses on diversity, but has a large education component:
What is the effect of the of the STC Program, and specific STC activities, on developing human capital, creating excellence in educational and research training settings, and fostering a variety of career paths that have been either taken or explored by program participants?
What educational and research activities have been and/or will be effective and successful in accomplishing the education and research training goals of the program?
How effective has the STC Program been in the recruitment, preparation, and retention of a diverse population of U.S. students in science and engineering?
Although the extant documents (i.e., annual reports, site visit reports, monitoring system data) are a rich source of information about educational settings, program activities, and the people who participated in them, the extant data do not provide information about participants’ educational or career trajectories or participants’ perspectives on the value-added of Center participation to their education and careers. Consequently, we will conduct a follow-up survey of Center student participants to answer parts of the research questions that cannot be answered through document analysis alone and also to provide greater insight into and understanding of the extant data.
In order to reduce respondent burden, internet-based surveys will be used to collect information from participants. This approach has become more and more commonly used in recent years. NSF tends to favor Web-based systems because they can facilitate respondents' data entry across computer platforms and information, once entered into the system, can be presented to the respondent for verification, thus reducing the respondent burden. Another valuable feature is that there can be a thorough editing of all submitted data for completeness, validity, and consistency. Editing is performed as data are entered. Most invalid data cannot enter the system, and questionable or incomplete entries are called to respondents' attention before they submit their survey.
Web-based surveys employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. In addition, survey skip patterns automatically move the respondent forward into the next appropriate section, creating less confusion and simplifying the survey-taking experience. This approach also allows for easy identification of non-respondents and facilitates follow-up. All these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
Email will be used to send respondents their invitations to complete the survey and follow-up with the non-respondents to ensure their participation. Data from the existing program monitoring databases, maintained under the Generic Clearance (OMB # 3145-0194), will be used to identify STC student participants. Centers records will provide information about the student participants who have graduated. Center records and internet searches will be used to update contact information.
This evaluation does not duplicate other NSF efforts. There are no other studies at this point that examine the initial and long-term impacts of the STC program on participating students. Project monitoring data for STC gathered via OMB 3145-0194 will be available to the contractor and the NSF staff working on this research. These data will be used to identify participants and pre-fill surveys as appropriate. Future data collection tasks likewise will also draw on whatever prior program data exists, thus preventing duplication.
No small businesses will be involved in this study.
Based on prior evaluations of the program, the National Science Board (NSB) approved the continuation of Science and Technology Centers through the establishment of new program solicitations and several new competitions. As part of the continuation, the NSB required the program to conduct a program evaluation of the outcomes and impact of the program seven years after the first new cohort of Centers were established (Memorandum to Members of the National Science Board, February 13, 1997). Education is one of the key goals of the program, and failure to collect the information proposed in this request will prevent NSF from assessing the role that the STC program plays in the professional preparation of students.
The project will fully comply with the guidelines of 5 CFR 1320.5. No special circumstances apply to this data collection.
Comments on this data collection effort were solicited in the Federal Register on June 11, 2009 (vol. 74, no. 111). A copy of the notice is included in Appendix A. During the first comment period prior to submission to OMB, one comment was received from
Roger Clegg of the Center for Equal Opportunity (attached) with a request for reassurance that the STC program “does not use quotas, numerical goals, or other discrimination or preferences.” NSF responded to inform Dr. Clegg that the program does not use quotas, numerical goals, or other discrimination or preferences on the basis of race, ethnicity, or sex. As such, NSF is proceeding with seeking approval from OMB.
Consultation on the study design was conducted by the research firm, Abt Associates Inc., contracted by NSF to provide evaluation technical assistance to the STC program. In addition, the proposed data collection instrument was shared with STC directors for feedback and was pilot tested with respondents drawn from the target populations. Respondents were asked to comment on the clarity and content of the questions. The duration of the data collections was recorded to help with an accurate estimation of time burden. Five current or former STC students completed the surveys. The median time to completion was 38 minutes. The survey was shortened by removing individual items and sections in order to reduce the respondent burden to 30 minutes. The principal investigator from the American Association for the Advancement of Science (AAAS) who will leading the program review was also consulted.
No payment or gift will be provided to respondents.
Respondents will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. Data collected are available to evaluation contractors, contractors hired to manage data and data collection software, and at the aggregate level to NSF officials and staff. Data are processed in accordance to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed, Reg. 264, 272 January 5, 1998). The system limits access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants, and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in court, or Federal administrative proceeding, if the government is a party.
Individuals surveyed will be assured that the information they provide will not be released in any form that identifies them as individuals and their responses will be kept confidential to the extent provided by law. The contractor will be expected to maintain the confidentiality, security, and integrity of the survey data. The web-based survey data and notes will be maintained on a secure server with appropriate levels of password and other types of protection.
All assurances of confidentially will be reviewed by the contractor’s Institutional Review Board prior to data collection.
The proposed surveys ask for demographic information (gender, race/ethnicity, and citizenship status) from participants on a voluntary basis, thus respondents may choose not to provide information that they feel is sensitive in nature. This information is being collected so that NSF can answer questions about how effective the STC program has been in the recruitment, preparation, and retention of a diverse population of U.S. students in science and engineering. All survey questions will be reviewed by the contractor’s Institutional Review Board prior to fielding.
The requested burden for this evaluation is 850 hours for 1700 respondents, which represents the universe of former students across the 17 active STCs.
The total number of respondents for surveys is estimated to be 1,700 and the estimated overall response burden for the surveys is estimated to be 850 hours over one year. The estimated average time per survey and interview is based on similar surveys and interviews conducted in earlier studies and the information gathered during pilot tests.
The chart below indicates the number of respondents to be surveyed for each category of respondent type and the time demand these surveys will place on each individual respondent, and then aggregated across all respondents.
Respondent Type |
Number of respondents |
Time per response (hours) |
Number of responses |
Total time burden (hours) |
Students |
1200 |
0.5 |
1200 |
600.0 |
Postdocs |
500 |
0.5 |
500 |
250.0 |
Total |
|
|
1700 |
850.0 |
Each respondent will receive the survey one time. The table below contains the same values as the table in section A.12.1 and shows the total burden hours. Because the survey is given to each respondent only one time, the annual burden and the aggregate burden are the same.
Respondent type |
Number of respondents |
Time per response (hours) |
Number of responses |
Total time burden (hours) |
Students |
706 |
0.5 |
706 |
353.0 |
Postdocs |
294 |
0.5 |
294 |
147.0 |
Total |
|
|
1000 |
500.0 |
The overall annualized cost to respondents is $29,750. The following chart shows the estimated total annual costs to each group of respondents over one year for the surveys.
Respondent Type |
Number of respondents |
Time Per Response (hours) |
Number of Responses |
Total Time Burden (hours) |
Hourly salary estimate |
Estimated cost per respondent |
Estimated overall cost |
Students |
706 |
0.50 |
706 |
353.0 |
35.00 |
17.50 |
12,355.00 |
Postdocs |
294 |
0.50 |
294 |
147.0 |
35.00 |
17.50 |
5,145.00 |
Total |
|
|
1000 |
500.0 |
|
|
17,500.00 |
There is no overall annual cost burden to respondents that results from this study other than the time spent responding to the survey.
The estimated cost to the Federal Government for the data collection activities included in this request for approval is $126,329. This cost estimate includes: instrument development and pretesting; student recruitment; data collection; and data processing.
This is a new collection of information.
To provide the STC expert peer review panel with an understanding of the progress the centers have made toward reaching their education and diversity goals, and the role of participation in STCs on the development of human capital, the contractor will prepare a brief report for the STC expert panel review that describes the study and findings. The report will characterize the educational setting (e.g., curriculum, courses, integration of research and education, mentoring structures, teamwork, opportunities for professional development/internships in industry, opportunities to work at other center campuses) in general, and for each group of students (e.g. graduate, and post-doc), what educational and career paths participants have taken, and how well prepared participants are for the next stage in their education/career.
Tables, charts, antidotal reports, and descriptive information will be organized to provide NSF and the expert peer review panel an understanding of the role the STCs play in developing human capital. Descriptive analysis of extant program data that provides information about the activities offered by the centers will be used to frame the findings of the study. Analyses will include simple descriptive analyses, including counts, measures of central tendency and frequency distributions that describe students’ participation in center activities, their other educational experiences, and their education and career paths.
Activity |
Schedule |
Program surveys |
1 months after OMB approval |
Update student contact information |
2-3 months after OMB approval |
Recruit survey respondents |
4-5 months after OMB approval |
Implement survey |
6-7 months after OMB approval |
Analyze data |
8-9 months after OMB approval |
Report findings |
10-12 months after OMB approval |
The data collection instruments will display the expiration date.
No exceptions are sought.
| File Type | application/msword |
| File Title | Section A |
| Author | NeishiK |
| Last Modified By | SUZANNE |
| File Modified | 2010-04-13 |
| File Created | 2010-04-13 |