Supporting Statment

OMB Control No. 0693-0033 – NIST Generic Clearance for Program Evaluation Data Collections

OUTCOME STUDY FOR THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY (NIST) SUMMER INSTITUTE FOR MIDDLE SCHOOL SCIENCE TEACHERS

1. Explain who will be surveyed and why the group is appropriate to survey.

The survey group will consist of middle school science teachers involved with the annual National Institute of Standards and Technology (NIST) Summer Institute for Middle School Science Teachers. The target group will be the teachers who were selected for the Summer Institute, with a comparison group of teachers who applied for the Summer Institute but were not selected due to oversubscription.

It is essential that NIST understand the best practices for developing and implementing high quality professional development for middle school teachers. The collection of the requested data will enable NIST program officers to enhance the quality and responsiveness of the professional development opportunities that are made available to future Summer Institutes participants. The survey of the teachers who applied but were not selected will provide a control group upon which to compare results.

This request for review asks for clearance to conduct an outcomes study of the NIST Summer Institute. The study will examine outcomes of teacher participation in Summer Institutes 2010, 2011, and 2012. Each year, participants will include a treatment group of participating middle school science teachers and a control group of non-participating middle school science teachers. Each year a maximum of 20 participants and 20 non-participants will comprise the treatment and control groups. The purpose is to obtain information that can be used to assess the program’s effectiveness and improve the professional development provided at future Summer Institutes.

Background: Reports on the current status of science and mathematics education in the U.S. indicate that a significant number of K-12 teachers of science and mathematics classes have neither a degree nor significant training in the subjects that they teach. In order to enable teachers to excite our next generation of scientists, mathematicians and engineers and keep them engaged in these important subject fields, scientific and technical agencies need to provide opportunities for teachers themselves to experience more real-world science. Such experiences can provide “ammunition” for teachers to be more informed and excited about their subjects and therefore impart that knowledge and excitement to their students.

To address these issues, NIST initiated a two-week Summer Institute for 12 Montgomery County Public School System (MCPS) science teachers as a professional development pilot program that was aligned with the school system’s science curriculum in 2007. The program has expanded to include additional school districts, with a second session conducted with 16 teachers in the

summer of 2008 and a third session was held with 15 teachers in the summer of 2009. The plan is to continue holding a summer session in the coming years—with a goal of reaching a new group of approximately 20 middle school (Grades 6, 7, and 8) science teachers per session.

This two-week long professional development workshop is designed to support middle school science teachers through a combination of hands-on activities, lectures, tours, and visits with NIST scientists in their laboratories. In alignment with county middle school curriculum, teachers are provided with resources and instructional tools for teaching math and science, with an emphasis on the measurement science used at NIST. Led entirely by NIST scientists, the Summer Institute translates cutting-edge research done in the laboratory to activities intended to be carried out in the classroom. Networking among NIST scientists and teachers provides long-term resources through on-going relationships for the teachers and their students. Return visits of the participants occur during the school year, with efforts focused on additional lessons and collaborative enrichment.

The format for the summer sessions is a combination of short interactive lectures and laboratory demonstrations/experiments, combined with time spent with selected NIST staff in the laboratory, for limited hands-on research. The Summer Institute is designed to allow teachers to see measurement research in a real-world environment, to give teachers practical laboratory examples suitable for classroom presentation, and to cultivate mentoring relationships with NIST scientists that the participants will draw upon in the future.

The Summer Institute strives to improve the participating middle school teachers’ content knowledge and instructional practices in scientific disciplines. It provides a chance not only for the teachers to gain ideas to bring back to their students, but to build mentoring relationships with NIST scientists that reinforce their confidence in their command of the subject matter. The program has significant potential to improve teacher content knowledge and enthusiasm in their subject matter.

Focus groups and informal feedback of teachers who have completed the program have provided the basis for improvements to the annual program to date. However a more formal approach of evaluation is necessary, both for suggestions for program improvement, and for a more rigorous assessment of the program itself.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The survey was developed by the contractor for the Summer Institute assessment, Westat, together with the Summer Institute coordinator and administrator.  The survey is the result of Westat’s experience in assessing education programs, their previous experience of conducting focus groups with the first three groups of teacher participants in the NIST Summer Institute (2007-2009), and results from the customer satisfaction survey that was conducted last year.  In addition to feedback from the Summer Institute’s coordinator and administrator, a draft survey was informally shown to returning teachers, who have participated in the 2007, 2008 or 2009 Summer Institutes during one of the recent return Science Afternoons at NIST, and their suggestions were incorporated into the surveys submitted here.

The information to be collected represents the minimum effort required to assess the effectiveness of the professional development provided by the NIST Summer Institute. It is expected that each pre-survey will take 15 minutes, and each post-survey will take one hour for each respondent to complete. In year 1, participants will complete one pre and one post-survey; non-participants will only complete a post-survey. In years 2 and 3, each participant and non-participant will complete both the pre and post-surveys. The pre-survey will be administered in the spring preceding the Summer Institute and the post-survey at the end of the school year following the Summer Institute.

The purpose of this effort is to determine the impact of the NIST Sumner Institute. Specifically, the survey will be used to assess whether there is an increase in the content knowledge and a change in the instructional practices of participating middle school science teachers. Westat, NIST’s contractor for evaluating the Summer Institute, has provided NIST with a series of reports on the implementation of the professional development program and its’ efficacy. While these reports have provided summary data of teacher experiences, they have not been able to delve more deeply into whether participating teachers made appropriate use of the NIST professional development in their classrooms. Nor have they explored the factors that influenced the extent to which they were able to make use of the knowledge and tools they received during the Summer Institute. The reports that result from these outcomes studies will be used to inform development and revise practices for future Summer Institutes.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

A maximum of 40 respondents per year will be surveyed. All respondents will be middle school science teachers who have expressed an interest in participating in NIST’s 2010, 2011, or 2010 Summer Institutes. Year 1 non-participants will only complete a post-survey, while participants will complete both a pre- and a post-survey. In years 2 and 3, both participants and non-participants will complete both a pre- and a post-survey.

For participants, the response burden is 15 minutes for the pre-survey and 1 hour for the post-survey. For non-participants, the response burden is 15 minutes for the pre-survey and 15 minutes for the post-survey. The difference in burden between participants and non-participants on the post-survey reflects additional questions that will be asked of participants about their experiences in the NIST Summer Institute.

Participants in year 1 will receive and complete a paper version of the pre-survey on the first day of the 2010 Summer Institute. Participants and non-participants in year 2 and year 3 will receive the pre-survey as part of the application process for the Summer Institute. All participants and non-participants will be contacted by NIST via email requesting that they complete an electronic version of the post-survey and send the completed form to Westat. Respondents can also print out the survey and mail it back to Westat. Upon completion of the post-survey, respondents will receive a $50 gift card from a local bookstore.

Individuals who do not complete the post-survey within the two-week timeframe will receive a follow-up phone call from either the NIST representative responsible for overseeing the Summer Institute, a representative from Westat, or both. The purpose will be to encourage their completion of the post-survey. In addition, as an incentive for completing the outcomes study, all respondents will receive a $50 gift card from a local bookstore when they have completed the post-survey. This incentive will be critical in obtaining a full response rate among teachers randomly assigned to the control group who were not provided an opportunity to participate in the Summer Institute.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The data collected will be analyzed for trends using statistical analysis, and will be presented in aggregate form. The surveys will not include questions of a sensitive nature. Respondents will be assured that their responses will not be shared with other individuals at the Summer Institute, with school staff, or with the NIST Summer Institute staff, and that all results will only be presented in an anonymous and aggregated form.

The results of the survey will be examined the second year to assess the efficacy of the program in improving participating middle school teachers’ content knowledge and instructional practices in science. This includes examining: to what extent did the professional development provided by NIST meet teachers’ needs; whether the NIST-based activities lead to the learning for which they were designed, in terms of both content knowledge and pedagogy; whether teachers incorporated the NIST-based activities into their classrooms; whether students found the NIST-based activities engaging; and, whether there are components of the program that need to be strengthened or improved.

4