3145-0217 Part B - December Revision

3145-0217

Section B

Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Survey Samples

The survey data collection will include all 104 recipients awarded Master Teaching Fellowships (MTFs) in the 2010-11 academic year. The study will field one wave of survey data collection of Noyce MTFs. (See Appendix A for Noyce MTF survey).

B.2. Procedures for the Collection of Information

The study team will field online surveys for Noyce Master Teaching Fellows. The study will use the same procedures for identifying the list of MTFs as discussed in the overall evaluation. The study team will obtain the list of names and e-mail addresses of PIs from the Noyce Monitoring System. Additionally, the monitoring data will provide us with the names of recipients and schools where past Noyce participants are currently teaching. The next step will be to locate respondents and obtain current contact information. PIs will be contacted to provide the contact information for these MTFs.

We expect to obtain complete information for these recipients given that they are all currently receiving support.

As with the other surveys, we will implement an accessible and intuitive survey. Prior to data collection, the study team will provide early notification to respondents, including a signed letter from NSF as well as from the Abt Team. This will permit respondents to allocate time to complete the surveys, and increase respondent “buy-in,” enabling them to ask questions before beginning data entry and allowing our team to establish rapport with any reluctant respondents. At the designated opening date, an e-mail message will be sent to respondents with a unique user name and password, detailed instructions, the closing date, and project staff contact information. Detailed on-screen instructions will be included and an extensive help functionality for survey items, including a Frequently Asked Questions section, glossary, and navigation instructions. Throughout the data collection cycle, a toll-free number and e-mail address will be available to ensure that potential respondents can easily and quickly obtain answers to questions or concerns. We will conduct up to five follow-ups per respondent.

Protecting the Confidentiality of Data Collected. Electronic project directories, files, and databases are accessible to project staff only and are protected by discretionary access control lists (ACLs), group memberships, passwords, and locking workstations. Access to the data processing area and database servers is limited to authorized personnel and building security staff all sites 24 hours, 7 days per week. To protect against data loss, Abt also uses automated, redundant backup procedures and file management techniques to ensure that files are not inadvertently lost or damaged. The web-based survey data will be maintained on a secure server with appropriate levels of password protection.

B.2.1. Statistical Methodology

Primary Data Collection

As described above, the study team will be surveying the census of MTFs that received Fellowships in the 2010-11 academic year.

The primary purpose of surveying the Noyce MTFs is to describe supports MTFs receive and the leadership activities they engage in, and to see how various program components affect MTF outcomes. Therefore, for this part of the study, a comparison group will not be surveyed.

Analyses of all survey data will include a detailed summary that utilizes appropriate descriptive statistics. For survey items using continuous scales, the study will calculate means and standard deviations to describe both central trends and variation across the samples of schools. Frequency distributions and percentages will be used to summarize answers given on ordinal scales.

The strength of the observed associations between program and recipient characteristics and outcomes will be estimated using multiple regression techniques to appropriately control statistically for other factors that could potentially affect recipients’ plans to stay in teaching other than the recipients’ respective IHE’s level of financial support (i.e., contextual factors such as climate of K-12 school, districts’ hiring policies due to changes in state or district economic circumstances).

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

One challenge of this study will be motivating participation and following-up with initial non-respondents to achieve our target response rate. The web approach will allow us to easily identify non-respondents for follow-up emails and phone contact to encourage participation, thus substantially increasing response rates. In an effort to increase overall survey response rate, follow-up with respondents will be multi-modal. Telephone and email follow-up will be used for non-respondents.

Prior to data collection, early notification will be provided to respondents, including a signed letter from NSF as well as from the study team. This will permit respondents to allocate time to complete the surveys, and increase respondent “buy-in,” enabling them to ask questions before beginning data entry and allowing our team to establish rapport with any reluctant respondents. At the designated opening date, the study team will send an e-mail message to respondents with a unique user name and password, detailed instructions, the closing date, and project staff contact information. Detailed on-screen instructions will be included with an extensive help functionality for survey items, including a Frequently Asked Questions section, glossary, and navigation instructions. Throughout the data collection cycle, the study will use a toll-free number and e-mail address to ensure that potential respondents can easily and quickly obtain answers to questions or concerns.

One challenge of this study will be motivating participation and following-up with initial non-respondents to achieve our target response rate. The web approach will allow us to easily identify non-respondents for follow-up, thus substantially increasing response rates. Letters and follow-up efforts will encourage participation by including information on the importance of participation to document effects and on having evidence to support the importance of the program.

B.4. Test Procedures or Methods

The survey instruments developed for this data collection were pilot-tested with seven Master Teaching Fellows to ensure that items were clear, that language was unambiguous, and that items were understandable to respondents. Based on responses to the pilot tests, the study team clarified ambiguous language and item formats, eliminated items that were not meaningful, and revised response scales.

The survey also draws from the National Center for Education Statistics Schools and Staffing Survey 2007-08 (SASS). Some items have been adapted from the national protocol to ask about more specific information necessary to understand MTFs’ educational background and professional development and leadership activities.

Finally, to ensure that items have been properly interpreted, we will compare survey results to results from data entered into annual monitoring system about related activities (note that the data entered into the monitoring system are more generic and less individualized than data that will be obtained from surveys that allow respondents to report individual experiences ).

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractors for collection and analysis of data in this study are Abt Associates Inc., and its data collection subsidiary Abt-SRBI. Staff from these organizations have knowledge of survey design and data collection, statistical methods, experience in evaluation of research programs, and expertise in scientific research.

Key personnel who will be involved in collecting and analyzing data include:

Abt Associates, Inc. Ellen Bobronnikov 617-349-2718

Beth Gamse 617-3492808

Cristofer Price 301-634-1852

Amanda Parsad 301-634-1791

Radha Roy 301-347-5722

Abt-SRBI Paul Schroeder 301-608-3883

Andy Weiss 301- 608-3883

Abt Associates Inc. Section B: Collection of Information Employing Statistical Methods B-3