CE 4 1 5 OMB Submission Supporting Statement B

Contract No: ED-IES-12-C-0007

Supporting Justification for OMB Clearance of a Study of Clinical Practice in Traditional Teacher Preparation Programs in Missouri

Section B

December 5, 2013

Submitted to: Submitted by:

Institute of Education Sciences REL Central at Marzano Research Laboratory

U.S. Department of Education

555 New Jersey Ave., NW 9000 E. Nichols Ave., Ste. 112

Washington, DC 20208 Centennial, CO 80112

(202) 219-1385 (303) 766-9199

Project Officer: Project Director:

Sandra Garcia Stephen Meyer

Table of Contents

Section B. Data Collection Procedures and Statistical Methods 2

B1. Respondent Universe and Sampling Methods 2

B2. Procedures for the Collection of Information 5

B3. Methods to Maximize Response Rates and To Deal With Non-Response 5

B4. Tests of Procedures or Methods to be Undertaken 6

B5. Individuals Consulted on Statistical Aspects of the Design 7

References 8

Attachment A: Public Law 107-279-Nov. 5, 2002 Education Sciences Reform

Attachment B: Printed Copy of Online Survey

Attachment C: Affidavit of Nondisclosure

Attachment D: Text for Invitations and Reminders

Section B. Data Collection Procedures and Statistical Methods

This study is designed to support needs in the Central Region by providing descriptive information about clinical practice in Teacher Preparation Programs (TPPs). The study purpose is to document current practice and provide a data collection tool and an approach for collecting data from TPP graduates that can be adopted and adapted as part of future research to be conducted by the Educator Effectiveness Research Alliance (Teacher Preparation Workgroup). Data from the study may be used to understand the extent to which clinical practice components vary within and across programs. This will inform conversations about emerging state standards for TPPs, such as what constitutes minimum implementation of priority components. These data will also inform conversations about TPP reform or redesign by identifying areas in which programs have less or greater emphasis.

B1. Respondent Universe and Sampling Methods

State Selection. Data to address the research questions will be collected from first-year teachers teaching in public schools in Missouri. The sample is focused on one state to provide a feasible scope of data collection, with the potential to apply the study methodology in other Central Region states based on their needs and priorities in subsequent years. Missouri was chosen as the Central Region state in which to collect data for the following reasons:

1. Missouri prepares more teachers each year than any other state in the region;

2. The Missouri Department of Elementary and Secondary Education (MO DESE) has a data system with the capacity to link value-added student achievement to TPPs and has collaborated with external researchers to analyze these data and publish results about the effectiveness of TPPs (for example, Ehlert, M. et al., 2011; Koedel, Parsons, Podgursky, & Ehlert, 2012). These data could be used in future work linking TPP characteristics to student achievement outcome data;

3. The MO DESE Office of Educator Quality is actively developing program standards for a comprehensive TPP assessment system with an annual program-level reporting component and is exploring new ways to collect data about TPPs;

4. Missouri is one of two Central Region states pursuing reportedly “innovative” models for clinical preparation as part of its participation in the NCATE State Alliance for Clinically-Based Teacher Preparation¹; and

5. Administrators in the MO DESE Office of Educator Quality (who are also members of REL Central’s Educator Effectiveness Research Alliance) have relationships with TPP administrators and have agreed to provide contact information for all first-year teachers in the state.

Focus on Teachers Prepared in Traditional Programs. Missouri offers six “routes to certification².” The study will include teachers prepared through Missouri’s “traditional route” for two primary reasons. First, most graduates of Missouri TPPs are trained through this route. Missouri’s 2012 Title II State Report³, shows that that 6,226 teacher candidates (86%) completed the “traditional” route at one of 38 institutions of higher education in the state during the 2010–11 academic year. Second, in traditional programs there is a clear distinction between teacher preparation and teaching—unlike in alternative certification routes in which many teacher candidates serve as the classroom teacher of record prior to completing their preparation program and becoming a certified teacher. Such a distinction is needed because a retrospective survey of clinical practice experiences cannot be meaningfully completed by practicing teachers who have not completed their preparation program. Only teachers who have completed their preparation can provide accurate and complete reports of their clinical experience. Further, a survey of the clinical experiences of teachers trained through alternative routes requires a different set of questions to address fundamental differences in definitions of terms such as “field experience” and “student teaching experience” which would provide information that could not be directly compared to that collected from students in traditional programs. In this first effort to collect comprehensive and individual-level data about clinical experience in teacher preparation, REL Central believes that study resources are most efficiently deployed through a study that focuses on the pre-service preparation activities of teachers in traditional programs. The study will generate findings that are relevant to the route through which most teachers are prepared in Missouri, providing information that can be meaningfully compared across programs.

Identification of Recent Teacher Preparation Program Graduates. To identify individuals who recently completed teacher preparation programs in Missouri, REL Central will work with MO DESE to collect names and contact information of all first-year public school teachers in the state who were prepared in traditional programs. Since 2007, MO DESE has administered an online First-Year Teacher Survey to all first-year teachers in the state. Each year, the state works with the Office of Social and Economic Analysis (OSEDA) at the University of Missouri to develop a comprehensive list of teacher names and contact information for this purpose. OSEDA has developed protocols to ensure inclusion only of first-year teachers, excluding those teachers who are new to a particular district or new to the state, for example. Use of MO DESE contact information for practicing first-year teachers was chosen in favor of an alternative approach in which program graduate contact information would be collected from each of the nearly 40 teacher preparation programs in the state. While collecting contact information from each Missouri TPP would help to ensure representation of all TPP graduates regardless of whether they chose to teach in the state, consideration of the following factors led REL Central to choose the former approach:

• Collection of graduate contact information from multiple TPPs—some of which might have additional requirements for data release or human subjects review—would require OMB approval which would require additional effort and lengthen the project timeline;

• Data collection from multiple TPPs would require contact with administrators at each TPP, increasing the overall data collection burden for respondents;

• The quality and consistency of data collected from multiple TPPs would likely vary, creating potential challenges for successfully contacting graduates; and

• Collection of information about the pre-service clinical practice experiences of first-year teachers in Missouri is potentially more useful in that: (1) state administrators and policymakers may be most interested in the experiences of teachers who actually go on to practice in the state and, (2) the data collected via REL Central’s survey can be more easily linked to data maintained by MO DESE in future research. For example, reports of teacher preparedness collected on the First-Year Teacher Survey could potentially be linked to reports of clinical practice experience in a future research project.

Representativeness of the Sampling Frame. Data from the prior-year state Title II report will be used to provide an indication of the percentage of all traditional TPP graduates who are included in the study sampling frame. Using the count of teachers trained during the 2011–12 academic year (from Missouri’s 2013 Title II State Report) and the number of first-year public school teachers who were trained in traditional Missouri TPPs (those in the study sampling frame), REL Central will compute the estimated percentage of TPP graduates included in the sampling frame.

Sampling Plan. REL Central used the Guidance on Agency Survey and Statistical Information Collections (Office of Management and Budget, 2006) to design the sampling plan for the study. Based on Missouri’s 2012 Title II State Report, eighteen teacher preparation programs had fewer than 100 graduates; twelve programs had between 100–200 graduates; and eight programs had more than 300 graduates. Among these programs, there are only four that are likely to have had more than 300 graduates who go on to teach in public schools in Missouri.

Based on OMB guidance, and given a confidence interval level of 95 percent, the minimum proportion of respondents needed (to achieve estimates with a 95 percent confidence interval no larger than plus or minus five percent) is listed in Table 1 for several teacher preparation program sizes. “Adjusted minimum” proportions are presented to account for a nonresponse rate of up to 20 percent.

Table 1: Minimum Proportion of Respondents Needed by Program Size

Program Size	Minimum Proportion	Adjusted Minimum Proportion
100	95%	100%
200	90%	100%
300	80%	100%
400	60%	75%
500	50%	62.5%

As Table 1 shows, OMB guidance suggests that all graduates of programs with 300 or fewer graduates should be included in the study sample to provide estimates with a 95 percent confidence interval of no larger than plus or minus five percent—accounting for an estimated survey nonresponse rate of up to 20%. Therefore, REL Central will include all first-year teachers who graduated from traditional teacher preparation programs in the study sample, resulting in an estimated 3,500 first year teachers in the sampling frame.

B2. Procedures for the Collection of Information

The study involves one data collection instrument: a researcher-developed teacher survey, which will be administered online during a 12-week period beginning either in April or September 2014⁴. REL Central will work with MO DESE to collect contact information for first-year teachers who teach during spring 2014 or fall 2014. Staff in the Educator Preparation Section at MO DESE have agreed to share the contact information for first-year teachers that is used for administration of the states’ First-Year Teacher Survey. MO DESE has agreed to provide the following data elements to REL Central: teacher identifier, teacher name, teacher email address, name of school in which they are teaching and physical address, current teaching assignment, certification route, name of preparation program, teaching certification(s), gender, and race/ethnicity.

Using the data provided by MO DESE, an invitation to complete an online survey will be sent via email to each first-year teacher who was prepared in a traditional program. The invitation will be sent by REL Central, describing the purpose of the study, the voluntary nature of participation, intended use of the data collected, methods for maintaining confidentiality, and the estimated amount of time needed to complete the survey (estimated to be 25 minutes). Customized invitations will be used with respondent identifiers embedded in the emailed survey links to track survey response. If email addresses are not available, the study team will communicate with respondents via U.S. Postal Service mail sent to their school address. Paper surveys will be mailed to respondents via USPS will contain unique identifiers for the same purpose. Text for invitations and reminders is presented in Attachment D.

B3. Methods to Maximize Response Rates and To Deal With Non-Response

MO DESE has offered to endorse the survey and to help REL Central solicit the support of state teacher associations. REL Central will seek endorsement of the study by the Missouri National Education Association, the Missouri State Teachers Association, and the American Federation of Teachers in St. Louis. The survey will include information about the purpose of the study, including its potential utility for improving teacher preparation in the state. All endorsements will be included in the study invitation sent to respondents. Potential respondents will also be informed of the importance of their feedback to understanding the range of experiences among first-year teachers.

MO DESE has been able to achieve a 55 to 60 percent response rate for the First-Year Teacher Survey by soliciting respondents by email only and sending two or three follow-up email messages to non-respondents. REL Central will use the same initial approach to collect data from first-year teachers—contacting the first-year teachers via email using contact information from MO DESE. Researchers will employ strategies identified in Dillman (2000) to seek response rates (for the entire sample and for each TPP) of a minimum of 80 percent, using the following approach.

• Initial email messages will be sent by REL Central, with information about the study purpose and organizational endorsements, as described above.

• Email messages will contain links to the online survey with embedded respondent identifiers that allow REL Central to track survey response and follow up with non-respondents. Paper surveys mailed to respondents via USPS will contain unique identifiers for the same purpose.

• REL Central will follow up with non-respondents for a total of up to seven occasions using three different modes: email, hardcopy via US Postal Service (USPS), and telephone. The follow-up sequence comprises the following contacts:

Contact 1: A postcard thank-you note and reminder will be sent to all participants after the first week of the email survey implementation.

Contacts 2–4: Survey non-respondents will receive another three emails⁵ that contain the survey at approximately 10-day intervals after the first contact.

Contact 5: A USPS package will be sent to each non-respondent. The package will contain the paper survey and a cover letter encouraging participation.

Contacts 6 & 7: REL Central will make the last two follow-up contacts via telephone. The researcher will call non-respondents during school hours, informing them that a survey was sent to them previously, asking if they have questions about the study, and encouraging them to complete the survey. If a non-respondent is not available for the phone call, the researcher will leave a voice message and make a follow-up phone call within two work days. During the calls, the researcher will give non-respondents the opportunity to complete the survey on the telephone or send another paper copy of the survey.

B4. Tests of Procedures or Methods to be Undertaken

Survey Pretest. Cognitive interviews⁶ were conducted by telephone with nine first-year teachers in Missouri in summer 2013. REL Central collected contact information for a random sample of potential respondents, who were graduates of nine randomly-sampled teacher preparation programs. Respondent participation was solicited via email by providing information about the survey and cognitive interview process, and the study for which the survey was developed. Participants in the survey pretest will be excluded from the sampling frame for the full study. Cognitive interviews were used to improve the relevance and clarity of survey items and to minimize response error (Tourangeau, 1984; Willis, 1999). The Theory of Response Process proposed by Tourangeau, Rips, and Rasinski (2000) was used as a basis for plans to understand how survey respondents answer questions and to identify sources of survey error. Respondents were asked to first complete the survey online and then contacted for a telephone interview. Retrospective verbal probing was used to review the draft survey content with each respondent and collect feedback to clarify and improve item wording (Willis, 1999). Information collected via cognitive interviews was used to create a final version of the instrument.

Survey Data Analysis. REL Central will present descriptive statistics for all respondents in the sample. To provide context for interpreting findings, respondent counts, survey response rates, and descriptive information about survey respondents will be presented (name of teacher preparation program, certification(s), current teaching assignment, gender and race/ethnicity). These variables will also be used to assess nonresponse bias. For continuous variables (such as number of hours or weeks), means and standard deviations will be presented. For categorical variables, the proportion of respondents selecting each response category will be presented. For “check all that apply” responses (such as subject areas taught), means and standard deviations based on the total number of responses selected will be reported, along with the proportion of respondents selecting each response category. Frequencies (expressed as percentages) of response categories will be presented in tables with categorical response options.

Nonresponse Analysis. As one of the four sources of error in sample survey research, nonresponse bias is a concern for response rates as high as 90 percent (Dillman, 2000; Miller & Smith, 1983). This study will assess the presence of nonresponse bias by comparing respondent and non-respondent characteristics based on available data for the entire sample (current teaching assignment, certification route, name of preparation program, teaching certification(s), gender, and race/ethnicity). Characteristics of respondents and non-respondents will be analyzed to identify significant differences. If nonresponse bias is identified, sample weights will be developed and incorporated into analysis plans with under the guidance of a sampling statistician. Either post-stratification weighting (for example, see Taylor, 2005; Oh and Scheuren, 1983) or propensity score adjustment (for example, see Duffy, Smith, Terhanian, & Bremer, 2005; Lee, 2006; Loosveldt & Sonck, 2008) will be used to develop sample weights.

B5. Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical, data collection, and analytic aspects of this study:

Name	Title	Organization	Contact Information
Bruce Randel	President	Century Analytics	[email protected]
Trudy Cherasaro	Senior Researcher	Marzano Research Laboratory	[email protected]

The following individuals will be involved in the study implementation:

Name	Role	Organization	Contact Information
Stephen Meyer	Principal Investigator	RMC Research Corporation	[email protected] 303-825-3636
Xin Wang	Research Analyst	RMC Research Corporation	[email protected] 303-825-3636
Matt Linick	Research Analyst	RMC Research Corporation	[email protected] 303-825-3636
Brandie Ward	Data Collection Oversight	RMC Research Corporation	[email protected] 303-825-3636
Gale Hairston	SEA Study Liaison	Missouri Department of Elementary and Secondary Education (MO DESE)	[email protected]

References

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: Wiley.

Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-to-face surveys. International Journal of Market Research, 47(6), 615–639.

Ehlert, M., Podgursky, M., Harrington, J. Xiang, P., Parsons, E., Frazelle, S., & Taylor Knight, L. (2011). Teacher and Leader Preparation in Relationship to Student Achievement in the Kansas City Region. Kansas City, MO: Kansas City Area Education Research Consortium.

Koedel, C., Parsons, E., Podgursky, M., & Ehlert, M. (2012). Teacher preparation programs and teacher quality: Are there real differences across programs? Washington, DC: CALDER/American Institutes for Research.

Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web surveys. Journal of Official Statistics 22(2), 329–349.

Loosveldt, G & Sonck, N. (2008). An evaluation of the weighting procedures for an online access panel survey. Survey Research Methods 2(2): 93–105.

Miller L. E., & Smith, K. (1983). Handling nonresponse issues. Journal of Extension On-line, 21(5). Retrieved on Apr 15, 2013 from http://www.joe.org/joe/1983september/83-5-a7.pdf.

National Council for Accreditation of Teacher Education. (2010, November). Transforming teacher education through clinical practice: A national strategy to prepare effective teachers. Washington, DC: Author.

Office of Management and Budget. (2006). Questions and answers when designing surveys for information collections. Washington, DC: Office of Information and Regulatory Affairs.

Oh, H.L. and Scheuren, F.J. (1983). Weighting adjustment for unit nonresponse. In W.G. Madow, I. Olkin, & D.B. Rubin (Eds.), Incomplete data in sample surveys, Volume 2: Theory and Bibliography (pp. 143–184). New York: Academic Press.

Taylor, H. (2005). Does internet research “work”? Comparing online survey results with telephone surveys. International Journal of Market Research, 42(1), 51–63.

Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines, pp. 73-100. Washington, DC: National Academy Press.

Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge: Cambridge University Press.

Willis, G.B. (1999). Cognitive interviewing: A “how-to” guide. Presentation at the 1999 Meeting of the American Statistical Association.

1The NCATE State Alliance for Clinically-Based Teacher Preparation creates partnerships between states and national experts to pilot diverse approaches to implementation and to bring new models of clinical preparation to scale. This alliance works with organizations such as the American Association of Colleges of Teacher Education, the Association of Teacher Educators, and teacher unions to share information among entities working to transform teacher education (NCATE, 2010).

2 See http://dese.mo.gov/eq/cert/routes-to-certification.htm for additional information.

3 Available here: http://title2.ed.gov/Title2STRC/DataTools/2011/Missouri.xls

4 Timing is contingent on OMB approval.

5 In the case that email addresses are not available or generate “bounce-back” messages, the study team will communicate with respondents via U.S. Postal Service mail sent to their school address. MO DESE indicated that email address information is typically missing for up to 5% of first-year teachers and that physical address information is available for all teachers in the sample.

6 Cognitive interviews were chosen as the approach to inform survey design in favor of a traditional pilot test to increase efficiency in terms of time and cost. REL Central expects that each cognitive interview will last approximately 60 minutes.