4.7 OMB SUPPORTING STATEMENT B final

SUPPORTING STATEMENT B:

REQUEST FOR CLEARANCE OF INFORMATION COLLECTION FOR

“Assessing the Role of Noncognitive and School Environmental Factors in Students’ Transitions to High School in New Mexico”

February 2015

Submitted to: Submitted by:

U.S. Department of Education SEDL

Institute of Education Sciences 4700 Mueller Blvd.
555 New Jersey Ave. NW, Rm. 308 Austin, TX 78723

Washington, DC 20208 Phone: (800) 476-6861

4700 Mueller Blvd. Austin, TX 78723

800-476-6861

www.relsouthwest.org

This publication was prepared for the Institute of Education Sciences (IES) under contract ED-IES-12-C-00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

Contents

Introduction 3

Research Questions 3

Data to be Collected 4

1. Respondent Universe and Sampling Methods 5

2. Description of Procedures for the Collection of Information 6

Data Sources 6

3. Description of Procedures for Maximizing Response Rates 8

4. Description of Tests, Procedures, and Methods 9

5. Individuals Consulted on Statistical Aspects of the Design and Key Staff 10

Attachment B-1. Parent Consent Form 12

Attachment B-2. Student Survey 13

Introduction

The U.S. Department of Education (ED) requests clearance under the Office of Management and Budget (OMB) clearance for activities related to the Regional Educational Laboratory (REL) Program. ED, in consultation with SEDL, examine relationships between measures of students’ noncognitive factors and school environments and grade 9 outcomes in New Mexico.

Although graduation rates have been on the rise, New Mexico consistently achieves one of the lowest graduation rates in the United States. Only about 70 percent of students in the 2012-13 graduating cohort earned a high school diploma, with graduation rates varying significantly by race/ethnicity (New Mexico Public Education Department, 2014). While 77 percent of White students in this cohort graduated in four years, only 68 percent of Hispanic and 64 percent of Native American students did the same (New Mexico Public Education Department, 2014). This study will use data from a minimum of twenty schools in New Mexico to examine how noncognitive factors (e.g., growth mindset, learning strategies, and self-efficacy) and school environmental factors (e.g., perceptions of school safety, supportive teachers and counselors, usefulness of academic work) relate to three measures that Allensworth and Easton (2007) identified as mattering most for a successful transition (i.e., a successful freshman year—overall freshman year GPA, number of course failures in all courses during freshman year, and freshman-year absences.

OMB approval is being requested for the use of the recruitment materials, survey data collection and extant data collection in New Mexico high schools, and extant data collection in associated New Mexico schools and districts, as well as from the New Mexico Public Education Department (NM PED). The survey is designed to collect data on students’ perceptions of their noncognitive skills and school climates. The extant student data will provide information about student demographics and achievement.

The results of this study will contribute to the body of knowledge used to inform practitioners and policymakers in New Mexico about the role of noncognitive and school environmental factors in students’ transitions to high school. It may also provide additional avenues to explore in future research that uses more rigorous designs or that explores new areas suggested by study results. These results can also help schools or districts to determine where they might want to focus some of their efforts with regard to helping students to make successful transitions to high school.

Research Questions

This study will address the following research questions:

1. How do grade 9 students in New Mexico perform on average on measures of noncognitive factors?

   1. Do students’ scores on measures of noncognitive factors differ significantly by race/ethnicity?

2. How supportive do grade 9 students in New Mexico perceive their high school environments to be?

   1. Do students’ scores on measures of the supportiveness of their high school environments differ significantly by race/ethnicity?

3. Are there relationships between measures of grade 9 students’ noncognitive factor and the supportiveness of their high school climates and successful transitions to high school¹?

   1. Do students’ scores on measures of their noncognitive factors and supportiveness of their high school environments differ significantly by race/ethnicity?

Data to be Collected

Data collection for this project consists of surveys to be administered to grade 9 students in participating high schools in New Mexico and extant data file collection from NM PED and participating schools and districts. Specifically, in this OMB clearance package, ED is requesting clearance for the following data collection approach and approval of recruitment materials:

• Recruitment materials for participating districts and schools

• A paper-and-pencil survey of grade 9 students in participating high schools in New Mexico

• Extant data collection consisting of:

  • Student records data (specifically, student ID and grade 9 GPA – information not collected by NM PED) to be obtained from the district offices associated with the high schools participating in the study

  • Classroom rosters containing student names and IDs from the high schools participating in the study

  • Data on student achievement and background characteristics from NM PED

Each research question will be answered using a combination of survey and extant data. ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to examine relationships between measures of students’ noncognitive factors and school environments and grade 9 outcomes in New Mexico.

The remainder of Part B addresses the following: respondent universe and sampling methods; description of procedures for maximizing response rates; description of tests, procedures and methods; and contact information for statistical consultants and key staff.

1. Respondent Universe and Sampling Methods

The student survey will be administered in a minimum of 20-25 public high schools, within five or six districts. ED’s contractor will partner with the New Mexico Public Education Department (PED) for recruitment activities. ED’s contractor plans to recruit the four largest school districts in New Mexico—Albuquerque, Santa Fe, Gallup, and Las Cruces – with the goal of two of these districts agreeing to participate. Because New Mexico school districts tend to be small, it is expected the remaining participating school districts will be relatively small—approximately one to two high schools. The goal for recruitment outside of the largest school districts is to include at least one predominantly Hispanic district; at least one predominantly Native American district; and at least one district with a mixed population of Hispanic, Native American, and White students in the project. The rationale for selecting these types of districts is to ensure that we have an adequate representation of American Indian and Hispanic students—the groups of students for which alliance members are interested in closing achievement gaps. It should be noted that while a goal of the study is to include a diverse set of students and schools, the final sample is not likely to be representative at the state level. The purpose of the sampling plan is to ensure an adequate sample size for the study analyses, particularly with regard to key subgroups of interest, including Native American and Hispanic students.

Starting in summer 2015, ED’s contractor will begin recruiting high schools to participate in the study. For recruiting purposes, with the exception of the four largest, school districts statewide will be stratified based on racial/ethnic composition—predominantly Hispanic, predominantly Native American, mixed—and recruited in groups. Within each strata, school districts will be stratified by size—from largest to smallest—and the five largest school districts in each strata will be targeted for recruiting. Additional rounds of recruitment activities will occur until at least one school district from each strata has agreed to participate. In all, there will be five or at most six districts that participate in this project.

Recruitment will be conducted via email and telephone. Once OMB clearance has been granted, ED’s contractor will send an email message to district superintendents introducing the study and inviting them to participate (see attachment A-4). The emails will outline the goals of the study, the content of the survey, and schools’ in the district’s role in survey dissemination and collection. A follow-up email will be sent to all non-responding superintendents one week after the initial email (see attachment A-5). One week after distribution of the follow-up email, another identical follow-up email will be sent to superintendents. If enough districts have not agreed to participate in the study, REL Southwest will begin conducting telephone recruiting calls (see attachment A-6). The REL Southwest researcher will ensure that district superintendents understand the nature of the study and the responsibilities of participating schools. Once superintendents have agreed to participate in the study, they will be asked to sign a memorandum of understanding/consent form indicating that their district agrees to participate in the study. The goal is to include 5 or 6 school districts for a total of 20-25 high schools².

After district superintendents have agreed to participate in the study. REL Southwest researchers will conduct telephone calls with principals from all of the high schools in each of the participating districts. The telephone calls will introduce principals to the study and familiarize them with the survey. At this time, schools will be invited to participate in the study (see attachment A-7). REL Southwest expects the process of gaining principal consent may require two phone calls. Principals will then designate a staff member to work with REL Southwest to further coordinate receipt of survey materials and dates for survey administration.

Although REL Southwest will stratify school districts by geographic location and race/ethnicity in order to ensure that we have adequate representation of the target racial/ethnic groups, the sample for this study will be nonprobabilistic. As such, the results of this study will not be generalizable beyond the sample of schools and students included in the study.

All entering grade 9 students in study schools who are eligible to complete the unmodified, English versions of the Partnership for the Assessment of Readiness of College and Careers (PARCC), which have been adopted as New Mexico’s state standards-based assessments, will be asked to complete a survey during their English class.

2. Description of Procedures for the Collection of Information

The study will utilize data from two sources: (1) a noncognitive and school environmental factors survey, and (2) student- and school-level datasets obtained from participating schools, participating district databases and NM PED. The noncognitive and school environmental factors survey will be administered to a sample of districts in New Mexico. The sample will consist of 20-25 high schools. Data will be collected from grade 9 students. REL Southwest will oversee the survey administration and data collection for this study.

Data Sources

ED’s contractor will collect both primary and secondary data for this study. The primary data will consist of a student survey that will be administered to students who will enter grade 9 in a New Mexico public high school in one of the participating districts during the 2015–16 school year. The student survey will be administered to grade 9 students early in October 2015 during their English class. October was selected for survey administration because by that time, students will have been enrolled in grade 9 for a little less than two months. This timing will allow us to determine students’ incoming noncognitive skills while also providing enough time for students to get a feel for their new peers, teachers, and school.

The secondary data consists of classroom rosters including student names and state IDs from participating schools, grade 9 GPAs collected from extant school district data, and student academic achievement and demographic data collected from extant NM PED data.³ We have a signed MOU from NM PED allowing us access to student-level data on demographic characteristics, grade 9 course failure, grade 9 attendance, and grade 8 state test scores. Table 1 provides a summary table of the data collection timeline.

Table 1. Data Collection Timeline

Data Collection	Purpose	Source	Sept/Oct 2015	June 2016
Student names and State ID numbers for grade 9 students	Distribute survey forms Reliably link survey data with extant student data files	School Data Manager	X
Student survey administration	Collect noncognitive factors and school environmental factors data	Students	X
Student GPAs	Outcome data for study analyses	School District Data Manager		X
Extant school and student data files from the state	Student achievement data Student background characteristics	State data manager		X

Student Survey

The student survey, entitled Beginning High School Survey, to be used in this study is divided into eight sections:

Your School

Your Schoolwork

Your 9^th Grade School Year

Your 8^th Grade School Year

Your Teachers

Your Peers

Your Parents

Your Future

The survey asks students to respond to a series of survey scale items measuring important noncognitive factors, such as grit/perseverance, growth mindset, motivation, and self-regulation, as well as school environmental characteristics, such as positive relationships with teachers and other school staff, school safety and order, teacher expectations for student success, and perceived challenge and relevance of coursework. The survey primarily contains validated survey scales shown to have good psychometric properties or scales closely based off of validated scales. A copy of the survey is included in appendix A-3.

3. Description of Procedures for Maximizing Response Rates

ED is committed to obtaining complete data for this evaluation. A large share of this descriptive study relies heavily on administrative data. ED’s contractor anticipates a 95-percent response rate from districts on all student variables in the administrative data. That is, ED’s contractor assumes that 5 percent or less of the data will be missing on key variables; considering that this data is used by the state to inform resource allocation, and to apply for federal resources it is to the state’s advantage to ensure its accuracy. A key to achieving complete administrative data is tracking the data components from participating districts with email and telephone contact to the appropriate parties to resolve issues of missing or delayed data files. ED’s contractor has considerable experience working with schools and districts to obtain administrative data. All administrative data files will be reviewed for consistency and completeness. If a data file has too many missing values or if an instrument in the implementation study has too few items completed to be counted as a response, REL Southwest will seek to obtain more complete responses by e-mail or phone.

Based on its prior experience with administering surveys to students in a variety of schools and districts, ED’s contractor expects the response rate for the survey to be at least 85 percent. A number of steps will be taken to maximize response rates. First, REL Southwest researchers will work with school administrative staff to coordinate dates for survey administration and receipt of survey materials. They will also work with schools to coordinate inclusion of information about the study on the school website and the eblast that will be sent to parents of eligible children. Additionally, to reduce burden on schools, REL Southwest researchers will be responsible for most aspects of data collection. REL Southwest researchers will distribute surveys (with teacher assistance) read survey instructions, and collect completed surveys. REL Southwest researchers will also be responsible for mailing survey materials to schools and returning completed surveys to REL Southwest. The survey materials will be mailed to schools for ease of distribution, since the REL researchers are not based in New Mexico. Survey materials will be waiting at the schools when REL researchers arrive.

Survey respondents will also receive assurances of confidentiality. Survey respondents will be informed that every measure will be taken to protect the confidentiality of the data collected and the data will be used for the purpose of the study. No one at the school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools⁴.

Teachers and school administrative staff will be provided with small incentives for assisting with preparation for survey administration. Both classroom teachers, who assist with survey distribution, and school administrative staff, who assist with collection classroom rosters containing student names and state IDs, will receive $25 Amazon.com gift cards. The gift cards will be distributed by REL Southwest researchers on the date of survey administration for each individual school.

In exchange for allowing students to complete the survey, REL Southwest will produce a report—Individual School Survey Results—for each of the high schools participating in the study. These reports are designed to provide participating schools with feedback on the status of students in their schools. They are also intended to serve as an incentive to encourage districts/schools to participate in the study, as well as to provide an incentive for schools to achieve higher response rates. The Individual School Survey Results will contain school-specific results from the non-cognitive and school environmental factors survey, including summaries of students’ responses to each of the survey items (presented as counts and percentages), as well as mean scores and standard deviations on each of the survey scales.

Finally, ED’s contractor will consider the most recent statistical literature and work with IES to determine the most appropriate method for handling missing data. They will use appropriate analytic methods to account for missing data, and consider options such as complete case analyses with regression adjustment, maximum likelihood methods, non-response weights, or fitting the models specified above in a Bayesian hierarchical framework. Upon obtaining the data, ED’s contractor will examine the extent of missing data overall and by treatment group.

4. Description of Tests, Procedures, and Methods

REL Southwest researchers worked with members of the New Mexico Achievement Gap Alliance, which includes practitioners and policy makers from several of the New Mexico Regional Education Cooperatives, the New Mexico Legislative Education Study Committee, three New Mexico universities, and the New Mexico Public Education Department, to create the student survey. To do so, REL Southwest researchers (1) conducted a webinar to provide alliance members with an overview of noncognitive factors and their relationships to high school transitions, (2) located and assembled potential survey scales from extant surveys, (3) reviewed potential survey scales with six alliance members at the survey workshop, and (4) created several versions of the survey for alliance member review and feedback.

To ensure that the survey contained relevant questions and used terminology familiar to students, REL Southwest researchers conducted semi-structured cognitive interviews with eight students in New Mexico—six in Las Cruces and two in the Acoma Pueblo as part of the survey development process. Five of the students were female, three were male, two were Hispanic, two were Native American, and four were White. Seven of the students had just completed grade 8 and will be freshmen in high school during the 2014/15 school year; one student had just finished grade 9. The purpose of the cognitive interviews was to ensure that the survey scales and items included on the survey are understandable and relevant to the survey target population.

Overall, students found the survey fairly easy to complete. Students indicated that the instructions were clear and they did not seem to have any trouble with the flow from one section to the other. No student had trouble completing the survey in a reasonable amount of time, indicating that the length is appropriate. However, some students did struggle with a few of the items. REL Southwest researchers flagged these items for discussion and edits. Edits were made to the survey based on this feedback. The final version of the survey, to be used in this study, is contained in appendix A-3.

The study recruitment and survey administration materials have been adapted from materials used in previous studies conducted by SEDL. Only minor modificiations have been made to these materials.

5. Individuals Consulted on Statistical Aspects of the Design and Key Staff

The following individuals were consulted on the statistical, data collection, and analytic aspects of this study through REL Southwest’s Technical Working Group (TWG):

Dan Goldhaber, Ph.D.

Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research)

Vice President, American Institutes for Research (AIR)

Director, Center for Education Data & Research (CEDR), University of Washington Bothell

Co-Editor, Education Finance and Policy

3876 Bridge Way N,Suite 201

Seattle, WA 98103

Ph: 206-547-1562

Fax: 206-547-1641

E-mail: [email protected]

Geoffrey Borman, Ph.D.

Professor of Education, University of Wisconsin—Madison

Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program

Senior Researcher, Consortium for Policy Research in Education.

348 Education Building
1000 Bascom Mall
Madison, WI 53706-1326

Ph: 608-263-3688

Fax: 608-265-3135

E-mail: [email protected]

Johannes M. (Hans) Bos, Ph.D.

Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program

American Institutes for Research

2800 Campus Drive, Suite 200
San Mateo, CA 94403

Ph: 650-843-8100

Fax: 650-843-8200

E-mail: [email protected]

W. Steven Barnett, Ph.D.

Board of Governors Professor and Director of the National Institute for Early Education Research

Rutgers University

73 Easton Avenue
New Brunswick, NJ 08901

Ph: 848-932-4350 x23132

Fax: 732-932-4360

E-mail: [email protected]

Attachment B-2. Student Survey

See attachment

1 In this study, a successful transition to high school is measured by students’ grade 9 GPAs, number of absences, and number of course failures.

2 All estimates contained in Supporting Statement A assume recruitment of no more than 6 districts and 25 high schools.

3 NM PED does not collect data on student GPA, hence why ED’s contractor will request this data separately from school districts at the end of the 2015-2016 school year. Districts typically do not maintain classroom roster information, hence why this information will be collected from participating schools; if participatingdistricts do have this information, we will request it from them to reduce the burden on participating schools.

4 District database managers may have access to student responses when merging survey responses with administrative data. However, database managers will not have access to survey codebooks, and all student responses will be coded numerically. No open-ended questions appear on the student survey. Moreover, item order will be changed so that items will appear in the dataset in a different order than on the survey. As such, dataset data will not be aligned with the survey. All variables in the dataset will be coded in a way that will not allow database managers to determine the proper item order.

2