RELSW 5.1.7.5 EAF OMB Supporting Statement Part A

The U.S. Department of Education (ED), through its Institute of Education Sciences (IES), requests clearance for the recruitment materials and data collection protocols under the Office of Management and Budget (OMB) clearance agreement (OMB No. 1850-NEW) for activities related to the Regional Educational Laboratory (REL) Southwest program under contract 91990023C0003.

By 2030, the Texas Higher Education Coordinating Board (THECB) expects that 60 percent or more of all new jobs in Texas will require some postsecondary education. However, in 2019, less than half of the Texas population aged 25–34 years (44.3 percent) had some type of postsecondary credential (THECB, 2021). To close this gap and support districts in meeting the state statute that requires schools to fully develop each student’s academic, career, personal, and social abilities (Texas Education Code §33.006), the Counseling, Advising, and Student Supports team (under the Division of College, Career, and Military Preparation) at the Texas Education Agency (TEA) established the Effective Advising Framework (EAF) in 2020/21. This framework expands access to effective college and career advising by streamlining and modernizing advising offerings and services for secondary and postsecondary students. The initiative aims to support students in making informed decisions about postsecondary education and careers and to offer professional development to educators and guidance counselors on advising services.

The REL Southwest developed a partnership with TEA’s Division of College, Career, and Military Preparation to inform the development of resources that meet identified needs and implement additional changes to improve the EAF. Therefore, the REL Southwest is requesting clearance to conduct a study that will examine the implementation of the EAF across the three cohorts of regional education service centers (ESCs) and school districts that are currently participating in the EAF pilot grant program. The pilot program supports the development and implementation of an individual student planning system within the context of a comprehensive school counseling program. Participation in the pilot program was determined through three grant application processes from TEA to the ESCs. The ESCs selected to receive grants were required to partner with two to four school districts within their geographic region. Because it is expected that districts are implementing the EAF in a variety of ways, this study will provide an in-depth look at the variation in implementation across districts, including an analysis of the factors that support or hinder implementation. The study will serve as an evaluation of progress toward the partnership’s medium-term goals that districts will use materials, tools, and resources based on effective or promising practices to conduct individual student planning; require Individual Career and Academic Plans (ICAP) for all students by grade 8; and increase districts’ ability to determine whether students are on track for postsecondary readiness by applying grade-level indicators of progress.

Description of the Effective Advising Framework

The EAF is designed to enhance both the quality and quantity of advising support that students experience throughout their time in K–12. The EAF outlines five “levers” of effective advising practices on which school districts are encouraged to base their ICAPs to improve the way they prepare students for success after high school. Lever 1 focuses on building strong program leadership and planning. Lever 2 aims to provide school counselors and advisors with the resources they need to be effective. Lever 3 is about promoting a culture of advising within schools that engages more personnel than just counseling staff in coordinated efforts to support students in meeting grade-level expectations. Lever 4 focuses on establishing effective external partnerships to fill what would otherwise be gaps in the capacity of schools to promote postsecondary readiness. Finally, Lever 5 concerns the selection and use of high-quality advising materials and assessments to enhance student preparation for life outside of K–12 schooling.

The purpose of the EAF is to systematically support districts and schools in effective advising efforts, with the ultimate goal of increasing college, career, and military readiness (CCMR) rates. TEA uses the acronym “CCMR” to make explicit the inclusion of military readiness along with college and career readiness. Policymakers and researchers more frequently use “college and career readiness” to describe students’ postsecondary preparation. Definitions of college and career readiness have been used by educators and policymakers when developing new initiatives, programs, or materials and can be used to monitor student outcomes (Byrd & Macdonald, 2005; Mishkind, 2014; Tierney & Sablan, 2014). Definitions of college and career readiness, circulated by researchers and policymakers, vary in length but emphasize a consistent set of skills, including academic content knowledge, critical thinking, communication and community-building skills, and self-management skills (Camara, 2013; Conely, 2008; Mishkind, 2014; Sampson et al., 2011). College readiness and career readiness are often conceptualized as the same thing, but there are meaningful differences between these two approaches to postsecondary readiness (Pérusse et al., 2017).

A1. Circumstances necessitating the data collection

This study will examine the implementation of the EAF across pilot districts and schools and will provide important insight to inform efforts to scale up implementation of EAF statewide. We anticipate that the leadership at TEA will be able to make use of the findings of this study to guide future EAF implementation and support ESCs and districts in implementation. In addition, we expect that the study results will inform the work of staff members at the ESC and district levels who are leading efforts to transform effective advising practices in their respective regions and districts. Similarly, the results of this study also may inform the efforts of school-level staff engaged in student advising.

The study is authorized through provisions in the Education Sciences Reform Act (ESRA) of 2002. Specifically, ESRA Part D, Section 174(4), describes the role of RELs and their mission and function. The study is motivated by one aspect of that role:

(4) in the event such quality applied research does not exist as determined by the regional educational laboratory or the Department, carrying out applied research projects that are designed to serve the particular educational needs (in prekindergarten through grade 16) of the region in which the regional educational laboratory is located, that reflect findings from scientifically valid research, and that result in user-friendly, replicable school-based classroom applications geared toward promoting increased student achievement, including using applied research to assist in solving site-specific problems and assisting in development activities (including high-quality and on-going professional development and effective parental involvement strategies). (ESRA, Part D, Section 171, f.4)

A2. Purpose and use of these data

The data collected in the study will be used to address the following research questions (RQs):

How are districts and schools using the EAF to enact district implementation plans?
1. What approaches and strategies are districts using to apply the EAF? How do approaches and strategies differ by district size and locale?
  1. What roles are district and school staff playing in implementing the EAF?
  2. What materials and resources are districts and schools using, and how?
  3. How do current approaches differ from past advising efforts (pre-EAF)?
2. How and to what extent have districts and schools planned and implemented an individual planning system as outlined in the EAF?
3. How are districts using the EAF to align advising practices across grade levels?
4. To what extent are districts aware of and have goals to increase their CCMR rates? What efforts are districts making to increase their CCMR rates?
What factors inhibit or facilitate successful implementation of the EAF at the district and school levels?
1. Which structures and practices by district implementation steering committees support successful implementation of the EAF?
2. To what extent are school staff aware of district EAF implementation plans? To what extent do school staff understand and effectively carry out the implementation plans?
3. To what extent are student-facing school staff trained on the EAF?
What are participating districts’ CCMR rates, and how have these rates changed over time?
1. How do CCMR rates differ by student gender, race/ethnicity, socioeconomic status, and English proficiency status as well as school size and locale?

To analyze the data, the research team will use a descriptive mixed methods approach, employing a concurrent triangulation design (Creswell et al., 2003) to provide an in-depth characterization of districts in terms of their implementation of the EAF and the preparation of their students for postsecondary success. Quantitative and qualitative data will be collected concurrently, analyzed separately, and then synthesized by comparing the results of each type of analysis and developing an overall set of findings that answer the RQs.

Data collection activities for which clearance is requested as part of this package

Overview

With approval from OMB, the REL Southwest will collect quantitative and qualitative data in the 2024/25 school year from the 12 ESCs and 34 of the 52 districts that received TEA’s EAF implementation grants between 2021/22 and 2023/24. Due to the time and effort that will be required to secure district approval to conduct the study and the likelihood that some districts will choose not to participate in the study, we set 34 total districts as a more reasonable target for our study sample than the total 52 districts involved in the pilot. We will use the term “cohort” to identify school districts based on the number of years each district has been implementing the EAF. Currently, a total of three cohorts of districts are participating in the EAF pilot program, including districts in their first, second, or third year of implementation (table 1). It is important to note that some EAF coaches will be providing support for districts in more than one cohort.

Table 1. Participating Effective Advising Framework districts, disaggregated by cohort

Cohort	Number of districts	First implementation year	Number of implementation years at start of data collection (2024/25)
Cohort 1	11	2021/22	3
Cohort 2	18	2022/23	2
Cohort 3	23	2023/24	1

Recruitment of districts

The REL Southwest researchers will begin the data collection process by contacting the 12 EAF coaches and inviting them to participate. The researchers will confirm with each coach which districts and district project leads they support. The next step will be to solicit participation in the project from all districts that received an implementation or planning grant. To do this, the research team will contact each district EAF project lead via personalized emails that state the purpose of this research, that it is supported by TEA and IES, and that participation is voluntary. We will then follow up as needed until we have recruited a representative sample that includes 34 of the 52 districts participating in the EAF pilot. The REL Southwest team will use TEA’s district type categorization to maximize the regional variation of districts represented in our sample. District type, as defined by TEA, is an eight-category variable that ranges from “major urban” and “major suburban” to “non-metropolitan: fast growing” and “rural.” To the extent possible, we will endeavor to recruit districts from all eight categories. This amounts to a purposive sampling of districts aimed at maximizing variation in district type. We will complete any research application procedures required by districts before recruiting participants or collecting data. Once these requirements have been met, we will reach out to the respective district EAF project leads and inform them of the timeline and process of data collection.

Extant data on school staff

After securing district commitment to participate and research approval (if required), we will recruit participants for individual data collection efforts. We will rely on cooperation from each participating district to send us lists of names and contact information for school principals, assistant/vice principals, and counselors, which we will use to send invitations to take the school staff survey.

Surveys

To answer RQs 1 and 2, the REL Southwest will administer online surveys in spring 2025 to all EAF coaches, district project leads, and school staff (school administrators and counselors). Because the surveys will be designed in such a way that they can be completed by individuals regardless of where their district is in the implementation process, all three cohorts will receive the same surveys at the same time. To collect survey data, the study team will electronically deliver a survey form to potential participants via Qualtrics. The content will remain largely constant across the three surveys; however, we will alter the text of each question so that the wording is appropriate for each respondent group (EAF coaches, district project leads, and school staff). We expect that this survey distribution will involve eliciting survey responses from 12 EAF coaches, 34 district project leads, and 1,397 school staff.

The survey items can be grouped into four conceptual categories according to their focus. Descriptions of each set of items are provided here. The RQs addressed by each set of items are noted in parentheses.

District EAF planning and implementation. A set of questions focused on understanding what districts and schools are doing and planning to do in terms of academic advising as it relates to the EAF, including the roles that survey participants play in implementation efforts.
School staff understanding and efforts. A set of questions that will gauge the extent to which school staff are aware of, understand, and carry out the district’s plans (RQ2b) and whether and to what extent they have received training regarding the EAF (RQ2c).
Factors that inhibit or facilitate successful implementation of the EAF (RQ2a). A set of questions in which participants will be asked to identify and reflect on factors that either limit or enhance EAF implementation in their district or school.

Draft survey communication materials are in attachment A. Draft survey questionnaires are in attachments B–D.

One-on-one interviews

After the survey administration is complete, we will recruit a subset of EAF coaches and district project leads to participate in virtual, one-on-one interviews with members of our research team. We will select eight EAF coaches to participate in interviews. Our team will then select 16 district project leads based on district characteristics, using a purposive sampling method. Specifically, we will begin by selecting districts from each TEA district type category. With the goal of selecting a sample that comprises a wide array of districts, we will identify districts that represent a range in the following categories: EAF cohort, geographic region, percentage of students identified as economically disadvantaged, percentage identified as emergent bilingual/English learner students, percentage of students who are Black, percentage of students who are Hispanic or Latino/a, and district locale type (rural, urban, or suburban). This diversity is important for the current study because it will help ensure that findings are not unduly influenced by certain types of districts. If an ESC or district declines to participate in qualitative data collection, we will select an ESC or district that is as similar as possible to replace it. We will repeat this process until we have a target sample of data collection participants. EAF coach interviews will take place in September 2025, and district project lead interviews will subsequently be conducted in October and November 2025.

Semistructured interview protocols will be organized by topical domains of interest for each participant group (Carspecken, 1996; Roulston, 2010). Domain topics for interviews with EAF coaches and district leads are described here, with the RQs addressed by each domain noted in parentheses. Draft communication materials for the interviews are in attachment E. Copies of the interview protocols are in attachments F and G.

Domains of interest for interviews with EAF coaches

Respondents’ CCMR background and role related to the EAF. EAF coaches will be asked to share their experiences with CCMR-related efforts (for example, student advising) prior to implementation of the EAF, their EAF training, and their current understanding of their role and responsibilities as EAF coaches (RQ1a).
Descriptions of district contexts. EAF coaches will be prompted to share their understanding of districts’ context, including factors such as size, locale, and political structure and their perceptions about how these contextual factors interact with the implementation of the EAF (RQ1d and RQ2a).
Accounts of EAF awareness and implementation. EAF coaches will be asked to explain how individual districts are approaching EAF implementation. Follow-up probes will include specific questions about materials, resources, and strategies used by districts to implement their EAF plans (RQ1a, RQ1b, RQ1c, and RQ1d).
Reflections on collaboration with district project leads. EAF coaches will reflect on their collaborative relationships with district project leads, including the EAF training process, the district EAF planning process, EAF implementation progress monitoring, and relational successes and/or challenges (RQ1b, RQ1d, RQ2a, and RQ2b).
Perceptions of district implementation efficacy. EAF coaches will report their perceptions of districts’ EAF implementation progress, their understanding of what supported or impeded EAF implementation, and districts’ progress toward their CCMR goals (RQ1b, RQ1c, RQ2a, RQ2b, and RQ2c).

Domains of interest for interviews with district project leads

Respondents’ CCMR background and role related to the EAF. District project leads will be asked to share their experiences with CCMR efforts prior to the EAF, their EAF training, and their current understanding of their role and responsibilities as EAF district project leads (RQ1a).
Descriptions of district context. District project leads will describe relevant district characteristics related to their reasons for participating in the EAF project, community beliefs about CCMR, past approaches to CCMR, and current CCMR goals (RQ1d and RQ2a).
Accounts of EAF implementation. District project leads will be asked to recount how their district implemented the EAF and ICAPs. Follow-up probes will include specific questions about developing a plan, communicating with district stakeholders about the plan, systemic implementation across grade levels, EAF resources and materials used, and implementation of specific EAF elements (RQ1a, RQ1b, RQ1c, and RQ1d).
Reflections on collaboration during the EAF pilot program. District project leads will have an opportunity to discuss their collaborative relationship with EAF coaches and schools (RQ1a, RQ1c, RQ2a, and RQ2b).
Perceptions of district implementation efficacy. District project leads will report their perceptions of districts’ EAF implementation progress, their understandings of what supported or impeded EAF implementation, and districts’ progress toward their CCMR goals (RQ1b, RQ1c, RQ2a, RQ2b, and RQ2c).

Focus group interviews

Approximately 64 school staff members will be recruited to participate in the focus groups, with the goal of including about four members representing at least two schools from each of the 16 districts participating in the one-on-one interviews. We will work in consultation with the district project leads to select school staff members who oversee advising efforts at the schools (such as counselors and administrators) or work directly with students (such as teachers). Focus groups with school staff members will take place in January and February 2026.

Semistructured focus group interview protocols are organized by topical domains of interest (Carspecken, 1996; Hall, 2020) and described here, with the RQs addressed by each domain noted in parentheses. A copy of the focus group interview protocol is provided in attachment H.

Domains of interest for school staff focus groups

Respondents’ CCMR background and role related to the EAF. School staff members will be asked to share their experiences with CCMR efforts prior to the EAF, their EAF training, and how they support students’ CCMR (RQ1a).
Descriptions of district and school context. School staff members will describe relevant district and school characteristics related to CCMR climate, student readiness, and CCMR programs (RQ1d and RQ2a).
Accounts of EAF awareness and implementation. School staff members will be asked to explain how familiar they are with the EAF, what they understand about the district’s implementation plan as well as the communication and training that went into developing their understanding (RQ2a, RQ2b, and RQ2C).
Reflections on collaboration and implementation. School staff members will be asked to describe how implementation of the EAF and ICAPs is going in their context. Follow-up probes will include specific questions about systems, activities, and practices used to facilitate implementation as well as roles and responsibilities regarding CCMR, with different prompts for counselors and noncounseling staff (RQ1a, RQ1b, RQ1c, RQ2a, RQ2b, and RQ2c).
Perceptions of district implementation efficacy. School staff members will be asked to reflect on how the EAF implementation process is going in their context. Specific follow-up probes will prompt participants to name successes and identify challenges experienced while implementing their districts’ EAF plan (RQ1a, RQ1b, RQ1c, RQ2a, RQ2b, and RQ2c).

Other data collection activities

Texas Education Research Center extant administrative data

To address RQ3, we will obtain access to secondary administrative data from the Texas ERC, a clearinghouse of public education data collected by the state. These data are broadly available to researchers who complete an application process. There is a two-step approval process to gain access to ERC data: First, the study team will submit to the ERC a 10- to 15-page proposal describing the research project and detailing the specific ERC data files to which we will need access. The ERC will screen the proposal and may ask for revisions. The study team expects ERC approval given the TEA’s expressed need for the study. Once approved, the ERC advisory board will schedule a set time during its regular meetings during which researchers will respond to questions from board members about their proposed research. The board will then signal approval, approval with adjustments, the need to resubmit, or denial of the proposal. If approved, access to the ERC data will be set up within a few weeks. This entire process will take place during spring 2024. At this time, we are finalizing the draft of our ERC proposal, and we intend to submit it in May.

Once the study team is granted access, the data can be analyzed via an online server from anywhere in the country. All variables of interest from the ERC are school- and district-level aggregates of student characteristics and student outcomes. This will require us to calculate these aggregate rates from student-level data. Our key outcome variables are CCMR rates. In addition to examining overall trends in CCMR rates, we will also assess whether rates differ across student subpopulations. To do this, we will compute CCMR rates for categories of students based on race/ethnicity, gender, economic status, and English proficiency and test whether rates vary significantly depending on these student characteristics. This information may be useful to the state in shaping policy around CCMR moving forward. For example, if English language learners (ELLs) were found to have lower CCMR rates than their peers, that information might direct TEA to examine how the ELL program interacts with CCMR efforts in schools or to consider whether the EAF should include explicit recommendations for how college- and career-ready materials should be produced in other languages in addition to English. We will compile year-by-year data for each of these variables and construct a longitudinal dataset from the 2015/16 school year to the 2023/24 school year so that we can investigate trends over time, before and after the impact of the COVID-19 pandemic.

A3. Use of technology to reduce burden

REL Southwest will use Qualtrics to collect a portion of the data used in this study. Documentation of Qualtrics’ accessibility can be found at this link: https://www.qualtrics.com/support/survey-platform/survey-module/survey-tools/check-survey-accessibility/. There are some components that are not accessible and those are identified. REL Southwest will use no Qualtrics question types that are not 508 compliant. Additionally, individual surveys that are developed on Qualtrics also go through an automated Section 508 accessibility review before they are released for use to ensure accessibility. Reports are provided to the system owner to correct if findings are identified. IES will assure all findings are resolved before release.

The REL Southwest will be responsible for the security of electronic data files downloaded from Qualtrics directly onto our secure server on Microsoft’s Azure system. The Qualtrics system allows us to contact school staff directly via email and prevents staff from having to complete any paper forms in order to participate in data collection. To reduce the burden on respondents, staff will be able to respond to the survey from a variety of devices, including computers, tablets, and smartphones.

Interviews and focus groups will be conducted via Microsoft Teams. If interviewees consent to being recorded for transcription, the interviewer will use the Teams recording feature as the recording and transcription tool. The REL Southwest team will ensure that all interview recordings and transcription files are secure. Documentation outlining the security features of Microsoft Teams are available here: https://learn.microsoft.com/en-us/microsoftteams/security-compliance-overview. The resulting automated transcriptions will then be reviewed and edited for accuracy prior to using them as the primary text for analysis. If the interviewee declines to be audio-recorded but agrees to participate in the data collection process, then a notetaker will be called in to take detailed, transcript-style notes in digital format of the interview protocol. These typed notes will then be saved and analyzed with the other interview transcripts.

The research team will access the ERC data remotely while complying with all ERC data security directives. All data and data summaries exported from the ERC servers will first be subjected to a review process by ERC staff to ensure compliance with Texas’s interpretation of Family Educational Rights and Privacy Act guidelines for the protection of personally identifiable information (PII). Prior to accessing ERC data. all research team members participating in such access will complete a data privacy training course as required by the ERC.

The REL Southwest will only use question types that are Section 508 compliant. IES will confirm that all electronic data collection materials being used in the study are Section 508 compliant before they are administered to participants.

A4. Efforts to avoid duplication

In an effort to avoid duplication of effort, this study will use extant administrative records collected from partnering school districts, where possible, to provide background information on interview and survey participants. This will allow us to avoid burdening interviewees and survey respondents with questions about their professional roles when this information is available in the data that school districts provide us. We are unaware of any other efforts to collect this information. TEA is not planning to collect any of this information from schools or districts.

A5. Methods to minimize burden on small entities

We aim to acquire administrative data on school staff from each school district that consents to participate in this data collection effort. Many of these districts are small, with few administrative resources. In order to minimize the burdens they face, we will prepare template spreadsheets that districts may use to make it as easy as possible for administrators to securely send us the data required for us to engage in data collection. For recruitment of interview and focus group participants, we will work to minimize the burden on smaller districts by directly providing them individual guidance as needed.

A6. Consequences of not collecting data

The Education Sciences Reform Act of 2002 states that the central mission and primary function of the RELs is to support applied research and provide technical assistance to state and local education agencies within their respective REL regions (ESRA, Part D, section 174[f]). If the proposed data were not collected, the REL Southwest would not be fulfilling its central mission to serve states in the region and provide support for evidence-based research. The systematic collection and analysis of the data described earlier is required to accomplish the goals of the research project approved by IES. Participation in all data collection activities is voluntary. This is a one-time study (not recurring), and therefore periodicity is not addressed.

A7. Special circumstances

There are no special circumstances involved with this data collection. Data collection will be conducted in a manner consistent with the guidelines in 5 CFR 1320.5.

A8. Federal Register announcement and consultations outside the agency

A 60-day Federal Register notice was published on April 23, 2024 (89 FR 30348). One comment unrelated to the collection was received. No changes were made in response to the comment. A 30-day notice was published on July 12, 2024 (89 FR 57138).

The following list describes the steps taken to ensure the availability of data, the soundness of the evaluation design for addressing evaluation questions, and the clarity of measures.

ED has consulted with the REL Southwest contractor that proposed this evaluation based on the needs identified in Texas.
The subject matter expert for this study is Kristina Zeiser, PhD, who is a principal researcher at AIR. Dr. Zeiser has more than 10 years of experience evaluating educational interventions and school models focused on college and career readiness and college success.
The REL Southwest contractor has consulted with former educators within its organization about surveys and interviews. These former educators reviewed the survey and interview questions for clarity of wording, the “loadedness” of questions (such as whether questions are written to elicit only one type of response), and the appropriateness of response options.
The REL Southwest contractor has also consulted with TEA staff who oversee the development and implementation of the EAF. These staff members reviewed the interview and survey questions and provided feedback on the accuracy and applicability of the items.
ED also has contracted with another organization to review the technical aspects of project plans and reports submitted by REL contractors. This external peer review contractor examines the rigor of the evaluation design, the analytic approach for determining the impact and fidelity of implementation, and the degree to which the findings address the evaluation questions and the conclusions are supported by the data. The study plan has been reviewed by the external review contractor and was approved by IES in January 2024.
AIR’s Institutional Review Board (IRB) approved this study on January 12, 2024.

A9. Payments or gifts

To encourage participation, we will offer incentives. For the EAF coach and district project lead surveys, we will inform potential participants that they will receive a $50 Amazon gift card if they complete the survey. For the school staff survey, we will also inform potential participants that 20 survey participants will be selected at random to receive a $50 Amazon gift card. Each one-on-one interview participant and focus group participant will receive a $50 Amazon gift card.

The remuneration of the gift cards was based on the rough hourly wage of school and district staff. To offer educators fair compensation for participating in REL toolkit development or original data collection for REL research, REL Southwest has consulted the Bureau of Labor Statistics (BLS) Occupational Outlook Handbook to identify the most current information (currently from 2021) about educator wages to calculate reasonable incentive amounts.

In the Occupational Outlook Handbook, there are several classroom teacher categories, including pre-school, kindergarten and elementary teachers, middle school teachers, high school teachers, and postsecondary teachers. Additionally, there is salary information for principals. Each category has its own associated median annual wage.

Across classroom educator (teacher) categories, the 2021 approximate annual wage is $61,500. Across principals, the approximate annual wage for 2021 is $98,420. To convert those wages to an hourly amount, each can be divided by 2080 hours. This yields:

Approximate teacher hourly rate: $30/hour
Approximate principal hourly rate: $47/hour

Per IES, RELs may not propose incentive rates lower than the rates indicated above for teachers (including pre-school teachers) and principals, and the minimum incentive, no matter how much time the research activity requires, is one full hour of wages.

A10. Assurances of confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. The REL Southwest will be following the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, which requires “all collection, maintenance, use, and wide dissemination of data by the Institute” to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act” (20 U.S.C. 1232g, 1232h). These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment.

In addition, for student information, ESRA states “The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act.”

Subsection (c) of section 183 referenced above requires the Director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.”

Subsection (d) of section 183 prohibits disclosure of individually identifiable information as well as making the publishing or communicating of individually identifiable information by employees or staff a felony.

The REL Southwest will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released publicly. Information from participating respondents will be presented at aggregate levels in reports. All surveys, focus groups, and interview responses will be kept strictly confidential and will only be used for the purpose of the study. No one at the school, district, or state will have access to data that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. No data on individual students will be collected. The study also will adhere to the requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony. No individually identifiable information will be maintained by the study team upon study completion.

The AIR Human Subjects Protection Committee, which serves as AIR’s Institutional Review Board (IRB00000436) approved this project (Study ID 8B336). AIR’s Federalwide Assurance for the Protection of Human Subjects (FWA00003952) serves as our assurance of compliance with federal regulations.

In addition to these safeguards, AIR, ED’s contractor for REL Southwest, routinely employs the following safeguards to assure confidentiality:

All employees engaged in data collection sign confidentiality agreements that emphasize the importance of confidentiality and specify employees’ obligations to maintain it.

PII is maintained on separate forms and files, which are linked only by sample identification numbers.
Access to a crosswalk file linking sample identification numbers to PII and contact information is limited to a small number of individuals who have a need to know this information.
Access to hard-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.
Access to electronic files is protected by secure usernames and passwords, which are available only to approved users. Access to identifying information for sample members is limited to those who have direct responsibility for providing and maintaining sample crosswalk and contact information. At the conclusion of the study, these data are destroyed.
The plan for maintaining confidentiality includes staff training on the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems.

The study team will make certain that all data are held in strict confidentiality, as just described, and that in no instance will responses or data be made available except in aggregate statistical form. All data collection instruments will include statements that participation is voluntary, answers will be kept confidential, and individual responses will not be attributed to individuals in any reporting.

The confidentiality and security of data will be maintained across all data collected through surveys and interviews. These datasets will be downloaded to the research team’s computers but will only ever be stored on secure servers—namely, Microsoft’s Azure system. The REL Southwest will be responsible for the security of electronic data files downloaded from Qualtrics directly onto our secure server on Microsoft’s Azure system. Although the identity of respondents will be available to the research team, we will create analytic datasets that anonymize this information and remove all identifiable data from the file. Survey data containing identifiable information will be replaced with a unique ID to protect the confidentiality of respondents.

A11. Justification for Sensitive Questions

There are no personally sensitive questions in this data collection. All individuals completing a survey or participating in interviews will only be asked to share information related to their work with the EAF and their professional interactions that relate to that program.

A12. Estimates of hours burden

There are three components for which the evaluation team has calculated hours of burden for this clearance package: recruitment activities, extant data provided by the state, and primary data collected from study participants. Table 2 shows the hourly burden overall and for all three components. The total burden associated with this study is 570.5 hours, with an annualized burden of 285 hours over two years. The recruitment burden is 20 hours, the extant data collection burden is 102 hours, and the survey and interview data collection burden is 448.5 hours. The annualized number of responses is 861.5 (for a total of 1,703 across the two years of data collection). We estimate that responding to recruitment emails will never take longer than 15 minutes, because the emails themselves are brief and only short yes/no responses are needed. Once districts agree to participate, we estimate that it will take them three hours to compile extant data that contain staff contact information, since this task could entail contacting another person at the district, locating the relevant files, and some light cleaning of the data in the files. Each of our surveys is designed to take 15 minutes, and no interviewee will be asked to spend more than one hour in an interview. Based on the estimated average hourly wages for participants, the total respondent cost associated with this one-time study is approximately $25,686, and the annualized respondent cost for the two years of data collection is $12,843.

Table 2. Estimated annual burden and respondent costs table

Information activity	Sample size	Respondent response rate	Number of respondents	Responses per respondent	Number of responses	Average burden hours per response	Total burden hours	Estimated respondent average hourly wage	Estimated monetary cost of burden
Recruitment
Initial recruitment of districts	52	65%	34	1	34	0.25	8.5	$47	$400
Effective Advising Framework (EAF) coach interview recruitment email	8	100%	8	1	8	0.08	0.64	$47	$30
District staff interview recruitment email	16	100%	16	1	16	0.08	1.28	$47	$60
District staff communication to determine focus group participants	16	100%	16	1	16	0.25	4	$47	$188
School staff focus group interview recruitment email	128	50%	64	1	64	0.08	5.12	$31	$159
Subtotal			138		138		20		$836
Extant Data Collection
Request school staff contact information from districts	34	100%	34	1	34	3	102	$47	$4,794
Subtotal			34		34		102		$4,794
Survey and Interview Data Collection
EAF coach survey	12	100%	12	1	12	0.25	3	$47	$141
District staff survey	34	100%	34	1	34	0.25	8.5	$47	$400
School staff survey	1,644	85%	1,397	1	1,397	0.25	349	$47	$16,403
EAF coach interviews	8	100%	8	1	8	1	8	$47	$376
District staff interviews	16	100%	16	1	16	1	16	$47	$752
School staff focus groups	128	50%	64	1	64	1	64	$31	$1,984
Subtotal			1,531		1,531		448.5		$20,056
Total			1,703		1,703		570.5		$25,686
Annualized totals			851.5		851.5		285		$12,843

A13. Estimates of cost burden to respondents

There are no additional respondent costs associated with the data collection for this study other than the hour burden accounted for in item A12. This is a one-time data collection effort and there are no plans for follow-up studies or other recurring data collections outside of what is being proposed in this package.

A14. Annualized cost to the federal government

The total cost to the federal government for work conducted over all five years is $997,517, and the estimated annualized cost to the federal government for each year of the study is $199,503.

Funding includes staff time for independent evaluators to recruit participants; implement the study; and collect, clean, and analyze data from the study. Also included are costs incurred by REL Southwest staff related to study preparation and submission of the study information to IES (including the establishment of the proposed research design, and reporting results).

A15. Reasons for program changes and adjustments

This is a new study.

A16. Plans for publication of results

Our work will not include any complex analytic techniques because the study is solely descriptive in nature. Results of quantitative analyses will include basic descriptive statistics. We will not create any Restricted Use File, Alt Data File, or Public Use File with the data we collect. The primary means for disseminating the results of this study will be a research report that will provide information on the study design and a detailed description of the findings. This format will best serve the needs of the project stakeholders and the broader research field because it can well accommodate the large quantity of information that typically accompanies descriptive research studies. We will meet with our TEA partners to discuss preliminary findings as we prepare for and begin drafting the report. These discussions will take place during our regular monthly partnership meetings or via additional meetings as needed. The report will be structured so that audiences can glean from it the level of detail that is most useful to them. In addition to the main report, the study team will develop a one-page snapshot that gives a high-level synopsis of the study and its findings as well as appendices that provide detailed information about the data sources and methods and that include copies of the data collection instruments. The research report will be published on the IES website, and the findings will be presented to TEA partners. Depending on the nature of the results and the needs of partners, additional products may be developed that succinctly communicate particular findings (for example, a research brief that outlines promising school district practices for supporting local implementation of the EAF). The REL Southwest will work with partners to plan for these potential additional dissemination activities.

The timeline for the activities in this project—including data collection, analyses, and reporting—is presented in Table 3.

Table 3. Project timeline

Activity/milestone	2023
Activity/milestone	01	02	03	04	05	06	07	08	09	10	11	12
Concept paper approved								x
Draft proposal								x	x	x
Submit proposal										x
Prepare OMB package										x	x	x

Activity/milestone

2024

Submit for provisional IRB review

Submit to OMB

Proposal approved

Draft ERC proposal

Submit ERC proposal

Defend ERC proposal at quarterly board meeting

ERC data cleaning and analysis

Activity/milestone

2025

ERC data cleaning and analysis

OMB approval received

Field-test data collection instruments

Recruit EAF coaches and districts

Administer EAF coach survey

Administer district project leads survey

Administer school staff survey

Analyze EAF coaches' survey results

Analyze district project leads survey results

Analyze school staff survey results

Conduct EAF coach interviews

Conduct district staff interviews

Analyze interview data

Activity/milestone

2026

Conduct school staff focus groups

Analyze interview and focus group data

Synthesize findings

Write final report

Submit final report

Activity/milestone	2027
Activity/milestone	01	02	03	04	05	06	07	08	09	10	11	12
Respond to final feedback	x	x	x	x	x	x	x
Publish final report								x

EAF = Effective Advising Framework; ERC = education service center; IRB = Institutional Review Board; OMB = Office of Management and Budget

A17. Approval not to display the expiration date for OMB approval

IES is not requesting a waiver for the display of the OMB approval number and expiration date. The surveys and notification letters will display the expiration date for OMB approval.

A18. Exception to the certification statement

This submission does not require an exception to the certificate for the Paperwork Reduction Act
(5 CFR 1320.9).

Attachments

Attachment A. Collected EAF survey communication materials

Attachment B. Survey of EAF coaches

Attachment C. Survey of district staff project leads

Attachment D. Survey of school staff

Attachment E. Collected EAF interview communication materials

Attachment F. Interview protocol for EAF coaches

Attachment G. Interview protocol for district staff project leads

Attachment H. School staff focus group protocol

References

Byrd, K. L., & Macdonald, G. (2005). Defining college readiness from the inside out: First-generation college student perspectives. Community College Review, 33(1), 22–37. https://eric.ed.gov/?id=EJ751688

Camara, W. (2013). Defining and measuring college and career readiness: A validation framework. Educational Measurement: Issues and Practice, 32(4), 16–27. https://doi.org/10.1111/emip.12016

Carspecken, F. P. (1996). Critical ethnography in educational research: A theoretical and practical guide. Routledge.

Conley, D. T. (2008). Rethinking college readiness. New Directions for Higher Education, 2008(144), 3–13. https://eric.ed.gov/?id=EJ824814

Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Sage.

Hall, J. (2020). Focus groups: Culturally responsive approaches for qualitative inquiry and program evaluation. Myers Education Press.

Mishkind, A. (2014). Overview: State definitions of college and career readiness. College and Career Readiness and Success Center at the American Institutes for Research. https://files.eric.ed.gov/fulltext/ED555670.pdf

Pérusse, R., DeRonck, N., & Parzych, J. (2017). School counseling: Partnering with a school district to provide postsecondary opportunities for first generation, low income, and students of color. Psychology in the Schools, 54(10), 1222–1228. https://eric.ed.gov/?q=a&pg=7934&id=EJ1161221

Roulston, K. (2010). Reflective interviewing: A guide to theory and practice. Sage.

Sampson, J. P., Hooley, T., & Marriot, J. (2011). Fostering college and career readiness: How career development activities in schools impact on graduation rates and students’ life success. International Centre for Guidance Studies. https://diginole.lib.fsu.edu/islandora/object/fsu:207200/datastream/PDF/view

Texas Education Code §33.006 (2021). https://statutes.capitol.texas.gov/docs/ED/htm/ED.33.htm

Texas Higher Education Coordinating Board. (2021). 2021 Texas public higher education almanac: A profile of state and institutional performance and characteristics. https://reportcenter.highered.texas.gov/agency-publication/almanac/2021-texas-public-higher-education-almanac/

Tierney, W. G., & Sablan, J. R. (2014). Examining college readiness. American Behavioral Scientist, 58(8), 943–946. https://psycnet.apa.org/doi/10.1177/0002764213515228

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Rich, Patrick
File Modified	0000-00-00
File Created	2024-07-21