Site Visit Protocols

Appendix C
Site Visit Procedures and Protocols

Site Visit Procedures and Protocols

Overview

This appendix offers an overview of protocols and procedures for planning and conducting case study site visits to treatment and control schools. The case study site visits will provide rich background and context for survey findings and opportunities to gauge the fidelity of implementation of the intervention. Site visit data will be collected from schools in both the treatment and control groups to allow the study team to examine how RTS in context might differ from other college and career planning services provided in schools in the state. The site visits are designed to help researchers understand how and why RTS may have impacted 11^th- and 12^th-graders’ perceptions of their college and career preparedness and their plans after high school. The following overarching research questions will guide case study data collection and analysis:

• What college and career planning assistance did students receive prior to their current year?

• What challenges and successes have instructors and students experienced in implementing the program?

• To what extent was the program implemented with fidelity to the model (e.g., how many of the components were introduced, for how long, to what cross-section of students, and in what ways)?

• How did college and career development strategies differ in the treatment and control schools?

Because we anticipate incorporating up to 47 treatment schools as part of the study, visiting each school would be cost-prohibitive; however, researchers plan to visit 14 of the treatment schools and 6 of the control schools to enhance the generalizability of site visit findings. Researchers will visit each of the selected sites once in the spring of 2016 (April or May). While the high schools will be the cases, several populations associated with RTS will serve as secondary cases or data sources in the treatment schools, including: 11^th- and 12^th-grade students, teachers, academic and career counselors, and administrators. The high schools will serve as the context in which collaboration between these groups around RTS occurs.

Information to be Collected

Researchers will conduct interviews with RTS teachers, academic teachers, counselors, and principals, as well as focus groups with 11^th- and 12^th-grade students and observations of RTS lessons. These strategies will allow study team members to collect information on various perspectives on RTS and triangulate across these data sources to develop a full, detailed picture of RTS implementation in schools and its effectiveness. Open-ended, exploratory questions will be presented to respondents and focus groups in order to prompt reflection and allow for elaboration beyond what is allowable within a structured, closed-ended survey design. Questions to be used as part of the site visits can be grouped into the following four areas pertaining to (1) prior college and career planning assistance; (2) challenges and successes with RTS; (3) fidelity of implementation; and (4) other college and career planning activities:

1. What college and career planning assistance did students receive prior to their current year?

• What college and career planning assistance did students receive prior to RTS, if any? How helpful was it? What aspects were most valuable?

• Did RTS participation change senior year course taking patterns? Why? How?

• Did RTS participation change students’ plans for applying to college and careers? Why? How?

• Did RTS participation change students’ college course taking plans and/or plans for college major? Why? How?

• How has RTS impacted teachers’ instructional practices or strategies?

• How has RTS impacted counselors’ practices or strategies? How?

• Has RTS impacted students’ preparedness for college and/or career in comparison to previous years? How?

2. What challenges and successes have instructors and students experienced in implementing the program?

• What assistance was provided in registering and preparing for workforce development assessments? How helpful was it? Have students’ plans changed as a result?

• What assistance was provided in identifying and applying for employment? How helpful was it? Have students’ career plans changed as a result? How?

• What assistance was provided in preparing a resume? How helpful was it? Have students’ career plans changed as a result? How?

• What assistance was provided in identifying and applying for sources of financial aid (FAFSA, scholarships, etc.)? How helpful was it? Have students’ postsecondary plans changed as a result? How?

• What assistance was provided in identifying colleges and preparing and submitting college applications? How helpful was it? Have students’ postsecondary plans changed as a result? How?

• What 11^th and 12^th grade RTS topic areas have students found most and least useful? Why?

• What 11^th and 12^th grade RTS topic areas have teachers found most and least effective? Why?

• How do principals, counselors, and teachers assess students’ level of preparedness for college and career?

3. To what extent was the program implemented with fidelity to the model?

• What college and career readiness topics were covered during RTS training?

• How many RTS components were introduced over the course of the academic year? For how long? To what cross-section of students? In what ways?

• How, if at all, does the RTS model advocate integrating college and career planning instruction with academic and CTE topics?

• What ongoing supports for RTS implementation, if any, do teachers receive during the academic year?

• What aspects of RTS training were most and least useful? Why?

4. How did college and career development strategies differ in the treatment and control schools?

• What college and career planning assistance do students receive? How helpful is it?

• What assistance is provided in registering and preparing for workforce development assessments?

• What assistance is provided in preparing a resume?

• What career exploration activities are provided?

• What assistance is provided in enrolling or planning to enroll in college?

• What assistance is provided in identifying and applying for sources of financial aid (FAFSA, scholarships, etc.)?

• What assistance is provided in identifying and applying for employment?

• What impact, if any, has participation in college and career planning activities had on students? How do participants know that these changes have occurred?

Site Visit Procedures and Protocols

Before the Site Visit

Site Selection: Researchers plan to visit 14 of the treatment schools and six of the control schools to enhance the generalizability of study findings. The study team will randomly select treatment schools from each of the four largest districts in the state (Bismarck, Minot, Fargo, and Williston), and randomly select the remainder of treatment schools from each of seven regional education associations (REAs¹) (1-2 additional treatment schools per REA). We plan to randomly select two control schools from among the four largest districts and the remaining four from other REAs. This will ensure geographic diversity and diversity of school size across our sites. Researchers will visit each of 14 treatment schools and six control schools once in the spring of the 2015-2016 academic year.

Site Visit Planning: A school coordinator to serve as main point of contact will be identified for each of the site visit schools. One member of the research team will serve as site visit lead for planning and conducting each site visit. Researchers will seek to provide ample advance planning time to school coordinators before the site visit. To facilitate site planning, the research team will share with the school coordinator several documents in advance of the site visit, including:

• A list of suggested interview and focus group participants and guidance for recruiting each.

• A one-page overview of the study, research questions, and information that will be sought during the site visit from each group of participants.

• A list of background documentation requested.

• A sample site visit agenda.

The site visit lead and school coordinator will collaborate to finalize the list of participants and schedule for each one-day site visit.

Site Visit Training: Prior to data collection, senior staff at FHI 360 will conduct a one-day training session to promote consistency in data collection, analysis, and reporting of qualitative data. The training will be offered in-person and via webcast. The schedule for site visits and all interview, focus group, and observation protocols will be made available to all participants two weeks prior to the one-day training session.

Notebooks prepared for the case study training will include background on RTS; the research design, including major research questions and other tasks; the project organization chart and staffing/role of site researchers; project abstracts and contact information for all sites to be visited; interview, focus group, and observation protocols; a checklist for site visit planning; outlines for individual case descriptions; and a college and career planning literature review.

Researchers will receive training on effective moderating practices for interviews and focus groups, such as establishing a comfortable environment, remaining neutral, using pauses and probes effectively, transitioning between major topics, and maintaining participants’ interest throughout the interview or focus group. Team members will also be trained on collecting observational data.

Documentation: Prior to the site visit, the site visit lead will work with the school coordinator to identify and collect appropriate background information on college and career curricula in the school. This documentation might include program descriptions, lesson plans, planning documents and additional relevant information associated with RTS and the students participating in the curriculum. Site visit team members will review background documentation prior to the site visit and come prepared with clarifying questions and requests for further information, as appropriate. Information gleaned from documentation will help researchers understand the school context for college and career planning implementation and inform analysis of data collected during the site visit.

During the Site Visit

Teacher and Staff Interviews: During each site visit, researchers will conduct standardized, open-ended interviews with RTS and academic teachers, principals, and school counselors. Because of the small size of most North Dakota high schools, in most instances there will be only one representative from each category. The site visit lead will work with the site coordinator in advance of the visit to identify up to three academic teachers with appropriate perspectives on 11th and 12th graders’ college and career readiness. Interview guides will ask for participants’ perspectives on behaviors and opinions related to how RTS is implemented, its impact on students, and why or why not it has been successful in their schools.

For each interview, researchers will follow generally accepted best practices for preparing and conducting interviews, including: selecting interview settings with few distractions; explaining the purpose of the interview to participants; addressing the terms of confidentiality; explaining the format of the interview and encouraging participants to offer candid answers and ask clarifying questions; indicating that the interview will take approximately one hour; providing contact information for the research team; asking neutral, open-ended questions one at a time; providing appropriate probes and prompts to encourage the participant; providing transitions between major topics; remaining neutral to responses; carefully checking the time to cover all questions in the allotted one hour; and thanking all participants. The last question will be to allow participants’ to provide any additional information.

With participant permission, all interviews will be audio recorded and transcribed to facilitate data analysis.

Student Focus Groups: The site visit lead will serve as focus group moderator, working with the school coordinator to recruit participants and establish a comfortable environment. Six to eight carefully selected students will be recruited for each of two 90-minute focus groups during site visits; focus groups of 11^th and 12^th grade students will be conducted separately. The site visit lead will ask the school coordinator in advance to identify and recruit students who will likely be able to share varied and comprehensive perspectives on RTS.

Researchers will follow generally accepted best practices for preparing and conducting focus groups, including: reducing barriers to participation by scheduling around participants’ previously-scheduled commitments; clearly explaining the purpose of the research; following a prepared script to welcome participants and establish ground rules; asking short, focused, open-ended, and unambiguous questions; encouraging participation by all members; using probes and prompts to clarify responses and elicit additional information; summarizing long or complex comments to check for understanding; and thanking all participants. The moderator will seek out varied opinions and ideas from all members of the group related to their participation in the college and career planning curriculum and the curriculum’s impact.

With participant permission, all focus groups will be audio recorded and transcribed to facilitate data analysis.

Classroom Observations: Observations allow researchers to see programs “in action,” and provide data to cross-check and validate information collected during interviews and focus groups. Teacher-student interactions are crucial to student outcomes. Observations will provide direct evidence of behavior and processes to enhance data on perceptions, attitudes, and beliefs collected through interviews. Observation protocols developed for this study will be descriptive rather than evaluative. Drawing on the RTS Program Manual, researchers will seek information on tools and materials shared with students through RTS as well as any supplemental information and how RTS lessons are carried out. The observation tool will ask for documentation of, for example, the lesson topic, instructional setting, instructor practice, student time on task, and evidence for students’ active engagement in the lesson.

Researchers will follow best practices for conducting classroom observations, including: providing adequate training on the observation tool; observing in consistent ways across classrooms; focusing on rich description of observable behaviors; and limiting inferences to the scope of the tool. Observers will also conduct a short debrief with teachers to gauge their perceptions of the lesson.

After the Site Visit

Data Analysis: Initial site-specific data analysis will begin during the data collection phase. All audio transcripts, observation notes, and documentation will be immediately uploaded to NVivo for initial coding to identify themes within and across cases. The initial themes derived from early analysis will inform the development of a code book to guide subsequent analyses. Early data analysis will also provide the research team with the opportunity to collect additional data that could inform study findings. Once all of the data has been coded, researchers will conduct a within-case analysis to compare informant responses at each high school to understand implementation of RTS, fidelity to the model, participant perceptions on implementation and outcomes, and program outcomes.

Reporting: A cross-case analysis will serve as the report for the case study phase of documentation. Researchers will conduct the cross-case analysis following completion of data analysis for individual sites. Analyzing the data across cases allows the research team to compare high schools in different contexts to identify common elements, themes, and patterns emerging across cases. The cross-site report will develop a thematic understanding of the challenges and opportunities for students and schools as a whole, in order to draw broader lessons about the constraints and considerations schools and sites face in implementing a college and career planning curriculum. Each researcher conducting a site visit will contribute to the cross-case report. All members of the research team will be expected to review and comment on the first draft of the cross-case report. The authors will elicit feedback on future drafts based on staff expertise.

1 As noted in our sampling plan, two of the smallest REAs would be combined and treated as one REA as part of the study.