Master Topic Guide for Staff Interviews

instrument #2

MASTER TOPIC GUIDE

This page has been left blank for double-sided copying.

Form Approved

OMB No. 0990-XXXX

Expiration Date:

In-Depth Implementation Study

Master Topic Guide¹

1. CONTEXT AND COUNTERFACTUAL

Community attitudes

• Perception of community/school needs and how they should be addressed

• Perception of the problems of teen pregnancy and risky behaviors that the program is being implemented to address

• Community members and/or organizations important to the program, and how their support is critical

• Steps taken to gain or maintain community and school support for the program

• Outside influences that may have affected the program outcomes targeted for change

Counterfactual

• Other programs and services with similar goals available to program participants and control group teens

• Number of program participants participating in these programs or receiving these services

• Number of teens participating in these programs or receiving these services

2. THE INTERVENTION

Program Development and Adoption

• Name and version of the program being implemented

• Process and stakeholders involved in adapting the program for schools

• The process used to assess the fit of the program with organizational or school needs

• Prevalence of current and past use of this program (state-level, county or district-level and/or community-based organizations that have purchased and/or implemented the program)

Target Outcomes/Theory of Change

• Primary goals of the program

• Outcomes targeted for change

• Population targeted for change

• Program’s approach to improving the targeted outcomes

• Program’s main messages

• Mechanisms by which program’s key features are expected to improve outcomes

Core Components of the intervention

• Primary components of the program

• Planned timeline for each session and overall program

• Planned staffing for program

  • Ideal background, qualifications, experience of staff

• Planned content for program

  • Program approach: self-efficacy and behavioral control beliefs

• Type and content of materials available from the program developer and used (curriculum materials, facilitator guides, staff monitoring and evaluation tools)

• Type and content of supplemental materials available

• Instructional strategies prescribed by the program

• Mandatory and optional components (14 required sessions plus optional topics)

• Minimum requirements for program completion

Fidelity benchmarks and guidelines

• Defined number and frequency of program sessions

• Maximum group sizes and/or adult: youth ratios planned during instructional activities

• Criteria for staff qualifications, training, and technical assistance

  • health educators

  • classroom teachers

• Criteria for participants to participate, and complete the program

• Minimum participation requirements and consequences for not meeting them

• Performance standards and monitoring by program developer (if applicable)

Planned and Unplanned Adaptations

• Changes to the curriculum or program model, and why

• Changes to the schedule, duration, or frequency of sessions to accommodate context, and why

• Changes to recommended program staffing to fit context, and why

• Supplemental materials used, and why

• Other changes to respond to needs of target population, and why

• Consultation with program developer about planned adaptations

• How decisions about adaptations were made and communicated to staff

• How adaptations were tested and what was learned

• Unplanned changes

Target Population

• Description of target population (specify ethnicity, age, gender, income, socioeconomic and geographical characteristics, risk groups, affiliation with or enrollment in government or community-based programs, etc.)

• Prevalence of key risk factors or risk behaviors among the target population

• Prevalence of strengths or protective factors among the target population

• Specific needs of target population and potential challenges in meeting these needs

Participant Eligibility and Recruitment

• Planned recruitment and identification strategies for implementation sites (schools)

• Program eligibility criteria for schools and participants

• Number of eligible youth in targeted communities and/or schools

• Actual enrollment

• School’s motivation for enrolling in the intervention; most attractive features of the intervention

3. IMPLEMENTING ORGANIZATIONS

Site Background (schools)

• Geographical locations of implementing schools

• Characteristics of the general population of youth in each school (risk profile, demographic, and socioeconomic characteristics)

• Characteristics of schools’ settings

  • Staff attitudes toward the program

  • Administration support for the program

  • How the program came to the school

  • “Program champions” within the school, etc.

• Previous programs aimed at preventing adolescent pregnancy that have been implemented in the schools and lessons learned

Lead organization (for Health Educators)

• Description of organization responsible for implementing the program in the evaluation

  • Mission, goals, history

  • Staffing characteristics, size, geographical location(s)

  • Budget and funding sources

• Organizational climate/culture

• Leadership support for implementing the new program

• Program champions and persons of influence

• History of implementing the program and other teen pregnancy prevention programs

• Experience with Making Proud Choices or other evidence-based TPP programs

Experience

• Respondent’s role in the organization and with the program

• Length and depth of experience

• Types of training received relevant to program topics and approach

• Respondent and other staff experience working with youth

• Experience working with MPC and other teen pregnancy prevention programs

Community Coordination and Partners

• Type and level of coordination with other community agencies to implement program activities or provide services to participants

• Coordination or links with schools or school-based activities (if any)

• Resources from partner agencies

• History of past relationship with (formal or informal) partner organizations or agencies

• Decision-making strategies and key decision-makers

• Mechanisms for cross-organization collaboration and communication

Staffing

• Staff recruitment process at lead organization for health educators

• Staff recruitment process in schools for classroom teachers

• Challenges in recruitment and hiring

  • Teachers

  • Health educators

• Staff positions and minimum qualifications for health educators and classroom teachers (request job descriptions for dedicated staff)

• Numbers and qualifications of staff

  • Health educators

  • Classroom teachers

• Number of staff (health educators, teachers, supervisors, relevant school administrators) who have left and been replaced since evaluation period began

• Reasons for turnover

• How turnover is/was managed

• How long vacancies have remained open and how work was managed

Decision-making and Collaboration

• Communication between senior leadership (such as school administrators and organizational leaders) and program staff (health educators and classroom teachers) about important decisions related to program implementation

• Strategies used to encourage staff buy-in, both within lead organization, and among classroom teachers and other school staff

• Delegation of responsibility

Polices that Affect Program Implementation

• State-level, county-level, city-level, school-level, or agency policies that affect program implementation

• Regulations that negatively impact the program’s implementation

Structural barriers

• Staff workloads or course-loads

• Available supports to health educators and to classroom teachers

Adequacy and sources of resources

• Financial resources

• Supervision and technical support resources

• Training resources

• Facilities

Attitudes towards the program

• Attitudes and perceptions of managerial and frontline staff (health educators and classroom teachers)

• Health educator and school teachers’ attitudes about implementing MPC or control program

• Particular topics or subjects that health educators and teachers support or especially like

• Staff satisfaction with the support they receive for implementing MPC

• Particular topics that health educators and teachers have expressed concerns about

• Steps that have been taken to address staff concerns

• Overall suitability of program with organization values and priorities

• Overall suitability of program with school’s values and priorities

4. PROGRAM IMPLEMENTATION

Program training and technical assistance

• Approach (didactic, participatory, role-play, etc.)

• Pre-service and in-service training required for health educators and classroom teachers in each site

• Any variations in the types of training accessible to health educators and classroom teachers

• Participation in required training sessions

  • Differences/similarities among health educators and teachers

• Usefulness of training for health educators and teachers

• Access to ongoing technical assistance in implementing the program

• Sources, amount, and topics of technical assistance received by health educators and classroom teachers

• Certification requirements, process, and cost

• Procedures for performance monitoring of staff

  • Process for providing feedback and technical assistance to health educators and classroom teachers based on data collected

Communication systems

• Formalized protocols and guidance for staff to follow

• Frequency of communication (staff meetings, one-on-one meetings, memos, reporting, etc.) between levels and types of staff

Tracking and Monitoring

• Collection of attendance data (frequency, methods, staff responsible, measures, etc.)

• Collection and tracking of participant outcome data

• Purpose and uses of process and outcome data

• How data are used to monitor implementation and identify challenges that need to be addressed

  • Frequency of data monitoring

  • Who is involved

  • How the data are used

  • Whether data are used to assess changes to implementation procedures

  • Monitoring of participation

  • Follow-up when attendance is low

• Whether the program has an ongoing CQI process in place and who is involved

• Data reporting procedures

Program Costs

• Total budget for implementing the program

• Budgeted cost per participant

• Whether actual cost was less than, about the same as, or higher than budgeted

• Major program costs (staffing, external staff training and technical assistance, program materials, other)

• Time spent by staff (on training, on service delivery in an average week)

Characteristics of Program and Control Group Participants

• Description of youth receiving the program

  • Cultural or ethnic backgrounds

  • Strengths or protective factors

  • Risk factors

  • Personal and family characteristics, home environments

  • Sources of support

  • Relationship characteristics

  • Educational characteristics, needs, and goals

• Perceived differences or variations among schools

Adherence to planned program (based on staff perceptions and MIS data)

• Actual timeline

• Actual number, frequency, and duration of sessions or visits

• Types of activities conducted and the staff who conducted them

• Actual training and technical assistance provided to health educators and classroom teachers

• Content/program materials covered

• Actual class sizes and adult: youth ratios during program sessions

• Perceived differences or variations between schools

Adherence to counterfactual plan

• Actual services offered and received

• Actual number, frequency, and duration of sessions or visits

• Types of activities conducted and the staff who conducted them

• Actual training and technical assistance provided

• Content/program materials covered

• Attendance and participation

• Actual class sizes and adult: youth ratios during program sessions or visits

• Perceived differences or variations between schools

• Quality of interactions during observed activity

  • How engaged are youth in the session

  • Nature of interactions between staff and youths

  • Extent to which health educators and classroom teachers are comfortable presenting the material

  • Extent to which youths are comfortable with material that is presented

Participation and engagement in program activities

• Perceived levels and patterns of attendance at key program sessions and activities

• Rates of program completion

• Reasons for youth’s low attendance levels (as relevant)

• Health educator and classroom teacher perceptions of participant engagement in program activities

Overall implementation experience

• Lessons learned

• Overall Challenges

  • Challenges in reducing risk behaviors

  • Challenges related to implementing program with fidelity

  • Challenges classroom teachers faced

  • Challenges health educators faced

  • Challenges related to implementing the program in a school setting

  • Challenges related to working with partner organizations or outside staff

• Aspects of the program most difficult to implement, and why

• Aspects of the program that worked well in the school setting, and why

• Accomplishments most proud of, and why

• Examples of successful implementation strategies

• Steps taken to address challenges and replicate effective strategies

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-xxxx . The time required to complete this information collection is estimated to average one hour per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

1 In this and other Expanding the Use and Understanding of Evidence-Based Teen Pregnancy Prevention Programs implementation study documents, the term “program” refers to the intervention that is being implemented. The term “site” refers to the organization and places in which the program (intervention) is being implemented. The term “location” refers to a specific place in which the program is implemented within a site.