Semi- structured interview

instrument #1

MASTER LIST OF TOPICS FOR IMPLEMENTATION STUDY

This page has been left blank for double-sided copying.

OMB Control No:
Expiration Date:

In-Depth Implementation Study

Master Topic Guide¹

1. CONTEXT AND COUNTERFACTUAL

Community attitudes

• Perception of community needs and how they should be addressed

• Perception of the problem that the program is being implemented to address

• Community members and organizations important to the program, and how their support is critical

• Steps taken to gain or maintain community support for the program

• Outside influences that may have affected the program outcomes targeted for change

Counterfactual

• Other programs and services with similar goals available to program participants and control group teens

• Number of program participants participating in these programs or receiving these services

• Number of teens participating in these programs or receiving these services

2. THE INTERVENTION

Program Development and Adoption

• Name and version of the program being implemented

• When, by whom, and how the program was developed

• The process used to assess the fit of the program with agency needs

• The process used to adopt the program

• Prevalence of current and past use of this program (state-level, county or district-level and/or community-based organizations that have purchased and/or implemented the program)

• Prior evidence of effectiveness or evaluations conducted (if any)

• Rationale for including the program in the Pregnancy Assistance Fund evaluation

Target Outcomes/Theory of Change

• Primary goals of the program

• Outcomes targeted for change

• Population targeted for change

• Program’s approach to improving the targeted outcomes

• Program’s main messages

• Mechanisms by which program’s key features are expected to improve outcomes

Core Components of the intervention

• Primary components of the program

(Adult-led instruction, peer-led instruction, counseling, home visits, mentoring, tutoring, school-based instruction, direct health services, community-based services, online resource and activities, group activities, etc)

• Components that involve other family members (parent, partner, child)

• Planned timeline for each component

• Number, frequency, and duration of sessions for each component

• Planned staffing for each component

• Activities planned under each component

• Content/program material for each component

• Type and content of materials available from the program developer and used (curriculum materials, facilitator guides, client assessment and tracking tools, staff monitoring and evaluation tools)

• Type and content of supplemental materials available

• Instructional strategies

• Mandatory and optional components

• Minimum requirements for program completion

Fidelity benchmarks and guidelines

• Defined number and frequency of program sessions

• Maximum caseloads, group sizes, and/or adult:youth ratios planned during instructional activities

• Criteria for staff qualifications, training and technical assistance

• Criteria for participants to attend, participate, and complete the program

• Minimum participation requirements and consequences for not meeting them

• Performance standards and monitoring by program developer

Planned and Unplanned Adaptations

• Changes to the curriculum or program model, and why

• Changes to the schedule, duration, or frequency of sessions to accommodate context, and why

• Changes to recommended program staffing to fit context, and why

• Supplemental materials used, and why

• Other changes to respond to needs of target population, and why

• Consultation with program developer about planned adaptations

• How decisions about adaptations were made and communicated to staff

• How adaptations were tested and what was learned

• Unplanned changes

Target Population

• Description of target population

(Specify ethnicity, age, gender, income, socioeconomic and geographical characteristics, risk groups, affiliation with or enrollment in government or community-based programs, etc)

• Prevalence of key risk factors or risk behaviors among the target population

• Prevalence of strengths or protective factors among the target population

• Specific needs of target population and potential challenges in meeting these needs

Participant Eligibility and Recruitment

• Planned recruitment and identification strategies for program participants

• Program eligibility criteria

• Number of eligible youth in targeted communities

• Expected enrollment relative to overall targeted for recruitment

• Actual enrollment

• Difficulties or challenges associated with recruitment and strategies used to address them

• Lessons learned about effective and ineffective recruitment strategies

• Youth motivation for enrolling in the intervention; most attractive features of the intervention

3. IMPLEMENTING ORGANIZATIONS

Site Background

• Definition of sites or sub-sites participating in the evaluation (community-based organizations, schools, state agencies, clinics, etc)

• Geographical locations

• Characteristics of the general population of youth in each site (risk profile, demographic and socioeconomic characteristics)

• Similarities and differences among sub-sites

• Organizational settings for program implementation

• Previous programs aimed at preventing adolescent pregnancy that have been implemented in the evaluation locations and lessons learned

Lead organization

• Description of organization responsible for implementing the program in the evaluation

• Mission, goals, history

• Staffing characteristics, size, geographical location(s)

• Budget and funding sources

• Organizational climate/culture

• Leadership support for implementing the new program

• Program champions and persons of influence

Experience

• Respondent’s role in the organization and with the program

• Respondent and other staff experience working with youth

• Experience working with teen pregnancy prevention programs

• Experience working with teen parents

Community Coordination and Partners

• Formal and informal partners and their roles in implementing the program

• Role of national or state agency in implementing the program in the evaluation

• Type and level of coordination with other community agencies to implement program activities or provide services to participants

• Coordination or links with schools or school-based activities (if any)

• Resources from partner agencies

• Referral relationships and partners

• History of past relationship with (formal or informal) partner organizations or agencies

• Agreements with current or potential community partners

• Designated staff contacts at partner organizations

• Decision-making strategies and key decision-makers

• Mechanisms for cross-organization collaboration and communication

• Conflicts or challenges associated with partner relationships

• Retention of partners during the program’s implementation

Staffing

• Staff recruitment and hiring plans

• Challenges in recruitment and hiring

• Staff positions and minimum qualifications for frontline staff positions (request job descriptions for dedicated staff)

• Other staff positions and minimum qualifications

• Numbers and qualifications of staff

• Positions filled with staff members from partner organizations

• Role of volunteers

• Number of staff who have left and been replaced since evaluation period began

• Reasons for turnover

• How turnover is/was managed

• How long vacancies have remained open and how work was managed

Decision-making and Collaboration

• Involvement of program staff in implementation decisions

• Teamwork and collaboration related to program implementation

• Communication between senior leadership and program staff about important decisions related to program implementation

• Strategies used to encourage staff buy-in

• Delegation of responsibility

Polices that Affect Program Implementation

• State-level, county-level, city-level or agency policies that affect program implemention

• Regulations that negatively impact the program’s implementation

Structural barriers

• Staff caseloads and/or workloads

• Available supports to staff

Adequacy and sources of resources

• Financial resources

• Supervision and technical support resources

• Training resources

• Facilities

Attitudes towards the program

• Attitudes and perceptions of managerial, supervisory, administrative, frontline staff

• Staff attitudes about implementing the new program

• Particular topics or subjects that staff support or especially like

• Staff satisfaction with the support they receive the implement the new program

• Particular topics that staff have expressed concerns about

• Steps that have been taken to address staff concerns

• Overall suitability of program with organization values and priorities

4. PROGRAM IMPLEMENTATION

Program training and technical assistance

• Approach (didactic, participatory, role-play, etc)

• Pre-service and in-service training required for staff and volunteers in each position

• Participation in required training sessions

• Training offered on a voluntary basis

• Participation in voluntary training related to role in program

• Usefulness of training

• Access to technical assistance in implementing the program

• Sources, amount, and topics of technical assistance received

• Certification requirements, process, and cost

• Procedures for performance monitoring of staff

• Process for providing feedback and technical assistance to staff based on data collected

Communication systems

• Formalized protocols and guidance for staff to follow

• Frequency of communication (staff meetings, one-on-one meetings, memos, reporting, etc) between levels and types of staff

• Reporting relationships

Tracking and Monitoring

• Data collection and monitoring procedures

• Data entry protocols (frequency, staff responsible, method)

• MIS systems

• Data collection procedures on program implementation

• Enrollment, program delivery, staff training, service receipt/dosage, service quality, etc

• Collection of attendance data (frequency, methods, staff responsible, measures, etc)

• Collection and tracking of participant outcome data

• Purpose and uses of process and outcome data

• How data are use to monitor implementation and identify challenges that need to be addressed

• Frequency of data monitoring

• Who is involved

• How the data are used

• Whether data are used to assess changes to implementation procedures

• Monitoring of participation

• Followup when attendance is low

• Whether the program has an ongoing CQI process in place and who is involved

• Data reporting procedures

Program Costs

• Total budget for implementing the program

• Budgeted cost per participant

• Whether actual cost was less than, about the same as, or higher than budgeted

• Major program costs (staffing, external staff training and technical assistance, program materials, other)

• Time spent by staff (on training, on service delivery in an average week)

Referrals

• Process and criteria for making referrals

• Types of referrals made

• Follow-up procedures

• Staff contacts and relationships

• Uptake rate for referrals

• Gaps in community resources

Characteristics of Program and Control Group Participants

• Description of youth

• Cultural or ethnic backgrounds

• Strengths or protective factors

• Risk factors

• Personal and family characteristics, home environments

• Sources of support

• Relationship characteristics

• Parenting needs and goals

• Educational characteristics, needs, and goals

• Characteristics of enrollees

• Perceptions of differences between youth targeted and youth enrolled

• Perceived differences or variations among sub-sites

Adherence to planned program (based on staff perceptions and MIS data)

• Actual timeline

• Actual number, frequency, and duration of sessions or visits

• Types of activities conducted and the staff who conducted them

• Actual training and technical assistance provided

• Content/program materials covered

• Actual caseloads and adult : youth ratios during program sessions or visits

• Perceived differences or variations between sub-sites

Adherence to counterfactual plan

• Actual services offered and received

• Actual number, frequency, and duration of sessions or visits

• Types of activities conducted and the staff who conducted them

• Actual training and technical assistance provided

• Content/program materials covered

• Attendance and participation

• Actual caseloads and adult: youth ratios during program sessions or visits

• Perceived differences or variations between sub-sites

• Quality of interactions during observed activity

• How engaged are youth in the session

• Nature of interactions as described by staff and youths

• Whether staff are comfortable presenting the material

• Whether youths are comfortable with material that is presented

Participation and engagement in program activities

• Mandatory and voluntary activities

• Perceived levels and patterns of attendance at key program activities

• Rates of program completion

• Reasons for youth dropout

• Staff perceptions of participant engagement in program activities

Overall implementation experience

• Lessons learned

• Overall Challenges

• Challenges in reducing risk behaviors

• Aspects of the program most difficult to implement, and why

• Accomplishments most proud of, and why

• Steps taken to address challenges and replicate effective strategies

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-xxxx . The time required to complete this information collection is estimated to average one hour per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

1 In this and other PEPY implementation study documents, the term “program” refers to the intervention that is being implemented. The term “site” refers to the organization and places in which the program (intervention) is being implemented. The term “location” refers to a specific place in which the program is implemented within a site.