Guide for community stakeholder

APPENDIX H

Evaluation of Teen Pregnancy Prevention Replications

Discussion Guide for Use with COMMUNITY STAKEHOLDERS

(Agency head, staff, caseworkers)

Program Name:

Program Location:

Sponsoring Organization:

Individual(s) Interviewed: (names and titles)

Contract Staff (as appropriate):

Date of Communication:

Instructions for Site Visitors

This visit has a set of very specific goals:

• To ensure that we have a complete and up-to-date understanding of the roles and responsibilities of the grantee and partners and their staff:

• To expand our understanding of aspects of readiness and preparation that would support strong replication of a program model (preliminary information has been abstracted from the grant proposal and other extant documents and is incorporated into the profile you received. Please make sure that you have read this and are thoroughly familiar with it so that you can probe for updated information and identify incorrect information);

• To understand the plan for replication of the program model, the adaptations that were approved and made, and the extent to which the replication was implemented as planned (again, you will have been given a summary of the replication plan as contained in the proposal and updated in the request for continuation funding. Your task will be to use discussion and direct observation to determine how the program is actually implemented and to determine the extent to which the various aspects of the replication were implemented as planned);

• To understand the local context in which the replication is being implemented (this includes the school or agency environment as well as the social structure ,behavioral norms, resources and services availability of the local community);

• To understand the ways in which the grantee changed or adapted aspects of the replication plan in response to local needs or pressures;

• To understand the challenges encountered in replicating the program model and in other aspects of implementation, the extent to which staff are able to address those challenges and the strategies they employ to address them; and

• To document the services provided to the members of the control group.

These goals will have been articulated in prior correspondence with the sites before the visit, but you should reiterate them at the beginning of any discussion with staff. You should hand the following statement to everyone you interview and an appropriate version of it to youth participants in focus groups.

Thank you for taking time to talk about (Name of Program). As you know, we are conducting the TPP Replication Evaluation. As an important part of that effort, we are visiting programs that are participating in the evaluation two or three times during their grant period, in an effort to understand and document the process of replicating an evidence-based model in real-world settings, the challenges that arise, and how staff on the ground respond to those challenges. The information we gather will serve two purposes: it will help future program operators and policymakers understand what is needed to replicate a program with fidelity, to implement it as intended, with the populations originally targeted; and it will be used to help us understand variations in program impacts, where they occur.

You may, of course, choose not to discuss any topic or end the discussion at any time. We will combine the information from this visit and subsequent visits, with information from your program documents and the performance and fidelity data you have collected, to create a narrative account of your program and the process of replicating the program model you selected. At the same time, we will combine the information about your program with information about other programs in the study to identify consistent themes that apply more generally across a range of program types and replication efforts. Neither your name nor the names of any individuals will be reported, and the notes we take about our discussions will not be shared with or provided to the federal government or anyone else except the members of the evaluation team.

A: Community Agency Background

A1. Demographic composition of youth populations served by the agency

Probes: Can you tell me something about the youth population who use your programs and services? Demographic characteristics. Truancy, school failure and dropout rates, protective factors (parent involvement, religious affiliation, close-knit school and/or external community), risk factors (drugs, gang activity, violence, sexual risk behavior), involvement with juvenile justice system

A2. Challenges facing youth

Probes: From your perspective, what are some of the challenges that your young people face? What are the prevailing attitudes towards adolescent sexual and other risk behaviors?

A4. Selection of program model for replication

Probes: Were you involved in the selection of the program model that is being used in your agency/organization? If not how did you come to be involved with the program? What aspects of the program attracted you, made you think it would be a good thing for your organization and the youth you serve?

D. Implementation: Putting the Program in Place

D1. Setting for the program

Probes: Is the agency a good setting for the program? That is, is it easy to fit into your schedule, are staff interested/enthusiastic about it? What have been the reactions from staff and parents? What if any challenges have you faced as a result of the decision to implement the program in the agency? How have you dealt with them?

D2. Staffing the program

Probes: Have you had the opportunity to observe any groups or sessions? What is your impression of the staff who deliver the program? Do you feel they are well-qualified to work with youth in this setting? What are the advantages and disadvantages of having outside staff deliver the program vs. your own staff?

D4. Schedule for program activities

Probes: How is the program delivered? When (after school, on weekends)? In how many sessions, of what length, and over what period of time? What challenges to scheduling the program did you encounter? How does scheduling affect retention of youth?

D6. Program components/activities

Probes: Which lessons/sessions seem to you to be most appropriate in terms of content and activities linked to the content? Are there any that you feel are inappropriate or that should be dropped? Is there content that you feel needed to be covered that was not covered?

D8. Satisfaction with program model

Probes: Overall, do you feel that the program is the correct choice for the youth population you are serving? If not, in what ways is it less than ideal? In retrospect, would you choose a different program? model to replicate?

D9. Response of participants

Probes: How engaged are youth in the activities/content of the program? What aspects of the program/activities/content are they most/least responsive to? Have you had any feedback from them about the program? What kinds of comments do they make about the program?

D10. Collaboration and conflict

Probes: Has the collaboration with the program been a smooth and satisfactory one? How have you and your staff managed the relationship? Have you experienced conflict? Did these problems occur in the early phase of the program or have they continued?

G. Community Context

G1. Other services or programs in the community that deal with the issue of teen pregnancy and sexual risk behavior

Probes: What kinds of programs or services exist in your community that aim to prevent risk behavior in teens) e.g., youth development programs, adolescent health clinics? Do students in your school use any of these – do any of them reach out to students in or out of school?

G3. Community attitudes toward the problem of teen pregnancy

Probes: What are the prevailing attitudes towards adolescent sexual and other risk behaviors? What are the beliefs about teen pregnancy (i.e. a large problem, a manageable problem)? Are teen sexual behavior and pregnancy perceived as problems by members of community?

G4. Visibility of the program and community response

Probes: Is this program (highly) visible in the community? What is the level of community support for and/or opposition to the program from schools/school supervisors/community leaders? What are the sources of support for and/or opposition to the program from schools/schools supervisors/community leaders? Have you received any positive or negative messages about the program? Are there particular components of the program that are perceived positively or negatively by the community?

H. Services/Classes/Activities Provided to youth in the Control Group

Probes: What services, activities or classes were provided to youth who were assigned to the control group? Were these the same for all students or did some students do different things? Who provided or supervised these activities? Did they occur on the same schedule and for the same amount of time as the program sessions?

Abt Associates Inc. Appendix H: Discussion Guide for Use with COMMUNITY STAKEHOLDERS