Instrument #8: Semi-structured interview topic guide

PACT – Site Visit Topic Guide OMB #: 0970-0403

EXPIRATION: XX/XX/201X

Parents and Children Together (PACT) Evaluation

INSTRUMENT (8)

SEMI-STRUCTURED INTERVIEW TOPIC GUIDE

(FOR PROGRAM STAFF)

Parents and children together (PACT)

TOPIC GUIDE FOR implementation study site visit interviews

The Parents and Children Together implementation study will include two multi-day site visits to each participating RF/HM program. Visits will occur in 2013 and 2014. Administrative and supervisory personnel and staff who have direct interaction with participants will be interviewed during both site visits. Interviews will be either one-on-one or small group, depending on staffing structure, roles, and the number of individuals in a role.

Introductory script:

Purpose and use of information: A key objective of the PACT Evaluation is to learn about the organizational context, design, and operations of healthy marriage and responsible fatherhood programs awarded grants from the Office of Family Assistance within the Administration for Children and Families (ACF). The purpose is to provide information to practitioners and policymakers within multiple levels of government as well as in other sectors. The information collected will be used to document the actual operational experiences of agencies and participants over the course of the grant period.

Authority of collection: Mathematica Policy Research is conducting the PACT Evaluation on behalf of ACF. ACF obtained Office of Management and Budget approval for the collection of this information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it has a current valid OMB control number. The number for this study is 0970-0403. If you have any questions, please contact [CONTACT NAME], at xxx-xxx-xxxx, regarding OMB-PRA (0970-0403).

Privacy: The information you provide will be combined with that of other respondents and discussed internally among the evaluation team and with federal staff at ACF. Descriptive information about programs may be included in reports, but no individuals’ identifying information will be disseminated publicly.

Voluntary Participation: Providing information for the purposes described is voluntary.

Estimated time: This discussion is expected to take about an hour.

Construct	Site Visit Interview Topic
Input: Program Logic Model
Design Process	Why organization applied for current ACF grant Design and planning process for RF/HM program and its implementation, including changes if program is a continuation of past efforts Key issues decided during RF/HM program design and planning phase
Target Population	Characteristics of individuals or couples the RF/HM program intended to serve Eligibility criteria for participants in the RF/HM program Anticipated strengths and needs of program participants
Recruitment and Retention Plan	Enrollment targets Planned outreach and recruitment strategies to identify potential RF/HM program enrollees and how strategies are expected to help meet established enrollment targets Intended geographic area for recruiting potential RF/HM program enrollees Planned activities or strategies to encourage initial and ongoing participation in RF/HM program services
Intervention Plan	Services the program initially planned to provide: Parenting education Healthy marriage and relationship skills education Economic stability or job and career advancement services Planned curricula for parenting, healthy marriage, and economic stability services Curriculum name, author, extent of use Modules or topics to be provided Intended format, length, frequency, and intensity of services Planned level of service standardization across participants Plan for who would deliver curricula Additional services to be available to participants within RF/HM program and through external service providers; anticipated level of need for these supplementary services Plan for conducting initial and ongoing assessments to assess participant needs and how participants would be linked to services Services anticipated to most contribute to change in participants’ lives
Plan for Involving Organizational Partners	Organizations outside of the grantee organization that were expected to be involved in service delivery Expected role of each outside organization in RF/HM program service delivery; why organization was selected for their role
Intended Outcomes	Goals for the RF/HM program Expected benefits for participants, how RF/HM program would contribute to these benefits, and anticipated length of program participation needed to reach benefits How program planned to determine whether participants obtained expected benefits
Input: Implementation Drivers
Organizational Characteristics	Mission and goals of organization leading the RF/HM grant Services provided by organization leading the RF/HM grant, other than RF/HM services Population served by organization leading the RF/HM grant, other than RF/HM population RF/HM grantee organization’s prior experience and investment in delivering parenting, healthy marriage, and economic stability services Coordination between the RF/HM grantee organization and other community organizations that provide either parenting, healthy marriage, or economic stability services
Leadership/ Management/ Champion	Organizational structure for the RF/HM grant program and why program structured in this way Decision-making process and extent to which decision-making authority for RF/HM program is shared across organizational leadership, program management, and supervisors or direct service staff Strongest advocate for program and how demonstrated; role of advocate and how individual emerged as advocate How RF/HM program helps to advance the lead organization’s goals and vice versa
Staffing Structure/ Supervisory Model	Staffing structure for the RF/HM program Responsibilities and expectations for each staff role Organizational affiliation of each person providing direct services for the RF/HM program; expected means of communication/coordination across all involved organizations Plan for providing supervision to direct service staff, including the intended frequency, duration, and focus
Service Delivery Supports	Training plan for new and ongoing RF/HM program staff, including the frequency, content, length, and format of training, and individual or organization providing the training Technical assistance or guidance available to RF/HM program staff from curriculum developers, OFA, ICF, and other entities to support program services; whether staff have accessed these resources and, if so, helpfulness of the services Plan for monitoring program performance and for tracking service delivery and quality, adherence to curricula or other programming, client engagement and participation, and participant outcomes
Output: Program As Implemented
Recruitment and engagement strategies	Outreach and referral Outreach strategies to identify potential participants for the RF/HM program; success/challenges of each strategy; and why more successful strategies are working Organizations and individuals involved in outreach and their roles Sources of referrals, length of relationship with these referral sources and how relationships were established Referral sources that consistently refer individuals that meet eligibility criteria and engage in the RF/HM program Process used by other agencies to refer potential participants to the RF/HM program Screening and enrollment Enrollment process for new participants in the RF/HM program, including process for screening potential participants for program eligibility and interest Challenges encountered with enrollment process; attempted strategies to address challenges Process for tracking progress in (1) outreach and referral, and (2) enrollment, including whether there are weekly or monthly goals, how often progress is monitored, and who is responsible for this monitoring
Frequency, intensity, and type of services offered	Actual services provided through RF/HM program related to parenting, healthy marriage, and economic stability; their sequencing, and how program determines which services participants receive and at what intensity Format, length, frequency, and intensity of parenting, healthy marriage, and economic stability services Curricula used Modules or topics provided Any adaptations made to curricula and why Level of service standardization across participants and how determined Frequency, mode, and purpose of contact between program staff and participants; how these vary across program staff within the same position and across positions Frequency, mode, and purpose of contact between program staff and other service providers on behalf of the client Initial and on-going assessments of fathers and/or couples to assess needs; formal instruments or tools used; which staff complete assessments; how assessments are used to guide service delivery and link participants to outside services Additional services available to participants within RF/HM program and through external service providers; whether participants with specific needs receive assistance for these needs from either the RF/HM program or an external service provider Challenges connecting to and engaging participants in services provided by external organizations; attempted strategies to address challenges, and success of these strategies
Adherence to implementation plans	Efforts to ensure that all staff carry out program activities in a consistent manner Barriers that impede the delivery of consistent services across program staff; how managers and staff try to address these factors Whether staff received the planned level of initial and ongoing training and guidance; adequacy of training and guidance in preparing and supporting staff to provide services; further training needs Efforts to track service delivery and adherence to curricula or other programming, and whether it aligns with intended plan; who completes tracking; what is tracked and how often; how information is used by staff
Modifications to planned implementation	Modifications or adaptations to the RF/HM program’s implementation plan that have occurred; timeline for and monitoring of roll-out of modifications or adaptations; why program implemented these modifications/adaptations; how changes affected service delivery Potential future modifications or adaptations to the implementation plan; how potential modifications or adaptations could improve service delivery; timeline for roll-out of these modifications or adaptations Unplanned modifications or adaptations to the implementation plan; how these came about; whether they became long-term changes; how changes affected service delivery
Quality of service delivery	Expectations by program management for high quality delivery of RF/HM program services; how program defines high quality delivery for parenting, healthy marriage, and economic stability services and why program defines service quality in this manner Efforts to monitor service quality, adherence to curricula or other programming, client engagement, participation, and participant outcomes; who completes monitoring; what is monitored and how often; how information is used by staff
Integration/ support of organizational partners	For organizations that collaborate with the RF/HM program on program activities, including the actual delivery of services to program participants: how and why these organizations are involved; contractual relationship with RF/HM program; when the grantee formed a relationship with each organization; and how the relationship developed and collaborative efforts How collaborations with community organizations support the goals of the organization hosting the RF/HM program and the RF/HM program itself; whether and how collaborations are groundbreaking for either the organization or community Community organizations that the organization leading the RF/HM grant has stopped working with; when collaboration ended and why; how ending this collaboration affected the RF/HM program
Attainment of program targets	Average number of fathers and/or couples that are enrolled each month in the program; whether the RF/HM program is meeting enrollment targets, what accounts for shortfalls (if occurring) Enrollment of fathers and couples into each component of the RF/HM program; whether enrollment in specific components aligns with expectations and what accounts for differences that may be observed How participants have benefited from program services that may suggest the RF/HM program is on the path to achieving its identified goals Challenges that have hindered how well the program operates, including challenges that affect the program’s ability to deliver high-quality services, obtain high levels of client engagement and participation, and achieve intended participant outcomes; strategies to address challenges Natural or man-made disasters or unexpected events that altered RF/HM program activities; how they affected the program and how they were addressed
Staff characteristics, retention, turnover	Type and level of education, experience, background, and characteristics of each direct service, supervisory, and management staff position; type and level of education, experience, background, and characteristics sought when hiring or assigning existing staff to each direct service, supervisory, and management position Extent to which RF/HM program staff “bought in” to the idea that providing parenting, healthy marriage, and economic stability services are valuable to fathers and/or couples Current staff vacancies; length of vacancy; efforts to fill vacancy Extent of staff turnover since initiating program operations; reasons for staff turnover (or the lack thereof); effects on remaining staff when turnover occurs; length of process to replace departing staff Effect of staff turnover on enrollment and service delivery; programmatic adjustments and accommodations as a result of turnover Efforts to prevent future turnover and retain current direct service staff and supervisors How supervisors assess and provide feedback to staff on their work; types and frequency of supervision provided to direct service staff; focus of supervision Common concerns discussed during supervision meetings; how supervisors address concerns raised by direct service staff Training for and resources available to supervisors to address concerns raised by direct service staff; further training needs of supervisors
Characteristics of enrolled parents	Characteristics of individuals or couples enrolled in the RF/HM program Strengths and needs of individuals or couples enrolled in the RF/HM program
Output: Participant Responsiveness
Enrollment and motivation for enrolling	How fathers and/or couples came to be in the RF/HM program Reasons that fathers and/or couples enrolled in the RF/HM program Expectations of fathers and/or couples for RF/HM program Factors that most often make a person or couple ineligible for the RF/HM program How fathers and/or couples respond to RF/HM program services
Program Participation	Activities or strategies used to encourage initial and ongoing participation; perceived success of each strategy and activity at encouraging early and sustained attendance; new strategies under consideration to encourage participation Incentives to encourage fathers and/or couples to participate in RF/HM activities initially and on an ongoing basis; when incentives are provided; and which incentives appear to most promote participation Point at which, and reasons that, RF/HM program staff decide a nonparticipating father or couple is unlikely to ever attend (and cease attempts to encourage them) Reasons provided by fathers and/or couples for never attending or discontinuing participation in the RF/HM program
Knowledge and skill acquisition	Examples of key skills or information that participants have learned; demonstration of specific skills and information taught in RF/HM program
Community Context
Socio-demographic and economic profile	State or local political climate and how it impeded or supported program development Other community organizations providing parenting, healthy marriage, or economic stability services; how the services provided by these organizations differ from the RF/HM program; whether and how these services may have affected the RF/HM program; and use of these other services by fathers or couples Physical, social, and economic characteristics of neighborhoods and communities in which the RF/HM program is provided

4