Implementation Survey Interview Topic Guide (State-level Respondents)

Personal Responsibility Education Program (PREP) Multi-Component Evaluation

5 Instrument 1_InterviewTopicGuide_2014-08-19-14

Implementation Survey Interview Topic Guide (State-level Respondents)

OMB: 0970-0398

Document [docx]
Download: docx | pdf

PREP MULTI-COMPONENT EVALUATION omb supporting statement: INSTRUMENT 1


iNSTRUMENT 1

Implementation Survey Interview Topic Guide



















A. Introduction

Thank you for speaking with me today. As you know, my organization, Mathematica Policy Research is leading the Personal Responsibility Education Program (PREP) Multi-Component Evaluation on behalf of the Administration for Children and Families. The study’s purpose is to provide information to policymakers and administrators about how PREP-funded programs are operationalized in the field, to collect and analyze program performance measure data, and to assess the effectiveness of selected PREP-funded programs at achieving program goals.

As part of the PREP Evaluation, the study team is gathering information from four states to produce a detailed description of the structures and supports that are in place to assist in PREP program implementation. To achieve this goal, we are speaking with staff from various organizations involved with PREP program implementation and support to learn about the different structures and practices designed to support and monitor the PREP program.

The purpose of this interview is to collect information on supports for program implementation; it is not an assessment or monitoring effort. Everything that you say will be kept strictly private within the study team and participation is voluntary. The study report will include a list of the states that participated in these interviews, but all interview data will be reported in the aggregate. In our reports, we will not identify a specific person, program, or state unless we are highlighting a promising practice. If you do not feel comfortable answering a particular question, please let me know and we will move on to the next one.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0398 and it expires XX/XX/2017.

This discussion should take about one hour. Do you have any questions before we begin?

[The research team will not ask all questions of all respondents. For example, some questions pertain only to the work of certain staff (such as the state agencies responsible for overseeing the PREP program, and so on). Therefore, interviews will also be customized based on each respondent’s job duties and knowledge.]



















B. Topic Guide for Telephone Interviews in Four Grantee States

Construct

Interview Topics

PREP Structure and Planning

Implementation structure

Characteristics of organizations involved in PREP implementation, such as mission and length of operation

Changes in staffing structures and responsibilities for the PREP program, by organizational affiliation, since initial design and planning interviews (summer of 2012)

Modifications or adaptations to PREP implementation plans that has occurred and why

Potential future modifications or adaptations to implementation plans based on experience, and why

Model fit for service providers

Lessons learned in the selection of PREP models at state grantee and provider levels

Lessons learned in the selection of providers to deliver PREP services

Level of experience (and success) in delivering the selected model by providers prior to PREP; extent of similar in any challenges experienced prior to and during PREP implementation

Level of experience delivering teen pregnancy prevention programs

Level of experience in serving the targeted population for PREP

Successes and challenges in transitioning to PREP implementation

Organizational characteristics and role in community and how characteristics and role provide advantages or challenges in delivering PREP services

Perceived degree of overall “fit” of PREP model(s) within local provider organization and community (the degree to which the model aligns with the organization’s objectives and service capacity)

Lessons from PREP planning

Time from planning to early implementation and factors that determined the timeframe

Lessons learned during planning and early implementation of PREP that informed program refinement

Retrospective view of approaches to PREP planning (and potential differences in decisions or process) based on experience

PREP Program Support: Training and Technical Assistance

Implementation support structure

Changes in the role of state grantee agency in supporting implementation; number and responsibilities of staff supporting PREP implementation since initial design and planning interviews (summer of 2012)

Use and number of intermediary organizations to support PREP training and technical assistance (T/TA); number and responsibilities of staff supporting PREP implementation

T/TA provider’s experience with and exposure to the PREP model they are supporting; experience working with grantee organization, and service providers implementing PREP

Role of PREP model developer(s) in supporting implementation

Strengths and weaknesses to the T/TA approach pursued to support PREP implementation

Adequacy of resources to support T/TA and changes in funding for T/TA over time

Training to support PREP

Training or guidance from model developer that informed implementation support for PREP

Provision, duration, format, and frequency of training for T/TA providers to prepare them to support PREP

Provision, duration, format, and frequency of training for facilitators and supervisors to prepare them to deliver PREP

Specific requirements or benchmarks that facilitators must meet to deliver PREP sessions

Sufficiency of training in preparing staff to deliver or support PREP, or what additional training may be needed

Helpfulness of training to prepare staff to deliver or support PREP

Successes, challenges, lessons learned about training

Ongoing support and technical assistance

Use of data or other methods to inform TA and implementation support needs (to include federal performance measures, fidelity data, supervision and performance assessment, state grantee activities)

Initiation, frequency, content, format and level of satisfaction of TA received from the model developer or national model purveyor

Initiation, frequency, content, format and level of satisfaction of TA received from the state grantee agency

Initiation, frequency, content, format and level of satisfaction of TA received from an organization designated to support PREP in the state (or from any other entities)

Timing and means of first introduction of TA provider to program providers

Level of satisfaction with ongoing level of communication between TA provider and program providers

Feedback loops following TA (information to state grantee agency; feedback from program provider to TA provider about usefulness of TA; follow-up from TA provider with program provider to track or assess use of TA in practice)

Areas of unmet needs for technical assistance or ongoing implementation support

Key successes in technical assistance for the PREP program

Implementation Drivers: Competency Drivers

Staff selection

Changes needed in organizational staffing to implement PREP; whether new roles or new staff were needed

Skills, abilities, and other qualifications sought for PREP staff; whether these differ for existing staff or new hires

Whether guidance was received to specify the qualifications for PREP staff

Methods for communicating expectations for PREP during recruitment or hiring process

Ease or difficulty in hiring staff with the qualifications for PREP; successful methods for getting staff in place

Qualifications of current PREP facilitators and consistency in qualifications across all facilitators

Key lessons from the initial staff selection and recruiting process

Staff turnover and retention

Degree of turnover among PREP staff at all levels

How the turnover for PREP compares to other programs within organization

Main reasons for degree of turnover

Methods and timing of filling vacated positions

Operational challenges due to staff turnover

Strategies for retaining staff and preventing future turnover

Successes and challenges in staffing the PREP program

Staff expectations and receptiveness

Methods of orienting staff to the requirements and expectations of the PREP program

Receptiveness of staff to use of formal curriculum and specifications of PREP

Aspects of PREP that staff are resistant to

Perceived level of consistency in which PREP staff (particularly facilitators) deliver PREP

Staff supervision and performance assessment

Expectations for supervision or oversight of PREP program facilitators; frequency and methods for supervisory activities

Degree to which expectations for supervision are met

Common concerns or issues raised by facilitators during supervisory activities and ability of supervisors to address them

Methods of performance assessment of PREP facilitators; use of observations, intended frequency, and focus

Feedback loop following performance assessment; types of discussions, usefulness for staff in improving skills and fidelity to the PREP model

Use of performance assessment information by administrators to support PREP implementation to improve practice and fidelity

Implementation Drivers: Organization Drivers

Decision-support data systems

Data collected (beyond performance measures) to inform or support PREP implementation; what is collected, why, and how

How data are used to support PREP implementation and benefits from use of data

Specific ways in which data has informed changes or improvements to PREP implementation or services

Procedures for promoting timeliness and consistency in data entry related to PREP implementation; frequency or timing of data entry after a PREP activity

Documentation, guidance, or training for data entry and use

Type of data system used to track PREP implementation and support

Perceived quality of data collected for the PREP program

Facilitative administration

Shared understanding among administrators and staff within organization about the goals of the PREP program and what it means to effectively implement an evidence-based program

Presence of champion within organization who promotes shared understanding of PREP goals and expectations

Methods of developing a shared understanding (or buy-in) of PREP among PREP staff as well as among other staff in the organization

Resistance to PREP within the organization

Perception of role of state grantee agency as a partner in supporting PREP implementation

Perceptions of commitment of each implementation partner in supporting the level of staffing, training, and ongoing assistance necessary to implement PREP

Willingness to adjust or develop new policies and procedures to support PREP model

Methods to identify and reduce administrative or operational barriers to PREP implementation

Communication, decision-making, and feedback loops

Description of lines of communication between service delivery staff and management within program provider organizations to facilitate or support PREP implementation

Participation of service delivery staff in developing or modifying procedures or processes or identifying implementation challenges; formality of their involvement and regularity of consultation or feedback

Examples, if applicable, of policies, procedures, or processes that have changed within program provider organizations as a result of service delivery staff input

Description of frequency and mode of communication between state grantees and program providers to support PREP implementation, and changes in this communication over time

Examples, if applicable, of policies, procedures, or processes that have changed at the state grantee level as a result of program provider input

Means of identifying issues or problems in PREP implementation across all entities involved in service delivery and support, and methods of developing strategies across entities to address issues

Systems interventions

Methods of connecting with other systems or partners to support PREP implementation at the state and local levels

Change in partners over time

Capacities that partners bring or help build to support PREP implementation

Identification of barriers or problems across partners and methods of addressing

Ability to gain buy-in and cooperation with organizational partners and within the community at large to support the PREP model

Resistance to the approach or content of the PREP program within the state/community or from particular organizations such as schools or other community organizations; ways this has affected implementation

Changes in existing service delivery structures across entities or within communities to support PREP implementation

Perception of PREP program contribution to infrastructure to support youth education programs to prevent teen pregnancy in state/community

Fidelity Assessment and Monitoring

Program modifications or adaptations

Description of PREP model modifications or adaptations that were implemented, including type and reasons needed, degree of adaptations and use of existing adaptation kit to guide adaptations

Process for obtaining approval of modifications or adaptations and entity that had final approval

Process for rolling out modifications or adaptations (if made after implementation began); and process for monitoring implementation of modification or adaptation

Collaboration with model developer, state grantee, or T/TA provider to inform modification or adaptation

Plans for additional modifications or adaptations to the PREP model

Type and degree of unplanned model adaptations that occurred (and why)

Guidance and support provided by state grantee or T/TA provider with regard to implementation of modifications or adaptations

Adherence to service model

Methods that state grantee agency uses to convey expectations regarding quality and consistency of PREP services

Efforts to ensure that all staff carry out program activities in a consistent manner

Obstacles that impede the delivery of consistent services across program staff; strategies to remove obstacles

Access to materials (such as curriculum, supporting materials, procedures manuals) to support ongoing program delivery and address questions or issues related to practice among PREP staff

Concerns by program model developer about the ability of program providers to provide the PREP model as designed

Easiest and most challenging aspects of the PREP program to implement

Monitoring service delivery

How high quality delivery of PREP services is defined, and by whom

Use of existing fidelity indicators (from model developer) or extent of development of indicators such as fidelity monitoring logs

Ways that fidelity data are used to support or improve implementation and by whom

Aspects of implementation that are monitored on a regular basis, with what frequency, how monitored, by whom, for what purpose, to whom reported

Methods of collection of fidelity data (self-report, observation, data system)

Documentation and tracking of planned and actual content and activities of each PREP session, and adaptations made

Methods of assessing quality of PREP sessions, frequency, by whom, and how is information used

Changes in PREP service delivery due to monitoring efforts

Lessons learned, successes or challenges identified through monitoring

Evaluation, Sustainability, and Lessons Learned

Evaluation capacity

Evaluation activities focused on PREP program

Role of external entities in monitoring and evaluating PREP implementation; number and type of organizations and staff involved

Methods of communicating evaluation information to PREP stakeholders at the state and local levels

Methods of using evaluation information to improve PREP implementation

Key findings from evaluation that are currently available

Level of interaction and communication between evaluators and state grantee agency, program providers and T/TA providers

Sustainability

Sustained readiness of organization (financial, organizational, logistical capacities) to implement PREP

Efforts to sustain the implementation infrastructure and funding to deliver PREP services after the federal grant period ends

Level of coordination in efforts to sustain the PREP program across partners at the state and local level

Opportunities and challenges in ability to sustain the implementation or supports for the PREP program

Perceptions and lessons learned about supporting PREP implementation

Opinion of current number and type of PREP program models as right for organization’s ability to implement (or support) PREP with quality and fidelity

Opinion of current number of program providers as right for organization’s ability to implement (or support) PREP with quality and fidelity

Factors that facilitate ability to implement PREP

Challenges that affect ability to implement PREP and how they have been addressed

Based on experience, how might the organization have done things differently to implement or support PREP

Advice for other organizations implementing evidence-based teen pregnancy prevention programs

Advice for other state and federal agencies about how to support evidence-based programs generally or PREP specifically





I1.4

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorGretchen Kirby
File Modified0000-00-00
File Created2021-01-21

© 2024 OMB.report | Privacy Policy