Form 1 Interview Topic Guide

Regional Partnership Grants National Cross-Site Evaluation and Evaluation Technical Assistance

I Interview Topic Guide

Frontline Staff Individual Interview

OMB: 0970-0444

Document [docx]
Download: docx | pdf

OMB No.: 0970-0444

Expiration date: 03/31/2017

Regional Partnership grants (RPG) cross-site evaluation

TOPIC GUIDE FOR implementation study site visit interviews

The core implementation study for the Regional Partnership Grants (RPG) cross-site evaluation will include two multi-day site visits to each grantee. Researchers will interview grantee program directors, managers, supervisors, and frontline staff who work directly with families during the site visits. Interviews will be conducted either one-on-one or in small groups, depending on staffing structure, roles, and the number of individuals in a role.

Topic

Sub-Topic

Informant Characteristics

Informant Characteristics

Job title

Education background/licensing qualifications

Years in current position and with agency

Role on RPG

Relevant prior experience with the RPG grant program, target population, and evidence-based programs (EBPs) being implemented by the grantee

Pre-Implementation

Selection of EBPs

Grantees’ prior experience with similar programs and how prior experience informed the RPG design

Knowledge of evidence-based practices (EBPs)

Involvement of partners and other community organizations/stakeholders in the planning and decision-making processes, and how concerns were addressed

Involvement of frontline staff in the planning and decision-making processes, and how concerns were addressed

Key design decisions made during the planning phases and rationale for those decisions

Process by which grantees selected the planned interventions, including:

  • Community need to be addressed by the EBP

  • Needs and resource assessment (including need for and availability of: space, technology, financial and other resources, including in-kind contributions by grantee and/or partners)

  • Alignment with planned target population

  • Assessment of organization capacity/readiness

  • Whether other programs were considered

  • Champions for certain EBPs

  • Need for adaptation

  • Alignment with grantee and partners’ goals and mission

Challenges encountered during the planning process and steps taken to address them

Referral Processes to RPG services

How and when grantee determined referral pathways

Sources of referrals, length of relationship with these referral sources, and how relationships were established, relative size of enrollment from each referral source

Referral sources that consistently refer individuals that meet eligibility criteria and engage in the RPG program

Process used by other agencies to refer potential participants to RPG

Any changes to outreach and referral strategies and why

Barriers and facilitators to establishing pathways and translating referrals into participation

Sustainability of referral pathways

Staff Selection and Hiring and Retention

Staffing structure for the RPG program, including frontline staff and those who support their implementation (program directors, managers, and supervisors)

Responsibilities and expectations for each staff role

Timeline and process for hiring new staff or reassigning staff to fill RPG roles

Re-assignment of existing staff to implement RPG services and/or support implementation

Whether job postings specified specific qualifications required for implementation of EBPs selected; recruitment methods used to identify likely qualified candidates; protocols and criteria used to identify qualified candidates

Number of full-time equivalent (FTE) staff in RPG program devoted to administration and direct service provision

Extent to which staff in the RPG program had other responsibilities in addition to RPG

Current staff vacancies; length of vacancies; efforts to fill vacancies

Extent of staff turnover since initiating program operations; reasons for staff turnover (or staff retention); effects on remaining staff when turnover occurs; length of process to replace departing staff

Effect of staff turnover on enrollment and service delivery; programmatic adjustments and accommodations as a result of turnover

Efforts to prevent future turnover and retain current direct service staff and supervisors

Expectation of continued rate of turnover for sustainability

Likelihood of identifying individuals with necessary qualifications for sustainability and/or scale-up

Pre- and In-service Training

Plan for and approach to providing supervision and training to direct service staff, including the intended frequency, duration, and focus

Initial and in-service training plan for new and ongoing RPG program staff, including the frequency, content, length, and format of training, and individual or organization providing the training (includes whether the EBP’s developer or purveyor was involved in training, whether training covers key components of EBPs and whether trainees were given time to practice implementation with feedback)

Grantee’s ability to provide sufficient training to all necessary staff, at start-up and for sustainability and/or scale-up

Staff perception on extent to which training(s) provided necessary information on theory of intervention(s), goals of RPG, as well as competencies needed to implement

Whether staff received the planned level of initial and ongoing training and guidance

Implementation Teams

Organizational structure for the RPG project

Development of implementation team; timing of development, relative to program implementation

How grantee determined members of implementation team; qualifications established for team membership; member characteristics

Roles and responsibilities of team and its members

Strongest advocate for RPG program and how demonstrated; role of advocate and how individual emerged as advocate

Existence and role of advisory committee and/or steering committee

Duration of operation of implementation team; frequency of meetings; forms of communication by team members;

Turnover of team membership; reasons for turnover; impact on implementation

Barriers and facilitators to fully installing implementation team in RPG project

Accomplishments of implementation team

Staff perception of usefulness of team

Sustainability of team for scale-up

Implementation Plans

Development of plans and procedures used to ensure that all staff carry out program activities as planned and in a consistent manner; what details were included in plan (e.g. types of tasks, timeline for activities, staff responsible for tasks)

Modifications to the grantee’s RPG implementation plan that have occurred since implementation began; reasons for modifications; whether they were planned or unplanned

Development of strategies to address barriers to the program’s ability to deliver high-quality services

Staff perceptions of whether implementation plan was communicated sufficiently, executed successfully, and useful in proactively identifying roadblocks to implementation

Barriers and facilitators to success of implementation plan

Sustainability of implementation plan as RPG programs adapt

Early and On-Going Implementation

Facilitative administrative support

Grantee oversight of RPG activities and partner services

Changes in the demonstration’s organizational structure

Changes in grantee, partner, or RPG program leadership staff that occurred during the demonstration and may have impacted the direction of the RPG program

Strategies to reduce administrative barriers, develop communication and feedback protocols, implement program improvement based on data or staff suggestions

Staff perception of availability of these strategies

Staff perception of administration’s commitment to supporting the implementation of EBPs

Sustainability of leadership approach

Facilitators and barriers to providing administrative support

Supervision and Feedback

Whether protocols were established for providing feedback

Use of staff performance assessments for frontline staff

Sources of data for performance assessments

Facilitators and barriers to supervisory and feedback mechanisms

Technical Assistance and Coaching

Grantee and partners’ capability to provide ongoing TA for duration of RPG and beyond

Use of external TA providers, including curriculum developers, Children’s Bureau, and other entities; whether staff have accessed these resources and, if so, helpfulness of the technical assistance

Extent to which grantee believed TA providers were aware of and receptive to goals for RPG

Whether TA and coaching led to further adaptation of program model

Whether TA and coaching led to need for further training sessions

Topics on which staff needed more training and technical assistance

Plans for when, how, and why TA or coaching would be provided

Sustainability of TA and coaching networks, including financial and other resources

Internal evaluation and continuous program improvement

Grantee expectations about the quality of services delivered through RPG; how grantee defines high quality delivery for core services, and why program defines service quality in this manner

Efforts to monitor service quality, adherence to curricula or other programming, client engagement, participation, and participant outcomes; who completes monitoring; what is monitored and how often; how information is used by staff

Strategies for identifying successes and challenges to implementation for purposes of continuous program improvement

Use of improvement cycles or other continuous quality improvement strategies

Dissemination of RPG implementation to policymakers: frequency of exchanges, extent to which such exchanges are purposeful and part of usual practice

Dissemination of RPG implementation to partners: frequency of exchanges, extent to which such exchanges are purposeful and part of usual practice

Dissemination of RPG implementation to researchers, other practitioners: plans to publish findings

Facilitators and barriers to ongoing evaluation and program improvement

Sustainability of systems needed to monitor and improve program implementation

Decision Support Data Systems

Use of data systems to monitor progress toward goals and partner performance

Plan for monitoring program performance and for tracking service delivery and quality, adherence to curricula or other programming, client engagement and participation, and participant outcomes

Data sources and frequency of data collection and analysis

How staff use the data to make program decisions

Dissemination plans for: partners/stakeholders, administrators, support staff, frontline staff; perception of purpose of feedback dissemination

Staff perception of the relevance and usefulness of program data, management information system

Sustainability of data systems

Barriers and facilitators to using systems and conclusions derived from data

Referral Processes from RPG services

Types of community services to which RPG program staff refer participants

Extent to which needed services are available and accessible in the community

Plan for conducting initial and ongoing assessments of participants’ needs and linking them to appropriate services

Extent to which participants follow up on referrals and take up the services

Process for tracking referrals, how often progress is monitored, and who is responsible for monitoring

Interventions with external systems

Strategies to engage external systems in provision of financial, organizational, or other resources

Types of external systems/organizations engaged

Staff perception of alignment of organizations with grantee’s goals for RPG services

Adherence/Fidelity

Fidelity


How grantee defines high quality delivery of core components of the EBP, and why grantee defines quality in this manner

The extent to which staff adhere to the EBP guidelines (during service delivery)

Consistency with which services are provided, per EBP guidelines

Grantee expectations about the quality of services delivered through the EBP

Attitudes expressed by staff towards the use of the EBP

Extent to which program staff think that the EBP will improve outcomes

Staff understanding of EBP’s theory of change (how program services are linked to desired outcomes)

Staff Attitudes Toward Implementation

Staff perceptions as to whether these multiple roles had an effect on their ability to implement the program as designed

Extent to which RPG program staff “bought in” to the idea that providing substance abuse treatment, family strengthening, parenting education, and/or in an integrated package would improve participant outcomes

Staff perceptions of the EBP’s fit to the target population, strengths, and weaknesses

Staff perceptions about how well the model has worked in practice; benefits and challenges of this approach to demonstration leadership

Staff perceptions of the strengths and weaknesses of the demonstration’s approach to leadership

Staff perceptions of the utility of an integrated approach to the provision of core services

Community, State, and National Context

State and Local Context

State or local policies and policy climate, and how they impeded or supported program development

Other state or community organizations providing parenting or employment services; how the services provided by these organizations differ from the RPG program; whether and how these services may have affected the RPG program; and use of these other services by participants

Role of the courts and willingness of family court judges to support and participate in RPG

Physical, social, and economic characteristics of communities in which RPG is offered

Unexpected events that altered RPG program activities; how they affected the program and how they were addressed


THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) Public reporting burden for this collection of information is estimated to average of 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

2

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorSamina Sattar
File Modified0000-00-00
File Created2021-01-23

© 2024 OMB.report | Privacy Policy