Topic guide for staff and stakeholder interviews

OPRE Evaluation: Strengthening Relationship Education and Marriage Services (STREAMS) Evaluation [Impact and Process Studies]

Instrument 1_STREAMS_Site visit topic guide_2-26-16

Topic guide for staff and stakeholder interviews

OMB: 0970-0481

Document [docx]
Download: docx | pdf



INSTRUMENT 1

TOPIC GUIDE FOR STAFF AND STAKEHOLDER INTERVIEWS





This page intentionally left blank for double sided copying

STREAMS

TOPIC GUIDE FOR PROCESS STUDY STAFF AND STAKEHOLDER INTERVIEWS

The STREAMS process study will include semi-structured interviews with staff and stakeholders for the HMRE programs selected to participate. This topic guide identifies the range of topics that will be covered during interviews. The guide will be used to develop respondent-specific guides that include only topics relevant to particular types of staff and stakeholders. Respondent-specific guides will also convert topics to questions that will be asked during interviews.

Most interviews will be conducted during a multi-day site visit to each participating program. Visits will occur approximately half-way through enrollment for STREAMS. Administrative and supervisory personnel and staff who have direct interaction with participants will be interviewed as well as representatives of partner agencies and other stakeholders. Interviews will be either one-on-one or small group, depending on staffing structure, roles, and the number of individuals in a role. Interviews will last one hour, on average.



OMB #:XXX-XXXX

EXPIRATION: XX-XX-XXXX



Strengthening Relationship Education and Marriage Services (STREAMS)

Staff and Stakeholder Interviews


GENERAL INSTRUCTIONS

Thank you for speaking with me today. On behalf of the U.S. Department of Health and Human Services (HHS), we are interviewing staff at each STREAMS evaluation site. The purpose of this interview is to learn how your HMRE program operates, your role in the program, and your experiences with the program. Your participation will help HHS learn how to better provide HMRE services.

Your name will not shared in our reports or any other publications resulting from this study, and your responses will remain private to the extent permitted by law. We want you to know that:

1. Your participation in this interview is voluntary.

2. The interview will take about 60 minutes to complete.

3. We hope that you will answer all the questions, but you may skip any questions you do not wish to answer.

4. The answers you give will be kept private to the extent permitted by law.

Shape1

THE PAPERWORK REDUCTION ACT OF 1995

Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The information requested in this study will be used to document how programs receiving HMRF grant funding operate and describe participant outcomes. The data gathered will allow ACF to better monitor grantee progress and performance. In accordance with the requirements of the Privacy Act of 1974, as amended (5 U.S.C. 552a), ACF/OPRE established system of records titled: 09-80-0361 OPRE Research and Evaluation Project Records, HHS/ACF/OPRE. A Federal Register Notice (80 FR 17893) announced the system.





Construct

Site Visit Interview Topic

Implementation Drivers: Competency Drivers

Staff Recruitment and Selection

Specification of staff skills and abilities and selection criteria

Required staff skills and abilities; pre-requisites for employment

Extent to which job descriptions define required skills, abilities, and staff expectations

Prioritization of skills and abilities when selecting candidates

Method for recruiting candidates

Process for identifying candidates to fill program positions

Sources for advertising open positions

Challenges recruiting candidates

Staff responsible for recruitment and selection

Lead person responsible for coordinating staff recruitment and selection

Challenges coordinating staff recruitment and selection

Process and protocols for interviewing candidates

Process for identifying interviewers who can assess applicant knowledge, skills, and abilities

Interview protocol content, including:

  • Assessment of required skills and capacity to complete job responsibilities

  • Assessment of applicant’s ability to receive feedback professionally and use it for improvement

Process for monitoring adherence to the interview protocol

Extent to which program uses training, turnover, and fidelity data and exit interviews to assess the effectiveness of staff selection

Training

Focus of content covered during training and regularity of training

Training content, including:

  • History, theory, philosophy, and values of the program

  • Instruction on delivering curricula

  • Opportunities to practice new skills (role plays, teach backs) and receive feedback

Timing and frequency of initial and on-going training; availability of booster training sessions

Staff or agencies responsible for ensuring implementers receive training

Lead person responsible for coordinating training

Challenges to ensuring regular training occurs

Identifying trainers; trainer qualifications

Collection and use of training data

Collecting training assessment data (pre-post tests)

Sharing training assessment data with supervisors and coaches to support trainee competency development

Soliciting feedback from trainees to improve future training activities

Coaching/Supervision

Development and use of a coaching/supervision plan

Developing staff coaching or supervision plans

Content of coaching or supervision plans, including:

  • Coaching/supervision requirements

  • Coaching/supervision frequency

  • Coaching/supervision methods

  • Coaching/supervision plan review

Staff responsible for providing coaching/ supervision

Lead person responsible for coordinating coaching/supervision

Selection of coaches/supervisors; required skills and abilities

Challenges coordinating coaching/supervision, identifying qualified coaches/supervisors, and providing coaching/supervision

Process for assessing coaching/supervision effectiveness and modifying coaching/supervision plan

Monitoring performance of coaching/supervision recipients (i.e. direct observations, video or audio recordings, feedback from colleagues and other stakeholders)

Obtaining feedback on coaching/supervision (i.e. satisfaction surveys of coaching recipients; coaching observations)

Providing feedback to coaches/supervisors

Using monitoring data and feedback to improve coaching/supervision

Fidelity Assessment

Development and use of plan and protocol for monitoring fidelity of service delivery

Protocol for assessing fidelity to HMRE curricula including:

  • Data sources and methods for assessing fidelity

  • Content of fidelity measures; assessment of content, staff competence to deliver content, and implementation context

  • Frequency of fidelity assessments

Use of fidelity data to improve curriculum delivery

Staff involved in developing fidelity assessment protocol and measures

Reviewing protocol for assessing fidelity

Staff responsible for assessing fidelity

Lead person responsible for coordinating fidelity assessments

Selection of staff to conduct fidelity assessments; required skills and abilities

Challenges coordinating and conducting fidelity assessments

Agency support for using fidelity protocol

Extent to which the agency supports the use of consistent fidelity measures

Implementation Drivers: Organization Drivers

Decision Support Data Systems

Use of data systems to support and monitor intervention delivery and fidelity, and assess overall program performance

Existing data systems prior to STREAMS evaluation

Data elements captured in pre-existing and nFORM systems:

  • Outcome measures

  • Participation data

  • Fidelity measures

  • Cost data

How data system supports decision making

Extent to which data collected are:

  • Reliable (standardized protocols, trained data collectors)

  • Valid (useful indicators of the concepts or practices being assessed)

  • Reported in a timely manner (when/to whom the data are most useful)

  • Built into regular practice routines

Extent to which agency staff have access to the data

Staff responsible for managing data system

Lead person responsible for coordinating content, quality, and timeliness of data collection

Selection and training of data entry staff

Challenges with data collection, data entry, and data system management

Process for using data to support decision making

Staff that receive data reports and their use for program improvement

Frequency and content of data reports

Usability of data reports

Sharing data reports and program improvement plans with key stakeholders

Facilitative Administration

Composition and role of leadership team

Leadership team within agency to guide HMRE program; how identified

Role of leadership team

Program leaders actively facilitate use of implementation supports

Extent to which leaders and managers make changes in organizational roles, functions, and structures to facilitate implementation

Extent to which leaders and managers make changes to organization policies and procedures to facilitate implementation of HMRE program

Extent to which leaders and managers use data to inform decisions and planning

Process for facilitating meetings

Use of meetings within the organization and HMRE program for program management

Staff roles and responsibilities during meetings

Process for ensuring meeting follow-up is completed and progress continues within designated timelines

Process for receiving and using feedback from staff, participants, and stakeholders

Feedback from staff, stakeholders, and program participants on program management and operations; frequency of feedback

How leaders and managers use feedback for program improvement

Systems Intervention

Engagement of larger delivery and funding systems to encourage program sustainability

Frequency and focus of meetings with ACF, other funders, and other service organizations

Sharing information about facilitators and barriers to program quality with ACF, other funders, and other service organizations

Engagement of stakeholders and partners to support program

Development and use of communication plans with key stakeholders, including schools, agencies serving at-risk youth, referring organizations, and other partner agencies

Challenges with stakeholder communication

Process for identifying partner organizations; role of organizations

Partner roles; criteria and process for selecting partners

Challenges in the selection of and work with partners

Factors that facilitate the selection of and work with partners

Strategies to obtain buy-in for program among stakeholders

Assessing stakeholder support for the HMRE program

Building stakeholder support for HMRE program

Implementation Drivers: Leadership Drivers

Identification of contextual issues related to implementation

Staff responsible for identifying community needs and how HMRE program can address these needs

Building community support for HMRE program; community barriers to and facilitators for implementation

Identification of barriers and challenges to implementation and staff involvement

Key challenges to program implementation; how addressed by program staff

Support from agency leaders to address implementation barriers, including allocation of needed time and resources

Communication process for raising and addressing implementation challenges

Decision-making authority to address implementation barriers

Fidelity of Implementation: Consistency

Regularity of offering HMRE curriculum

Frequency, format, length, and intensity of HMRE curricula offered through individual sessions and full curriculum

Formats used for delivering curriculum; reasons for offering multiple formats

Challenges and successes with using different formats for delivering curriculum

Process and frequency of starting new cohorts

Challenges with offering curriculum with intended regularity and starting new cohorts

Regularity of content delivery included in HMRE curriculum

Plan for providing all content specified in HMRE curriculum

Process for tracking whether facilitator delivers all content

Challenges with delivering all content

Availability of ancillary services within the HMRE program

Types and providers of ancillary services available to program participants

Frequency of participant engagement in ancillary services

Monitoring participant take-up of ancillary services

Frequency, mode, and purpose of contact between program staff and other service providers on behalf of the client

Challenges connecting participants to and engaging participants in ancillary services; attempted strategies to address challenges and success of these strategies

Need for adaptation of HMRE curriculum or ancillary services; adaptations (if made)

Process for identifying needed adaptations to HMRE curriculum or ancillary services

Working with curriculum developers to make adaptations; extent to which curriculum developers are included in the adaptation process

Adaptations made to HMRE curriculum and ancillary services; rationale for changes

Lessons and challenges offering HMRE curriculum and content and ancillary services

Challenges to providing HMRE curriculum as designed by curriculum developer; strategies for overcoming challenges

Challenges to providing supplemental HMRE content (parenting skills, financial management, conflict resolution, and job and career development) and ancillary services; strategies for overcoming challenges

Fidelity of Implementation: Satisfaction

Buy-in for HMRE program

Alignment between leadership and front-line staff on strategies for improving participant outcomes

Views about how HMRE program advances organization’s goals

How program components seem to contribute to achieving expected outcomes

Gaps in services that may prevent participants from achieving expected outcomes

Enthusiasm for providing HMRE curriculum

Perceived importance of HMRE services to achieving expected outcomes

Willingness of agency staff to implement program services

Enthusiasm for providing ancillary services

Perceived importance of ancillary services to achieving expected outcomes

Willingness of agency staff to provide or refer participants to ancillary services

Fidelity of Implementation: Quality

Receipt of initial and booster trainings on HMRE curriculum

Receipt of initial and ongoing training and guidance on HMRE curriculum

Access to or attendance at initial and ongoing trainings

Perceived benefit of initial and booster trainings

Participation in coaching or support systems related to curriculum delivery

Availability of and participation in coaching meetings and other support activities

Perceived availability of coaches and supervisors to provide support to program staff

Perceived benefit of coaching meetings and other support activities

Knowledge of HMRE curriculum

Identification of core HMRE curriculum components such as topics and activities

Ability to articulate program goals and goals of HMRE curriculum

Understanding of how HMRE curriculum aligns with agency goals and mission

Knowledge of community resources available to participants requiring additional support

Ability to identify community resources to which participants can be referred

Participant Responsiveness

Enrollment and motivation for enrolling

How participants learn of and come to be in the program

Factors that most often make a participant ineligible for the program

Participation rate for curriculum sessions, ancillary services, and entire program

Challenges to participants attending HMRE sessions at expected frequency

Challenges to participants receiving ancillary services

Challenges to participants receiving any services offered by the program

Most common barriers to participation

Strategies to support participation and reduce barriers

Average dosage for curriculum sessions, ancillary services, and entire program

Definition of program completion

Challenges to participants receiving the intended dose of the HMRE curriculum and other services

Strategies to support program completion

Satisfaction with program services

Services participants seem to respond to the most

Services participants seem to respond to the least

Services that are challenging to get participants to engage in

Knowledge and skill acquisition

Observed changes in participant behavior that indicates acquisition of new knowledge or skills

Whether pre-and post-tests indicate acquisition of new knowledge and skills

Examples of key skills or knowledge that participants have gained; demonstration of specific skills and information taught in the program

Community Context

HMRE program awareness

Extent to which this HMRE program is known in the community

Availability of other HMRE or similar programming

Whether there are similar services available in the community (i.e. availability of other HMRE or related curricula through other organizations)

Whether other community agencies offer similar ancillary programming in the community



File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorHeather Zaveri
File Modified0000-00-00
File Created2021-01-15

© 2024 OMB.report | Privacy Policy