Program Manager Group Interview 12 Month

OMB Control No: _____

Expiration Date: ______

Length of time for instrument: 1.50 hours

ATTACHMENT 21: MIHOPE PROGRAM MANAGER GROUP INTERVIEW_12 MONTH

5/29/2012

Program Manager Group Interview - 12 Month

Protocol

Instructions: The following request will be sent to local program managers about one month before the site visit:

Before we meet to discuss your MIECHV program, please provide us with information about program costs for the program’s most recent fiscal year. Please provide the information about aggregate program costs broken out by your agency’s usual cost categories. If possible, we would like aggregate cost information for personnel, equipment and supplies, space and utilities, and direct support and benefits to clients. However, if your agency breaks out costs in different categories, we will accept those categories. We would also like back up information about what goes into each cost category, including salary levels for home visiting program staff.

Thank you very much for agreeing to participate in this discussion. Your participation is very important to the Maternal Infant Home Visiting Program Evaluation (MIHOPE) study. I’m _____________ and I work for _______________, an independent research firm/university.

We are conducting a study for the Department of Health and Human Services to learn about home visiting programs funded by the Maternal, Infant, and Early Childhood Home Visiting program. We want to talk to home visiting program managers to learn about your experiences managing your home visiting program and the support you receive for doing so. Our reports will describe the experiences and viewpoints expressed, but comments will not be attributed to specific individuals or programs. No individuals will be quoted by name.

I would like to tape-record our discussion. I am taping our discussion so I can listen to it later when I write up my notes. No one besides our research team will listen to the tape. If you want to say anything that you don’t want to be taped, please let me know and I will be glad to pause the recorder. Do you have objections to being tape-recorded?

The discussion will take about 75 minutes.

Once again, thank you for participating. Do you have any questions before we get started?

A. INTRODUCTION

1. So that we all have a sense of one another’s role, can you each identify your job title, how much of your total effort is allocated to your role as your home visiting program’s manager, and your primary responsibilities in that role?

Our discussion today will focus on three things: influential organizations, the home visiting program service model, and the home visiting program implementation system. First, I’m going to define each of these. Then, I’ll ask questions to guide our discussion of these.

Influential organizations play a role in defining the the home visiting program site’s service model and implementation system. These influential organizations might include: the implementing agency; its parent organization if it is part of a larger system; its funders; the national home visiting model developers; professional groups; public agencies at the national, state and local levels; and the community-based organizations from which it receives referrals, to which it makes referrals, and with which it coordinates services for families.

The service model is how the program looks on paper. It is the program site’s intended goals and outcomes for families, its philosophy and theory of change, its definition of the families it wants to target, the services it intends to provide to these families, and its intended staffing, that is, its definition of staff qualifications and competencies and how it expects different staff members to work together.

The implementation system is the set of resources that are used to carry out the service model. We think of five main parts to the implementation system. The first is professional development, which includes staff recruitment, hiring, training, and supervision. The second part is clinical supports for staff, which can include access to experts and other professionals. The third part is administrative supports, such a data systems to make it easy to document and monitor service delivery for each family, to monitor overall program performance and to remind staff of tasks to carry out with each family. The fourth part is the implementing agency’s culture and climate. The fifth part is the program’s relationships with other community organizations and services for family referral and for service coordination.

B. INFLUENTIAL ORGANIZATIONS

1. What organizations have influenced your program site’s service model and implementation system?

2. From your perspective, how well do these organizations work together?

PROBE: Are they usually aligned in how they want to define your program’s service model, or do they sometimes differ in ways that make it hard for you to define your service model and implement your program? What are some examples?

3. Do you have any insights on why some influential organizations are aligned and others are not, and how influential organizations can become more aligned?

C. SERVICE MODEL - INTENDED OUTCOMES FOR FAMILIES

Let’s start by talking about the outcomes your program site intends to achieve with families.

1. How well do influential organizations agree on what your program’s intended outcomes should be and on how to prioritize outcomes?

PROBE: Can you give me some examples where influential organizations differ in what they feel your program’s outcomes should be?

2. How have you dealt with these different opinions, and how has this affected program operations for better or worse?

3. How clear are your program’s intended outcomes and priorities to staff? That is, if one were to ask staff members what outcomes your program aims to improve, and which outcomes are its top priority, how much do you think staff members would agree or disagree?

PROBE: Can you give examples of outcomes on which staff members agree? Examples where they might disagree?

4. Do you have any insights on why staff might agree more in their understanding of some outcomes than others?

5. Can you offer any insights on how to improve staff members’ shared understanding of intended outcomes and priorities among outcomes?

D. SERVICE MODEL – THEORY OF CHANGE

Let’s talk about your program’s theory of change, that is, its philosophy for how services will improve those intended outcomes.

1. Does your program site specify any theories of behavior that are the foundation for how its services will improve family outcomes? What theories does it specify?

2. How well do influential organizations agree on your program’s theory of change?

PROBE: Can you give examples where influential organizations differ in what they feel your program’s theory of change should be?

3. How have you dealt with these different opinions, and how has this affected program operations for better or worse?

4. FOR EACH THEORY: How well do you think home visitors understand what this theory means? How do they vary in their understanding of it? Can you give some examples?

5. Do you have any insights on why staff might vary in their understanding of this theory of change?

6. Any insights on how to improve staff members’ understanding of a home visiting program’s theory of change?

E. SERVICE MODEL – TARGETED FAMILIES

Let’s talk a bit about the families your program intends to serve.

1. How does your program define the families it intends to serve? What inclusion and exclusion criteria does it use?

2. How well do influential organizations agree on how you define your target population?

PROBE: Can you give examples where influential organizations differ in what they feel your program’s target population should be?

3. How have you dealt with these different opinions, and how has this affected program operations for better or worse?

F. SERVICE MODEL – INTENDED SERVICES

ASPECTS OF INTENDED SERVICES FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Intended services means are the specifics about when services are to be provided, visit content, and approaches to interacting with families. Each group interview will focus on issues felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys and logs. Candidate topics include:

• Timing of Services: When families are to enroll, how often they are to be visited, how long visits are to last, and how long families are to remain enrolled

• Visit Content: Distribution of time for activities to address different intended outcomes

• Visit Content: Topics determined by program vs. selected by home visitor vs. selected by family

• Approach: Active screening for family needs and response to these vs. response only to needs parents choose to express

• Approach: Reinforcement of observed positive parenting behaviors vs. reinforcement of observed positive parenting behaviors and direct response to observed ineffective parenting behaviors by noting these and suggesting, demonstrating, and reinforcing positive alternatives

• Approach: Actions to build mother’s reflective capacity, and to dispel inaccurate parenting beliefs and negative parenting attitudes as ways to improve parenting behavior vs. only parenting education to promote parenting knowledge

EXAMPLE OF QUESTION SET FOR TIMING OF SERVICES:

1. Think about your program’s requirements and preferences for when families enroll, how often they are visited, how long visits last, and how long families remain enrolled. How well do influential organizations agree on what these should be for your program site?

PROBE: Can you give examples where influential organizations differ in their opinions on this?

2. How have you dealt with these different opinions, and how has this affected program operations for better or worse?

3. How well do you think home visitors understand and adhere to your program’s requirements and preferences for when families enroll, how long visits last, and how long families remain enrolled? How do they vary in their understanding of and adherence to this policy?

PROBE: Can you give examples?

4. Do you have any insights on why staff might vary in their understanding of and adherence to policies for when services are to be provided?

5. Any insights on how to improve staff members’ understanding of and adherence to when services are to be provided?

G. INTENDED STAFFING

ASPECTS OF INTENDED STAFFING FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Intended staffing has three aspects: competencies, how staff work together as a team, and caseload limits. Each group interview will focus on those issues felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys and logs.

EXAMPLE OF QUESTION SET FOR STAFF COMPETENCIES:

1. First, how do you define the competencies your staff must achieve to be effective in their roles?

2. How vocal are influential organizations on what these competencies should be, and how much do they agree on this?

PROBE: Can you give examples where influential organizations are silent on competencies, and example of where they differ in their opinions on this?

3. How have you dealt with lack of information and on differing opinions, and how has this affected program operations for better or worse?

4. How well do you think staff understand their personal achievement of competencies? How do they vary in their understanding of this?

PROBE: Can you give examples?

5. Do you have any insights on why staff might vary in their understanding of their achievement of competencies?

6. Any insights on how to improve staff members’ understanding of how well they have achieved competencies?

H. IMPLEMENTATION SYSTEM – PROFESSIONAL DEVELOPMENT

ASPECTS OF PROFESSIONAL DEVELOPMENT FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Professional development has five main aspects: staff recruitment and hiring; staff training, staff supervision including reflective supervision; coaching; and peer learning and support. Each group interview will focus on those issues felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys and logs.

EXAMPLE OF QUESTION SET FOR STAFF RECRUITMENT AND HIRING:

1. What are your site’s policies and procedures for staff recruitment and hiring?

2. How vocal have influential organizations been on what your site’s policies and procedures should be for staff hiring and recruitment, and how much have they agreed on this?

PROBE: Can you give examples where influential organizations have been silent on this, and examples where they differ in their opinions?

3. How supportive have influential organizations been in providing your program site with infrastructure – such as tools and registries – to help you recruit and hire staff?

4. How have you dealt with lack of information, differing opinions, and inadequate support for staff recruitment and hiring, and how has this affected program operations for better or worse?

I. IMPLEMENTATION SYSTEM – CLINICAL SUPPORT

ASPECTS OF CLINICAL SUPPORT FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Clinical support has five main aspects: parenting curricula; protocols and tools for identifying parent needs; protocols for responding to family needs; and accessibility of experts to advise the home visitor; and accessibility of experts to work directly with families together with the home visitor. Each group interview will focus on those issues felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys and logs.

EXAMPLE OF QUESTION SET FOR PARENTING CURRICULA:

1. What is your site’s policy for requiring or recommending home visitors’ use of specific parenting curricula and for using other parent training materials of their own choosing?

2. How vocal have influential organizations been on what your site’s policy should be? And how much have they agreed in their views on this?

PROBE: Can you give examples where influential organizations have been silent on parent training materials and examples where they differ in their opinions?

3. How have you dealt with influential organizations’ lack of guidance and differing opinions with regard to requiring, recommending, and allowing home visitor discretion in choosing parent training materials? How has this affected program operations for better or worse?

4. How well do you think staff understand and adhere to your policy regarding parenting curricula? How do they vary in their understanding and adherence?

PROBE: Can you give examples?

5. Do you have any insights on why staff might vary in how well they understand and adhere to your policy on parent training materials?

6. Any insights on how to improve staff members’ understanding of and adherence to your policy?

J. IMPLEMENTATION SYSTEM – ADMINISTRATIVE SUPPORT

ASPECTS OF ADMINISTRATIVE SUPPORT FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Administrative support has five main aspects: management information systems; staff service monitoring; program monitoring; continuous quality improvement; and advisory groups . Each group interview will focus on those issues felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys and logs.

EXAMPLE OF QUESTION SET FOR MANAGEMENT INFORMATION SYSTEMS:

1. What management information system or systems does your site use for documenting service delivery and outcomes?

2. How active have influential organizations been in specifying what your site’s management information system(s) should be and in supporting development and maintenance of your system(s)? How much have they agreed in their views? And as a group, how adequately have they supported your development and maintenance of your management information system(s)?

PROBE: Can you give examples where influential organizations gave too little guidance and support for setting up and maintaining your management information systems, or were too prescriptive, or had conflicting expectations for your management information system(s)?

3. How have you dealt with issues of too little guidance and support, overly prescriptive expectations, and conflicting expectations in setting up and maintaining your management information system(s)? How has this affected program operations for better or worse?

4. What does your site expect of supervisors, home visitors and other staff in terms of entering and using data in your management information system?

5. How well do you think staff carry out their roles in entering and using data in your management information system? How do they vary in this?

PROBE: Can you give examples?

6. Do you have any insights on why staff might vary in how well they enter and use data in your management information system?

7. Any insights on how to improve staff members’ data entry and use?

K. IMPLEMENTATION SYSTEM – ORGANIZATIONAL CULTURE AND CLIMATE

ASPECTS OF CULTURE AND CLIMATE FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Organizational culture is defined as expectations for the way things are done. It has three main dimensions: rigidity, proficiency and resistance. Organizational climate is defined as an employee’s perceptions of his or her own psychological well-being and functioning in the organization. Organizational climate has three main dimensions: engagement, functionality and stress. Each group interview will focus on those dimensions felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys.

EXAMPLE OF QUESTION SET FOR ORGANIZATIONAL RIGIDITY:

1. Programs vary in how much discretion they afford staff in have in how they carry out their jobs, in how much input staff can have in key management decisions, and in the extent to which staff must follow rules and regulations. How would you characterize your program on these features?

2. Which organizations have had the greatest influence on these aspects of your program? Have any organizations been in conflict in how they have tried to influence these aspects of your program?

PROBE: Can you give examples of how specific organizations have influenced these features of your program? Example of how specific organizations have been in conflict in influencing these features of your program?

3. How have you dealt with these influences and with conflicts across influential organizations? How has this affected program operations for better or worse?

L. IMPLEMENTATION SYSTEM – SYSTEMS CHANGE

ASPECTS OF SYSTEMS CHANGE FOR WHICH A STANDARD SET OF QUESTIONS COULD BE ASKED: Systems change involves policies and procdures for three main aspects of program operations: family referral; service coordination; and reimbursement for services. Each group interview will focus on those dimensions felt to be most relevant for the participants of that group, based on their sites’ responses to the web-based surveys.

EXAMPLE OF QUESTION SET FOR SERVICE REFERRAL:

Let’s consider different types of services, one at a time: referrals for prenatal and primary care; referrals to address risks for poor parenting, such as parental mental health, substance use, and domestic violence; referrals to education and employment resources to promote family economic self-sufficiency; and referrals to assess and address possible child developmental delay.

1. What agreements do you have with other organizations for referring your families to them?

2. Which organizations and other forces have influenced your ability to establish referral agreements?

PROBE: Can you give examples of how specific organizations and other forces have helped or hindered your program in establishing referral agreements with community resources?

3. How have you dealt with challenges to establishing referral agreements? How has this affected program operations for better or worse?

M. PARTICIPATION IN MIHOPE

Let’s talk about your program’s participating in MIHOPE.

1. Please describe your program’s procedures for identifying eligible families and coordinating with the MIHOPE field staff to collect baseline data and conduct random assignment.

1. How well is this working?

2. What problems have come up, and how well have you been able to resolve them?

2. How does your program notify families assigned to the control group of their status?

1. Who is responsible for notifying control families?

2. How well is this working?

3. Have any other operational challenges come up related to the evaluation?

1. Web-based surveys?

2. Home visitor and supervisor logs?

3. Records requests?

4. Baseline data collection visits?

5. Home visit video recording?

6. Other?

4. Have you been able to resolve these challenges? If not, how can the evaluation team help?

N. WRAP UP

Thank you so much for taking the time to talk with me today. Is there anything else you would like to add before we end the discussion?