9. MUSE Local Program Evaluator Survey 7.9.18 CLEAN

DRAFT 7/9/18 OMB Control No.: xxxx-xxxx

Expiration Date: xx/xx/20xx

Length of time for instrument: 30 minutes

MULTI-SITE IMPLEMENTATION EVALUATION OF HOME VISITING (MUSE)

LOCAL PROGRAM EVALUATOR SURVEY

This collection of information is voluntary. Public reporting burden for this collection of information is estimated to average 30 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-XXXX, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Kate Lyon, James Bell Associates; 3033 Wilson Blvd. Suite 650, Arlington, VA 22201; [email protected].

MUSE Local Program Evaluator Survey

Thank you for taking part in the Multi-Site Implementation Evaluation of Tribal Home Visiting (MUSE). The purpose of this study is to learn about tribal home visiting program implementation and the experiences of families receiving home visiting services.

We are asking you to complete this survey because you are an evaluator working with one of the home visiting programs participating in MUSE. Your answers will help us understand your role in your home visiting program and your perspective on the program.

Your participation in this survey is voluntary. If you choose to participate, it will take about 30 minutes to complete this survey. If you are unsure how to answer a question, please give the best answer you can instead of leaving it blank.

Your answers will be kept private. Only the MUSE study team will have access to this information. Your answers will not be shared with anyone at your program or any other agencies. We will not report information collected in this study in a way that could identify you or your program.

We would appreciate your response by MM/DD/YYYY. If you have questions about the survey or at any time during the study, please call Tess Abrahamson at James Bell Associates at ### or email ____.

A. BACKGROUND AND WORK EXPERIENCE

1. What was the highest level/degree you completed in school?

• Some high school, no diploma

• High school/GED

• Some college/no degree

• Technical training or certification

• Associate’s degree (e.g. AA, AS, ADN)

• Bachelor’s degree (e.g. BA, BS, BSN)

• Master’s degree or higher (e.g. MA, MS, MSW, MSN, PhD)

2. Field(s) of study (for example, psychology, sociology, education) (List all degrees completed): ___________________

3. How many total years of experience do you have working as a program evaluator?

   • Less than 1 year

   • 1-2 years

   • 3-5 years

   • 6-10 years

   • More than 10 years

4. Other than as an evaluator, do you have experience working with children and families? (e.g. in home visiting, social work, etc.)?

   • No

   • Yes. Please describe ___________________________________________

B. CURRENT POSITION

1. When did you begin your present role as an evaluator for the Tribal Home Visiting program? Please enter the month and year in numeric format.

Month _______ Year ________

2. Have you worked with the agency or tribe on other projects?

• No [ SKIP TO Question 3]

• Yes [ GO TO Questions 2a & 2b]

2a. [If Question 2= yes] If so, in what capacity? ___________

2b. [If Question 2= yes] How many years did you work with your home visiting program in that capacity?

• Less than 1 year

• 1-2 years

• 3-5 years

• 6-10 years

• More than 10 years

3. Are you employed by the organization that administers the Tribal Home Visiting program?

• No [ GO TO Questions 3a & 3b]

• Yes [ SKIP TO Question 4]

3a. [If Question 3 = No] Which of the following best describes your employer?

• University

• Private company

• Self-employed

• Other, please describe _________

3b. [If Question 3 = No] My employer is very supportive of my work with the Tribal Home Visiting program.

 Strongly agree

 Agree

 Disagree

 Strongly disagree

4. On average, how many hours per month do you work as an evaluator for your home visiting program?

Hours: __________

5. Is the time you have allocated to this project adequate to fulfill your assigned duties?

• Yes

• No

6. On average, how often are you in contact with your home visiting program staff?

• Daily

• 2-3 days a week

• Weekly

• A couple times a month

• Less than once a month

7. Which option best describes your relationship with your home visiting program?

• Just getting to know the staff and program

• Know the program and staff pretty well

• Fully involved in the staff team and program

8. How would you rate the amount of time you spend working as an evaluator on the home visiting program as compared with other similar projects?

• More time than other similar projects

• Roughly the same amount of time as other similar projects

• Less time than other similar projects

• NA/unsure

9. How likely is it that you will continue in your current evaluator role 6 months from now?

• Very likely [ SKIP TO Section C]

• Somewhat likely [ GO TO Question #9a]

• Somewhat unlikely [ GO TO Question #9a]

• Very unlikely [GO TO Question #9a]

9a. [If somewhat likely, somewhat unlikely, very unlikely] What factors affect whether you will stay in this role? [CHECK ALL THAT APPLY]

• The pay I receive for this project

• The funding for my role on this project is uncertain

• Another project requires more of my time

• Opportunities for advancement within my organization

• Personal reasons (e.g., health, family obligations, change in career)

• Retire or stop working

• Moving out of the area

• Challenging work environment with the home visiting program

• Other ________________________

C. PERCEPTIONS OF PROGRAM

Instructions: In this section, we would like to learn how staff members perceive their program’s intended outcomes. In general, a program outcome is a benefit to a child, parent, or family. For example, some programs might see the improvement of prenatal health as an important outcome.

Below is a list of possible outcomes for home visiting programs. We know your program may care about all of these benefits, but we would like to know which is MOST important, which outcomes you think your program believes may be more important than others.

To help you decide on how to rank each outcome, think about whether it is discussed routinely in meetings, training and supervision and whether there is a focus on measuring these outcomes and improving them.

1. How much of a priority is each of the following outcomes for your program, on a scale of 0 to 10?

0 = Not a Priority

5 = Moderate priority

10 = Highest priority

		0	1	2	3	4	5	6	7	8	9	10	Not sure
	Supporting prenatal health and obtaining prenatal care												
	Supporting postpartum health and obtaining postpartum care												
	Supporting breastfeeding												
	Supporting physical health outside of pregnancy and postpartum health												
	Supporting good nutrition and physical activity												
	Supporting family planning												
	Preventing and reducing alcohol, tobacco, and other drug use												
	Preventing and reducing mental health problems or stress												
	Supporting healthy relationships												
	Preventing and reducing domestic violence												
	Increasing social support												
	Supporting caregivers in getting a job, or getting a new job												
	Furthering a caregiver’s education and job training												
	Supporting caregivers in budgeting and making ends meet												
	Meeting basic needs like food, utilities, health care, and housing												
	Ensuring appropriate child care arrangements												
	Addressing unresolved issues from past trauma												
	Connecting to community and culture												
	Supporting child health												
	Supporting parenting to promote child development												
	Supporting parent-child interaction												
	Supporting positive discipline and behavior management												
	Supporting caregivers in feeding children (including formula and solids)												
	Helping caregivers to establish and maintain developmentally appropriate care/routines												
	Supporting effective co-parenting												
	Supporting child and home safety												

2. Overall, how effective do you believe your program is in MAKING A DIFFERENCE for families in the following areas?

		Not at all effective	Slightly effective	Somewhat effective	Mostly effective	Very effective
	Prenatal health/prenatal care					
	Postpartum health/postpartum care					
	Breastfeeding					
	Physical health (outside of pregnancy and postpartum health)					
	Nutrition and physical activity					
	Family planning					
	Alcohol, tobacco, and other drug use					
	Mental health or stress					
	Healthy relationships					
	Domestic violence					
	Social support					
	Getting a job, or getting a new job					
	Furthering a caregiver’s education and job training					
	Budgeting/making ends meet					
	Meeting basic needs like food, utilities, health care, and housing					
	Child care					
	Trauma (things that happened in the past that affect caregiver or family today)					
	Connecting to community and culture					
	Child health					
	Child development					
	Parent-child interaction					
	Discipline/behavior management					
	Feeding children (including formula and solids)					
	Developmentally appropriate care/routines					
	Co-parenting					
	Child/home safety					
	School readiness/early learning					

D. EVALUATOR ROLE

1. Next are some skills and knowledge that evaluators working on Tribal MIECHV-funded grants may need. Based on your experience working with the home visiting program, please rate each of these areas from not at all important (1) to extremely important (5).

		1	2	3	4	5
	Study design					
	Developing a data collection protocol					
	Conducting data analysis					
	Developing data collection forms					
	Training staff					
	Managing program data					
	Database engineering					
	Conducting data quality checks					
	Building relationships with program staff					
	Project management skills					
	Meeting facilitation/presentation skills					
	Translating evaluation terminology into lay terms					
	Negotiating multiple perspectives in planning					
	Advocating on behalf of the program					
	Influencing program decisions					
	Federal requirements					
	Home Visiting Services					
	Evaluation methodology					
	Data quality and cleaning procedures					
	Community protocols					
	Developing an IRB application					

2. How often are you involved in overall program decision making for the home visiting program?

• Always

• Often

• Occasionally

• Rarely

• Never

3. Have you previously worked with the organization that administers Tribal Home Visiting on any other programs or projects?

• Yes

• No

4. How often do you participate in your home visiting program’s team meetings? Please include in-person and telephone meetings.

• Once a week or more

• Every other week

• Once a month

• Less than once a month

• Never

5. How often do you come on site to meet with the program?

• I am located in the same building as the program staff

• Once a week or more

• Every other week

• Once a month

• Once every few months

• Twice a year

• Once a year

• Never

6. How often do you interact with the following home visiting program team members? Please include in-person meetings, telephone and email communication?

	Every day	2-3 times a week	Once a week	2-3 times a month	Once a month	Less than once a month
Program Director						
Program Coordinator/Manager						
Home Visitors						
Other Program Staff						

7. Have you ever observed a home visit?

• Yes [ GO TO Question 7a]

• No [ SKIP to Question 8]

7a. [If Question 7 = Yes] How many home visits have you observed? ___

8. Have you ever attended a family group event?

• Yes

• No

9. Please tell us about your involvement with your home visiting program. To what extent are you involved in the following activities:

		I am not involved	I consult as needed	I am substantially involved	I lead this effort
	Data quality checks				
	Developing data collection protocols				
	Data analysis				
	Developing data collection forms				
	Training home visitors to collect data				
	Data management and cleaning				
	Database changes				
	Running reports				
	Interpreting data				
	Sharing data with staff				
	Sharing findings/presenting data to stakeholders				

E. DATA SYSTEMS

1. How much experience did you have with data systems prior to working with your home visiting program?

• A lot [ GO TO Question 1a]

• Some [ GO TO Question 1a]

• A little [ GO TO Question 1a]

• None [ SKIP to Question 2]

1a. [If Question 1 = A lot, some, or a little] How adequate was your prior experience with data systems in preparing you for the demands of this project?

• More than adequate

• Adequate

• Not quite adequate

• Not at all adequate

2. Which of the following reflect your involvement with your team’s data system? (SELECT ALL THAT APPLY)

• I oversee all aspects of the data system

• I provide training on the data system to incoming staff

• I develop policies or procedures relevant to the data system

• I supervise staff who do data entry

• I do data entry

• I work with IT or vendor to make changes to the data system

• I access the data system for running reports

• Other_______________________

3. Based on all sources of support, how would you rate the level of support your home visiting program receives related to their data system?

• The support we receive fully meets our needs

• The support we receive meets most of our needs

• The support we receive meets some of our needs

• The support we receive meets very few of our needs

• The support we receive does not meet our needs

• Unsure

4. Which kinds of data systems support have been most helpful to the home visiting program team? Please rate each of these supports from not at all helpful (1) to extremely helpful (5).

		1	2	3	4	5	Did not receive data system support from this entity
	Support from the data system developers (including trainings)						
	Support from your TEI capacity building specialist						
	Support from the home visiting organization’s internal IT department						

5. Overall, does the primary data system meet the home visiting program’s needs?

• Yes

• No

• Not sure

6. How would you rate the usefulness of the home visiting program’s data system?

• The system is extremely useful for meeting all of our reporting requirements

• The system is useful but does not meet all of our data reporting needs

• The system has some useful features but requires us to do a lot of extra work to accomplish our data reporting requirements

• The system makes it harder to meet our data reporting requirements than counting cases by hand

• Not sure

7. How easy to use is the home visiting program’s data system?

• Very easy

• Easy

• Difficult

• Very difficult

• Not sure

8. If possible, would you prefer an alternative data system to the one currently being used?

• Yes

• No

• Don’t know

9. Has your role in managing the data system changed over time?

• Yes, I have more responsibility

• Yes, I have less responsibility

• Yes, I have the same amount of responsibility, but my tasks have changed

• No

F. PERFORMANCE MEASUREMENT

1. We are interested in learning how performance measurement has impacted your program. Please rate your level of agreement with the following statements:

		Strongly Agree	Agree	Disagree	Strongly Disagree	Not Sure
1	Planning for performance measurement has helped program leadership ensure high quality implementation.					
2	Discussions about performance measurement have helped program staff develop common priorities.					
3	Program staff see performance measurement as more of a chore than a help.					
4	Performance measurement requirements have taken away from important program work.					
5	Performance measurement data have helped home visitors see how they are making a difference.					
6	Looking at performance measurement data has shown us where we can make improvements.					
7	Collecting so much performance measurement data makes it hard to spend enough time providing services to families.					
8	This grant’s data reporting requirements are too burdensome.					
9	Other programs in this agency have improved their data collection systems or the way they use data as a result of the home visiting program’s work on performance measurement.					
10	Technical assistance providers play a key role in helping the home visiting program carry out high quality performance measurement.					
11	Performance measures are not aligned with our program priorities.					
12	Technical assistance adds additional burden to our planning efforts.					

G. CONTINUOUS QUALITY IMPROVEMENT

1. To what degree are you involved in the home visiting program’s continuous quality improvement (CQI) activities? Continuous quality Improvement is using data and information to improve performance and practice.

• I lead the home visiting program’s CQI activities

• I am substantially involved

• I consult on these activities as needed

• I am not involved

• Our program does not do CQI activities [ SKIP to next Section]

2. The number of hours I have allocated to this project allows time for me to support CQI initiatives.

• Yes

• No

3. How have you supported the home visiting program’s CQI activities? (Please check all that apply)

• Leading CQI efforts

• Consulting with program staff

• Training staff on CQI

• Providing or analyzing data for CQI

• Creating/running trend charts

• Monitoring data collection

• Analyzing CQI data

• Presenting QI info to stakeholders

4. Does the program hold meetings dedicated to reviewing data and CQI?

• Program holds regular meetings focused solely on data and CQI

• Program holds regular meetings where data and CQI are on the agenda for part of the meeting

• Program holds regular meetings but data and CQI are only occasionally discussed

• Data and CQI are not discussed at meetings

5. How many CQI projects has the home visiting program worked on since receiving a Tribal MIECHV grant?

• 0

• 1-3

• 4 or more

6. How have CQI efforts changed the home visiting program’s service delivery?

• They have made a large positive change

• They have made a small positive change

• They have made little to no change

• They have made a small negative change

• They have made a large negative change

H. ROLE SATISFACTION

1. The following questions ask how you feel about your role with the home visiting program. Please state how often you feel this way.

		None of the time	A little of the time	Some of the time	Most of the time	All of the time
1	The work I do with the home visiting program is satisfying.					
2	My work with the home visiting program is boring.					
3	My work with the home visiting program allows me to be creative.					
4	I feel respected in my work with the home visiting program.					
5	My work with the home visiting program is frustrating.					
6	My work with the home visiting program gives me a sense of accomplishment.					
7	My work with the home visiting program is interesting.					
8	The work I do with the home visiting program is important.					
9	My work with the home visiting program is overwhelming.					

2. We are interested in learning about how your job relates to the community where you provide home visiting services. Indicate your agreement or disagreement with each statement below.

		Strongly Agree	Agree	Disagree	Strongly Disagree	Not Sure
1	The local community is very involved in shaping the home visiting services the home visiting program provides.					
2	Being connected to this community is critical to my success as an evaluator.					
3	I worry that what I have to do to help with data requirements negatively impacts how I am perceived in the community.					
4	General distrust and/or unfamiliarity with data makes my job harder on this project.					
5	What I do as an evaluator is meaningful to people in this community.					
6	What I do as an evaluator makes a positive difference in the local community					
7	What I do as an evaluator is contributing to a brighter future for the local community					

I. ORGANIZATIONAL CULTURE AND CLIMATE

The following questions ask you to think about how your home visiting team does its work. We want to know how your team works together, takes in information, and makes decisions about the team’s approach to home visiting. When answering questions about your team, please think about the staff that make up your home visiting program. This would include other home visitors, program managers, supervisors, evaluators, data managers and anyone else that might work closely with your program.

1. Please state whether you agree or disagree with the following statements.

		Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree	Not Sure
1	This team gets all the information it needs to do our work and plan our schedules.						
2	It is easy for this team to obtain expert assistance when something comes up that we don't know how to handle.						
3	This team is kept in the dark about decisions that impact day-to-day work and what may happen with the program and its staff in the future.						
4	This team lacks access to useful training on the job.						
5	Excellent work pays off in this organization.						
6	It is clear what this team is supposed to accomplish.						
7	This team spends time making sure every team member understands their role and responsibilities.						
8	The team has invested plenty of time to clarify our goals.						
9	If you make a mistake on this team, it is often held against you.						
10	Members of this team are able to bring up problems and tough issues.						
11	People on this team are expected to conform to the group.						
12	It is safe to try something new on this team.						
13	It is difficult to ask other members of this team for help.						
14	No one on this team would deliberately act in a way that undermines my efforts.						
15	Working with members of this team, my unique skills and talents are valued and utilized.						
16	Achieving this team's goals is well within our reach.						
17	This team can complete work as assigned without being required to put in unreasonable time or effort.						
18	With focus and effort, this team can do anything we set out to accomplish.						
19	Most people in this team have the ability to solve the problems that come up in our work.						
20	All members of this team have more than enough training and experience for the kind of work they have to do.						
21	Certain individuals in this team lack the special skills needed for good team work.						
22	We regularly take time to figure out ways to improve our team's work processes.						
23	This team tends to handle differences of opinion privately, rather than addressing them directly as a group.						
24	Team members go out and get all the information they possibly can from others-such as families, community members, and other program partners.						
25	This team frequently uses information and data that leads us to make important changes.						
26	In this team, someone always makes sure that we stop to reflect on the team's work process.						
27	People on this team often speak up to test assumptions we might have.						
28	People on this team are encouraged to think outside the box.						
29	We invite people from outside the team to present information or have discussions with us.						
30	This team uses data to see if our processes are leading to the results we want.						
31	Members of this team are encouraged to try new strategies to see if they will work.						
32	Members of this team support each other as we work to master new skills.						
33	The quality of work provided by this team is improving over time.						

2. Tribal home visiting programs have a lot of different leadership structures. For the next few questions, think about the person in YOUR home visiting program who serves as the team leader. Please rate your level of agreement with the following statements.

		Strongly agree	Agree	Neutral	Disagree	Strongly disagree	Not sure
1	The team leader initiates meetings to discuss the team's progress.						
2	The team leader is available for consultation on problems.						
3	The team leader is engaged in the team’s day-to-day work.						
4	The team leader manages crises in a calm and dependable way.						
5	The team leader helps us get through challenges we face in our work.						
6	The team leader handles personnel issues thoughtfully.						
7	The team leader would go to bat for us.						
8	The team leader has enough training and experience to be an effective leader.						
9	The team leader treats all team members fairly.						
10	The team leader doesn’t really understand what our team needs to do its job well.						

J. CONNECTION TO COMMUNITY SERVED

1. Do you live in the community or neighborhoods the home visiting program provides services to?

• Yes [ GO TO Question 1a]

• No [ SKIP TO Question 1b]

1a. [If Question 1 = Yes] In total, how many years have you lived in the community or neighborhoods the home visiting program provides services to?

• Less than 1 year

• 1-2 years

• 3-5 years

• 5-10 years

• More than 10 years

1b. [If Question 1 = No] If you ever previously lived in the community or neighborhoods the home visiting program provides services to, how long did you live there?

• I never lived there

• Less than 1 year

• 1-2 years

• 3-5 years

• 5-10 years

• More than 10 years

2. Do you consider yourself a member of the tribal or urban Indian community the home visiting program provides services to?

• Yes

• No

• Somewhat

3. In general, do you feel as though you and the families the home visiting program serves share a similar cultural background?

• Yes, with most families

• Yes, with some families

• Yes, with a few families

• No

K. DEMOGRAPHICS

1. What is your Ethnicity?

• Hispanic or Latino

• Not Hispanic or Latino

2. What is your Race? (Select one or more)

• American Indian or Alaska Native [ GO TO Question 2a]

• Asian [ SKIP TO Question 3]

• Black or African American [ SKIP TO Question 3]

• Native Hawaiian or Other Pacific Islander [ SKIP TO Question 3]

• White [ SKIP TO Question 3]

2a. [If Question 2 = American Indian or Alaska Native] What is your tribal affiliation and/or identity?

_____________________________________________________________________

1. What is your age?

• 25 and under

• 26-29

• 30-39

• 40-49

• 50-59

• 60 or older

[NEXT SCREEN]

THANK YOU FOR TAKING THE TIME TO COMPLETE THIS SURVEY.

Please click NEXT to exit the survey.

Multi-Site Implementation Evaluation of Tribal Home Visiting OMB Supporting Documents: Local Program Evaluator Survey

2