Site visit and key informant data collection: Sustainability survey (Appendix D)

APPENDIX D

sustainability survey

THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) Public reporting burden for this collection of information is estimated to average 20 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

OMB Approval number: 0970-0527, Expiration Date: XX/XX/XXXX.

Regional Partnership Grants Cross-Site Evaluation Sustainability Survey

January 4, 2019

THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) Public reporting burden for this collection of information is estimated to average 20 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

OMB Approval number: 0970-0527, Expiration Date: XX/XX/XXX.

CONTENTS

Section Page

i. consent and screener 2

A. Organization characteristics 6

B. PLANS FOR SUSTAINING RPG PROJECT 9

Ca. Implementation Supports to Improve RPG services 12

Cb. Implementation Supports to SUSTAIN RPG SERVICES 14

D. Funding and Resources for Sustainability 16

E. Federal, State, and Local Context 20

The Regional Partnership Grant (RPG) program supports interagency collaboration and program integration designed to increase the well-being, improve the permanency, and enhance the safety of children who are in, or at risk of, out-of-home placements as a result of a parent or caretaker’s substance abuse. The Children’s Bureau within the U.S. Department of Health and Human Services, Administration for Children and Families has contracted with Mathematica Policy Research to complete the national cross-site evaluation of the program. The evaluation will describe the services that were implemented, the nature of the partnerships, and participant outcomes.

You are being asked to complete this survey because you were identified as a representative of an organization working on an RPG project who is familiar with improvement activities and planning for sustainability (meaning the continued implementation of a service or program after a defined period of time). Representatives from RPG project organizations are asked to complete this survey to provide information about their organizations’ involvement in plans and activities to improve services during and after the grant period, and to sustain the RPG project after the grant ends. The length of this survey is different for different people, but on average it should take about 20 minutes.

Your participation in this survey is important and will help us understand more about the current improvement activities and plans for sustainability for RPG projects. You will be asked questions both about your organization, [Grantee or PARTNER ORGANIZATION], and your [RPG project] as a whole. If you are unsure of how to answer a question, please give the best answer you can rather than leaving it blank.

Your responses will be kept private and used only for research purposes. They will be combined with the responses of other staff and reported in the aggregate; and no individual names will be reported. Participation in the survey is completely voluntary and you may choose to skip any question. The reports prepared from the information provided as part of this survey will be summarized across RPG projects and individual responses will not be available to anyone outside the study team, except as required by law.

If you have questions about the survey, please contact the team at Mathematica by emailing [email protected] or calling 866-627-9538 (toll-free).

Please read and answer the statement below and then click the “Submit Page and Continue” button at the bottom of the page to begin the survey.

ALL

I1. I have read the introduction and I understand that the information I provide will be kept private and used only for research purposes. My responses will be combined with the responses of other staff and no individual names will be reported.

 I agree with the above statement and will complete the survey 1

 I do not agree with the above statement and will not complete the survey 0 GO TO END 2

 NO RESPONSE M

SOFT CHECK: IF i1=0; You have indicated that you will not complete the survey. Please check that this is correct and either keep your answer or change your answer below.

To keep your answer without making changes, click the “Submit and Continue” button.

HARD CHECK: IF i1=NO RESPONSE; Please indicate whether you agree to complete the survey and click the “Submit and Continue” button.

IF I1=1

I2. Are you planning to sustain the project?

Select only one

 Yes 1 GO TO S1

 No 2

IF I2=2

I3. Why did you decide not to sustain the project?

PROGRAMMER: CODE ONE PER ROW

	YES	NO
a. Lower referrals or enrollment than expected.	1 	2 
b. Inability to enroll intended target population	1 	2 
c. Staffing challenges, such as finding or retaining qualified grantee or partner organization staff for implementing services.	1 	2 
d. Inability to access training for clinical or other staff.	1 	2 
e. Challenges implementing services.	1 	2 
f. Challenges sharing information or data with RPG partners	1 	2 
g. Challenges coordinating case management or services with partners or other entities.	1 	2 
h. Other challenges collaborating with RPG partners.	1 	2 
i. Challenges engaging program participants.	1 	2 
j. Challenges retaining program participants.	1 	2 
k. Contextual issues, such as broader policies or community factors .	1 	2 

These next questions are about your organization’s participation in the use of data to improve RPG services and in planning for sustainability of the RPG project.

IF I1=1 and i2=1

FILL ORGANIZATION NAME FROM SAMPLE FILE

S1. Does [ORGANIZATION NAME] participate in planning for sustainability? By sustainability, we mean the continued implementation of a service or program after RPG funding ends.

Select only one

 Yes 1

 No 2

IF I1=1 and i2=1

FILL ORGANIZATION NAME FROM SAMPLE FILE

S2. Does [ORGANIZATION NAME] participate in activities that use data to improve RPG project services?

For example, reviews of referral data to increase referrals of eligible families or reviews of service data to increase retention of families in services.

Select only one

 Yes 1

 No 2

s.2 programmer box

if i2=1 and s1=2 and s2=2 GO TO END1

if i2=1 and S1=1 AND S2=1 GO TO a1

if i2=1 and s1=1 and s2=2 go to A1, SKIP SECTION C

if i2=1 and s1=2 and s2=1 go to a1, skip sections b, D & e

In this section, we would like to learn about your role with [ORGANIZATION NAME] and [ORGANIZATION NAME]’s services and role in RPG.

ALL

A1 What is your current job title?

Revised from Staff Survey

Select only one

 Mental health administrator/manager 1

 Substance abuse disorder treatment administrator/manager 2

 Child welfare administrator/manager 3

 Child development administrator/manager 4

 Health administrator/manager 5

 Other (Specify) 99

Specify (STRING 60)

NO RESPONSE M

SOFT CHECK: If A1=99 AND Specify=EMPTY; Please specify your job title in the space provided.

ALL

[ORGANIZATION NAME] FROM ORG_NAME

A2. How long have you been employed at [ORGANIZATION NAME]?

Partner Survey

Please include the total time you have been employed at the organization, not just the time you have been in your current position. Your best estimate is fine.

. NUMBER OF MONTHS OR YEARS

(0-99) (0-11)

PROGRAMMER: ALLOW FOR PARTIAL MONTHS OR YEARS

YEARS 1

MONTHS 2

NO RESPONSE M

All

A3. Which of the following best describes your organization?

Partner Survey

Select only one

 Child welfare services provider 1

 Substance abuse disorder treatment provider 2

 Mental health services provider 3

 School district, school, or early childhood education or services provider 4

 Housing/homeless services provider 5

 Medical or dental services provider 6

 University 7

 Court/judicial agency 8

 Corrections or law enforcement agency 9

 Home visiting services provider 10

 Department in state or tribal government 11

 Department in local government 12

 Foundation 13

 Research/evaluation organization 14

 Other (Specify) 99

Specify (STRING 60)

NO RESPONSE M

SOFT CHECK: If A3=99 AND Specify=EMPTY; Please specify your organization type in the space provided.

all

A4. What is your organization’s role in the RPG project?

Select all that apply

 Grantee organization (the organization awarded the grant) 1

 Referral source to RPG services 2

 Recipient of RPG referrals 3

 Direct service provider to RPG participants 4

 Contributor of in-kind resources (e.g. office space, office supplies, staff time) 5

 Contributor of financial resources 6

 Advisory/Planning 7

 Other (Specify) 99

Specify (STRING 60)

NO RESPONSE M

SOFT CHECK: If A4=99 AND Specify=EMPTY; Please specify your organization’s role in the RPG project in the space provided.

programmer box:

If i2=2 or [s1=2 and s2=1] go to Ca1

In this section, we would like to learn more about sustainability planning for your RPG project.

if i2=1

The first set of questions covers the involvement of your organization and other partners in the planning and decision making for sustaining the RPG project.

B1. How would you describe the extent of sustainability planning for the RPG project? Would you say it was extensive, moderate, minimal or has the project not done any planning?

Select only one

 Extensive planning 1

 Moderate planning 2

 Minimal planning 3

 No planning 4

if i2=1 and b1 ne 4

B2. [Fill A-B] Would you say your organization was very involved, somewhat involved, slightly involved or not at all involved?

PROGRAMMER: CODE ONE PER ROW	VERY INVOLVED	SOMEWHAT INVOLVED	SLIGHTLY INVOLVED	NOT AT ALL INVOLVED
a. How involved has your organization been in the planning for sustaining the RPG project?	1 	2 	3 	4 
b. How involved has your organization been in the decision-making process for sustaining the RPG project?	1 	2 	3 	4 

Now thinking of your RPG partner organizations, [FILL C-D] Would you say your partner organizations were very involved, somewhat involved, slightly involved or not at all involved?

PROGRAMMER: CODE ONE PER ROW	VERY INVOLVED	SOMEWHAT INVOLVED	SLIGHTLY INVOLVED	NOT AT ALL INVOLVED
c. How involved have other partners been in the planning for sustaining the RPG project?	1 	2 	3 	4 
d. How involved have other partners been in the decision-making process for sustaining the RPG project?	1 	2 	3 	4 

if i2=1

B3. Which organization will lead the partnership after RPG funding ends?

Select only one

 My organization 1

 A different partner organization 2

• Not yet decided …...3

 Other (Specify) 99

Specify (STRING 60)

NO RESPONSE M

SOFT CHECK: If B1=99 AND Specify=EMPTY; Please specify the organization name in the space provided.

if i2=1

fill [core services] from data system = yes

B4. Is your RPG project planning to continue providing [CORE SERVICES] after the grant period ends?

Select one per row

	YES	NO	NOT YET DECIDED	NO RESPONSE
a. Case management or service coordination	1 	2 	N 	M 
b. Support group or workshop	1 	2 	N 	M 
c. Therapy or counseling	1 	2 	N 	M 
d. Parenting training/home visiting program	1 	2 	N 	M 
e. Mentoring	1 	2 	N 	M 
f. Screening or assessment	1 	2 	N 	M 
g. Medication assisted treatment	1 	2 	N 	M 
h. Medical care or appointment	1 	2 	N 	M 
i. Employment training	1 	2 	N 	M 
j. Academic education (child or adult)	1 	2 	N 	M 
k. Housing	1 	2 	N 	M 
l. Transportation	1 	2 	N 	M 
m. Court or legal	1 	2 	N 	M 
n. Financial or material support (such as vouchers or stipends)	1 	2 	N 	M 
o. Child care	1 	2 	N 	M 
p. Something else?	1 	2 	N 	M 
IF B4p=1 q. Specify (STRING 60)	1 	2 	N 	M 

if i2=1

B5. What data did the RPG project review to determine which core services should be sustained?

Select all that apply

 Data about the needs of the community (children and families) 1

 Data about referrals to core services 2

 Data about enrollment in core services 4

• Data about retention in core services 5

• Data about implementation of core services (such as, fidelity data) 6

• Data about participants’ outcomes 7

 Data and evidence from the research literature on the effects of core services 8

 Other (Specify) 99

Specify (STRING 60)

• NO RESPONSE M

if i2=1

Adapted from SAPR (Semi Annual Progress Report)

B6. Which of these are potential service-related barriers to sustainability of the RPG project?

PROGRAMMER: CODE ONE PER ROW

	YES	NO
a. Lower referrals or enrollment than expected.	1 	2 
b. Inability to enroll intended target population	1 	2 
c. Staffing challenges, such as finding or retaining qualified grantee or partner organization staff for implementing services.	1 	2 
d. Inability to access training for clinical or other staff.	1 	2 
e. Challenges implementing services.	1 	2 
f. Challenges sharing information or data with RPG partners	1 	2 
g. Challenges coordinating case management or services with partners or other entities.	1 	2 
h. Other challenges collaborating with RPG partners.	1 	2 
i. Challenges engaging program participants.	1 	2 
j. Challenges retaining program participants.	1 	2 
k. Contextual issues, such as broader policies or community factors .	1 	2 

PROGRAMMER BOX:

if s1=1 and s2=2 GO TO D1

Ca. Implementation Supports to Improve RPG Services

i2=1 or 2

Pretest – all respondents will answer this section

The questions in this section are about current project activities to improve RPG services.

Ca1. To what extent are the following implementation supports currently in place within the RPG project? Are they fully in place, partially in place or not in place? [FILL A-C]. By partially we mean the activities have not been completed but some activities are underway.

PROGRAMMER: CODE ONE PER ROW	FULLY IN PLACE	PARTIALLY IN PLACE	NOT IN PLACE
a. A team that is responsible for managing the implementation of RPG services.	1 	2 	3 
b. A process to resolve barriers to implementation of RPG services (such as, inadequate referrals, inadequate staff training).	1 	2 	3 
c. A process to assess quality of RPG services.	1 	2 	3 

The next set of questions ask about your RPG project’s current plans for data related to referrals, enrollment, screenings, assessments, treatment, and outcomes.

i2=1 or 2

Ca2. Has your project analyzed [Fill a-i] data for program monitoring and improvement?

PROGRAMMER: CODE ONE PER ROW	YES	NO
a. referrals into service	1 	2 
b referrals out to other services	1 	2 
c. enrollment	1 	2 
d. screening for service eligibility	1 	2 
e. participant needs assessment	1 	2 
f. participation in services	1 	2 
g. participant outcomes	1 	2 
h. participant feedback	1 	2 
i. fidelity monitoring	1 	2 

i2=1 or 2 and IF Ca2 a-i=1

Ca3. Has your project determined how [Fill a-i] data will be shared?

PROGRAMMER: CODE ONE PER ROW	YES	NO
a. referrals into service	1 	2 
b referrals out to other services	1 	2 
c. enrollment	1 	2 
d. screening for service eligibility	1 	2 
e. participant needs assessment	1 	2 
f. participation in services	1 	2 
g. participant outcomes	1 	2 
h. participant feedback	1 	2 
i. fidelity monitoring	1 	2 

Programmer box:

if I2=2 GO TO e1

Cb. Implementation Supports to Sustain RPG Services

if i2=1 and [s1=1 and s2=1]

The questions in this section are about project plans to continuously improve RPG services after the grant period ends.

Cb1. To what extent are plans for the following implementation supports in place for the RPG project after the grant period ends? Are they fully in place, partially in place or not in place? [FILL A-C]. By partially we mean the activities have not been completed but some activities are underway.

PROGRAMMER: CODE ONE PER ROW	FULLY IN PLACE	PARTIALLY IN PLACE	NOT IN PLACE	NOT PLANNING FOR IT
a. A team that will be responsible for managing implementation of the sustained RPG services.	1 	2 	3 	NA 
b. A process that will be used to resolve barriers to implementation of the sustained RPG services.	1 	2 	3 	NA 
c. A process that will be used to assess quality of the sustained RPG services.	1 	2 	3 	NA 

The next set of questions ask about the RPG project’s sustainability plans for data related to referrals, enrollment, screenings, assessments, treatment, and outcomes.

if i2=1

Cb2. After the grant period ends, will your RPG project collect data about [FILL A-I]?

PROGRAMMER: CODE ONE PER ROW	YES	NO
a. referrals into service	1 	2 
b referrals out to other services	1 	2 
c. enrollment	1 	2 
d. screening for service eligibility	1 	2 
e. participant needs assessment	1 	2 
f. participation in services	1 	2 
g. participant outcomes	1 	2 
h. participant feedback	1 	2 
i. fidelity monitoring	1 	2 

if i2=1

Cb3. For [Fill Cb2a-i=1], has your project determined [FILL A-E] ?

PROGRAMMER: CODE ONE PER ROW	YES	NO
a. the methods that will be used to gather data after the grant period ends	1 	2 
b. who will record or gather the data after the grant period ends	1 	2 
c. where data will be entered and stored after the grant period ends	1 	2 
d. how data will be organized and analyzed after the grant period ends for program monitoring and improvement	1 	2 
e. how data will be shared after the grant period ends	1 	2 

Programmer box:

if I1=1 AND [s1=2 and s2=1] go to END 1

if I2=2 GO TO e1

if i2=1

The following questions are about funding and resources for sustaining RPG services after the grant period ends.

D1. Has the RPG project conducted the following activities to plan and prepare for financing RPG services after the grant period ends? Would you say yes, no or partially?

By partially we mean the activities have not been completed but some activities are underway.

PROGRAMMER: FILL RESPONSE CATEGORY “NA” FOR D1e, f, & g ONLY

PROGRAMMER: CODE ONE PER ROW	YES	NO	PARTIALLY	NA
a. Determined annual costs to sustain RPG services.	1 	2 	3	NA 
b. Identified possible funding source(s) for personnel to carry out RPG services.	1 	2 	3	NA 
c. Identified possible funding source(s) for other resources necessary to carry out RPG services.	1 	2 	3	NA 
d. Secured or awarded financing to sustain services.	1 	2 	3	NA 
e. Identified new organizations that will be working with the partnership after the grant ends.	1 	2 	3	NA 
f. Executed agreements with new organizations that will be working with the partnership after the grant ends.	1 	2 	3	NA 
g. Extended or renewed agreements with existing partners to continue work after the grant ends.	1 	2 	3	NA 
h. Identified strategies to engage external systems (e.g., health, education, housing) for financial, organizational, and other support.	1 	2 	3	NA 

if i2=1

RPG created

D2. Does your organization plan to contribute financial support to the RPG project after the grant period ends? Yes or no.

Select only one

 YES 1

 NO 2

if i2=1 and d2=1

[RPG NAME] from RPG_NAME

RPG created

D2a. How much are you planning to contribute? Please provide your best estimate for this question.

Select only one

10,000-19,999 1

20,000-29,999, 2

30,000-39,999, 3

40,000-49,999, 4

50,000-59,999, 5

60,000-69,999, 6

70,000-79,999, 7

80,000-89,999, 8

90,000-99,999, 9

100,000 or more, 10

DON’T KNOW d

REFUSED r

SOFT CHECK: If D2a= response 10; Please confirm the amount your organization is planning to contribute.

if i2=1

[RPG NAME] from RPG_NAME

Partner Survey

D3. Does your organization plan to contribute the following as in-kind resources to the partnership after the grant period ends. [FILL a-g].

PROGRAMMER: CODE ONE PER ROW	YES	NO
a. Staff time	1 	2 
b. Office space	1 	2 
c. Office supplies	1 	2 
d. Program materials	1 	2 
e. Computer/Internet, telephone, or fax service	1 	2 
f. Transportation	1 	2 
g. Something else	1 	2 
IF D2g=1 i. Other (SPECIFY) (STRING 100)	1 	2 

if i2=1

D4a. What funding sources will your organization use to pay RPG project staff after the grant period ends?

Select all that apply

 Federal funding 1

 State funding 2

 Local funding 3

 Foundations 3

 Other (Specify) 99

Specify (STRING 60)

DON’T KNOW D

if i2=1

D4b. What funding sources will your organization use to cover indirect costs such as computers, training and travel for RPG project staff after the grant period ends?

Select all that apply

 Federal funding 1

 State funding 2

 Local funding 3

 Foundations 3

 Other (Specify) 99

Specify (STRING 60)

DON’T KNOW D

Programmer Box:

If s1=2 and s2=1 go to end1

E. Federal, State, and Local Context

if s1=1 and s2=1 or if i2=2

E1. We would like to understand how federal, state, and local policies and media reporting have affected plans for maintaining the RPG project.

Revised from Guide to Developing, Implementing and Assessing an Innovation

How have plans for sustaining the RPG project been affected by [FILL A-H]?

PROGRAMMER: CODE ONE PER ROW	VERY POSITIVELY	SOMEWHAT POSITIVELY	NOT AT ALL	SOMEWHAT NEGATIVELY	VERY NEGATIVELY
Child welfare
a. the federal policy climate about child welfare	1 	2 	3 	4 	5 
b. the state policy climate about child welfare	1 	2 	3 	4 	5 
c. the local policy climate about child welfare	1 	2 	3 	4 	5 
d. media reporting about child welfare	1 	2 	3 	4 	5 
Substance use disorder treatment programs
e. the federal policy climate about substance use disorder treatment	1 	2 	3 	4 	5 
f. the state policy climate about substance use disorder treatment	1 	2 	3 	4 	5 
g. the local policy climate about substance use disorder treatment	1 	2 	3 	4 	5 
h. media reporting about substance use disorder	1 	2 	3 	4 	5 

if s1=1 and s2=1 or if i2=2

E2. How has the pattern of substance use changed in your service area since the grant period started?

Code only one

 Increase in use 1

 Decrease in use 2

 No change 3

 DON’T KNOW D

 NO RESPONSE M

if s1=1 and s2=1 or if i2=2

E3. Is there anything else you would like share about the effect of federal, state, or local policy or the media on your plans for sustaining the RPG project?

(STRING 1000)

NO RESPONSE M GO TO END 1

if i2=1 and [s1=1 and s2=1] or if [s1=2 and s2=1]

END1. Thank you for taking the time to complete this survey. We appreciate your participation.

IF i2=2 and [S1=2 and s2=2]

END2. Thank you for this information. There are no further questions at this time. We appreciate your participation.