Site Visit Interview Guide

OMB No. xxxx-xxxx

Expiration Date: xx/xx/xxxx

National Evaluation of Project LAUNCH

Site Visit Protocol and Interview Guide

The Site Visit Protocol and Interview Guide is a data collection instrument that will be used for the national evaluation of Project LAUNCH (Linking Actions for Unmet Needs in Children’s Health) to collect data from grantees during on-site visits. The site visits have two primary objectives.

First, the national evaluation seeks to understand grantees’ progress in program implementation and service delivery. During site visits to Project LAUNCH communities and s, members of the national evaluation team will collect information about the implementation of services within each of the five service categories of interest to Project LAUNCH: 1) mental health consultation; 2) developmental assessments across settings; 3) integration of behavioral health programs into primary care; 4) family strengthening and parenting skills training; and, 5) home visitation. Data collected during site visits will expand and aid in explaining data reported by grantees in the web-based Project LAUNCH data reporting system.

Second, the national evaluation will collect data during site visits about Project LAUNCH activities targeted at infrastructure development and systems change at both the state and community/Tribal levels. We will seek to understand how these activities are implemented and document system change outcomes. Interviews of grantee staff and partners within Project LAUNCH communities/Tribe will be conducted face-to-face during site visits. Interviews of state-level staff and partners will be conducted by telephone, unless these individuals are located within close proximity to the Project LAUNCH community.

Two members of the national evaluation team—a lead site visitor and associate site visitor—will conduct each site visit.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  The OMB control number for this project is xxxx-xxxx.  Public reporting burden for this collection of information is estimated to average 1.25 hours per interview, including the time for reviewing instructions, searching existing data sources, gathering the data needed, and completing and reviewing the collection of information.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to INSERT NAME AND ADDRESS.

General Instructions for Site Visitors

Interviews should be scheduled with the following individuals:

• Child Wellness Coordinator (community/Tribal level)

• Chair of Child Wellness Council (community/Tribal level)

• Other Child Wellness Council Representatives

• Providers

• Evaluator

• Child Wellness Council Coordinator (state level)

• ECCS Coordinator

• Chair of Child Wellness Council (state level)

Depending on the individual’s specific role on Project LAUNCH, not all questions in the interview guide may be relevant. Please refer to the crosswalk table at the end of the interview guide for the questions likely to be relevant to individuals in each of these categories.

At the start of the interview, notify the informant approximately how long the interview will take. The interview may be scheduled in two parts if the informant is unable to dedicate the full time to the interview at one sitting.

Interviews should be scheduled prior to the site visit. However, it is also important to be flexible. Most informants are service providers or managers and their schedules may change, necessitating you to re-schedule an interview.

You can save the informant time and reduce respondent burden by requesting and reading pertinent materials prior to the site visit.

Site Visit Interview Guide

1. GENERAL INFORMATION

1. What is your name?

2. What is your position title?

3. What organization or agency do you work for?

4. What are your responsibilities in addition to those of Child Wellness Coordinator?

5. What is your educational background or training?

6. How long have you been working with the Project LAUNCH program?

7. Are you a member of the Child Wellness Council?

2. COMMUNITY/TRIBAL CONTEXT

8. What specific conditions (e.g., environmental, social, economic, etc.) within the community/ have positive or negative impacts on the health and well-being of young children and families? Please explain.

9. How available are jobs that provide enough income to support the caregivers and/or families targeted by Project LAUNCH?

10. What specific economic conditions positively or negatively affect the development and implementation of Project LAUNCH? Probe for: budget crises, new funding for services, unemployment, staff turnover due to loss of funding.

11. How has the state and local political climate positively or negatively affected the development and implementation of Project LAUNCH? Probe for: mistrust of government-sponsored programs, and/or policy-related factors such as new local or state laws.

12. What other factors have positively or negatively affected the implementation of Project LAUNCH? Probe for: community stigma associated with mental health problems; agency re-organization; etc.

3. STRATEGIC PLANNING PROCESS

13. Ask this question during the first site visit only: What agencies, organizations, and individuals participated in the strategic planning process and how were they identified? How were they engaged in the strategic planning process?

14. Looking back on your strategic planning process, how useful was it to you and your community [or organization]? Please explain.

15. Has the strategic plan for Project LAUNCH been modified since it was first developed? If so, please explain. What factors prompted these modifications?

4. CHILD WELLNESS COUNCIL AND PROGRAM OVERSIGHT

16. Please describe the composition of your Child Wellness Council. Probe for: # of members, types of organizations represented, # of family members represented, how members of the Child Wellness Council were selected. Note: Some information may be available in the annual web-based report; verify that it is correct and complete.

17. Describe the diversity in membership on the Council in terms of representation of the target population for Project LAUNCH. Please explain.

18. What formal policies exist requiring cultural and ethnic/racial diversity on the Child Wellness Council?

19. How many members of the Child Wellness Council are decision-makers (that is, people who can influence funding, policy, and programming of the organization that they represent)?

20. How much turnover has there been in Council membership over the past year? If there has been turnover, please explain the extent of the turnover. How are these members replaced?

21. What organizations and agencies do you believe should be represented on the Council, but are not currently? If any, why aren’t these organizations and agencies represented? Probe for whether they have been invited to participate and whether there are plans to engage them in the future.

22. Please describe the functions of the Child Wellness Council. Probe for whether the Council makes decisions about Project LAUNCH or is advisory. Note: Some information may be available in the annual web-based report; verify that it is correct and complete.

23. How has the role of the Child Wellness Council changed or evolved over time?

24. Was the Council created for Project LAUNCH, or did it exist previously? If the latter: What was the role of the Council prior to Project LAUNCH? How has the composition of the Council changed since Project LAUNCH?

25. What are the bylaws for the Council? If this is the first visit or the bylaws have changed, ask for a copy of the bylaws.

26. How often does the full Council meet?

27. What subcommittees or workgroups of the Council have been formed? What is the purpose of these subcommittees or workgroups? How often do the subcommittees or workgroups meet?

28. How are decisions or recommendations by the Council made? Probe for whether decisions are made by consensus or majority vote, whether all members have an equal role in the decision-making or advisory process.

29. To what extent are the decisions or recommendations of the Child Wellness Council acted upon or implemented? Please provide examples.

30. Overall, how well would you say that the Child Wellness Council is functioning? From your perspective, are there specific areas where operations and effectiveness of the Council could be improved?

31. From your perspective, what have been the most important or significant activities and accomplishments of the Child Wellness Council?

32. In what efforts, if any, has the Child Wellness Council been less successful? How has the Council addressed these challenges?

33. What organizational factors, if any, limit the ability of the Council to be effective?

34. Are there contextual factors that have challenged, or limited, the Council’s effectiveness (e.g., members have no history of collaboration, competing agendas of agencies represented, unfriendly political scene)? Are there contextual factors that have contributed to the Council’s success?

35. For grantees where the Council oversees more than Project LAUNCH: Are there other areas/programs that have required more immediate attention from the Council?

36. If informant’s agency is represented on the Council: What is the value of membership on the Council to the organization or agency that you represent? Has your organization’s involvement in the Council (or with Project LAUNCH) changed the way your organization does business? If so, how?

37. How do the local and state Child Wellness Coordinators interact with one another? Please explain.

5. PROJECT OUTREACH/AWARENESS

38. What outreach/awareness strategies have been used to identify and encourage participation by children and families in Project LAUNCH-funded programs and services? Have some outreach/awareness strategies been more effective than others? Please explain. Probe for differences in strategies by target group or population, or other background characteristics.

39. Who has been involved in these outreach/awareness strategies?

40. How successful has Project LAUNCH been in reaching children and families that are the target audience for programs and services? Please provide examples

41. Has participation in each service or program met the goals set out for Project LAUNCH?

42. Once families and children begin to receive services and programs funded by Project LAUNCH, what strategies are used to maintain their participation in the program for its full duration?

43. Have some of these retention strategies been more effective than others? Please explain.

44. What factors (e.g., environmental, social, economic, etc.), unrelated to Project LAUNCH, prevent children and families from initially participating in Project LAUNCH programs and services? Probe for: issues related to transportation, economics, childcare, culture, values, etc.

45. What factors contribute to program attrition (i.e., children and families leaving the program before it is over)? What factors contribute to program retention? Please explain. Probe for: issues related to transportation, economics, childcare, culture, values, etc.

46. Have you been involved in any other outreach activities related to Project LAUNCH? Please explain. Probe for: outreach activities related to raising awareness of young child wellness, informing other agencies/organizations of Project LAUNCH, etc.

47. What challenges, if any, have you encountered in conducting outreach activities related to Project LAUNCH? How have you overcome these challenges?

6. PROJECT LAUNCH SERVICE DELIVERY

48. How are you involved, if at all, in service delivery?

49. What methods do you employ to ensure adherence to the program model in service delivery? Who is tasked with ensuring adherence?

50. Overall, what challenges, if any, have you experienced in delivering Project LAUNCH-funded services? Probe for: challenges related to organizational capacity, staffing, program participation, fidelity to the program model, cultural appropriateness, coordination with other programs, and data collection.

How have you addressed or overcome these challenges?

51. From your perspective, what have been the most significant accomplishments so far for your Project LAUNCH program overall? To what do you attribute these accomplishments? Probe for: community/Tribal/organizational norms, values, policies, community/Tribal/organizational leadership, and coordination across services.

52. What services in your community or organization, if any, were discontinued after the start of Project LAUNCH? Please explain.

Services or Programs Supported by Other Sources of Funding

We’re also interested in learning about other services offered within the /community that fall within the Project LAUNCH categories of 1) mental health consultation; 2) developmental assessments across settings; 3) integration of behavioral health programs into primary care; 4) family strengthening and parenting skills training; and 5) home visitation, but that are not funded by Project LAUNCH. The next set of questions is about these other services.

53. Does the community provide any services or programs within these categories that are funded from sources other than Project LAUNCH? If so, what are these services? Complete the bottom half of Table 1 during the first site visit. On subsequent site visits, refer to and verify information collected during first site visit. Ask each of the following as a probe to verify the information in Table 1, as programs may change, be added, or be eliminated over time.

• Into which category does each of these programs, or services, fall into? Probe for: 1) mental health consultation; 2) developmental assessments across settings; 3) integration of behavioral health programs into primary care; 4) family strengthening and parenting skills training; and 5) home visitation.

• What evidence-based model are you using for each of these programs? Note: Ask this question only for services that are in the following categories: developmental assessments across settings; family strengthening and parenting skills training; and home visitation. Also, some key informants may not be able to provide this information. If this is the case, Skip to next question.

• What age children are targeted by each of these other services or programs?

• What group of parents is targeted by each of these other services or programs?

• What are the sources of funding for these other services or programs? Note: Some key informants may not be able to provide this information. If this is the case, skip to next question.

• How are you involved, if at all, in delivery of these other services or programs?

54. To what extent are Project LAUNCH-funded services coordinated with these other services and programs within your community/ [or organization]? Please explain.

55. Approximately what percentage of funding for all child wellness programs in your community (i.e., those funded by Project LAUNCH and the other services and programs provided) are supported by the Project LAUNCH grant? Probe: refer to your environmental scan.

7. WORKFORCE DEVELOPMENT & CAPACITY BUILDING

56. What activities has Project LAUNCH sponsored at the community level to build the capacity of providers and/or organizations and agencies to serve the needs of young children and families? Probe for: who has participated in these activities; what results, if any, have you observed from these activities

57. What process have you gone through to decide on the type of workforce development (professional development and training/technical assistance) activities you’ve conducted and who would participate? Probe for: how were topics selected, selection of participants, etc.

58. What workforce development (professional development and training/technical assistance) activities are planned over the next year of Project LAUNCH? Will these activities include newly-hired service delivery providers, previously-hired service delivery providers, or both? How do these activities differ from what you have done previously?

59. Does the agency [or community] have a workforce development plan? If so, how are current and future activities addressed in your workforce development plan? Please describe. Ask for a copy of the plan.

8. SYSTEM COORDINATION

60. To what extent do agencies (and organizations) in the community/ collaborate on system building activities? Probe for: Cross-referral, shared funding, shared training, co-location of staff, common intake forms, etc.

Please describe and provide examples of collaboration activities and what agencies/ organizations) have been involved.

61. How has Project LAUNCH assisted in establishing or maintaining these system building activities?

62. To what extent are services for young children and families coordinated across agencies (and organizations) in the community/ – for example, coordination among providers of primary care, other specialty medical treatment services, and mental health services? Please describe and provide examples of coordination activities and what agencies (organizations) have been involved.

63. How has Project LAUNCH helped to develop or expand the service coordination activities you just described?

64. To what extent is there a mechanism for coordination and services integration among programs serving young children and families – for example, early intervention and special education, social services, and family support services? Please describe coordination and services integration activities and what agencies (organizations) have been involved. Probe for: common data protocols, common referral form, etc.

65. How has Project LAUNCH assisted in developing these mechanisms and processes for coordination and services integration?

66. What mechanisms or procedures are in place to coordinate services across various service providers and organizations?

Probe for:

• Memoranda of agreement (MOAs)

• Contractual agreements

• Cross-system training

• Joint meetings of multiple organizations

• Meetings of the Young Child Wellness Council

• Common referral form

• Follow-up procedures after referral

• Shared data systems

• Record sharing

• Other

67. If any mechanisms are in place: When were these mechanisms put in place? Did Project LAUNCH have a role in putting them in place? Please explain.

68. How would you characterize the level of coordination and collaboration among services? Across systems?

69. What challenges have you or your agency (organization) experienced when trying to coordinate across services and systems? How have you addressed these challenges?

70. How well do other programs communicate with you about your clients? Probe: Is communication with staff in programs funded by Project LAUNCH different from communication with staff from non-LAUNCH-funded programs?

71. How much do you trust the information you get about your clients from staff working at other programs? Please explain.

9. CULTURAL COMPETENCE

72. To what extent does your agency (or organization) ensure the cultural competence of programs and services for young children and families?

Probe for:

• Perform needs assessments with diverse groups

• Assess what’s important to program participants with regard to language, beliefs, values

• Provide translation services

• Hire a culturally diverse staff

• Design services to meet the needs of culturally diverse groups (e.g., use of traditional healers, flexible times of services, language services)

• Provide cultural competence training

• Have policies and procedures regarding translation of consent forms, materials, and other information in formats that meet the literacy needs of program participants

• Other

73. Has Project LAUNCH helped agencies to ensure cultural competence in programs and services? If so, how? Do specific policies exist within agencies (and organizations) related to cultural competence? If so, please provide examples.

74. Has Project LAUNCH helped agencies develop and incorporate cultural competence policies within their organizations? If so, how?

75. How does Project LAUNCH ensure the cultural competence of service providers?

76. What types of cultural competence training, if any, have been offered to service providers and others in the community/? Who provided this training? Who has participated in the training?

10. QUALITY MONITORING / QUALITY IMPROVEMENT

77. Describe the process for monitoring the quality of services funded by Project LAUNCH? Probe for: quality monitoring process, individuals/agencies conducting the monitoring, how data on service quality are collected and from whom, the types of data collected, how data are used.

78. What types of data are collected to monitor service quality? Probe for: grievance and complaint tracking, client satisfaction survey data, data from client charts, focus group data, outcome data.

79. What service elements are assessed as part of the quality monitoring process? Probe for: quality and content of family/child assessment, program fidelity, qualifications/training of program staff, accessibility of programs/services, cultural appropriateness, family centeredness, program/service outcomes, participant satisfaction.

80. From whom are the data collected?

81. How are the data used? Please explain. Probe for: Tribal/state reporting requirements, Federal reporting requirements, board of health or other government body reporting requirements, service improvement.

11. PROGRAM SUSTAINABILITY AND REPLICATION

82. What efforts or strategies have been undertaken to date to ensure the sustainability of Project LAUNCH? Note: Some information may be available in the annual web-based report; verify that it is correct and complete.

Probe for:

• Seeking other types of funding

• Blending funding streams across agencies

• Implementing a fund development strategy

• Developing specific collaborative efforts

• Working to implement local/Tribal/state laws or policies

• Working to ensure third-party reimbursement for services

• Other

83. Have there been any policy-related activities related to sustaining Project LAUNCH? If so, please explain and provide examples. Probe for: discussed policy needs, developed a policy agenda, informed policy decisions, developed new policies.

84. What role, if any, has the Child Wellness Council had in sustainability planning?

85. What do you believe are the challenges to sustaining Project LAUNCH?

86. From your perspective, what conditions or factors appear to be essential for replicating an initiative like Project LAUNCH in other communities?

Name of Service or Program	Program or Service Category	Program Model Provide Name	Program Adaptations Yes/No	Target Population Provide Age of Children and/or Parent Group	Funding Source Type of Funding (LAUNCH, other Federal, State, Local, or Private)
	(1) MH Consultation (2) Developmental Assessments (3) Integration of Behavioral Health Programs into Primary Care (4) Family Strengthening and Parenting Skills Training (5) Home Visitation Insert # for Service/ Program Category
Project LAUNCH-funded Services or Programs (pre-fill with data reported by grantee in Project LAUNCH Data Reporting System)
Other Services or Programs in Community/Tribe

Key Informant	General Information	Community/Tribal Context	Strategic Planning Process	Child Wellness Council and Program Oversight	Project Outreach	Project LAUNCH Service Delivery	Workforce Development and Capacity Building	System Coordination	Cultural Competence	Quality Monitoring / Quality Improvement	Sustainability and Replication
COMMUNITY/TRIBE
Child Wellness Coordinator	All	All	All	All	All	All	All	All	All	All	All
Chair of Child Wellness Council	1 – 6	All	All	All	All	All	All	All	All	All	All
Other Representative(s) from Child Wellness Council	1 – 6	―	―	All	―	All	―	―	All	―	All
Providers	All	―	―	―	All	48 – 54	―	―	All	All	All
Evaluator	1 – 6	All	―	19, 20, 23, 29 – 34, 37	38 – 45	―	―	61 – 68	―	All	All
STATE
Child Wellness Coordinator	All	All	All	All	All	―	All	60 – 69	All	All	All
ECCS Coordinator	All	―	―	17, 19, 21, 23, 28 – 37	―	―	―	60 – 69	73 – 74	―	―
Chair of Child Wellness Council	1 – 6	All	All	All	All	―	―	60 – 69	73 – 74	All	All
Evaluator (if applicable)	1 – 6	All	―	23, 28 – 35, 37	―	―	―	60 – 69	―	All	All

2