Early Childhood Training and Technical Assistance (T/TA) Cross-System Evaluation Project

9

^{Early Childhood T/TA Cross-System Evaluation} Project

Appendix B1: Semi-structured Interview Protocol for National Center Evaluators

INTRODUCTION: Thank you for taking time to meet with us. As we explained in our email, we contacted you on behalf of the Administration for Children and Families (ACF), which in partnership with us at NORC is designing and implementing a utilization-focused evaluation for the specific purpose of documenting, assessing, and enhancing ACF’s Early Childhood Training and Technical Assistance (EC T/TA) System. The goal of this interview is for us to learn more from you about your Center’s goals, Stakeholders, T/TA services, desired outcomes, dissemination practices, evaluation data quality and utility, T/TA effectiveness, data systems, data systems users, Center evaluation needs, and general impressions. We are still in a preliminary stage of gathering information to inform the design of the national cross-system evaluation that can also support your Center’s continuous quality improvement efforts. We started by reviewing existing work plans, RFPs, and other publically available materials relevant to your Center’s activities. In the course of this interview, we may ask you to confirm or elaborate on our findings from our review of existing documents describing your Center’s activities. We expect this conversation to take no more than 90 minutes.

Please know that your participation is voluntary and choosing not to participate will have no bearing on your participation and access to the research resources developed as a result of these discussions. The information we collect through this process will not be released to the public. It will be used for internal purposes to develop the evaluation design and evaluation support activities, and therefore, we will share what we learn with ACF and the National Center Evaluators’ Group. We will protect individually identifiable information and will summarize responses. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.

Thank you again in advance for sharing your knowledge and insights with us.

1. Goals

We want to begin the discussion today by hearing about your Center’s short- and long-term goals.

1. What are your Center’s intended short-term goals?

   1. Do your Center’s short-term goals differ by T/TA recipient group (e.g., Regional T/TA providers, CCDF Administrators, HS grantees, etc.)?

2. What are your Center’s long-term desired goals?

   1. Do your Center’s long-term goals differ by T/TA recipient group (e.g., Regional T/TA providers, CCDF Administrators, HS grantees, etc.)?

3. The goals of the broader EC T/TA System are to support early care and education programs and early educators in delivering quality services to children and their families, through high quality professional development, practical resources and approaches that build early childhood program capacity, support adoption and implementation of early childhood program best practices, and promote consistent practices across communities, states, tribes, and territories. With these goals in mind:

   1. How do your Center’s goals align with those of the broader EC T/TA System?

   2. How do your Center’s goals differ from those of the broader EC T/TA System?

4. How do other stakeholders within the EC T/TA System (e.g., other National Centers, OHS Regional T/TA providers, OCC State Capacity Building Center, etc.) support your Center’s goals? PROBE: Are there other things you’d like to see the EC T/TA System do to support your goals?

2. Stakeholders

Next we want to learn about your Center’s stakeholders; specifically, your partners in providing T/TA and the recipients of your T/TA services, and how they might inform or shape your services.

1. Who are your Center’s primary T/TA partners (e.g., other National Centers)?

   1. How do you obtain information about your partners’ views and needs?

   2. How do your partners’ views and needs affect your Center’s goals?

   3. How do your partners’ views and needs affect the services your Center provides?

   4. How are your Centers’ resources allocated to meet identified partner needs?

2. Who are the primary recipients of your Center’s T/TA services (include anyone you consider a direct consumer/user or secondary beneficiary, including but not limited to CCDF Administrators, HS grantees, Regional-Level T/TA providers, OCC State Capacity Building Center, etc.)?

   1. Does your Center differentiate services provided between recipients of direct T/TA services and consumers/users of your Center’s products? IF YES, how? [Alternatively, IF YES, can you say more?]

   2. How do you assess your Center T/TA recipients’ needs?

   3. How do you assess your Center T/TA recipients’ readiness for change?

   4. How do your service recipients’ needs affect your Center’s goals?

   5. How do your service recipients’ views and needs affect the services your Center provides?

   6. Can you share with us the protocols that you use to assess their needs (e.g., forms, descriptions, instruments, illustrative examples, results summaries)?

   7. What other information, if any, do you collect from T/TA recipients?

      1. Does the information you collect vary across recipients?

      2. If so, how?

3. T/TA Services

Now we would like to learn more about the specific T/TA services your Center provides.

1. What T/TA services, products or tools does your Center currently provide?

2. How does your Center determine what T/TA services to provide?

3. In your view, what are the biggest obstacles or barriers to your Center’s development of T/TA services? PROBE: Any other barriers?

4. In your view, what are the biggest obstacles or barriers to your Center’s provision of T/TA services? PROBE: Any other barriers?

5. What records do you keep on services provided? PROBE: What format do you use to record these services?

6. Do you keep a record of service requests? PROBE: What format do you use to record these requests?

7. Do you coordinate T/TA requests, service planning and delivery, and data or record keeping with other T/TA providers in the EC T/TA System (e.g., other National Centers, State Capacity Building Center, OHS Regional T/TA providers)?

   1. If so, how?

   2. How are records kept in such instances?

   3. Do you collect any direct feedback from recipients about the T/TA services? IF YES,

      1. What kind of feedback (i.e., specific topics or issues)?

         1. How do you do this?

      2. Do you record this feedback? PROBE: How?

   4. How do you use this information?

   5. Are these records available? IF MENTIONED: Are these summaries, analyses, etc. available?

4. T/TA Effectiveness

Next we would like to learn about how you evaluate the effectiveness of your Center’s T/TA efforts.

1. How do you assess the effectiveness of your Center’s T/TA?

POTENTIAL PROBES:

1. What information do you use to evaluate your Center’s effectiveness?

2. Do you collect information regularly? How frequently? From whom?

   1. Could you share any of that information with us?

3. Do you consult data collected by anyone else (e.g., other Centers, T/TA providers, T/TA recipients)?

   1. If so, do you use it to assess the extent to which your Center’s T/TA is having the intended effects? How?

1. How effectively is your Center meeting Center goals?

   1. On what evidence do you base your opinion?

2. How effectively is your Center meeting EC T/TA System goals?

   1. On what evidence do you base your opinion?

5. Outcomes

Next, we want to learn about the specific outcomes you are interested in or the information you use to measure progress toward Center goals.

1. What information does your Center collect to measure progress towards Center goals? PROBE, IF NECESSARY: Do you measure any outcomes or outputs?

PROBES:

1. How is information collected?

2. How frequently is it collected?

3. How is it recorded?

4. How do you use this information?

5. Do you analyze the data you collect? IF YES:

   1. What kinds of analyses do you complete?

6. Do you report on these analyses? If so, how? With whom?

1. Do you have different outcomes or information for specific types of Center T/TA provided (e.g., webinars, workshops, train-the-trainer, institute, structured peer learning, tailored approaches, etc.)?

   1. How do you use different metrics for specific types of Center T/TA? PROBE, IF NECESSARY:

      1. Do you analyze these?

      2. Can you give me an example of a recent analysis you completed?

6. Dissemination

Now we want to explore how your Center disseminates information.

1. What are your Center’s dissemination goals?

2. What specific dissemination methods does your Center use (e.g., ECLKC, CCTAN, other website(s), Webinars, conference calls, in-person briefings/seminars, publications/newsletters)? PROBES:

   1. Do methods differ by dissemination goal/outcome desired? IF YES: How do they differ?

   2. Do methods differ by audience? PROBES:

      1. How does your Center disseminate information to T/TA partners (i.e., other National Centers)?

      2. How does your Center disseminate information to T/TA recipients (i.e., Regional T/TA providers, CCDF Administrators, HS grantees, child care networks)?

   3. What methods do you use most? Why?

   4. What methods do you use least? Why?

3. What information do you use to guide your Center’s dissemination planning or quality improvement efforts?

4. Do you have summary information on the numbers and types of Center products/tools distributed? Or on who accesses them? IF YES:

   1. What kinds of information?

   2. How do you use the information?

5. Do you collect any direct feedback from recipients about your Center’s dissemination methods?

   1. If yes, how do you do this? PROBES:

      1. How often?

      2. From whom, exactly?

   2. What do you do with that information? PROBE:

      1. Do you have procedures for recording this feedback?

      2. Do you analyze the feedback in any way?

      3. Do you summarize or report the feedback in any way?

      4. Could you share any of that feedback with us?

      5. How do you use this information?

6. How do you assess the effectiveness of your Center’s dissemination methods? PROBES:

   1. What information do you collect?

   2. Do you collect information regularly? How frequently? From whom?

      1. Could you share any of that information with us?

   3. Do you consult data collected by T/TA recipients? IF YES:

      1. How do you use it?

      2. Do you use it to assess the effects of your Center’s T/TA?

7. Overall, how effective are your Center’s dissemination methods? PROBE: What are your most successful? What are your least successful?

   1. Are some approaches more successful with certain target groups?

   2. Do you have any specific metrics relating to your Center’s dissemination methods?

7. Data Quality & Utility

Now we would like to hear your thoughts about the quality and utility of the evaluation data your Center uses.

1. Of all the evaluative information you collect or consult, what do you find most informative? How/why? PROBES, AS NECESSARY:

   1. What do you measure best?

      1. What do you find most informative? Why?

      2. What do you think is most reliable? Why?

   2. What do you measure least well?

      1. What do you find least informative? Why?

      2. What do you find least reliable? Why?

   3. Is there any additional information/data that could enhance your Center’s products and services? [ALTERNATIVELY: What are you not assessing/measuring that you would like to?] PROBES:

      1. What information do you need to enhance your Center’s T/TA provision?

      2. What additional information would you want for management purposes?

      3. What additional information would you want for evaluation purposes?

      4. Does your current data meet all of these needs?

8. Data Systems & Information Technology

This next set of questions asks you to describe the data systems and information technology your Center uses to make available and document the use of your Center’s T/TA services, tools and resources.

1. What data systems (e.g., HS Enterprise System, TA Tracker, Center-proprietary system, websites) do you currently use to make available Center services, tools, and resources? What other purpose/s, if any, does the system serve?

2. What data systems (e.g., HS Enterprise System, TA Tracker, Center-proprietary system) do you currently use to document the use of Center services, tools, and resources? What other purpose/s, if any, does the system serve?

3. Do you use any web analytics or information from websites that distribute your Center’s products?

   1. From what websites do you obtain web analytics?

   2. Do you use ECLKC or CCTAN web analytics?

   3. How do you use the information?

4. Does your Center have one integrated data system or separate systems for different purposes [PROBE: documentation of needs, characteristics of T/TA recipients, services requested/provided, tools and resources accessed by T/TA recipients, data on recipient outcomes, etc.]? [ALTERNATIVELY: Are you able to link or look across the different types of information/data you use?]

5. Are your files computerized, and if so, how? (e.g., Excel spreadsheets, case records for each recipient, database of services provided, etc.)

6. Do you have a separate computer system for accounting (e.g., income and expenses)? IF YES,

   1. Is it integrated with T/TA data (e.g., by linking costs to specific activities)?

7. What do you like about the current T/TA data systems? 

8. Does your current data system meet your evaluation needs? If no, please describe what additional features would be useful.

9. Data System Users

Now we would like to learn more about the people who use these systems.

1. Who accesses your data system(s)?

2. Can Regional T/TA providers access your data system(s)?

   1. If so, which system(s) do they use and for what purpose?

   2. If not, is this something you would consider doing in the future?

3. Do you collect feedback about the data system(s) from data system users?

   1. If so, how is this feedback used?

10. Center Evaluation Needs

An important goal of the national cross-system evaluation is to support stakeholders’ continuous quality improvement efforts. We want to know what research, data resources or tools could support your Center’s efforts.

1. What research, data resources or tools could support your Center’s evaluation activities?

2. What research, data resources or tools could support your Center’s efforts to understand the reach of your products?

3. What research, data resources or tools could support your Center’s efforts to understand the effects of your services?

11. General Impressions

Our last few questions are about your general impressions of the T/TA system as well as your ideas for designing a maximally useful national evaluation.

1. What is important to retain or build upon amidst the transformation of the T/TA System?

2. What do you most want to know about the T/TA system and its functioning?

3. What research question would you prioritize for this evaluation?

4. What haven’t we asked about that surprises you?

5. What else should we ask or know about before trying to design an evaluation?

CLOSING: Thank you for participating in this discussion today. We appreciate your willingness generously share with us your expertise and time. If you have any additional information, comment or questions, please do not hesitate to email or call us at [INSERT INTERVIEWER INFO HERE].