50514_07_Appendix A_Semi-Structured Interviews_Revised_121118_Clean

APPENDIX A

CMI DATA LINKAGES PROTOCOL FOR KEY INFORMANT INTERVIEWS

This page has been left blank for double-sided copying.

Child Maltreatment Incidence Data Linkages

Protocol for Key Informant Interviews

Note: This protocol includes questions for all respondents. Not all questions will be asked of all respondents; questions will be selected and tailored according to the respondent’s position and role in the Child Maltreatment Incidence Data Linkages (CMI Data Linkages) project. In addition, not all questions will be asked in each round of interviews. The questions included in each interview will depend on the status of the site’s project at the time of the interview. In the body of the protocol, R1 designates sections that are likely to be addressed in the first round of interviews, and R2 designates sections that are likely to be addressed in the second round of interviews. Some sections may be addressed in both round of interviews.

Tailored interviews will fall within the estimated length of time (90 minutes). At each site, in addition to the principal investigator, we plan to interview administrators, directors, or managers at agencies that provide administrative data to the site; information technology or data managers; research staff; and legal staff. We will talk to eight or fewer people (including the principal investigator) at each site.

Introductory script

Note to interviewer: The following script should not be read verbatim. As the interviewer, you must be sufficiently familiar with the script to introduce the study and the focus of the interview. Please familiarize yourself thoroughly with the text before you conduct interviews.

Thank you for taking the time to talk with me today. Your participation is important to this study. I am [insert interviewer name], and I work for Mathematica Policy Research, an independent social policy research company.

As you know, your agency/organization/research group is participating in the Child Maltreatment Incidence Data Linkages (CMI Data Linkages) project sponsored by the Office of Planning, Research, and Evaluation and the Children’s Bureau in the Administration of Children and Families (ACF). The primary goal of the project is to identify and enhance existing practices of using linked administrative data to explore the incidence of child maltreatment and related risk and protective factors. The following research questions guide this part of the project:

1. What are promising methods or practices, within and across the sites, for using linked administrative data to inform the incidence of child maltreatment and related risk?

2. What contextual and organizational factors promote or impede enhancement or scaling of existing data linkages?

We would like to talk with you about your CMI Data Linkages project and your experiences in each phase of the data linkage and analytic process. We are interested in learning about the agencies, organizations, and research groups that are implementing the project and the activities your team has completed, including strategies or practices that have helped you move through the data linking and analysis process, challenges that have arisen, and any lessons learned. We also will be talking with [tailor to exclude the category of respondent participating in the current interview] the principal investigator; administrators, directors, or managers at agencies that provide administrative data to the site; information technology or data managers; research staff; and legal staff.

The questions are not intended in any way to judge you, your work, or your agency/organization/research group. There are no right or wrong answers. You are the expert, and we want to learn from you.

Your participation in the interview is voluntary. Your responses will be used only for research purposes, and notes, recordings, and write-ups of notes will not be shared with ACF or any other entities outside the research team, except as required by law. In the reports we prepare for the Administration of Children and Families, we will discuss and compare the experiences of individual sites participating in the CMI Data Linkages project, but we will not name individuals.¹

We expect that the discussion will last 90 minutes. I would like to record our discussion with a digital recorder so I can listen to it later when I write up my notes. No one besides our research team will listen to the recording. If you want to say anything that you do not want recorded, please let me know and I will be glad to pause the recorder. We will destroy the recording once project reports are released. Do you have any concerns about being part of this interview or my recording our discussion?

We want to reiterate that being part of this discussion is up to you, and you may choose not to answer a question, if you wish.

Do you have any questions before we get started?

Note to interviewer: We expect that background information about some respondents and project plans and activities will be available in documents reviewed. Interviewers should familiarize themselves with these documents and tailor questions to confirm or clarify information previously received, as appropriate.

In addition, questions should be tailored to emphasize the type(s) of enhancements that a site is planning to implement. CMI Data Linkages sites will implement one or more types of enhancements to data linkage and analysis: (1) replicating/scaling to implement a data linkage in a new location, (2) using an innovative method for linking and analyzing data, and (3) linking and analyzing new data sources.

1. Respondent’s background

To begin, I would like to hear about your background and responsibilities.

1. What is your official job title?

2. How long have you worked in your current position? How long have you worked with your agency/organization/research group in any capacity?

3. What are your primary responsibilities?

4. What is your role on the CMI Data Linkages project?

5. In general, how would you describe your background or level of experience with linked administrative data or applied data analytics? (By linked administrative data, we mean data sets from different departments or agencies whose records are linked for analysis based on common identifiers or fields.)

2. Site characteristics

Next, I would like to learn more about the characteristics of the agencies/organizations/research groups participating in this project and their partnership.

Participating agencies, organizations, and research groups (R1)

1. Please describe the role of your agency/organization/research group in the CMI Data Linkages project.

2. Is your agency/organization/research group a standalone agency/organization/research group or is it a part of a larger agency/organization/research group? For example, do you work within a division of an overarching agency/organization/research group?

Agency/organization/research group’s previous experience with linking and analyzing data (R1)

Now, I would like to hear about your agency/organization/research group’s previous experience with linking and analyzing administrative data.

[The following questions are intended for principal investigators and administrators or managers from data providers. If the site did not identify a respondent identified with the data provider or acquired data before the start of the CMI Data Linkages project, ask the principal investigator to (1) respond/confirm information about his/her experience and (2) share his/her understanding of the data provider’s previous experience with administrative data linkage and analysis.]

3. Prior to the CMI Data Linkages project, what previous experience, if any, did your agency/organization/research group have with linking and analyzing data? Has your agency/organization/research group been involved in a partnership to link and analyze data previously? [If no previous experience is indicated, go to the following section.]

4. For how many years has your agency/organization/research group been involved in projects related to linking and analyzing administrative data?

5. What was the purpose of your agency/organization/research group’s previous efforts to link and analyze data (for example, research on incidence and risk factors, support policy development, support ongoing service provision, other)?

6. What topics did your agency/organization/research group explore through your previous efforts to link and analyze data (for example, child maltreatment or another topic)? Are there any relevant citations or reports produced through these previous efforts that you could share with us (beyond those you may have already shared)?

7. Generally speaking, how is the CMI Data Linkages project similar to or different from your agency/organization/research group’s previous efforts to link and analyze data?

Type of enhancement attempted (R1)

[Ask the questions in this section of the principal investigator and research staff. For all other respondents, skip to the following section.]

Now, we would like to discuss the data linking and analysis enhancements you pursued through your project. These might have included implementing a proven approach to data linking in a new locality or jurisdiction, using new methods for data linking and analysis, or using a new data source.

8. I understand your project (is attempting to/attempted to) [replicate/scale a linkage to a new locality, use new methods for linking/analysis, and/or use a new data source]. Is that correct?

Throughout the interview, we would like you to think about how the process of accomplishing this enhancement(s) may have differed, if at all, from your previous data linking and analysis experiences.

Replication site selection (R1)

[If the site is replicating/scaling a proven approach, ask the principal investigator the questions in this section. If not, skip to the next section.]

9. Why did you choose your replication site? Do you have a previous relationship with the site? Did the site contact you about scaling/replicating your approach?

10. What about the site made you think it had the potential for successful replication? What criteria did you use, if any, to assess the site’s potential?

Partnership history (R1)

[Ask questions in this section of the principal investigator, the administrator/director at the data provider(s), and representatives of other partners, as appropriate. Ask the principal investigator to respond about each of the agencies/organizations providing data for the project. Ask the administrator/director at the data provider, and representatives of other partners, as appropriate, about his/her partnership with the lead agency/organization/research group for the CMI Data Linkages project (i.e., the principal investigator’s agency/organization/research group).]

Now, I would like to learn more about the partnership involved in implementing your CMI Data Linkages project. [These partnerships could be internal collaborations between different departments within a single agency or external partnerships between different agencies/ organizations/research groups. In some sites, partnerships might be limited in scope and include only data sharing agreements. Interviewers should tailor the questions in this section accordingly.]

11. I understand that your partnership with [agency/organization/research group] is [a new partnership established as part of your work on your CMI Data Linkages project/an existing partnership that was active prior to your CMI Data Linkages project]. Is that correct?

[If it is an existing partnership, ask questions 12-13.]

12. How many years has the partnership been in place?

13. What is the main purpose of the existing partnership (for example, for data sharing, data linking or analysis, or another purpose)?

14. How is the partnership structured? Do you have one primary contact at the partnering agency/organization/research group or do you interact with multiple people from the agency/organization/research group?

15. What were some of the key steps in establishing the existing partnership? [Probe:]

• Did you develop a partnership agreement? If yes, who was involved in developing and reviewing the partnership agreement before approval?

16. About how many months did it take to establish the existing partnership? That is, how long was it from the time you first approached the agency/organization/research group about partnering and the time partnership was formalized (through a partnership and/or data sharing agreement)?

17. For the CMI Data Linkages project, did you add agencies/organizations/research groups to the existing partnership? If so, what agencies/organizations/research groups were newly added?

18. [Sites could have both a partnership agreement and a data-sharing agreement or they could have an agreement that covers both the purpose and structure of the partnership and data sharing. If there is only a single agreement, here we are interested in agreements related to the purpose and structure of the partnership.] Was there a written partnership agreement [an agreement distinct from a data sharing agreement] in place at outset of the CMI Data Linkages project? [If yes, probe:]

• How, if at all, did you modify the partnership agreement for your CMI Data Linkages project?

• What was the reason for any modifications?

• Who was involved in revising and reviewing the modified agreement before approval?

[If it is a new partnership or new partners were added to existing partnership, ask questions 16-19]

19. Has your agency/organization/research group worked previously with the [other or newly added] agencies/organization/research groups that are participating in your project? [If yes, probe:]

• Please briefly describe your previous collaboration with the agencies/organizations/ research groups.

20. Why did you pursue a partnership with [new partner]? [If not mentioned, probe:]

• To facilitate access to data?

• To facilitate access to analytic expertise?

• To facilitate stakeholder involvement?

• Other reasons?

21. What were some of the key steps in establishing your partnership with [new partner]? [Probe:]

• Did you develop a partnership agreement? If yes, who was involved in developing and reviewing the partnership agreement before approval?

• [If partnership established to facilitate accomplishment of enhancement, ask:] Did the process of establishing this partnership differ from the partnering process involved in other data linking and analysis efforts?

22. About how many months did it take to establish your partnership with [each new partner]? That is, how long was it from the time you first approached the agency/organization/research group about participating in the project and the time partnership was formalized (through a partnership and/or data sharing agreement)?

23. In general, what makes it easier to initiate partnerships for these kinds of projects?

24. What challenges, if any, did you face in establishing partnerships for this project?

[Ask the following questions about both new and existing partnerships]

25. What strategies, if any, do you use to maintain the buy-in and engagement of your partners in the project? [Probe regarding specific strategies:]

• Do you communicate regularly with partners? If so, how often?

• Do you have a collaborative process for making decisions?

• What strategies have worked well?

• What strategies, if any, have been less successful, and why?

26. How, if at all, do your efforts to maintain your partnerships differ for individual partners?

3. Context in participating jurisdictions

[The questions in this section are intended to be asked of the principal investigator, the administrator/director from the child welfare agency, and/or the administrator/director from the data provider. If the site did not identify a child welfare and/or data provider respondent, ask the principal investigator to share his/her understanding of the child welfare system and/or the data provider’s infrastructure for data linkage.]

Next, I would like to hear briefly about the context in which you are implementing your project.

Characteristics of child welfare system (R1)

1. Have there been any major changes to the child welfare system within the past five years (for example, leadership changes or reform efforts)? [If there have been major changes, probe:]

• What were the major changes to the child welfare system? Were these state-level or local-level changes?

• Approximately when did these major changes occur?

How, if at all, have these changes affected the implementation of your CMI Data Linkages project? [Ask if the site is using data from the child welfare agency.] Now, I’d like to ask about the information system in the child welfare agency in your jurisdiction.

2. How, if at all, is your agency responding to the new Comprehensive Child Welfare Information System rule? [Agencies have the option to (1) transition their existing system to a CCWIS, (2) continue to use their existing system, or (3) build a new CCWIS.] [If transitioning to or building a new CCWIS, probe:]

• Have changes to the agency’s system affected whether and how data can be linked and analyzed? If so, how?

• How, if at all, have your agency’s CCWIS plans affected the implementation of your CMI Data Linkages project?

Relevant child welfare policies and definitions (R1)

[If the respondent is affiliated with the child welfare agency, ask questions 3 and 4. If the site did not identify a respondent affiliated with the child welfare agency, ask about the principal investigator’s impression of the child welfare agency.]

3. How would you describe the value and importance the child welfare agency in your jurisdiction places on the use of linked administrative data? Would you say it is (1) not a priority at this time (because of too many other pressing issues, lack of resources); (2) an emerging priority; or (3) highly valued and integral to agency work? Please explain.

4. To what extent, if at all, has the value the child welfare agency in your jurisdiction places on analysis of linked administrative data changed in recent years or over the course of the project?

• [If changes occurred, probe:] In what ways has the value the child welfare agency places on analysis of linked data changed?

5. How, if at all, did state and local child welfare policies affect the design and implementation of your CMI Data Linkages project? [Probe:]

• For example, did policies or laws about expunging records of unsubstantiated reports affect the project’s approach? [If yes, ask:] Were these state or local policies or laws?

• For example, did policies around mandated reporters affect the sources of data you considered for your project? [If yes, ask:] Were these state or local policies or laws?

• Did other state or local policies or laws affect the design and implementation of your project? [If yes, ask:] Were these state or local policies or laws?

6. [Ask the principal investigator:] Were you required to get approval from the child welfare agency to conduct the CMI Data Linkages project? If so, what steps were involved in obtaining the approval? How, if at all, did the need to obtain approval affect the design and implementation of your project?

State/local statutory context for data use (R1)

7. In your state or local area, are there laws or regulations (in addition to federal regulations) that govern access to and use of administrative data (for example, privacy laws)? [If yes, ask:]

• How, if at all, did these laws affect the design and implementation of your CMI Data Linkages project?

Infrastructure for data linkage/integration (R1)

8. Does your agency/organization/research group participate in a collaborative or other structured effort(s) to promote data sharing across state or local agencies, organizations, or research groups? [If yes, ask:]

• What is the name and main purpose of the collaborative or other effort(s)?

• How, if at all, has the collaborative or other effort(s) affected the design and implementation of your CMI Data Linkages project?

Replication site context (R1)

[If the site is replicating/scaling a proven approach, ask the principal investigator the questions in this section. If not, skip to the next section.]

9. How, if at all, does the context for data sharing in the replication site differ from the original site? [If not mentioned, probe:]

• Are there different child welfare policies and definitions?

• Does the replication site have a different infrastructure for data linkage/integration?

• Are there different state or local area laws or regulations that govern access to and use of administration data?

• Are there other differences in context?

• How, if at all, did this difference(s) affect the design and implementation of your project?

4. Financial and human resources

Next we would like to discuss the resources that are available to conduct your project. For these questions, please consider both staff resources and other types of resources (for example, information technology).

Sufficiency of resources for the project (R1, R2)

1. Would you say that the project has staff with sufficient time and sufficient expertise to accomplish the work? [If no, probe:]

• Which staff resources are insufficient?

• What additional resources would be helpful for completing the project?

• Do you face any challenges or barriers in accessing those resources? If so, what challenges or barriers exist?

2. Would you say that the project has adequate materials and funds to successfully complete the work? [If no, probe:]

• For what activities are the materials/funds inadequate? What additional resources (other than staff time) would be helpful for completing the project?

3. [Ask the principal investigator and administrators/managers in agencies providing data for the project:] At this point in your project, has the level of effort required been about what you had expected or has it required more or less effort than expected? [If more or less effort, probe:]

• What activities have required [more/less] effort than you had expected?

5. Organizational factors/capacity

Now we would like to hear about some of the organizational capacities it takes to implement a project like your CMI Data Linkages project. I’ll be asking questions about administrative leadership, partnerships, technical expertise, and the ability to communicate and use findings from studies like the CMI Data Linkages project.

Administrative leadership to promote data linking (R1)

[The questions in this section are intended to assess the administrative support available in the data provider’s agency/organization. Respondents affiliated with the data provider should answer the questions about the administrative leadership in their agency/organization. If an administrator or manager from the data provider is not among the interview respondents (for example, if the project team received and linked data before the start of the CMI Data Linkages project), the interviewer should ask the principal investigator for her/his perspective on the administrative leadership of the data provider and customize the protocol accordingly.]

First, we would like to discuss the role of an agency or organization’s leaders in promoting the efforts of the project. By “leaders,” we mean people who are decision makers within the agency and organization and who can help provide the resources and support necessary to complete the project.

1. From your perspective, what were your agency/organization’s main motivations for participating in the CMI Data Linkages project?

2. To what extent do leaders in your agency/organization support the project? [Probe:]

• Is there a leader within your agency/organization who has championed this project? If yes, what is this person’s title and role in the agency/organization?

• How has this leader demonstrated support for your CMI Data Linkages project?

3. To what extent are the leaders of your agency/organization involved in your project? Would you say highly involved, moderately involved, or not very involved? Please explain your response.

4. [If the respondent is the principal investigator, ask:] Does the level of leadership support and/or involvement necessary to accomplish your attempted enhancement differ from what was needed to accomplish your other data linking and analysis efforts? If yes, how?

Partnership quality (R1, R2)

Next, we would like to hear more about the quality of the partnerships involved in your CMI Data Linkages project.

[If the respondent is the principal investigator, ask questions 5-7.]

5. [Ask about the lead agency/organization/research group’s relationship with each agency/organization participating in the partnership.] How would you describe the quality of your partnership with [agency/organization] to implement the CMI Data Linkages project? [Probe:]

• To what extent do the members of the partnership know, understand, and agree on the goals of the project?

• To what extent do members of the partnership understand their roles and responsibilities?

• To what extent are members of the partnership involved in project decision making?

• How would you describe the quality of communication among partners? How is information about the project communicated?

• To what extent do the members of partnership trust and respect each other?

• To what extent are members of the partnership committed to the project?

• To what extent have members of partnership invested the time necessary to accomplish the project in the allotted time?

6. Please tell me a little more about your overall assessment of your partnership with [specific agency/organization]. What do you see as its main strengths and weaknesses?

7. [If the respondent is the principal investigator, ask:] Has the implementation of your enhancement required a different level of support and/or involvement from your partners than what was needed to accomplish your previous data linking and analysis efforts? If yes, how has the support or involvement needed from partners differed?

[If the respondent is the administrator/director in an agency/organization providing data for the project, ask questions 8 and 9.]

8. Thinking about your partnership with [the principal investigator’s agency/organization/research group] on the CMI Data Linkages project, how would you describe the quality of the partnership? [Probe:]

• To what extent do the members of the partnership know, understand, and agree on the goals of the project?

• To what extent do members of the partnership understand their roles and responsibilities?

• To what extent are members of the partnership involved in project decision making?

• How would you describe the quality of communication among partners? How is information about the project communicated?

• To what extent do the members of partnership trust and respect each other?

• To what extent are members of the partnership committed to the project?

• To what extent have members of partnership invested the time necessary to accomplish the project in the allotted time?

9. Please tell me a little more about your overall assessment of your partnership with [the principal investigator’s agency/organization/research group]. What do you see as its main strengths and weaknesses?

Technical infrastructure and expertise (R1)

Now I would like to learn about the technical infrastructure and expertise available to support your project. The following questions ask about legal support, data management infrastructure, analytic expertise, and project management support.

Legal and regulatory infrastructure and support

[Ask the questions in this section of principal investigators and administrators/managers in agencies providing data for the project. If there are no respondents from agencies providing data, ask the principal investigator for her/his assessment of the agency providing data.]

10. For the CMI Data Linkages project, who is responsible for addressing legal and regulatory issues related to data sharing and use? What are the roles of the different agencies and organizations in the partnership with respect to these issues?

11. How would you assess your agency/organization/research group’s capacity to address the legal and regulatory issues involved in data sharing? [Probe:]

• Does your agency/organization/research group have data sharing policies and protocols in place? If yes, are these policies and protocols reviewed regularly?

• Does your agency/organization/research group have a process for handling specialized research requests?

12. What type of legal and regulatory support does your agency/organization/research group have in place to support data sharing (for example, internal staff, consultants, other resources)?

13. To what extent are/were legal and regulatory staff (from your agency/organization/research group or providing support to your agency/organization/research group) involved in planning and implementing your CMI Data Linkages project? [Probe:]

• Would you say they are/were highly involved, moderately involved, or not very involved?

• Who was involved? What are the individuals’ agency/organization/research group roles and responsibilities?

• What activities are/were they involved in for the CMI Data Linkages project?

14. Overall, would you say that your agency/organization/research group has the necessary legal and regulatory expertise and support to meet the needs of your CMI Data Linkages project? [Probe:]

• What additional expertise, if any, is needed?

• Has your agency/organization/research group experienced any turnover in the legal and regulatory staff supporting the project? [If yes, ask:] How, if at all, did the turnover affect implementation of the project?

15. [If the respondent is the principal investigator, ask:] Did the level of legal and regulatory support required to accomplish this project differ from the support needed to accomplish your other data linking and analysis efforts? If yes, how?

Data management infrastructure and expertise

Now I would like to hear about your agency/organization/research group’s data management infrastructure and expertise.

[Unless otherwise noted, ask the questions in this section of (1) the principal investigator and information technology or data security staff from the agency, organization, or research group leading the CMI Data Linkages project and (2) administrators/managers and information technology or data security staff from the agency or organization providing data.]

16. [If the respondent is affiliated with a data provider, ask:] How would you describe the quality and sophistication of your data management infrastructure? [Probe:]

• To what extent and how do staff use the data system?

• To what extent does the system have data security measures in place?

• To what extent, if at all, does the system support data sharing?

17. What type of support for information technology and data management, and data security support does your agency/organization/research group have (for example, internal staff, vendors, consultants, other)?

18. To what extent are/were information technology, data management, and data security staff from your agency/organization/research group involved in planning and activities related to your CMI Data Linkages project. Would you say highly involved, moderately involved, or not very involved? Please explain.

19. Overall, would you say that your agency/organization/research group has the necessary information technology, data management, and data security infrastructure or expertise to meet the needs of the CMI Data Linkages project? [Probe:]

• What additional infrastructure or expertise, if any, is needed?

• Has your agency/organization/research group experienced any turnover in the information technology, data management, or data security staff supporting the project? [If yes, ask:] How, if at all, did the turnover affect implementation of the project?

20. [If the respondent is the principal investigator or information technology staff affiliated with the lead agency/organization/research group in the site, ask:] Did the information technology, data management, and data security infrastructure or expertise required to accomplish this project differ from the infrastructure and expertise needed to accomplish your other data linking and analysis efforts? If yes, how? [Probe:]

• Did the agency or organization providing data appear to have the information technology infrastructure and expertise required to provide the data necessary for your CMI Data Linkages project? How easy or difficult was it for that agency or organization to do so?

Analytic expertise

Now I would like to hear about your agency/organization/research group’s analytic expertise.

21. [Ask this question of respondents affiliated with a data provider. If there are no respondents from agencies providing data, ask the principal investigator for her/his assessment of the agency providing data.] How would you assess your agency/organization’s typical capacity for administrative data linking and analysis? [Probe:]

• Does you agency/organization typically use data from multiple/outside sources for planning or decision-making?

• Does you agency/organization typically incorporate analysis of linked data as part of regular decision making?

• To what extent does your agency/organization have the capacity to manipulate data from multiple/outside data sets, merge data sets, and analyze data?

22. What type of support for data linking and analysis does your agency/organization/research group have (for example, internal staff, consultants, research collaboratives, other)? [If internal staff, ask:] Are these individuals dedicated research staff?

23. T o what extent have staff from your agency/organization/research group with expertise in data analysis been involved in planning and implementing your CMI Data Linkages project? Would you say highly involved, moderately involved, or not very involved? Please explain.

24. Overall, would you say your agency/organization/research group has the necessary data linking and analysis expertise to meet the needs of CMI Data Linkages project? [Probe:]

    • What type of expertise would you say is essential for this project?

• What additional expertise, if any, would be helpful for your agency/organization/ research group to have?

• Has your agency/organization/research group experienced any turnover in the analytic staff supporting the project? [If yes, ask:] How, if at all, did the turnover affect implementation of the project?

25. [If the respondent is the principal investigator, ask:] Has the necessary data linking and analysis expertise required to accomplish this project differed from expertise needed to accomplish your other data linking and analysis efforts? If yes, how?

Project management expertise

[Ask questions in this section of the principal investigator and research staff.]

The final type of infrastructure and expertise I would like to hear about is your agency/ organization/research group’s project management support.

26. How would you assess your agency/organization/research group’s capacity to manage projects, including monitoring the budget, staffing, and schedule? [Probe:]

    • Does your agency/organization/research group have policies and procedures for addressing special projects?

• If yes, to what extent do the set of the processes ensure smooth, effective functioning of projects?

27. What type of project management support does your agency/organization/research group have for the CMI Data Linkages project, specifically (for example, internal staff, consultants, training, other)?

[If staff or consultants are identified, ask:]

28. To what extent have staff with responsibility for project management been involved in planning and activities related to your CMI Data Linkages project? Would you say they have been highly involved, moderately involved, or not involved? Please explain.

29. Overall, would you say your agency/organization/research group has the necessary project management support to meet the needs of the CMI Data Linkages project? [Probe:]

    • What type of project management support would you say is essential for this project?

• What additional project management support, if any, would be helpful for your agency/organization/research group to have?

• Has your agency/organization/research group experienced any turnover in project management staff supporting the project? [If yes, ask:] How, if at all, did the turnover affect implementation of the project?

30. [If the respondent is the principal investigator, ask:] Did the project management support necessary to accomplish this project differ from support needed to accomplish your other data linking and analysis efforts? If yes, how?

Translation capacity (R1)

Now I have a few questions about your agency/organization/research group’s past experience with communicating and using findings from projects like the CMI Data Linkages project.

[Questions 29-32 are for administrative leaders from child welfare agencies and agencies/organizations that provided the data.]

31. [If applicable:] Please tell me about your general experience with the communication of findings from past projects like the CMI Data Linkages project (that is, projects involving data linking and/or analysis). How, if at all, were project findings shared with you?

32. How would you rate the clarity of the findings? Were the findings very easy to understand, somewhat easy to understand, somewhat difficult to understand, or very difficult to understand? Please explain your response. [If somewhat or very difficult to understand, ask:] What would have improved the clarity of the findings?

33. How would you assess the relevance of the findings to your agency’s policies and practices? Would you say they were highly relevant, moderately relevant, slightly relevant, or not relevant? Please explain your response.

34. How, if at all, were the findings used to inform policy and practice in your agency? What changes, if any, did they prompt?

[Questions 33-34 are for the principal investigator.]

35. Please tell me about your general experience with the communication of findings from past projects like the CMI Data Linkages project (that is, projects involving data linking and/or analysis). How do you typically share findings from this type of study? With whom?

36. How, if at all, have you shared findings with representatives of agencies that provided data for the analyses? How, if at all, did you communicate findings with practitioners or other stakeholders?

37. How did these audiences respond to the findings?

38. From your perspective, did the findings seem to inform policy and practice? If so, how?

6. Activities to link and analyze data

The next set of questions focuses on activities necessary to conduct your CMI Data Linkages project. We will ask some questions about different phases of the project, from identifying research questions though reporting findings.

Exploring research questions and data sources (R1, R2)

[The questions in this section are intended to be asked of the respondents involved in planning the project, such as the principal investigator and possibly the administrator/director of the data provider(s).]

First, we would like to ask about some activities that helped lay the foundation for your CMI Data Linkages project. These might include (1) developing research questions, (2) exploring potential data sets and sources, and (3) developing a project plan.

Process for specifying research questions

Let’s start by talking about how you specified the research questions you originally proposed for your project in the project plan. I see from your project plan that you proposed the following questions: [list questions from the plan]. If you have revised your research questions, we will ask about that later. First, we want to focus on how you developed your original research questions. [Note to interviewer: Some sites might have identified two sets of research questions, a set of preferred questions, and a set of “backup” questions if the site is unable to access expected data sources. Interviewers should confirm the set of questions that the site currently focuses on.]

1. Who was involved in developing the research questions for your project? What was each person’s role?

2. How did your project team agree on what the research questions would be? [Probe:]

• Would you say one person or organization was the main driver of the research agenda? If so, which person or organization was the main driver?

• Would you describe the process as a collaborative effort among project partners? If so, how did that process work?

• [If the site is replicating/scaling a proven approach, probe:] To what extent was the replication site involved in specifying the research questions for your project?

3. What factors influenced the selection of research questions for your project? [Probe:]

• Were the selected research questions based on the child welfare agency’s priorities? If so, how?

• Did the selected research questions build on current or previous work done by your agency/organization/research group? If so, briefly describe the other efforts upon which this project builds.

• [If the site is replicating/scaling a proven approach, ask:] Are the research questions for your replication project the same questions you addressed in your original project? If no, how do they differ and why?

• Did you select research questions based on the known availability of relevant data or a data set you were interested in exploring? Please explain.

4. What changes, if any, have been made to the research questions for your project since the project plan was developed? [See note to interviewer above. If changes were made, probe:]

• Why were those changes made?

• When were those changes made?

1. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to specify the research questions for this project?

Approach to exploring data sets and sources

Next, we would like to hear about how you identified the types and potential sources of administrative data you would need for your project. [Note to interviewer: Some sites might have access to datasets at the outset of the CMI Data Linkages project.]

6. How did you identify and select the data sets for your study? For example, did you select the data sets primarily based on the relevance of data elements within the data set, the accessibility of the data set, your previous experience with the data set, your interest in exploring a new data set, or on some other factor? Please explain.

7. What types or sources of data were considered or explored that you did not ultimately use for your project, if any? [If data not used, probe:]

   • Why were these types or sources of data not used?

8. [If the site is replicating/scaling a proven approach, ask:] How, if at all, did the data sets you proposed for the replication site differ from those in the original site?

9. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to explore potential data sets for this project?

Approach to developing project plan

Now we would like to hear about your experience developing your plan for the CMI Data Linkages project.

10. Did you encounter any challenges in developing the project plan? If so, how did you address them?

11. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to develop a plan for this project?

Practices that facilitate or hinder exploration

Now, we would like to hear about any challenges you experienced during the exploration phase of the project and practices that helped you specify research questions, explore data sets, and develop the project plan. [For the questions in this section, probe separately, as appropriate, about the specific tasks involved in the exploration phase: specifying research questions, exploring data sets, and developing the project plan.]

12. Were you able to complete all the tasks in this phase of the project specifying research questions, exploring data sets, and developing the project plan) according to your planned schedule? [If no, probe separately about each task:]

• Which task(s) posed challenges and why?

• How, if at all, were you able to resolve the challenges?

• What, if any, challenges were insurmountable and why?

13. What practices, if any, made it easier to specify research questions, explore data sets, and develop project plans? [Probe:]

• What makes it easier to specify research questions for projects involving administrative data linkage?

• What makes it easier to identify data sets and sources for these kinds of projects?

• What makes it easier to develop plans for these kinds of projects?

Sharing/acquiring data (R1, R2)

[The questions in this section are generally intended for the principal investigator, the administrator/director at the data provider(s), legal staff, and IT and data security staff.]

Now we would like to hear about your process for and experiences acquiring data for your project. Your activities in this phase of the project might include developing data sharing agreements, taking steps to protect data confidentiality and privacy, and securing approvals from review boards or other authorities. [Note to interviewer: Some sites might have acquired data before the CMI Data Linkages project began. If that is the case, ask questions in this section about the process for acquiring that data.]

Approach to developing data-sharing agreements

First, we have a few questions about how you developed data-sharing agreements to support this project.

14. [If project plan indicates a DSA or MOU was not yet in place] I see from your project plan that you were working on establishing a data-sharing agreement/memorandum of understanding with [partner(s)]. Has the agreement(s)/MOU(s) been executed? If not, what is the status of your data-sharing agreement(s)?

15. Who is the data-sharing agreement with (between which agencies/organizations/research groups and which divisions or departments within the agencies/organizations/research groups)?

16. Who was primarily responsible for developing the agreement? What was each partner’s role in developing the agreement? Who from each agency/organization/research group had to review and approve the agreement? [If the site is replicating/scaling a proven approach, probe:]

    • Who was primarily responsible for securing the agreements in the replication site?

17. What issues or concerns, if any, were/have been raised in developing the agreement? [Probe:]

• [If not addressed earlier in interview:] What state laws and agency policies, if any, needed to be considered in drafting the agreement?

• How, if at all, did states’/agencies’ interpretation of federal regulations affect the process of developing the agreement?

• Was it necessary to determine whether the data could legally be shared? If so, how was that determination made and by whom?

• What, if any, provisions of the data-sharing agreement required substantial negotiation or were considered but left out of the agreement?

18. When did you begin developing the agreement(s)? [If agreement has been finalized, ask:] About how long did it take to finalize the agreement, once you began discussing it?

19. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to develop a data sharing agreement for this project? [If the site’s project involves linking a new data set, ask:]

• Thinking specifically about the new data set that your project is using, what made it easier or to develop a data sharing agreement? What challenges did you face?

Approach to protecting confidentiality and privacy

20. In the project plan you described the following approach to projecting the confidentiality and privacy of data used in your analysis. [Summarize approach]. Were you able to implement your proposed approach? Did you have to implement any additional protections?

21. Were methods for protecting data confidentiality and privacy tailored for each database used? If so, how?

22. Have you faced any challenges in implementing your planned approach to protecting data confidentiality and privacy?

23. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to protect the confidentiality and privacy of data used in this project? [Probe:]

    • [If new data set linked, ask:] Were there any unique hurdles (or facilitators) to protecting data confidentiality and privacy particular to the new data set?

    • [If the site is replicating/scaling a proven approach, probe:] Compared to the original site where you implemented your data linking analysis approach, did you have to put into place any new or different privacy or security measures in the replication site?

Approach to securing approvals

24. Was Institutional Review Board (IRB) approval (or another type of approval) required for your project? [If project builds on an existing project:] Was this project covered by the IRB agreement for your existing project or was a modification to the IRB agreement required?

[If IRB approval was required, ask:]

25. Were there any special considerations or questions that arose during the approval process?

26. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to get approval for this project? [Probe:]

• [If the site is replicating/scaling a proven approach, probe:] How, if at all, did the approval process for the replication site differ from that required, if any, for the original site?

• [If new data set linked, ask:] Did the new data set require any additional or different approval processes?

Practices that facilitate or hinder data acquisition and sharing

Now, we would like to hear about any challenges you experienced during the data acquisition phase of the project and practices that facilitated data sharing. [For the questions in this section, probe separately, as appropriate, about the specific tasks involved in the data sharing phase: developing data sharing agreements, developing and implementing procedures to protect confidentiality and privacy, and securing approvals.]

27. According to the project plan, you originally proposed to use the following types of data. [Name types of data or data sets listed in plan.] Were you able to acquire these types of data? If not, which ones were you unable to acquire? Why were you unable to acquire them?

28. Were you able to complete all the tasks in this phase of the project (developing data-sharing agreements, developing and implementing procedures to protect confidentiality and privacy, and securing approvals) according to your planned schedule? [If no, probe separately about each task:]

• Which task(s) posed challenges and why?

• How, if at all, were you able to resolve the challenges?

• What challenges were insurmountable, if any, and why?

29. What practices or circumstances, if any, facilitated data sharing? [Probe:]

• In general, what makes it easier to develop data-sharing agreements for projects involving administrative data linkage?

• What resources (for example, legal consultation or expertise, published guidance, or agreement templates) were most helpful or important in developing your data-sharing agreement?

• Were there any supports or resources not currently in existence that would have facilitated the development and execution of your data-sharing agreement?

30. What makes it easier to develop and implement procedures to protect confidentiality and privacy for these kinds of projects?

31. What makes it easier to secure IRB approval for these kinds of projects?

Preparing and linking data (R1, R2)

[These questions are generally intended to be asked of the principal investigator, IT and data security, and/or research staff.]

Next, we would like to discuss your experience preparing and linking the data for your project. [Note to interviewer: Some sites might have already linked data before the CMI Data Linkages project began. If that is the case, ask questions in this section about the process for linking that data.]

Approach to assessing data and feasibility of linking

First, we would like to hear about how your project team assessed the data you intended to use for your analysis.

[Ask the following questions about the data set(s) being used.]

32. What preliminary information, if any, did you obtain about the data set and/or the technological infrastructure of the managing agency/organization? What did you feel was important to learn about these topics and why? [Probe whether the site explored the following topics:]

• Does the agency/organization manage the database or contract with a vendor to manage the database?

• What procedures are in place for ensuring data quality (for example, whether the database supports algorithms to check and clean the data)?

• Did you determine if [data partner(s)] uses the desired data itself on a regular basis for program management or research purposes (to provide insight into what data would be of higher quality)? If so, how?

• Is there a data dictionary or codebook for the data set(s)? [If yes, for each data set ask:] Who developed it? What information does it include?

33. How, if at all, did you assess the quality and completeness of the data? [Probe:]

• What issues, if any, did you encounter related to data quality and completeness? How, if at all, did you address them?

• What concerns, if any, arose about biases embedded in the data sources? [If concerns identified, ask:] How, if at all, were you able to address those concerns (for example, through weighting, data cleaning, etc.)?

34. What other steps, if any, did you take to assess the feasibility of using the data set for linkage? [Probe:]

• If a preliminary assessment was conducted, who in your organization/agency/research group conducted it?

• For example, did you assess factors such as variable overlap, variable definitions and interrelationships, limitations on use of the data, and so on?

35. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to assess the quality and feasibility of linking data for this project?

• [If the site is replicating/scaling a proven approach, probe:] How, if at all, did the approach used to assess the quality and feasibility of linking the data differ from the approach used in the original site?

• [If new data set linked, ask:] Were there any issues unique to the new data set that arose in the process of assessing the quality and feasibility of linking the new data set?

Approach to processing and cleaning the data

[If the site is using more than one administrative data set, ask the following questions about each data set:]

36. What issues, if any, did you encounter in processing and cleaning the data? [Probe:]

    • Were the data available in existing data sets or did you have to create a new data set to pull the requested data? If someone else had to create the data set, was there a cost associated with doing so?

    • [If issues encountered] How, if at all, were you able to resolve the issues encountered?

37. How, if at all, did your approach to pulling or processing data for this project differ from the approach you have used on other projects?

38. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to process and clean the data for this project?

• [If the site is replicating/scaling a proven approach, probe:] How, if at all, did the approach used to process and clean the data and any issues that arose in the replication site differ from the approach used and issues encountered, if any, in the original site?

• [If new data set linked, ask:] Were there any issues unique to the new data set that arose during the processing and cleaning of the new data set?

Approach to data linking

The next set of questions focuses on the approach or methods you used to link the data sets for your CMI Data Linkages project.

39. [If project builds on an existing project and/or previously linked data:] I understand from your project plan that some data sets you are using for the CMI Data Linkages project were linked before the project began. Is that correct?

40. [If project builds on an existing project and/or previously linked data:] Thinking about the methods used to link these data set(s), please tell me why these methods were used. [Probe:]

• What made these methods appropriate for linking the data sets?

• What challenges, if any, did you encounter when linking the data sets? How were they addressed?

[If the site is using more than one administrative data set, ask the following questions about each data set:]

41. From the project plan, I understand that you proposed to use the [deterministic, probabilistic, machine learning, other] method to link data from [names of data sets]. Why did you propose this method?

42. [If new linking method proposed, ask:] How does this method differ from previous methods you have used? What about this method made you think it had the potential to provide new information on child maltreatment incidence? What does it add?

43. Since the CMI Data Linkages project began, have you revised or refined your method for linking data? [If yes, ask:] How and why have you revised it?

Facilitators and challenges to preparing and linking records

Now, I would like to hear about what strategies or practices were helpful for preparing and linking data for your project and what obstacles you may have faced. For the next few questions, please think about your experience assessing the feasibility of linking data sets, processing and cleaning the data, and completing data linkages. [For the questions in this section, probe separately, as appropriate, about the specific tasks involved in preparing and linking records: assessing the feasibility of linking data sets, processing and cleaning the data, and completing data linkages.]

44. Were you able to complete all the tasks in this phase of the project (assessing the data, processing and cleaning the data, and linking the data) in a timely fashion? [If no, probe separately about each task:]

    • Which task(s) posed challenges and why?

    • How, if at all, were you able to resolve the challenges?

    • What, if any, challenges were insurmountable and why?

    • In what ways, if any, did the tasks in this phase of the project differ from similar projects you have undertaken?

45. [If not already discussed above, ask] What challenges, if any, did you encounter in preparing and linking the data for your project?

46. What practices or strategies, if any, made it easier to assess the data, process and clean the data, and link the data? [Probe:]

• Did the data documentation provided, if any, facilitate the data preparation process?

• What tools, if any, facilitated linking the data sets? For example, did the National Information Exchange Model framework or specific types of software facilitate linking the data sets?

• What practices or strategies made it easier to assess the quality of the data?

• What practices or strategies made it easier to clean the data?

• What practices or strategies made it easier to link the data?

Analyzing data (R2)

[These questions are generally intended to be asked of the principal investigator and research staff.]

Next, we would like to hear about your process for analyzing the data and reviewing the findings.

Analytic methods

47. Based on your project plan, I understand you planned to analyze your linked administrative data by [describe methods]. Why did you plan to use this (these) method(s)? [Probe:]

    • Have you or members of your team used this method before? If so, who performed the analysis? What education or training did that person have to develop and/or implement the method?

    • Was your choice of method influenced by characteristics of the data?

    • Was your choice of method influenced by the research questions?

    • Were you interested in testing a method of analysis that had not been widely used in this field?

48. [If new analytic method proposed, ask:] How does this analytic method differ from previous methods you have used? What about this method made you think it had the potential to provide new information on child maltreatment incidence? What does it add?

49. What changes, if any, have you made to your analytic approach over the course of the CMI Data Linkages project? Why did you make those changes?

50. Compared to your previous experiences with data linking and analysis, what made it easier or more difficult to analyze the data for this project?

51. [If the site is replicating/scaling a proven approach, ask:] In the process of replicating your approach in the new site, did you have to make any adjustments to the analytic method(s) you used in the original site? If yes, what did you have to do differently?

52. [If new analytic method proposed, ask:] What additional analytic capacity, if any, is necessary to implement this new method compared to other methods you have used? What would another entity need to have in place to replicate this analytic method?

Results of analysis

53. Was the method successful in addressing your research questions? Would you say you were able to fully address your research questions, partially address them, or not able to address them at all? Please explain your response.

Facilitators of and challenges to implementing analytic approaches (if applicable to site)

Now, I would like to hear about what factors facilitated your analysis and what obstacles you may have faced.

54. Were you able to complete all the tasks in the analytic phase of the project in a timely fashion? [If no, probe:]

• What challenges, if any, did you encounter conducting your analysis?

• [If challenges identified:] How were you able to resolve the issue(s)? What, if any, challenges were not resolved and why not?

55. What would you say were the strengths and limitations of the analytic methods you used? Please explain.

56. What factors, if any, helped you implement your analytic approach? [Probe:]

• What makes it easier to analyze data for projects involving administrative data linkage?

Reporting findings (R2)

Finally, we would like to hear about your process for sharing the results of your project and building on the findings.

Format, frequency, and audiences for reporting

57. How have you shared or do you plan to share the findings of your study? [Probe:]

• What types of materials have you produced or will you produce (for example, journal articles, reports, briefs, presentations)?

58. Who was/will be the target audience for your findings? [If multiple audiences, probe:]

• Was the same information shared (or will the same information be shared) with each audience?

59. At what points in your project did you share or do you plan to share the results of your analyses?

60. How, if at all, did you share preliminary findings with your study partners or any other stakeholders? [If findings shared:] What was shared and with whom? [Probe:]

• What preliminary or interim findings, if any, did you share or do you plan to share with the child welfare agency?

61. [If findings shared:] What feedback, if any, did you receive? How, if at all, did the feedback affect your next steps?

Facilitators of and challenges to reporting findings

Now, I would like to hear about and practices that made it easier to report your findings and any challenges you faced in reporting.

62. What practices or strategies, if any, made it easier to report your findings in a timely way?

63. What practice or strategies, if any, made it easier to report the findings in a clear and understandable way?

64. What challenges, if any, did you encounter in effectively reporting the findings of the project? Did you face any challenges related to the timeliness, format, or clarity of your reporting? [If challenges were encountered, probe:]

• How were you able to resolve the issue(s)?

• If you were not able to resolve the issue(s), why not?

G. Cross-site collaboration, peer support, and technical assistance (R2)

Next, I would like to hear about your experience related to the cross-site network and the technical assistance provided by the CMI Data Linkages team.

Perceived benefits of cross-site collaboration and technical assistance

1. Did you participate in any cross-site learning network activities or receive any technical assistance provided through the CMI Data Linkages project?

[Ask the following questions of respondents who participated in cross-site learning network activities.]

2. To what extent was the cross-site learning network helpful in completing your project? Would you say it was very helpful, moderately helpful, slightly helpful, or not helpful? Please explain your response. [Probe:]

• Did you identify any specific topics for the cross-site learning network or request any specific assistance from the network? [If the site requested a topic or assistance, ask:] Did the cross-site learning network activities adequately address your issue?

3. What activities provided by the cross-site learning network were the most helpful? The least helpful? [Probe:]

• What topics addressed by the network were the most helpful? The least helpful?

4. Did you access other technical assistance or support from the CMI Data Linkages team as you completed your project? If so, to what extent was the technical assistance or support you received helpful in completing your project? Would you say it was very helpful, moderately helpful, slightly helpful, or not helpful? Please explain your response.

5. Did your project team access support or technical assistance to complete your project, other than the CMI Data Linkages cross-site learning network or technical assistance? If so, what type of support or technical assistance did you access? Who provided it? [Probe:]

• How, if at all, did this additional support or technical assistance help you implement your CMI Data Linkages project?

6. What additional external supports, if any, would have been useful to your team as you implemented your project?

H. Assessment of enhancements to data linking and analysis (R2)

[These questions are generally intended to be asked of the principal investigator and research staff.]

Now, we would like to discuss your assessment of the data linking and analysis enhancements you pursued through your project.

Extent to which enhancements are accomplished

1. [If the site’s project involved more than one enhancement, ask about each enhancement.] To what extent would you say your project successfully [describe enhancement]? [Probe:]

• Would you say the project was able to fully meet its objectives related to the enhancement, partially meet its objectives, or was not able to meet its objectives? Please explain your response.

2. Would you recommend [new data sets and/or new analytic methods] be used by others interested in understanding the incidence of child maltreatment and risk or protective factors? [Probe:]

• Why or why not?

• [If new analytic method used] What changes, if any, would you make to the approach if you applied it in a future analysis?

Factors that facilitate/inhibit enhancements

[Note to interviewer: Only ask the questions in this section if the topics have not already been addressed by the respondent in previous sections.]

3. Are there characteristics of your agency/organization/research group or partnership that you think helped you accomplish [intended enhancement; ask about each enhancement attempted]? If so, what were the most important? [If not mentioned, probe:]

• Was support from leadership of agencies/organizations/research groups important to your ability to implement [intended enhancement]?

• Was the quality of your partnership with other agencies/organizations important?

• Was available technical infrastructure and expertise (legal/regulatory, data management, analytic, project management) important?

4. Are there characteristics of your agency/organization/research group or partnership that you think got in the way of accomplishing [intended enhancement; ask about each enhancement attempted]? If so, what got in the way the most? [If not mentioned, probe:]

• Did a lack of support from agency/organization/research group leaders get in the way of your ability to implement [intended enhancement]?

• Did difficulty with collaboration among partners get in the way?

• Did a lack of technical infrastructure and expertise (legal/regulatory, data management, analytic, project management) get in the way?

5. Did the characteristics of the local agency or system context help you implement [intended enhancement; ask about each enhancement attempted]? If so, what aspects of the local context were most important? [If not mentioned, probe:]

• Did characteristics of the child welfare system support your ability to implement [intended enhancement]?

• Did local child welfare policies?

• [If applicable:] Did the existence of a central clearinghouse for public agency data or other broader system for linking administrative data?

• [If applicable:] Did the existence of an agency/organization/research group collaborative to support data sharing?

• Did state/local laws that govern access to and use of administrative data?

• Did other aspects of the local context?

6. What contextual factors, if any, got in the way of implementing [intended enhancement; ask about each enhancement attempted]? [If not mentioned, probe:]

• Did characteristics of the child welfare system get in the way of your ability to implement [intended enhancement]?

• Did local child welfare policies get in the way?

• Did state/local laws that govern access to and use of administrative data get in the way?

• Did other aspects of the local context get in the way?

7. What general lessons related to enhancing data linkages have you learned that might be useful to others attempting similar projects?

I. New information on child maltreatment (R2)

[The questions in this section are generally intended to be asked of the principal investigator and the administrator/director from the child welfare agency or other partner with expertise in the child welfare field.]

Finally, I would like to discuss your assessment of the project’s findings.

Extent to which findings add to knowledge base on child maltreatment incidence and related risk and protective factors

1. To what extent do you think the project yielded new and novel findings about the incidence of child maltreatment and/or related risk and protective factors? Would you say to a great extent, to a moderate extent, to some extent, or not at all? Please explain your response.

Use of findings and expected topics for future research

2. What would you say are the main policy or other implications of the findings from your project? [Probe:]

• What are the implications for agencies/organizations/research groups in your partnership? For the broader child welfare field?

3. How, if at all, are the data and data analysis findings currently being used?

4. How, if at all, will the approach be used in the future (for example, is this project likely to be a one-time analysis or do you hope to repeat the analysis after a period of time)?

5. How, if at all, have these findings influenced the research agenda of your organization/agency/research group or partnership? [Probe:]

• What additional or follow-up questions or topics for future research did this project generate, if any?

• What, if any, plans have you established for additional research?

1 If respondents have any questions about their rights as research participants, they can contact the New England Institutional Review Board at (800) 232-9570 or (617) 243-3924 or [email protected]. For any other questions, please refer respondents to Matthew Stagner, the director of the CMI Data project, at [email protected] or (312) 994-1044, or Christine Fortunato, the federal project officer, at [email protected] or (202) 401-5560.