Using Evaluation Results to Drive Decision-Making

use of evaluation to drive policy decision-making, phase 2 MATHEMATICA POLICY RESEARCH

OMB #0990-0421

Expires October 31, 2020

Using Evaluation Results – Phase 2 INTERVIEW Instrument

Research questions

The goal of this study is to conduct interviews with thought leaders and experts in the fields of evaluation, dissemination, and using evidence to drive policy and program decision-making. Through these interviews, we would like to understand the extent to which decision-makers use evaluation and research results (i.e. evidence) to make decisions and identify opportunities for improvements in the decision-making process. The following research questions reflect that goal and served as the starting point for developing the interview questions listed below.

1. What are the barriers and facilitators to using research and evaluation results to inform decision-making?

2. Where are there actionable opportunities to improve the use of evaluation and research results in decision-making?

• Do these opportunities differ depending on the organizations, policy areas, and/or level of government? If so, how?

• What other stakeholders (i.e. foundations, advocacy groups, etc.) have a role in creating actionable opportunities?

3. Which government organizations/agencies have successfully used research and evaluation results to inform policies or decisions?

• Why were they successful?

• How can these successes be replicated?

Interview questions

We will conduct one hour interviews with each respondent. We will use the questions listed below to guide the discussion and help capture data to answer the research questions listed above. Please note that there may be instances where we tailor probing questions based on the respondents’ response and/or work experience.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0421. The time required to complete this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer.

Introduction

Thank you for agreeing to speak with us today. As you know, we are working closely with ASPE to gain a better understanding of whether research and evaluations results are being used to help inform decision-making, and the reasons why it may or may not be used. We wanted to speak to you specifically because of your role as an [researcher/evaluator/evidence producer OR decision-maker] and wanted to capture your insights on this important topic. [Add in additional reasoning based on the respondent’s background and experience].

During this interview, we will focus on capturing your insights on what may be facilitating or hindering the use of research and evaluation results in decision-making within the government, your thoughts on what strategies can be used to improve the use of evaluation results and research evidence, and lastly, what organizations have successfully used evaluation results and research evidence to make important policy decisions.

If there are any questions you feel you do not have the expertise to address, or need to get back to us about, please let us know and we can move on. We’ll be speaking with multiple people about these topics to develop a balanced view of how evidence is being used to make decisions.

We will not reveal the names of our interview respondents or attribute comments to specific individuals in any reports, or share our notes with anyone outside of our research team. If it is okay with you, we’d like to record our discussion to ensure that we capture your comments accurately. If there is anything you share that you wish not to be recorded, please let us know so that we can take additional measures to keep it confidential. The recording will not be shared with anyone outside of Mathematica’s and ASPE’s research team.

Do you have any questions before we begin?

A. Warm-up

Please tell me about what you do here at [insert organization name].

B. Feedback on Barriers and Facilitators to Using Evidence

Now we’d like to spend some time discussing barriers and facilitators to using evidence to make decisions. We describe evidence as research, analysis, or evaluation results.

1. There has been increased attention on evidence-based decision-making and we conducted a literature review to understand, given this shift, what barriers still exist to using evidence. Our review of the literature provided numerous suggestions for why there may be a divide between policy/program decision-making and the use of evidence.

Before we share our findings from the literature review, we want to know what you think have been the main barriers to using research and evaluation results in the decision-making process?

2. Here are some of the barriers we identified in our literature review. Which of these findings resonate with you?

• Researchers do not gear their research towards decision-makers (e.g. researchers often choose research designs and methods not well suited to the needs of decision-makers; researchers’ questions, timelines, and objectives seldom align with those of decision-makers; funders may dictate the topics and types of studies researchers conduct, and funders’ and decision-makers needs may not align)

• The complicated process of policy decision-making can crowd out the use of research (e.g. decision-making is not linear; decision-makers use many inputs; anecdotal evidence rather than more rigorous research may significantly influences decision-making)

• Linking research to evidence can be resource-intensive, and government agencies have limited capacity

• Few researchers and decision-makers work to establish mutually beneficial relationships

3. What other barriers do you think exist but we have not discussed? Probe on whether any of the barriers are faced exclusively by decision-makers, evidence producers, or both. Also probe on why these barriers exist.

4. Which of these barriers do you think most influences the way in which decision-makers use evidence? Why?

5. Which of these barriers do you think most influences the way in which researchers and evaluators conduct research? Why?

6. Thinking about [insert barrier mentioned above], how can decision-makers/those who produce evidence overcome this barrier? Probe on other barriers discussed in question B1 and B2.

7. What factors must be in place to promote successful use of research and evaluation results to drive decision-making at the federal, state, and local government level? Probe on whether these factors exist at all three levels of the government, the degree of interaction between researchers and decision-makers, and how decision-makers are made aware of the availability of evidence.

C. Opportunities to Improve Use of Evidence

Now that we have discussed the existence of certain challenges to using research or evaluation results, we wanted to spend some time discussing what solutions can be used to address these challenges.

1. Besides the responses mentioned earlier when I asked how barriers could be overcame, what other solutions do you believe can help address the challenges with using research and evaluation results to make decisions?

2. Our review of the literature also identified specific solutions to overcome some of the barriers we discussed earlier, such as:

• Designing tools to help decision-makers navigate complex policy issues

• Making information more accessible to decision-makers (i.e. disseminating research findings through the mass media and at networking activities)

• Mixing research and anecdotes because evidence can be more persuasive to decision-makers when it is compelling and tucked into a narrative

• Establishing formal capacity for accessing relevant research within government agencies (i.e. assign a high-level official who is responsible for program evaluation within government agencies)

• After an evaluation, establish a working group of key stakeholders (i.e. researchers, policymakers in governmental agencies that address the issues at hand, and individuals from the public at large) to meet and review the results of an evaluation and the any issues identified by the evaluation, and choose one to five major issues that require evidence-informed policy change

Which of these solutions seem feasible to bridging the gap between evidence and decision-making?

3. What strategies can be used to build relationships between decision-makers and researcher and evaluators, both in the government and in private research organizations, so that decision-makers lean on research or evaluation results more regularly? Probe on which stakeholders would be involved in implementing these strategies. Also probe on whether knowing the information needs of decision-makers can help improve the use of evidence.

   1. How can researchers and evaluators better disseminate their research findings? How can decision-makers make their research needs more well-known? Probe on how organizations like ASPE can support decision-makers.

4. One of the barriers we noted earlier was that funders may dictate the topics and types of studies researchers conduct, and funders’ and decision-makers’ needs may not align. What role can funding agencies play in increasing the use of evidence? What other stakeholders can play a role? Probe on the role of advocacy groups or organizations promoting the evidence-based decision-making.

D. Successful Use of Evidence

Now we would like to discuss those federal and/or state agencies that have successfully used evidence to inform decision-making.

1. What organizations or federal and/or state agencies do you think have been effective in using evidence for decision-making? Why have they been effective? Probe on relationships between researchers and decision-makers at chosen organization, specific organizational characteristics, such as a “culture of learning and evidence-based decision-making,” and the types of tools that have played a role in making organizations or state/federal agencies successful at using evidence (e.g. discussions/panels with experts, contracted evaluations, databases, data visualization tools, systematic reviews of evidence). Probe on the consistency of using evidence (one-time vs. ongoing use).

2. How would you replicate these strategies in other agencies/policy areas? Probe for what elements are necessary to start and/or maintain a “culture of learning and evidence-based decision-making.”

E. Wrap-up

E.1 [All respondents] What haven’t we covered today that you think is important for us to know?

Thank you for your time and thoughtful feedback. Our plan is to analyze the data we captured today and in other interviews and put together a report for ASPE. The report will outline some opportunities and challenges with using evidence for decision-making/policymaking. We’re also hoping to put together a technical expert panel on this topic sometime this fall. So stay tuned for more information about the report and panel.

DRAFT 3