OPRE Research Dissemination Project

OMB Control Number: 0970-0401, Attachment H

Interview Protocol for State Policymakers

Introduction to be read to participants:

Thank you again for volunteering your time to help us understand the informational needs and preferences of state policymakers like you.

Because this interview is being conducted as a federal research interview, it is important for you to know the regulations that we are following:

• An agency may not conduct and a person is not required to respond to an information collection unless it has a currently valid OMB control number. The OMB control number for this information collection is 0970-0401.

• Your participation is voluntary, and you may refuse to answer any questions or to end the interview at any time.

• A summary of information from all of the interviews we conduct will be shared with OPRE and ASPE to inform and guide their research and communication efforts.

• Your name or personally identifiable information will not be associated with any responses, and information from this interview will not be shared publicly. Your honest answers are of great value to us, so we hope the privacy we offer will allow you to respond frankly to our questions.

• We expect this interview to take one hour or less of your time.

Do you have any questions about these regulations? [Wait for response]. We will be recording this call for quality control purposes and to ensure that we accurately capture your responses.

During this call we want to: 1) learn about the kinds of human service information you use, 2) understand how you access that information, and 3) discuss how we can most effectively communicate the information we produce to meet your needs in the future. As we shared with you in our preliminary outreach, we’re talking today as part of a project we are doing in partnership with the two research offices in the U.S. Department of Health and Human Services that are focused on human services: the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families and the Office of Human Services Policy in the Office of the Assistant Secretary for Planning and Evaluation (ASPE). These two offices work alongside the federal program offices that administer Head Start, TANF, Child Support, etc. to help inform the direction of policy, programs, and research.

Let’s get started. At any time throughout the interview, please feel free to expand upon your answers.

General Questions

First, it would be helpful to learn a bit about your professional background and interests:

1. *Describe your professional role and responsibilities.

2. How long have you worked in your current role? How long have you worked in government?

3. What are your areas of expertise or interest within the field of human services?

4. What is your age?

Now, I have some general questions to ask you:

5. *Where, or to whom, do you go when you are looking for reliable information that helps you to better understand an issue or concern related to human services policy?

Follow up questions:

1. Why do you turn to that source(s)?

2. What kind of information do you seek from that/those source(s)?

3. How do you usually acquire that information?

6. Describe a recent instance when you sought information to better understand a human services issue.

Now I’d like to know a little about your connection to human services research specifically. Because I will be asking you a lot of questions about research in this interview, I’d like to know what that means to you.

7. *What comes to mind when you think of the term “research” or “research-based information”?

For the remainder of the questions, I will be referring to research, which I define broadly as a detailed, careful study of a subject, involving data that is systematically collected and analyzed. And I define research as able to be communicated in many different ways, including but not limited to reports, briefs, briefings or conference presentations, face-to-face conversations.

Do you have any questions before we proceed?

8. *I’d like for you to think about a recent example of research-based information that you received in any form. (Make sure they have something specific in mind—a report, finding, statement, graph, etc.)
   Probes:

1. What was it about?

2. How did it come to your attention?

3. Who produced it?

4. What form was it in?

5. Did you share it with someone else?

6. How have you used it in your own work?

7. What would have made it more useful for you?

[If written form (report, brief, article, e-newsletter, etc.):]

8. How lengthy was it?

9. Did you scan or look for anything in particular (e.g., key words, diagrams, tables)?

10. Did you read the whole thing or only certain sections? If sections only, which ones?

9. Describe a recent instance in which you used some form of research-based information to inform a policy or launch or improve a service/practice in your state. (Note to interviewer: Ask this question if the research referred to in #8 was not used to inform a policy or launch/improve a service/practice).
   
   Probe: How did that process unfold?

Value of Research

Now I’d like to ask for your thoughts on the value of research:

1. *What criteria do you use to determine if you trust or value a research finding or report?
   
   Probe: What do you mean by (____)? Tell me more. [Ask for specifics especially related to the trust or value criteria such as who gave it to them, the author, timeliness, how useful the research is to them, etc.]

2. *Some parts of a research study may be more important to you in your work. Tell me how important each part of the research study is to you when you review a report or a brief, using the following description choices:
   “Not Important,” “Somewhat Important,” “Important,” or “Very Important”

[Note to interviewer: If interviewee responds with “Not Important” or “Very Important,” probe to find out why.]

1. Executive Summary ________

2. Context/Background (in other words, context for the issue being researched) _________

3. Methodology (in other words, how the research was conducted) _________

4. Results or Findings (in other words, results of the research) _________

5. Recommendations or implications for policy and/or practice _________

6. Recommendations for additional research _________

7. Other part or aspect [Probe for specifics if answered affirmatively] _________

2. *Now, please tell me about the importance of the following features of research information you consume using these description choices:
   “Not Important,” “Somewhat Important,” “Important,” or “Very Important”

1. Credibility and prestige of the source _________

2. Readability of the information _________

3. Clarity of the findings ________

4. Ability to readily utilize the conclusions or recommendations _________

5. Relevance and applicability of the research to the services or programs that concern you_________

6. Visual appeal of reports (e.g., graphics, color, packaging) _________

Specifically, what makes a research report visually appealing to you? ________

7. Length of the research information/report _________

Possible probe: What length do you prefer?

(Internal note: Question adapted from Landy, Lamari and Amara’s nine dimensions of adaptation of products index. Source: Landry, R., Lamari, M., & Amara, N. (2003). The extent and determinants of the utilization of university research in government agencies. Public Administration Review, 63, 192-205.)

4. Which type(s) of research are you most interested in and why? _________

[Use the examples below as probes if needed, note what they offer]

• Qualitative studies (in other words, findings from case studies, focus groups, the process of implementing a new program, etc.)

• Quantitative studies (in other words, involving statistical analyses using administrative data sets, survey data, program participation rates, etc.)

• Evaluation (in other words, systematic studies to assess how well a program is working—these can be qualitative and/or quantitative)

• Theory building (in other words, findings that confirm or disprove a theory of human service practice)

• Reviews or lists of evidence-based practices (in other words, practices showing the best research evidence in providing health, behavior, and education services)

• Cost-benefit analyses of programs

• Literature reviews or meta-analysis (in other words, studies that review findings from several other studies looking for overall themes or effects)

• Other [Probe: What specifically?]

4. There are lots of organizations that fund and conduct research projects. For you, does the funding source for the research matter? _________

[If Yes]:

Why does this matter? Which sources do you trust/value most? Why?

Possible probes: University-based research, government-funded studies, private/philanthropic studies, etc.

What does it mean to you if the research is federally funded?

Research Sources and Networks

Now I’d like to ask you some questions specifically about who you trust or rely on for research information:

1. Are there people with whom you work who are responsible for tracking or sharing the latest human services research with you?

If Yes:
Probe:

1. What are their names/titles or specific office or role (if name isn’t available or appropriate)?

2. How frequently do you get information from these individuals?

3. How do they share information with you?

2. *External to your organization, which specific people, organizations, or state or national networks do you most frequently turn to for research that influences your policymaking related to child and family-related issues?
   
   

[For probing purposes, we are interested in particular people’s names and job titles, and/or particular names of organizations or networks/associations. Organization examples may include Children’s Trust, County Welfare Director’s Association, National Governors Association, National Conference of State Legislatures, Council of State Governments, American Public Human Services Association, Casey Family Programs, Pew, etc. Be sure to probe for organization names or other more specific identifying information, if initial responses are vague (e.g., “teachers’ union,” “medical association”).]

Probes:

1. Why do you turn to these people or organizations?

2. What do they offer you? (such as conferences or professional events, e-newsletters, webinars, etc.)

3. How frequently do you engage with them?

4. Why do you trust these individuals or organizations?

(If they mention conferences or events):

Probes:

5. Why do you go to these events?

6. What do you like or dislike about the events?

7. Do you ever engage with federal human service organizations (or staff) at these events? If so, at what events do you remember engaging with them?

3. *Who do you identify as opinion leaders for state policymakers interested in human services? (These may be people inside or outside your organization, people others trust, listen to, with some level of influence; national experts, innovative practitioners, etc. Opinion leaders may be individuals or organizations.)

Probe: Why do you consider those individuals or organizations to be opinion leaders?

Accessing Research Information

Now I’d like to ask you some questions specifically about how you access information:

1. *For the following question, I’ll describe ways one might access research information. Please rate each of the statements as to how likely you are to access research to inform your decision making in these ways, using the following response choices:

Not likely; Less likely; Likely; Very likely

1. I read original research myself, including research reports and other scholarly documents ____

2. I read research summaries or syntheses that give context to the research _____

3. I rely on staff members to stay current with research findings and to brief me on information relative to our work _____

4. I have a group of researchers and/or research organizations that I respect and I closely follow their work ______

5. I count on my peers and colleagues around the country to share what they are learning about research _____

6. Are there any other ways that you are likely to access research?
   
   Probe: Which of these possible ways do you engage in most often?

2. *What is the greatest challenge for you in accessing research?

Possible prompts: time constraints, not necessary to my position, the language is too technical, don’t have access to a research library, don’t subscribe to journals, etc.

3. *Research information is communicated today in a variety of ways. I’m about to read a list of different methods of information sharing. The list is a bit long, but it should go quickly. For each method I list, tell me how useful you think these methods are for sharing research-based information with state policymakers, regardless of whether or not you personally use them for this purpose. Please use one of these responses for each method I share with you:

“Not useful” “Useful” “Very useful”

(Interviewer note: If respondent says the method is useful, ask why they think it is useful/what is important about it.)

1. Printed material (such as a policy brief, newsletter or magazine) that is mailed or received on a regular basis __________

2. Personal discussions with trusted sources through a scheduled face-to-face meeting or phone call __________

3. Personal relationships with researchers __________

4. Personal email from a trusted source __________

5. Audio broadcasts such as a radio show, podcast, etc. __________

6. Webinars, webcasts or web-based presentations __________

7. Twitter __________

8. Facebook __________

9. Linked In __________

10. Other social media (list) __________

11. Newspapers __________

12. E-newsletters or listservs __________

13. RSS or blog feeds __________

14. Media campaigns __________

15. Video presentations such as YouTube __________

16. Professional conferences __________

17. Books __________

18. Research clearinghouses (e.g., What Works Clearinghouse, Self-Sufficiency Clearinghouse, Child Welfare Information Gateway, Child Care and Early Education Research Connections, etc.)__________

19. News outlets (i.e. CNN, MSNBC, New York Times, Washington Post, etc. __________

20. Specific websites where new research findings and information are prominently featured

21. Trade journals __________

22. Some other way not mentioned (probe for specifics if affirmative) __________

4. *With all of the research information you get, what is it that draws you in to read or listen to something? (Probe with examples: topic, presentation style, trust of author/speaker, length of resource, the layout, and relevance to current questions/issues you’re wrestling with, etc.)

5. *Regarding research information, when you receive an email, newsletter, or review a summary online, what drives your decision to click on a link to read more about the research?

Possible probes: Your desire to learn more? An interesting summary statement? Content of the research seems applicable to your work? Etc.

6. *How frequently do you do any of the following things related to accessing research information? Please respond using one of the following choices:

“Daily,” “Several times a week,” “Once a week,” “Once a month,” “Less than once a month,” “Not at all”

1. Reading news stories or magazine articles in print or online _____

2. Reading a 1-2 page policy or issue brief, executive summary, or fact sheet in print or online _____

3. Reading policy or issue briefs that are about 5-10 pages long _____

4. Reading a work-related publication that is 20 pages or more _____

5. Reading a comprehensive research report that is 100+ pages _____

6. Reading an issue-specific book or journal _____

7. Talking with professional colleagues, issue-area experts, or your staff about evidence or research findings _____

8. Seeking research information (asking an assistant to find information for you, doing a web search, etc.) _____

9. Reviewing an online video presentation (such as YouTube or TED videos) _____

10. Listening to audio recordings (such as online podcasts or through a MP3 player) _____

11. Reading e-newsletters that come to your e-mail inbox _____

7. How many e-newsletters do you receive weekly that contain human services-related information?

Probes:

1. Do you read most of the e-newsletters that you receive? Why or why not?

2. What makes you more likely to read a specific e-newsletter (e.g., a compelling subject line, the credibility of the organization, concise content, the visual appeal of the e-newsletter, scanning the e-newsletter and seeing a specific topic you are interested in, etc.)?

3. Which organization’s e-newsletters are you most likely to read?

8. *Do you use any form of social media such as Twitter, Facebook, or LinkedIn either personally or professionally?

No Yes

(If Yes, clarify type) Twitter

Facebook

LinkedIn

Other, please specify

Probe:

1. Of these, which do you use most?

2. What do you like about it?

3. Which of these do you use for professional purposes?

(If not using any or any one of these for professional purposes):

*Tell me more about why you don’t use __________________ professionally (fill in the blank for each type of social media they do not use professionally).

Are you interested in using any of these forms of social media as part of your work in the future?

OPRE- and ASPE-Specific Questions

Now I’d like to ask you some questions about your awareness of two human services research offices in the federal government, specifically the Office of Planning, Research and Evaluation (OPRE) or the Office of the Assistant Secretary for Planning and Evaluation (ASPE).

1. Prior to participating in this project, had you ever heard of either of these specific offices?

OPRE: Yes No ASPE: Yes No

2. *Have you ever visited the Office of Planning, Research and Evaluation (OPRE) website? Have you signed up for OPRE’s mailing list? Have you ever received OPRE’s e-newsletter?

2. *Have you ever visited the Assistant Secretary for Planning and Evaluation (ASPE) website? Have you signed up for ASPE’s mailing list? Have you ever received ASPE’s e-newsletter?
   
   

OPRE			ASPE
Website	Mailing list	E-Newsletter	Website	Mailing list	E-Newsletter
YES NO	YES NO	YES NO	YES NO	YES NO	YES NO

(If yes) Probe:

1. How did you first learn of [OPRE or ASPE]?

2. Tell me about your most recent experience accessing information from either of them (note if access was through their websites, email, etc.).

3. What did you like/find useful?

4. What did you dislike/find less useful?

5. Did you find the information you were looking for?

6. Was information organized in a way that was easy to navigate? Etc.)
   
   

2. Can you think of a recent OPRE- or ASPE-funded project or study? This could be something you read, heard about at a conference or from a colleague, information from a web-based clearinghouse, or something on their website(s), etc.

(If yes) Probe:

1. What do you remember about that resource or study?
   (Try to get the name or at least the subject matter)

2. Where did you get it?

3. How did you use it?

Conclusion

As we wrap up, I have just three more questions for you.

1. From your perspective, what is most valuable to you about the research done by the federal government regarding human services?

2. Based on the work you do, what specific research projects or questions would you like federal research organizations to address to further inform you and others in state government about human services?

(Internal Note: Question adapted from National Child Welfare Workforce Institute Citation: Munson, S. (2010). Disseminating Child Welfare Workforce Knowledge to the Field: A Briefing Paper. Washington, D. C.: National Child Welfare Workforce Institute, 1-57.)

3. In closing, what additional thoughts or comments would you care to share with OPRE or ASPE related to creating or sharing research information?

Thank you so very much for your time today. Your responses will be used to improve the dissemination practices of these organizations in the hopes that their work can support your work. Again, we appreciate your time and have a great day/afternoon!

16

* Indicates priority questions