OPRE Research Dissemination Project

ID# OMB Control Number: 0970-0401
Expiration Date: 01/31/2015

OMB Control Number: 0970-0401, Attachment G

Interview Protocol for Intermediary Organizations

Introduction to be read to participants:

Thank you again for volunteering your time to help us to better understand your organization’s informational needs and preferences.

Because this interview is being conducted as a federal research interview, it is important for you to know the regulations that we are following:

• An agency may not conduct and a person is not required to respond to an information collection unless it has a currently valid OMB control number. The OMB control number for this information collection is 0970-0401.

• Your participation is voluntary, and you may refuse to answer any questions or to end the interview at any time.

• A summary of information from all of the interviews we conduct will be shared with OPRE and ASPE to inform and guide their research and communication efforts.

• Your name or personally identifiable information will not be associated with any responses, and information from this interview will not be shared publicly. Your honest answers are of great value to us, so we hope the privacy we offer will allow you to respond frankly to our questions.

• We expect this interview to take one hour or less of your time.

Do you have any questions about these regulations? [Wait for response]. Just one [or two—if multiple interviewees] more procedural piece of information: We will be recording this call for quality control purposes and to ensure that we accurately capture your responses.

[For multiple interviewees only]: Because there is more than one person in the room and I would like to stay within the time frame allotted for this interview, it would be helpful for you to determine who will be the primary responder to these questions based on area of expertise or knowledge. It is not necessary for all interviewees to respond to each question. Therefore, for clarity purposes, before answering any question, please state your name so I will know who is responding.

Finally, as a reminder of why we’re calling today, during this call we want to: 1) learn about the kinds of research information you or your organization use, 2) understand how you and your organization access that information, and 3) discuss how your organization shares research with your constituents. As we shared with you in our preliminary outreach, we’re talking today as part of a project we are doing in partnership with the two research offices in the U.S. Department of Health and Human Services that are focused on human services: the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families and the Office of Human Services Policy in the Office of the Assistant Secretary for Planning and Evaluation (ASPE). These two offices work alongside the federal program offices that administer Head Start, TANF, Child Support, etc. to help inform the direction of policy, programs, and research.

At any time in the interview, please feel free to expand upon your answers.

Let’s get started. I look forward to learning from you.

General Questions

First, it would be helpful to learn a bit about your professional background and interests (ask these for each interviewee):

1. *Describe your professional role and responsibilities within your organization.

2. How long have you worked in your current role?
   
   

3. What are your areas of expertise or interest within the field of human services?

4. What is your age?

Now I have some questions to ask about your organization and its constituents.

5. Describe your organization’s specific or primary areas of expertise.

Probe specific areas of expertise for the organization: child welfare (abuse/neglect/foster care/adoption), early childhood development (child care/ Head Start/Early Head Start/school readiness), welfare/TANF/employment, youth services, marriage/fatherhood/parenting, or cross-cutting (involving multiple areas).

6. *Who are the primary audiences (use his/her terms) for your organization’s work?

Probe: (If multiple): If you had to pick one, which one of these is the priority?

7. Tell me about your organization’s relationship with your audiences.

Probe: How do you interact with your audience? [Note: They may have different processes with different audiences; we would be most interested in audiences that overlap with OPRE/ASPE audiences. Before moving on, clarify how they refer to their audiences and use their terms (i.e. constituents, members, clients, affiliates, etc.)]

How do you assess their information needs? How do you respond to their needs?

8. *Describe the role your organization plays in communicating or brokering information in general about human services programs and/or policy to your audiences.

Probe: What types of information do you share? Are you mostly focused on policy, research, or program/service-related activities?

Probe: Does your organization ever partner with government organizations? [If yes]: For what purpose/to do what? In what ways do you partner?

9. Where, or to whom, does your organization go when looking for reliable information that helps it better understand an issue or concern related to human services?

Follow up questions:

1. Why does your organization turn to that source(s)?

2. What kind of information is sought from that/those source(s)?

3. How do you usually acquire that information?

4. Describe a recent example of when you requested information from another source.

Now I’d like to know a little about your connection to human services research specifically. Because I will be asking you a lot of questions about research in this interview, I’d like to know what that means to you.

10. *What comes to mind when you think of the term “research” or “research-based information”?

For the remainder of the questions, I will be referring to research, which I define broadly as a detailed, careful study of a subject, involving data that is systematically collected and analyzed. And I define research as able to be communicated in many different ways, including but not limited to reports, briefs, briefings or conference presentations, or face-to-face conversations.

Do you have any questions before we proceed?

Value of Research

(Interviewer note: The focus of this section is primarily on the organization, not on the respondent. If the respondent’s answers seem to reflect personal/individual perspectives/practices, record them, but then probe: Does this reflect the view/opinion of your organization as well? If not, what/how does your organization view this differently?)

Now I’d like to ask for your thoughts on the value of research:

1. *What criteria does your organization use to determine if a research finding or report is trustworthy or of value?*

Probe if necessary: What do you mean by ( )? Tell me more. (Ask for specifics, especially related to the trust or value criteria, such as where they got it, the author, timeliness, how useful the research is to them, etc.)

2. *Some parts of a research study may be more important to your organization’s work. Tell me how important each part of the research study is when you review a report or a brief, using the following description choices:
   “Not Important,” “Somewhat Important,” “Important,” or “Very Important”

[Note to interviewer: If interviewee responds with “Not Important” or “Very Important, probe to find out why.]

1. Executive Summary ________

2. Context/Background (in other words, context for the issue being researched) _________

3. Methodology (in other words, how the research was conducted) _________

4. Results or Findings (in other words, results of the research) _________

5. Recommendations or implications for policy and/or practice _________

6. Recommendations for additional research _________

7. Other part or aspect [Probe for specifics if answered affirmatively] _________

Follow up: Do you think your answers to this question would be typical for other people in your organization?”

2. *Now, please tell me about the importance of the following features of research information you consume using these description choices:
   “Not Important,” “Somewhat Important,” “Important,” or “Very Important”

1. Credibility and prestige of the source _________

2. Readability of the information _________

3. Clarity of the findings ________

4. Ability to readily utilize the conclusions or recommendations _________

5. Relevance and applicability of the research to the services or programs that concern you _________

6. Visual appeal of reports (e.g., graphics, color, packaging) _________

Specifically, what makes a research report visually appealing to you? ________

7. Length of the research information/report _________

Possible probe: What length do you prefer?

(Internal note: Question adapted from Landy, Lamari and Amara’s nine dimensions of adaptation of products index. Source: Landry, R., Lamari, M., & Amara, N. (2003). The extent and determinants of the utilization of university research in government agencies. Public Administration Review, 63, 192-205.)

2. What feature of research do you think is most important to your constituents/target audiences? Please explain.

Probe: Does the funding source of the research matter to your constituents?

If Yes:

Why does this matter?

What does it mean to you if the research is federally funded?

Accessing Research Information

(Interviewer note: The focus of this section is primarily on the organization, not on the respondent. If the respondent’s answers seem to reflect personal/individual perspectives/practices, record them, but then probe: Does this reflect the view/opinion of your organization as well? If not, what/how does your organization view this differently?)

Now I’d like to ask you some questions specifically about how your organization accesses research information:

1. *How does your organization find the research that it distributes or shares with its audience? (i.e., personal contacts, library searches, e-newsletters, blogs, social media sites, listservs, researchers send information directly, conferences & meetings, etc.) List specific sources when possible.

Probe: Why does [organization] turn to those sources?

2. *Are there people in your organization responsible for tracking/sharing the latest research?

If Yes:

Probe: What are their names/titles or specific offices or roles (if names aren’t available or appropriate)?

How frequently do you get information from this/these individual(s)?

How is that information shared with you?

What happens with the information that gets tracked or shared?

What do you or your organization do to ensure that you stay current on the latest research in your field?

3. What does your organization look for when it is searching for research-based information to share with your audience(s)? Why?

4. *External to your organization, which specific people, organizations, or state or national networks do you most frequently turn to for research or promising practices related to [organization’s primary issue area(s)]? (Internal note: For probing purposes, we are interested in particular people’s names and job titles, and/or particular names of organizations or networks/associations, pertinent to each issue area for respondents with multiple areas of interest/expertise (i.e., on child welfare issues, org goes to A, B, and C for information; in youth services, they go to D and E). Organization examples may include Association for Public Policy Analysis and Management (APPAM), National Council on Family Relations (NCFR), National Association of Social Workers, Welfare Peer TA Network, alumni network, etc. Respondents may have mentioned some of these in #1 above.)
   
   

Probes:

1. Why do you turn to these people or organizations?

2. What do they offer that you find useful (such as conferences or professional events, e-newsletters, webinars, etc.)? Get specifics.

3. How frequently do you engage with them?

4. Why do you trust these individuals or organizations?

5. *Are these individuals or organizations your constituents also access, or do you consider yourself an intermediary between this organization/its work and your constituents?

(If they mention conferences or events above):

6. What conferences or events do you or your audiences typically attend?

7. What about these events is particularly useful?

8. What is particularly frustrating?

9. How do you use the information gathered at them?

10. Do you ever engage with federal human service organizations (or staff) at these events? If so, at what events do you recall engaging with them?
    
    

5. *Who do you or your audiences identify as opinion leaders in [ISSUE AREA]? (These may be people inside or outside your organization, people others trust, listen to, and have some level of influence; national experts, innovative practitioners, etc. Opinion leaders may be individuals or organizations.)

Probe: Why do you or your audiences consider those individuals or organizations to be opinion leaders?

Who? Why?

1. _______________ ________________________________________

2. _______________ ________________________________________

3. _______________ ________________________________________

6. *Who or what would your audiences identify as the single best resource for research-based information?

To end this section, I’d like to ask you a few questions about your personal access to research-based information.

7. *Do you typically read original research, including research reports and other scholarly documents?

If No: Why not?

If Yes: How frequently? How do you access them?

8. Do you read research briefs?

If No: Why not?

If Yes: How frequently? How do you access them?

9. *With all of the research information you encounter, what is it that draws you in to read or listen to something? (Probe with examples: topic, presentation style, trust of author/speaker, length of resource, the layout, and relevance to current questions/issues you are wrestling with, etc.)

10. *When you receive an email, newsletter, or review a summary online, what drives your decision to click on a link to read more about the research?

(Possible probes: Your desire to learn more? An interesting summary statement? Content of the research seems applicable to your work? etc.)

11. *What is the greatest challenge for you in accessing research?

Possible prompts: time constraints, not necessary to my position, don’t have access to a research library, don’t subscribe to journals, etc.

Now I’d like to ask you a few questions about your use of social media:

14. *Do you use any form of social media such as Twitter, Facebook, or LinkedIn either personally or professionally?

No Yes

(If Yes, clarify type and personal/professional use: )

Twitter ________

Facebook ________

LinkedIn ________

Other ________

Probes:

1. Would you describe yourself as:

1. An observer—I mostly watch what others post

2. An occasional user

3. An active, frequent user

2. Of these types of social media, which one do you use most personally?

3. What do you like about it?

4. Which one do you use most professionally?

5. What do you like about it?

(If not using any or any one of these for professional purposes):

1. Tell me more about why you don’t use _______________ professionally (fill in the blank for each type of social media they do not use professionally.)

2. Are you interested in using any of these forms of social media as part of your work in the future?

Communication

Now I’d like to ask you some questions specifically about your organization’s methods of using and sharing research:

1. Approximately how many people in your organization are responsible for communicating, sharing or disseminating information with your audiences? This may be someone who manages a schedule of webinars, writes the organization’s newsletter or briefs, etc.
   
   

What is your role in this process? (i.e., host webinars, manage web page content, write e-newsletter articles, write briefs, etc.?)

2. * What do you or your organization do to make research more useful to your audience? [If clarification or prompts are necessary: in communicating to the org’s different audiences, it might be necessary to repurpose or repackage research information—e.g., synthesize multiple studies, summarize key findings, attach a testimonial, or invite a researcher to speak on a specific issue.]

1. Do you generally translate or simplify information when repurposing findings from a research report for your audiences?

If Yes: How does this process happen?

2. How do you/does your organization determine what information needs to be “translated” versus choosing to simply distribute or pass along a report in its original form to your target audience?

3. What tools do you use to engage your audience in discussing research (i.e., webinar, panel discussion, etc.)?

4. How do you decide between sharing information in a more interactive way with your audience (i.e., webinar, panel discussion, etc.) or a more passive way (i.e., posting on your website, distributing a report, including it in an e-newsletter, etc.)?

3. Which [issue area] organizations do you think excel at sharing or communicating research?

• Do you ever partner with this organization(s) to distribute information (for example, co-sponsoring an event, cross-linking, or allowing them to post their information on your website)?

4. *Research-based information is communicated today in a variety of ways. For the following list, tell me how useful you think these methods are for sharing research-based information with your audience(s), regardless of whether or not you personally use them for this purpose. Please use one of these responses for each method I share with you:

“Not useful” “Useful” “Very useful”

(Interviewer note: Ask interviewee why they think it is not useful or very useful. What is important about it? If limited on time constraints, only probe on “very useful.”)

1. Printed material (such as a policy brief, journal, newspaper, newsletter or magazine) that is mailed or received on a regular basis __________

2. Electronic versions of similar materials (such as a policy brief, journal, newspaper, e-newsletter, list-serve or magazine) that are received on a regular basis ________

3. Personal discussions with trusted sources through a scheduled face-to-face meeting or phone call __________

4. Personal relationships with researchers __________

5. Personal email from a trusted source __________

6. Audio broadcasts such as a radio show, podcast, etc. __________

7. Webinars, webcasts or web-based presentations __________

8. Twitter __________

9. Facebook __________

10. Linked In __________

11. Other social media (list) __________

12. RSS or blog feeds __________

13. Media campaigns __________

14. Video presentations such as YouTube __________

15. Professional conferences __________

16. Books __________

17. Research clearinghouses (i.e. the Self-Sufficiency Clearinghouse, Child Welfare Information Gateway, Child Care and Early Education Research Connections, etc.)__________

18. News outlets (i.e. CNN, MSNBC, New York Times, Washington Post, etc. __________

19. Specific websites where new research findings and information are prominently featured (If necessary to explain: website could be a news source, a site of trusted advocacy or research organization, etc.) __________

20. Some other way not mentioned (probe for specifics if affirmative) __________

5. *Based on your experience, why do you feel those methods you identified as useful or very useful work? (Ask for specific examples)

6. *Are these the same methods you would suggest a federal research organization use to deliver information to you/your organization? (Probe: Why or why not?)

7. How do you know that your efforts to share research information were effective? Describe how you measure success? (Only if probe is needed: Do you track the number of website visitors, blog users, number of copies sent, etc.?)

8. *When your organization shares research information with your target audience(s), what is it hoping to accomplish? In other words, if someone attends your webinar, reads your brief, cites your paper, etc. what do you want them to do with the information they just gathered? (i.e., build their knowledge base, educate others? about current issues, use the information to inform policy or practice, etc.)

9. *What do you think are the most successful outreach activities that your organization conducts in terms of distributing or sharing research-based information? (Publication, conferences, legislative briefings, newsletters, webinar, host conferences or special events, etc.)

Probe: What makes this so effective?

10. If a research organization such as OPRE or ASPE put out a 100-page research report today, what criteria would your organization use in determining whether to share this information with your audiences?

a. What kind of repackaging of this type of research report would your organization do before distributing the information to your audiences (e.g., distill the 100-page report into a series of briefs, highlight the information in a presentation, etc.)?

b. How do you think your audience(s) best receive(s) this kind of information?

c. [If written formats are identified:] What length of written information is most appropriate for your audiences?

4. [If non-written formats are identified, such as webinar, video, presentation, event, etc.:] What duration/length of [format identified by respondent] is most appropriate for your audiences?

OPRE- and ASPE-Specific Questions

Now I’d like to ask you some questions about your awareness of two human services research offices in the federal government, specifically the Office of Planning, Research and Evaluation (OPRE) and the Office of the Assistant Secretary for Planning and Evaluation (ASPE).

1. Prior to participating in this project, had you ever heard of either of these specific offices?

OPRE: Yes No ASPE: Yes No

2. *Have you ever visited the Office of Planning, Research and Evaluation (OPRE) website? Have you signed up for OPRE’s mailing list? Have you ever received OPRE’s e-newsletter?

2. *Have you ever visited the Assistant Secretary for Planning and Evaluation (ASPE) website? Have you signed up for ASPE’s mailing list? Have you ever received ASPE’s e-newsletter?
   
   

OPRE			ASPE
Website	Mailing list	E-Newsletter	Website	Mailing list	E-Newsletter
YES NO	YES NO	YES NO	YES NO	YES NO	YES NO

(If yes): Probe:

1. How did you first learn of [OPRE or ASPE]?

2. Tell me about your most recent experience accessing information from either of them. (Note if access was through their websites, email, etc.)

3. What did you like/find useful?

4. What did you dislike/find less useful?

5. Did you find the information you were looking for?

6. Was information organized in a way that was easy to navigate?
   
   

2. Can you think of a recent OPRE- or ASPE-funded project or study? This could be something you read, heard about at a conference, learned about from a colleague, found information about on a web-based clearinghouse or website(s), etc.

(If yes) Probe:

1. What do you remember about that resource or study?
   (Try to get the name or at least the subject matter)

2. Where did you get it?

3. How did you use it?

Conclusion

As we wrap up, I have just three more questions for you.

1. From your perspective, what is most valuable to you about the research done by the federal government regarding human services?

2. Based on the work you do, what specific research projects or questions would you like federal research organizations to address to further inform you and others in the field of human services?

(Internal Note: Question adapted from National Child Welfare Workforce Institute Citation: Munson, S. (2010). Disseminating Child Welfare Workforce Knowledge to the Field: A Briefing Paper. Washington, D. C.: National Child Welfare Workforce Institute, 1-57.)

3. In closing, what additional thoughts or comments would you care to share with OPRE or ASPE related to creating or sharing research information?

Thank you so very much for your time today. Your responses will be used to improve the dissemination practices of these organizations in the hopes that their work can support your work. Again, we appreciate your time and have a great day/afternoon!

17