User Experience Interviews Guide 20210212

February 4, 2021

Assessment of Audience Demand for Data Products for the Public Libraries Survey (PLS) and State Library Administrative Agency (SLAA) Survey 

User Experience Interviews:
Screener and Moderator Guide

Prepared for

Matthew Birnbaum, PhD

Office of Impact Assessment and Learning

Institute of Museum and Library Services

955 L’Enfant Plaza, Suite 4000
Washington, DC 20024

Prepared by

RTI International

3040 E. Cornwallis Road

Research Triangle Park, NC 27709-2194

_________________________________
RTI International is a registered trademark and a trade name of Research Triangle Institute.

Contents

Assessment of Audience Demand for Data Products for the Public Libraries Survey (PLS) and State Library Administrative Agency (SLAA) Survey 1

Participant Screener 3

Interview Guide 4

IF PARTICIPANT USES: Public Use Data Files and Documentation 6

IF PARTICIPANT USES: PLS Search and Compare Data Tool 7

IF PARTICIPANT USES: PLS Supplementary Tables 8

IF PARTICIPANT USES: Annual Reports, Topical Briefs, and Summary Products 8

Participant Screener

1. How did you hear about this opportunity? 

   1. Professional organization (e.g., American Library Association) 

   2. IMLS newsletter or email 

   3. Colleague 

   4. Facebook 

   5. Twitter 

   6. LinkedIn 

   7. Other: [specify] 

2. Are you aware of the Institute of Museum and Library Services (IMLS)? 

   1. Yes  

   2. No  

   3. Not sure 

3. Which of the following best describes your area of work or study? 

   1. Library Administration  

   2. Library Services 

   3. Social Science or Education Research 

   4. Data Science 

   5. Journalism or Media [skip to 5]

   6. Other: [specify]

4. Do you currently use data about public libraries as part of your work?

   1. Yes [skip to 7]

No [terminate]

5. Do you currently use data about public libraries as part of your work?

   1. Yes [skip to 7]

   2. No 

6. Are you interested in using data about public libraries?

   1. Yes [skip to “confirmation” screen] 

   2. No [terminate]

 

7. Have you accessed any of the following content on the Institute of Museum and Library Services (IMLS) website? Check all that apply. 

   1. PLS or SLAA public use data files  

   2. Annual reports, topical briefs, and summary products

   3. Supplementary tables 

   4. Library Search and Compare data tool  

   5. None of the above [terminate] 

 

Interview Guide

Welcome and Verbal Informed Consent [5 min]

Welcome! Thank you for participating in this interview today. My name is [name] and I work for RTI International, a non-profit research organization headquartered in Research Triangle Park, NC. We are working with the Institute of Museum and Library Services (IMLS) to better understand current and potential users of their library data. As part of this effort, we’re conducting a series of interviews with people who are familiar with these data products to understand their experiences, including the benefits and challenges of using the products. The purpose of our project is to help IMLS improve the usability and impact of their online data products.  

This interview will last up to 75 minutes.

Before we get started, I am going to review a few details and gather your consent to participate:

• Audio Recording. To make sure that we capture everything you say, we are making an audio recording of this discussion. This recording is for the use of this study and will not be shared with anyone else outside of the study. Is it okay for us to start recording? [if yes, start record; if no, proceed without recording]

• Participation. Your participation in this session is voluntary, and you can stop participating at any time. If you are uncomfortable with any question, you can choose not to answer.  

• Privacy. Your name and contact information will not be given to anyone else, and no one will contact you after our discussion is over unless you give permission. We will not collect any personal or family health information from you. This activity is not considered research with human subjects. 

• Reporting. As part of this study, we will write a report summarizing what we learned from these interviews. We will not include your name or any identifying information in the report. The information that you give us will be anonymized and combined with the responses of other participants. This report will be shared with staff at the Institute of Museum and Library Services and their Library Statistics Working Group. 

• Compensation: You will receive a $50 Amazon gift card for taking part in the study. If you cannot accept this gift card, please let us know now. [document if declining] 

Do you have any questions before we begin?

Do I have your consent to participate? [if a verbal “yes” then continue]

INTERVIEWER: SIGN AND DATE THIS FORM TO INDICATE THAT THE SUBJECT HAS PROVIDED CONSENT TO PARTICIPATE:

Subject Name: _____________________________

Interviewer Name: _____________________________

Interviewer Signature _______________________________ Date ______________

Logistical Items

• My colleague [name] is listening to our conversation and may chime in occasionally.

Background of Participant and Data Needs [10-15 min.]

Objective: Understand participants’ typical job functions and data needs or tasks

Activity Type: Open-ended questions

First, I would like to get a better sense of your job, what kind of information you need to do your job well, and challenges you often run into.

1. To start off, what is your job title?

   1. In a few sentences, can you walk me through a typical day in your job?

Thank you for sharing that. Today we are going to focus on the information, particularly data, you may need to help you succeed at your job.

2. For which parts of your job do you need to access data about libraries?

   1. What are your go-to tools and resources? [If participant only mentions IMLS data, probe to ask what other tools and resources they use to ensure holistic picture of data tools]

   2. Do you go to different places for different information/data needs?

   3. What kinds of questions are you looking to answer with the data?

   4. What format are these data in? Full data sets that you download or PDF reports of summary findings?

   5. To what extent do you need to compare your own library to others in your state?

   6. To what extent do you need to compare libraries in your own state to those in other states?

   7. How important is trend information to you?

3. What do you like about that data source or tool? [Ask about each tool the participant mentions above]

   1. Is there anything particularly frustrating? If yes, what is that?

4. What information about public libraries have you looked for but been unable to find?

   1. What questions have you been unable to answer?

[Jot down the top tasks that the participant is trying to accomplish with the data and translate these into up to 3 jobs-to-be-done (the job they are “hiring” the data product to complete).]

5. I would like to read back to you what I heard as your top needs for public library data. As I read them back to you, please let me know if I have captured these appropriately.

[Revise these jobs-to-be-done and top tasks based on the participant’s feedback.]

6. Which of these needs or tasks is the most important to you? Why is that?

7. On a scale of 1-5, with 1 being very unsatisfied and 5 being very satisfied, how you categorize your current solutions to having each of these needs met?

   1. What makes you say that?

Current Data Product Use [30 min.]

Objective: Understand current use of IMLS data products

Activity Type: Open-ended questions

Now I would like to better understand how you use the data resources from IMLS in particular.

1. For which parts of your job do you access Public Libraries Survey (PLS) and State Library Administrative Agency (SLAA) survey data from IMLS?

   1. What kinds of questions are you looking to answer with that data?

   2. [If participant doesn’t use SLAA data] You mentioned that you don’t use SLAA data, can you tell me why? Is there anything that would make it useful for your job?

2. Which specific IMLS data, reports, or tools do you use to answer those questions?

   1. [If the participant does not mention any of the PLS or SLAA data products, probe to see if they use PLS or SLAA data files, Library Search and Compare tool, PLS Supplementary Tables, and/or annual reports and topical briefs; NOTE which ones the participant uses. Have the participant open the Data Catalog landing page and browse that to jog their memory.]

3. What do you do with the data? [Ask these questions for each product they mention]

   1. What information from this product do you use?

   2. For what purpose do you use the data?

   3. How often do you use this product?

   4. How did you learn about it?

   5. Which of your needs that we discussed previously do you feel like it helps you meet?

   6. Which ones does it not meet?

[Tailor the following questions based on the IMLS product(s) the participant says they use. Do not ask questions of tools they do not use.]

IF PARTICIPANT USES: Public Use Data Files and Documentation

We are going to spend a few minutes talking about the [PLS and/or SLAA] Public Use Data Files and Documentation in particular, since you said you use that product.

4. What do you like about these data files and documentation?

   1. What are things you never want to change?

   2. Is there anything particularly frustrating? If yes, what is that?

5. What are your thoughts on the timing of the release of national data files?

   1. Why do you think the datafiles take a long time to be released?

   2. How might IMLS improve their communication about the steps involved to prepare and finalize the data?

   3. Are you aware of the supplementary tables that are released after the Public Use Data Files and before the Annual Report? Do these suffice for your needs while you await the full report?

6. How important is it for you to have data for more than a single state/territory?

7. Do you use any of the supporting materials with these files? (For example, the datafile documentation.) If so, how?

   1. Are the current materials appropriate, useful, and clear?

   2. What additional supporting files would be useful to you?

8. If you could wave a magic wand to make this the most useful data possible, can you tell me what it would be like?

   1. What additional pieces would it have?

   2. What would it not have?

   3. Would it look different? How so?

9. Any other thoughts on this tool that we have not talked about?

IF PARTICIPANT USES: PLS Search and Compare Data Tool

We are going to spend a few minutes talking about the PLS Search and Compare Data Tool in particular, since you say you use that product. Please go ahead and open the tool in your web browser and share your screen so we can review the tool together.

10. When was the last time you used the Search and Compare tool?

11. What do you like about the Search and Compare tool?

    1. What are things you never want to change?

    2. Is there anything particularly frustrating? If yes, what is that?

12. How important is it for new data to be quickly integrated into Search and Compare?

    1. Please note that there are some dependencies IMLS cannot change. For example, the Fiscal Year 2020 data are currently being collected, with the data collection period ending in August of 2021. After that, data quality checks and data like locale and other geographic information are added before the data are uploaded to the Search and Compare Tool.

    2. How might IMLS improve their communication about the steps involved to prepare and finalize the data?

13. Would it be useful for you to have a similar tool for the SLAA survey data? Why or why not?

14. Similarly, how useful would it be to have just one tool that allows you to access both PLS and SLAA surveys?

    1. Are there other datasets within IMLS or outside IMLS that would be useful to include in the tool? Why or why not?

15. If you could wave a magic wand to make this the most useful tool possible, can you tell me what it would be like?

    1. What additional pieces would it have?

    2. What would it not have?

    3. Would it look different? How so?

16. Any other thoughts on this tool that we have not talked about?

IF PARTICIPANT USES: PLS Supplementary Tables

We are going to spend a few minutes talking about the PLS Supplementary Tables in particular, since you say you use those.

17. What do you like about the PLS Supplementary Tables?

    1. What are things you never want to change?

    2. Is there anything particularly frustrating? If yes, what is that?

18. What are your reasons for using these tables?

    1. What is the most common question you are trying to answer by using these tables?

    2. How are you using the data from the tables?

19. Do you prefer any other public library data products over these tables? If so, which one(s)?

    1. Are there new products you would like to see?

    2. Are there other PLS data you would like to see in the supplementary tables?

    3. Are there other PLS indicators you would like to see in the State Ranking Tables?

20. If you could wave a magic wand to make this the most useful set of tables for your purposes, can you tell me what it would be like?

    1. What additional pieces would it have?

    2. What would it not have?

    3. Would it look different? How so?

21. Any other thoughts on this tool that we have not talked about yet?

IF PARTICIPANT USES: Annual Reports, Topical Briefs, and Summary Products

We are going to spend a few minutes talking about the Annual Reports, Topical Briefs, and Summary Products, since you say you use one or more of those.

22. Which of the specific Annual Reports, Topical Briefs, and Summary Products do you use (e.g., PLS or SLAA Annual Reports, State Profile pages, 2013 Research Brief on Small and Rural Libraries)?

23. What do you like about these reports and/or briefs?

    1. What are things you never want to change?

    2. Is there anything particularly frustrating? If yes, what is that?

    3. How useful are the various types of reports and briefs?

    4. Who do you share these reports with?

24. For what purpose do you use these reports?

25. Are these reports time sensitive for your job? How so?

    1. When are you most likely to use these reports?

    2. How do you use the reports? Walk me through the steps.

26. In your role do you prefer reports that summarize the trends over time in PLS or SLAA data or reports that focus on particular topics that also integrate data sources outside of PLS or SLAA, such as Census data (e.g., the American Community Survey) or education system data (e.g., Common Core Data)?

    1. If you were forced to pick between traditional, uniform analyses of PLS or SLAA indicators or briefs on emerging or timely topics, which would you choose?

27. What specific topics or information would you like covered in these reports or briefs?

    1. How would you use that information?

    2. Which of these topics you just listed would you say is most important to you?

28. If you could wave a magic wand to make these the most useful reports possible, can you tell me what they would be like?

    1. Would it these reports look different? How so?

    2. What additional products would you like to support these reports?

    3. What would these reports not have (e.g., no technical jargon)?

29. Any other thoughts on this tool that we have not talked about?

[Screenshare tools and data products that participant did not state they used and ask the following questions.]

On the screen is another data product that IMLS offers.

30. Were you aware of this?

    1. If so, why have you not used this product?

31. Looking at this product now,

    1. Would this help address any of the needs we discussed earlier, such as [insert needs they discussed at beginning of interview]?

    2. How is the way the information is presented helpful or not helpful to you? Why? What would make it better for you?

Usability Testing of IMLS Product [20 min]

Objective: Assess an IMLS Product through task-based scenarios

Activity Type: Task completion using the IMLS website

Now I would like to shift our focus into seeing you interact with these data products. I am going to ask you to go to the IMLS website and then start sharing your screen. Please close down any programs that you want to make sure I do not see during our time.

[Only guide participant to walk through scenarios based on tools they use. If participant uses more than two products, prioritize two for this activity.]

I am going to give you a scenario and a particular task to complete with a tool. I want you to try to think out loud as you work through the task. When I say, “think out loud”, I want you to speak your thoughts, letting me know what you are looking for, what you think particular buttons will do, what you find confusing, what feature is useful etc. There are no right or wrong answers, I am truly trying to get a peek into how this tool is working or not working for you. As you are working through the task, I may remind you to keep thinking out loud. Let’s start with the first task.

Public Use Data (PUD) Files and Documentation

[Task: Determine the correct data elements for your analysis based on the PLS Public Use Data File Documentation]

Scenario: Imagine in your job you need to compute an indicator for your library that is similar to the ones that IMLS includes in its national reporting. Specifically, you need to be able to write a report for a library board member who wants to know how the visits per person at your library compare to the national level statistic, which was 4.06 per person in FY2018. Open the PLS data file documentation and walk me through the steps you would need to take to determine this?

Supplementary Tables

[Task: Use the PLS Supplementary Tables to compare the number of visitors to public libraries in 2015 compared with 2018 in the participant’s home state.]

Scenario: Imagine in your job you need to find the number of visitors to public libraries in your state in 2015 and compare that with 2018. You know that this information is typically found in the PLS Supplementary Tables. Sharing you screen and talking out loud, can you walk me through how you would find that information and let me know what that number is?

Annual Reports and Topical Briefs

[Task: Use the FY2016 Annual Report to find the number of individuals who attended a public library program.]

Scenario: Imagine in your job you need to find the number of individuals who attended a public library program in FY2016 to add to a report advocating for continued investment in public library programs. You know that this information is typically found in the Annual Reports and Topical Briefs section of the IMLS website. Sharing you screen and talking out loud, can you walk me through how you would find that information and let me know what that number is?

PLS Search and Compare Data Tool

[Task: Use the Library Search and Compare tool to find the total operating expenses for Charleston County, SC and Chapel Hill, NC public libraries.]

Scenario: Imagine in your job you want to understand differences in operating expenses for public libraries, particular comparing the difference in two Southern areas: Charleston County, South Carolina. and Chapel Hill, North Carolina. You know that this information can typically be found through using the PLS Search and Compare tool on the IMLS website. Sharing you screen and talking out loud, can you walk me through how you would find that information and let me know what those total operating expenses are?

[Note where user struggled, where they got slowed down.]

32. How easy was this task for you to complete, on a scale from 1-7. What makes you say that?

33. Are there any changes you would like to see to make this process easier?

34. After using these tools, are you more or less likely to use them? What makes you say that?

Wrap up [5 min]

Objective: Assess

Activity Type: Open-ended questions

Thank you so much for conducting those scenarios today. I have just one closing question.

35. Based on your own experience, on a scale of 1-10, how likely are you to recommend each product to a colleague? [Ask for each product they say they use.]

That’s all we have time for today. Is there anything else you would like to share before we close?