Cognitive Interview Guide

National Museum Survey Cognitive Interviews Interview Guide

OMB No. XXXX-XXXX Exp. MM/DD/YYYY

National Museum Survey Cognitive Interviews

Interview Guide

Verbal Script

• Hello. My name is [name], and I work for the American Institutes for Research, or AIR. My colleague, [note taker], is also on the line to take notes during our call today. Thank you for taking the time to meet with us today.

• AIR is developing a National Museum Survey for the Institute of Museum and Library Services, or IMLS, and we are conducting interviews with museum administrators like you to gather feedback on some of that survey’s draft questions. Your input will help to identify any potential issues that need to be addressed prior to piloting the survey in 2023.

• Our interview will last 60 minutes and is completely voluntary. There are no right or wrong answers to the questions I’ll ask you or to those that you’ll see on the screen. As we talk, please keep in mind that we are interested in hearing your thoughts about the questions, including whether any terminology is confusing or unclear, and how you arrive at your answers. Your institution’s answers to the questions themselves, while interesting, are not the focus of this interview.

• Everything that you say today will be confidential, so please speak freely. The AIR team will combine the findings from all interviews in our report to IMLS, and we will not attribute any quotes to you or use your name or the name of your museum in any of our reports. The feedback you provide today will not affect any past, current, or future relationship you might have with IMLS in any way.

• With your consent, we will be recording today’s session to ensure that we capture everything you say and fully understand your thoughts. Only the AIR research team will have access to the recording. We will not share the recording or notes from the interview with IMLS.

• With that, is it okay if I begin the recording?

  • [If yes: Begin recording.]

  • [If no: Restate confidentiality measures for the survey from bullets above and state that interviews can be completed with the camera off, in which case only audio will be recorded. Ask whether interview could be recorded at another time and if so reschedule for that time. If respondent will not consent to recording and cannot/will not reschedule, thank respondent for their time and end the interview.]

• [If video camera not on] Do you have a video camera that we could use?

  • [If no: Continue to Think-aloud Practice]

  • [If yes] Would you mind turning it on? I’ve found that having the video on to see each other is helpful during interviews.

• Before we continue, do you have any questions about what I just said? [Answer any questions]

Think-Aloud Practice

• During the interview, I will show you one survey question at a time and ask you to read the question and any response options out loud. Then, I’d like for you to “think aloud” — that is, tell me what you are thinking — as you determine your answer or describe how you would determine your answer.

• Hearing you talk through your process for answering the questions will help us understand how to make the survey better. If you are able to answer a question after “thinking aloud,” please do. Just keep in mind that I am more interested in how you arrive at your answers than in your actual answers to the survey questions.

• Would you like me to show you how to think aloud? [Continue if no]

[MODERATOR: If yes, show question and model thinking aloud] How many glasses of water have you had today?

• Now, I’d like for you to try. Go ahead and read the question out loud and talk me through your process for answering.

• Great! You’re ready to go.

• Before we begin, we do have a lot to get through today, so please don’t take it personally if I move us forward at any point. We value your input and want to be sure to get your feedback on as many questions as possible.

Item Testing

[MODERATOR: Ask GENERAL PROBES as appropriate in response to any signs of confusion, misinterpretation, or hesitation during the think-aloud portion.]

General Probes
Problematic instructions or items	What is this instruction telling you? In your own words, what is this question asking? What were you thinking when you answered this question?
Unclear/confusing words or terms	What does [term] mean to you? Tell me what you were thinking when you read [term].
Unclear/confusing definitions	What were you thinking as you read that definition? How could we make the definition clearer or less confusing?
Inadequate response options	Are there any response options that are missing from this question? How could we expand these response options to better reflect your experience?
Other cues	I noticed you hesitated when answering this question. Can you tell me why that was? I noticed that you changed your answer to this question. What was the reason?

[MODERATOR: Repeat the instruction below as needed for each new item until the participant begins to read new items aloud without prompting. Where applicable, ask the probe(s) next to the response option the participant selects before asking overall probes.]

• Okay, let’s get started. Please read the information on the screen out loud and then think aloud as you [determine your answer, describe how you would determine your answer].

Question 1 – Governance Structure

Q1. How would you describe your institution in terms of its operational structure?
An independent institution (with no sub-institutions or parent organization)	What does the term “sub-institution” mean to you?
An institution that is part of a museum system	What does the term “institution” mean to you? Tell me about your museum system. Do some or all institutions within the museum system function as independent museums?
An institution that has a parent organization (e.g., university, nonprofit, city/county government)	What type of parent organization does your institution have? Is your parent organization also a museum? [As needed] What does the term “institution” mean to you?
Not sure	Tell me more about why you selected “not sure.” Is there a response option not listed here that describes your institution’s governance structure?
Was there anything confusing or unclear about this question? Please describe. Would you make any changes to improve this question? If so, what changes would you make?

Question 2 – Operational Data

[SHOW IF Q1 = b] Q2a. Can you report operational data (e.g., finances, human resources, visitation) for your institution that is separate from the data for other institutions in your museum system?
Yes	Are you able to report all operational data separately for your institution? If not, what types of data are combined across institutions?
No	Tell me more about why you would be unable to report operational data for your institution only. How is your institution situated within the museum system? Are all operational data combined across the museum system or would you be able to report some operational data for your institution? If so, what types of data could you report?
Was there anything confusing or unclear about this question? Please describe. Would you make any changes to improve this question? If so, what changes would you make?
[SHOW IF Q1 = c] Q2b. Can you report operational data (e.g., finances, human resources, visitation) for your institution that is separate from the data for your parent organization?
Yes	Are you able to report all operational data separately for your institution? If not, what types of data are combined across institutions?
No	Tell me more about why you would be unable to report operational data for your institution only. What would prevent you from doing so? Are all operational data combined with your parent organization or would you be able to report some operational data for your institution? Would your parent organization be able to report operational data separately for your institution?
Was there anything confusing or unclear about this question? Please describe. Would you make any changes to improve this question? If so, what changes would you make?

Question 3 – Building Space

Q3. What is the total square footage of your institution’s public and non-public building spaces? Estimate to the best of your ability as needed. For mixed-use spaces, please divide your responses across categories as appropriate.

Public building space (square footage)

Non-public building space (square footage)

• How would you define “public” and “non-public” building spaces?

• Does your institution have mixed-use spaces? If yes, how would you think about reporting the square footage of those spaces?

• Was there anything confusing or unclear about this question? Please describe.

• Would you make any changes to improve this question? If so, what changes would you make?

Question 4 – Exhibition Space

Q4. What is the total square footage of your institution’s interior and exterior exhibition spaces?

Total exhibition space (square footage)

• In your own words, what is this question asking?

• Was there anything confusing or unclear about this question? Please describe.

• Would you make any changes to improve this question? If so, what changes would you make?

Question 5 – Condition of Facilities

Q5. Overall, how would you rate the condition of your institution’s facilities?
Excellent	Public and non-public grounds and buildings easily and readily support their intended uses.
Satisfactory	Public and non-public grounds and buildings provide adequate support for their intended uses. Some minor modifications/updates may be desired to improve the suitability.
Conditional	Public and non-public grounds and buildings require limited renovations to support their intended uses on a continued basis.
Development required	Public and non-public grounds and buildings require significant renovation to support their intended uses on a continuing basis. The space significantly inhibits program delivery.
Unsatisfactory	Public and non-public grounds and buildings are unsatisfactory and/or not functional for their intended uses. Renovating would not be cost-effective.
What are some of the reasons you selected [rating]? Does your institution have multiple buildings in various conditions? If yes, how did you summarize to your “overall” rating? In general, how easy or difficult was it to select a response to this question? Please describe. Was there anything confusing or unclear about the definitions? Please describe. Would you make any changes to improve this question? If so, what changes would you make?

Question 6 – Employees and Vacancies

Q6. What were the total numbers of full-time and part-time employees and position vacancies at your institution at the end of the most recently completed fiscal year?

Please do not include independent contractors or consultants. Also do not include support staff that are not employed directly by your institution or its parent organization (e.g., security officers).

Please use your organization’s definitions of full-time and part-time employment.

FT Employees For reference, the IRS defines full-time work as 30 or more hours per week or 130 or

FT Vacancies more hours per month

PT Employees For reference, the IRS defines part-time work as less than 30 hours per week or less

PT Vacancies than 130 hours per month.

• What does the term “position vacancies” mean to you?

• Based on the question, what is the specific point in time at which you would report these counts? Probe as needed for institution’s fiscal year.

• Was there anything unclear or confusing about the instructions?

• Would you make any changes to improve this question? If so, what changes would you make?

Question 7 – On-site Visits

Q7. What was the total number of on-site visits to your institution for your most recently completed fiscal year? Please report the total number of paying and non-paying visits by individuals, including individuals visiting as part of a PreK–12 or other group.

Total on-site visits

• In your own words, what is this question asking?

• How would you report on groups in your response to this question?

• Was anything confusing or unclear about this question or the instructions? Please describe.

• Would you make any changes to improve this question? If so, what changes would you make?

Question 8 – Group Visits

Q8. How many groups visited your institution during your most recently completed fiscal year?

Please report the number of groups, not the number of group members.

Number of PreK–12 groups (e.g., field trips)

Number of other visiting groups (refer to group sales)

• In your own words, what is this question asking?

• What does the term "groups" mean to you in the context of this question?

• How easy or difficult would it be to report the number of groups that visit your institution?

• Was anything confusing or unclear about this question? Please describe.

• Would you make any changes to improve this question? If so, what changes would you make?

Question 9 – Education Programs and Services

Q9. Other than PreK–12 field trips, did your institution provide any of the following programs or services during your most recently completed fiscal year?
School-based programs or services (e.g., in-school programs, lesson planning or materials, programs for homeschoolers)	What does “school-based programs or services” mean to you?
Out-of-school programs or services (e.g., camps, afterschool programs, internships, teen programs)	How would you define “out-of-school” programs or services? Where would these take place? Probe whether these may occur on school grounds.
Teacher professional development
Was anything confusing or unclear about this question? Please describe. Would you make any changes to improve this question? If so, what changes would you make?

Question 10 – Free General Admission Days

Q10. On how many days during the most recently completed fiscal year did your institution offer free general admission to all visitors for at least part of the day? Estimate if you are unsure.
Number of days
In your own words, what is this question asking? What does the phrase “for at least part of the day” mean to you? How would you calculate the number of days in response to this question? Was anything confusing or unclear about this question? Please describe. Would you make any changes to improve this question? If so, what changes would you make?

Question 11 – Content Focus

Q11. Is your institution’s content focused on a specific ethnic, cultural, or other demographic identity?

If your institution’s exhibitions and/or programming represent a range of different cultural traditions or demographic groups, please answer “no” to this question.

• In your own words, what is this question asking?

• What are some of the reasons you responded [yes, no] to this question?

• Was anything confusing or unclear about this question? Please describe.

• Would you make any changes to improve this question? If so, what changes would you make?

Question 12 – Demographic Identity

[IF NO TO Q11, MODERATOR STATES]

• Based on your answer to the previous question, your institution would not be asked this question. However, I’d still like to hear your thoughts about its content.

Q12. What specific ethnic, cultural, or other demographic identity/identities are the focus of your institution’s content?
African-American or Black
Asian
European
Hispanic/Latino(a)
Indigenous/Native
Individuals with disabilities
Middle Eastern
Military or veterans
Gender identity
Religious identity
Sexual orientation
Other
[If yes to Q11] Tell me more about why you selected [response(s)]. What is your institution’s content focused on? [If no to Q11] What are your thoughts about this question? [All] Looking at this list, are there any options missing? [All] Was anything unclear or confusing about this question? [All] Would you make changes to improve this question? If so, what changes would you make?

Closing

Those are all of the items I wanted to get your feedback on today. Before we end, I have two quick questions.

• First, do you have any other feedback about the survey questions that I showed you today?

• How has seeing some of the proposed National Museum Survey questions affected your thoughts about participating in the NMS?

Thank you so much for taking the time to speak with me today. The insights you provided will help to ensure that the NMS is the best data collection possible for institutions like yours. Enjoy the rest of your day!

Page 5 of 5