Draft Protocol

Quarterly Survey of Public Pensions (QSPP) Proposed Form 2018

Draft Cognitive Interview Protocol, 10/18/18

(Kristin Stettler)

Research Questions:

• Do respondents understand what information they are being asked to provide with these questions?

• Do respondents understand the terminology used in the questions?

• Are respondents able to answer the questions that are being asked?

• Is the requested information available in respondent’s records?

• Are the instructions helpful to respondents?

Expected Length of Interview: 30-60 minutes

Materials Needed:

• Draft questions

• Consent forms

• Digital recorder

General probes that may be used throughout the interview:

• Reflect back on respondent’s answer: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question?

• What records (if any) did you look at? What line or lines were of interest?

• Specifically, what did you include in this number? What did you exclude?

• Would you consult other people to obtain this answer?

INTRODUCTION

• Explain purpose of meeting: get feedback on potential redesigned questions for the Quarterly Survey of Public Pensions.

• There may be times throughout this interview that this seems like a test of you, I assure you that it is not. The purpose of these interviews is to ensure that we are communicating the data requests clearly with you, so in that way, these interviews are actually a test of us. On that note, we want your honest feedback throughout, and if it’s not clear to you how to answer, please do your best. There will be ample opportunities for you to let us know how to make improvements so that you could do a better job answering.

ABOUT THE RESPONDENT(S)

First, I’d like to learn a bit about each of you, and your roles at your organization.

• What is your job title? Background?

• What is your role in completing government surveys?

• How much experience do you have completing the Quarterly Survey of Public Pensions?

• Do you work with anyone else in your organization to get the data for the Quarterly Survey of Public Pensions?

  • How are those other people involved?

  • How many people are involved?

OVERALL QUESTIONS

Before we get started with our draft questions, I want to ask you to about your financial records.

• What sorts of financial records are available?

• What type of financial records systems do you have?

• Are you able to show us your system(s)?

COGNITIVE INTERVIEW QUESTIONS

Thank you for that background information! Now I’d like to have you review the current version of the Quarterly Survey of Public Pensions, just to refresh your memory.

• What was your experience completing this survey?

• Do you have any comments or suggestions for us?

Next, I will show you the questions that we have drafted.

Please take a minute to review the form.

• Glancing through the form, what is your first impression?

OK, now let’s look at the first section…

[For each section, ask:]

• In your own words, what is being asked for in this section?

• Let’s go through these items one by one.

[For each item, ask:]

• In your own words, what are we asking for here?

• How would you answer this question?

• Where would the data come from in your records?

• Do you have this information in your records?

• What difficulties might you have retrieving this data?

• What do you think of the description in the third column?

• Is there any additional information you would like to see provided here?

• [Probe on any phrases of concern.]

• [For the Sum item at the bottom of each section, ask:]

  • Do you have this total in your records or would you have to sum up the earlier items to get it?

  • Is this item meaningful to you?

  • Is there anything else included in the Sum item that we have not included above?

  • Are all of the items above clear and mutually exclusive?

  • Would you prefer that this be auto-summed and you can correct it, or would you prefer to enter the data?

Z01—Total Cash and Investments

• In your own words, what are we asking for here?

• How would you answer this question?

• Where would the data come from in your records?

• Do you have this information in your records?

• What difficulties might you have retrieving this data?

• What do you think of the description in the third column?

• Is there any additional information you would like to see provided here?

• Do you have this total in your records or would you have to sum up the earlier items to get it?

• Is this item meaningful to you?

• Is there anything else included in this Sum item that we have not included above?

• Would you prefer that this be auto-summed and you can correct it, or would you prefer to enter the data?

Z80—Securities and Lending Collateral

• In your own words, what are we asking for here?

• How would you answer this question?

• Where would the data come from in your records?

• Do you have this information in your records?

• What difficulties might you have retrieving this data?

• What do you think of the description in the third column?

• Is there any additional information you would like to see provided here?

• Does this item fit in with the other items above?

• Would you prefer to have a transition or introductory text above this item?

General Instructions

Did you review the General Instructions on the first page before looking at the questions?

• What do you think of the instructions?

• Are there other instructions you think would be helpful?

• Are there any instructions you don’t think are necessary?

• Do you have any suggestions for improvement?

• [Probe on specific instructions as needed.]

WRAP-UP

Wrap-up

We are almost done, and I just have a few wrap-up questions.

• Overall, what did you think about the questions we reviewed?

• Do you have any suggestions for improvement?

• Do you think these questions would be easy to answer, difficult to answer, or somewhere in between? Why?

• Could you provide an estimate of how long you think it would take you to answer the questions we reviewed today?

• Compared to the current version of the QSPP form, what do you think of these redesigned questions? Would this version be easier or harder for you to complete?

• Is there anything else we haven’t touched on today, that you think is relevant?

• Do you have any questions for us?

Thank you very much for your time today!

6