Washington DC Client Integrity Cognitive Testing Protocol

Washington DC Client Integrity Message Testing Protocol

Participant ID: ______ Interviewer initials: _____ SNAP? __Current __Past __Never

Interview Date: ___ / ___ /2023 Start Time: ____________ End Time: ____________

Read or paraphrase the following intro text:

Hello. My name is ________________, and I work for RTI International. We are working with the U.S. Census Bureau to conduct these interviews. Thank you for agreeing to participate in our study.

Today we are talking to people about the information they receive from the SNAP food assistance program. The information should be clear and easy to understand, so people who are eligible receive their benefits know about rules and requirements to continue participating in SNAP. We have found that the best way to see whether information is clear is to meet with people in person and get their feedback as they read letters and things like that. You will be helping us evaluate some of the information that the District of Columbia provides to people who participate in SNAP.

To ensure we capture notes of your feedback during today’s interview we will be connecting to a virtual meeting so that my colleague [NAME] can take notes.

[INTERVIEWER: START TEAMS MEETING. ONCE CONNECTED, INTRODUCE NOTETAKER TO PARTICIPANT]

Before we start, there is a form I would like you to read and sign. [HAND PARTICIPANT CONSENT FORM]

This is a consent form. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to audio record this session. We do this to get an accurate record of your feedback for our notes. Our session today is completely confidential. If you currently receive or are applying for benefits, this interview will not have any impact on your benefits. Your participation in this study is completely voluntary. You may decline to answer any particular question and you may stop the interview at any time. We do not share information with anyone. Only the people on the project have access to the recording and we do not use any names in our reports. Take a look and let me know if you have any questions. Please sign it when you are finished, and then I will also sign it.

[SIGN CONSENT FORM AFTER PARTICIPANT SIGNS]

Think Aloud: As you read the materials today, I am interested in the thoughts and reactions that go through your mind as you process the messages. So I would like you to tell me everything that you are thinking and feeling as we go through these materials.

Practice: Let’s practice before we start. [SHOW PARTICIPANT PRACTICE PARAGRAPH]

Please read this short paragraph and say what comes into your mind as you’re reading it.

Practice probes: In your own words, can you tell me what this paragraph is saying?

Did you find any part of it difficult to understand?

Thanks; that was great. So as we go through the SNAP materials, I will ask you questions like these. I really want to hear your opinions and reactions, so don’t hesitate to speak up whenever something is unclear, or not easy to understand. We are going to look at some materials and I’m going to ask you to questions about each one. Do you have any questions before we begin?

[ INTERVIEWER TURN ON RECORDER. CHECK BATTERY LEVEL. NOTETAKER START RECORDING IN CENSUS VDI.]

General questions

• The SNAP program provides information about various rules, instructions, and things like that to people receiving or applying for food assistance. Do you recall receiving any of these kinds of information?

  • (If yes) What kinds of rules or instructions do you recall receiving?

• In your opinion, would you say you have a good understanding of the SNAP rules? Why do you say that?

• Do you recall getting any of these kinds of information that was not clear?

  • (If yes) Please tell me about that. What wasn’t clear about it?

  • Were you able to figure out what that information really meant? If so, how did you figure it out?

• If you had a question about your SNAP benefits, where would you go for more information?

Topic 1 - Card management

Exhibit 1: Protecting Benefits Guide

[HAVE PARTICIPANT REVIEW IN ITS ENTIRETY AND ADMINISTER GENERAL PROBES. THEN POINT PARTICIPANT TO RELEVANT SUBTOPIC EXCERPTS AND PROBE.]

This is a flyer provided by the DC SNAP program. Please take a few minutes to read this and then we will talk about it. If necessary: Please remember to think aloud as you read.

Now let’s look at this section.

• Please tell me in your own words what a PIN is.

• In your opinion, is this information useful or not useful? Why do you say that?

• Is anything here confusing or unclear? Why do you say so?

• Is there any other information you would want to see that was not here?

Exhibit 2: SNAP Approval Notice (Page 1)

[HAVE PARTICIPANT REVIEW PAGE 1. THEN POINT PARTICIPANT TO RELEVANT SUBTOPIC EXCERPTS BELOW AND PROBE.]

Here is some information that someone would get when they are approved for SNAP. Please take a few minutes to read this page and then we will talk about it. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what this page is saying?

• Is there any information that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• Did you see anything on this page that you didn’t know before?

• Was there anything that you expected to see in this page that was not here?

I now have some questions about specific parts of the letter. [POINT PARTICIPANT TO SELECTED SECTIONS.]

• What does this section mean to you?

• This section says to keep your PIN secure. What does that mean to you?

  • (if needed) What are some ways someone might keep their PIN secure?

• Are there any situations where someone might want to give their PIN to others?

  • (if yes) What are some examples of those kinds of situations?

• What would you do if someone knew your PIN and you didn’t necessarily want them to know it?

Now how about this section?

• How would you describe this section in your own words?

• (if needed) What does it mean to you to “keep your EBT card secure”?

Thank you for sharing your thoughts on that. Now let’s take a look at another flyer from the D.C. SNAP program. Some of the questions that I ask you may sound repetitive, so thank you for your patience as we talk about these materials.

Exhibit 3: Out of State Travel Flyer

[AFTER REVIEWING EXCERPTS FOR TOPIC TWO, POINT PARTICIPANT TO RELEVANT SUBTOPIC EXCERPTS BELOW FOR TOPIC 3 AND PROBE.]

Please take a few minutes to read this page and then we will talk about it. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what this page is saying?

• Is there any information that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• Is there any other information you would want to see in this page that was not here?

• This flyer talks about protecting yourself from fraud. What does that mean to you?

  • (if needed) Please tell me in your own words what you think they mean by “fraud”.

Topic 2 - Eligible/ineligible purchases

Exhibit 1. Protecting Benefits Guide

[POINT TO “SNAP IS FOR FOOD” SECTION]

Now let’s go back to the first flyer. Please take a moment to re-read this section. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what this section means to you?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• What does “pre-cooked meals” mean to you?

• What does “household supplies” mean to you?

• [POINT TO SECTION OF FLYER WITH QR CODE] This page has a QR code. Have you ever used a QR code like this one?

  • (if needed) Do you know what a QR code is for?

    • (if P does not know) A person with a cell phone can point their camera at a code like this and open up a web page to get more information.

  • (if no) Do you think you would ever use a QR code?

Exhibit 4. DC SNAP Website

Now please take a look at this information from the D.C. government website. If necessary: Please remember to think aloud as you read.

[HAVE PARTICIPANT REVIEW SPECIFIC SECTIONS OF WEBSITE BELOW FOR THIS TOPIC.]

• What does this information mean to you?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• How would you describe the kinds of food that you are allowed to buy with SNAP?

• How would you describe the kinds of foods and other items that you are NOT allowed to buy with SNAP?

• Did you see anything in this section that you did not know before?

• Is there any other information you would want to see on a webpage like this?

• [PRESENT “PROTECTING BENEFITS GUIDE” AND POINT TO “SNAP IS FOR FOOD” SECTION] Thinking back to the flyer we looked at a moment ago, is there any information from the website that you think should go in the flyer?

Topic 3 – Trafficking

Exhibit 2. SNAP Approval Notice (Page 2)

[HAVE PARTICIPANT REVIEW SPECIFIC SECTIONS OF PAGE 2 BELOW FOR THIS TOPIC.]

Now let’s take a look at another section from the approval letter. If necessary: Please remember to think aloud as you read.

• Please tell me in your own words what this section means to you.

• In your opinion, is anything about this page confusing or unclear?

  • If yes: Which parts were not clear?

• What does the first sentence mean to you?

  • (if needed) What does “barred from the SNAP” mean to you?

• What does “do not trade or sell SNAP benefits” mean to you?

  • (if needed) Can you tell me an example of trading or selling benefits?

• [POINT TO THE 3^RD BULLET] How would you put this sentence into your own words?

  • Do you know of an example of how someone might alter a SNAP card?

• Did you see anything in this section that you did not know before?

• Is there any other information you would want to see in a page like this?

Exhibit 5. SNAP Integrated Application

[HAVE PARTICIPANT REVIEW PAGE IN ITS ENTIRETY. THEN POINT PARTICIPANT TO RELEVANT SUBTOPIC EXCERPTS BELOW AND PROBE.]

Now please take a few minutes to read this section which is part of the application for benefits in D.C.

• In your opinion, what is the purpose of this page?

  • (if needed) This section asks for a signature. Why do you think that is?

• In your opinion, is anything about this page confusing or unclear?

  • If yes: Which parts were not clear?

    • Do you have any suggestions for making this page clearer or easier to read?

• [POINT TO 5^TH BULLET] This sentence mentions SNAP trafficking.

  • What does this mean to you?

  • [IF NEEDED] Can you name an example of SNAP trafficking?

• Did you see anything in this section that you did not know before?

• Is there any other information you would want to see in a page like this?

Exhibit 6. Multiple Card Replacement Letter

[HAVE PARTICIPANT REVIEW THE MULTIPLE CARD REPLACEMENT LETTER IN ITS ENTIRETY.]

Now please take a look at this letter. If necessary: Please remember to think aloud as you read.

• Is there any information that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part of this letter difficult to understand?

  • If yes: Which parts were difficult?

    • Do you have any suggestions for making this page clearer or easier to read?

• In your opinion, what is the main point of this letter? [INTERVIEWER: DETERMINE WHETHER P ATTENDS TO THE 2^ND AND 5^TH PARAGRAPHS (“DPDT & QA’S RECORDS…” AND “HOWEVER, IF…”)]

  • (if needed) [POINT TO 2^nd PARAGRAPH] The recipient of this letter is being told that they ordered four replacement cards. What does that mean to you?

  • [POINT TO 5^th PARAGRAPH] Please read the paragraph at the bottom that starts with “However.” What does that mean to you?

  • Do you recall ever hearing anything about someone asking for too many replacement EBT cards? If so, what do you remember about that?

  • In your opinion, is the information about asking for too many replacement cards clear or not clear?

    • (if not clear) Do you have any suggestions for making this letter clearer?

• [POINT TO 1^st PARAGRAPH] Let’s go back to the first paragraph. What does that mean to you?

  • In your opinion, is this information useful or not useful?

Topic 4 - Fraud reporting

Exhibit 1. Protecting Benefits Guide

Please take a moment to review this section. As needed: Please remember to think aloud as you read.

• What does this section mean to you?

• (if needed) This section mentions getting help with protecting your benefits or reporting misuse. What does that mean to you?

[POINT TO “WE’RE HERE FOR YOU” PAGE OF THE FLYER] Now let’s take a look at this section. As needed: Please remember to think aloud as you read.

• This section mentions SNAP fraud. What does that mean to you?

  • (if needed) What is an example of SNAP fraud?

• Have you ever thought about filing a fraud report related to SNAP or other assistance programs?

  • (If yes) You don’t have to tell me if you don’t want to, but did you file the complaint?

    • (If no) What made you decide not to?

• If you were to call to report SNAP fraud, would you have any concerns about providing your name if they asked for it?

Those are all the documents. Thank you for sharing your thoughts with me about them.

Debriefing Questions:

Now I just have a few more questions before we are finished.

• Thinking about all the materials we have reviewed today, is there anything you would like to add that we didn’t get a chance to talk about yet?

• In the past, have you had any issues with your benefits from the Food Assistance or SNAP program?

• If you are confused about your SNAP benefits or have a question, what do you do?

  • Would you call someone or look up information somewhere?

• Government agencies collect data when people fill out forms for programs, such as applying for food assistance benefits. Do you think this data should be shared with other government agencies?

• Do you have any other last thoughts you’d like to share?

Thank you for your help today!

[ASK PARTICIPANT TO FILL OUT VOUCHER FORM. GIVE PARTICIPANT INCENTIVE]

Thinkaloud Practice Paragraph

Our solar system was formed about 4.5 billion years ago. It consists of our sun and eight main planets: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune. Mercury, Venus, Earth and Mars are mostly made of rock and metal and are called “terrestrial planets.” Jupiter, Saturn, Uranus and Neptune are called “gas giants” because they are mostly composed of gas.

10