Pennsylvania Client Integrity Cognitive Testing Protocol

Pennsylvania Client Integrity Message Testing Protocol

Participant ID: ______ Interviewer initials: _____ SNAP? __Current __Past __Never

Interview Date: ___ / ___ /2023 Start Time: ____________ End Time: ____________

Read or paraphrase the following intro text:

Hello. My name is ________________, and I work for RTI International. We are working with the U.S. Census Bureau to conduct these interviews. Thank you for agreeing to participate in our study.

Today we are talking to people about the information they receive from the SNAP food assistance program. The information should be clear and easy to understand, so people who are eligible receive their benefits know about rules and requirements to continue participating in SNAP. We have found that the best way to see whether information is clear is to meet with people in person and get their feedback as they read letters and things like that. You will be helping us evaluate some of the information that the state of Pennsylvania provides to people who participate in SNAP.

To ensure we capture notes of your feedback during today’s interview we will be connecting to a virtual meeting so that my colleague [NAME] can take notes.

[INTERVIEWER: START TEAMS MEETING. ONCE CONNECTED, INTRODUCE NOTETAKER TO PARTICIPANT]

Before we start, there is a form I would like you to read and sign. [HAND PARTICIPANT CONSENT FORM]

This is a consent form. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to audio record this session. We do this to get an accurate record of your feedback for our notes. Our session today is completely confidential. If you currently receive or are applying for benefits, this interview will not have any impact on your benefits. Your participation in this study is completely voluntary. You may decline to answer any particular question and you may stop the interview at any time. We do not share information with anyone. Only the people on the project have access to the recording and we do not use any names in our reports. Take a look and let me know if you have any questions. Please sign it when you are finished, and then I will also sign it.

[SIGN CONSENT FORM AFTER PARTICIPANT SIGNS]

Think Aloud: As you read the materials today, I am interested in the thoughts and reactions that go through your mind as you process the messages. So I would like you to tell me everything that you are thinking and feeling as we go through these materials.

Practice: Let’s practice before we start. [SHOW PARTICIPANT PRACTICE PARAGRAPH]

Please read this short paragraph and say what comes into your mind as you’re reading it.

Practice probes: In your own words, can you tell me what this paragraph is saying? Did you find any part of it difficult to understand?

Thanks; that was great. So as we go through the SNAP materials, I will ask you questions like these. I really want to hear your opinions and reactions, so don’t hesitate to speak up whenever something is unclear, or not easy to understand. We are going to look at some materials and I’m going to ask you to questions about each one. Do you have any questions before we begin?

[INTERVIEWER TURN ON RECORDER. CHECK BATTERY LEVEL. NOTETAKER START RECORDING IN CENSUS VDI.]

General questions

• The SNAP program provides information about various rules, instructions, and things like that to people receiving or applying for food assistance. Do you recall receiving any of these kinds of information?

  • (If yes) What kinds of rules or instructions do you recall receiving?

• In your opinion, would you say you have a good understanding of the SNAP rules? Why do you say that?

• Do you recall getting any of these kinds of information that was not clear?

  • (If yes) Please tell me about that. What wasn’t clear about it?

  • Were you able to figure out what that information really meant? If so, how did you figure it out?

• If you had a question about your SNAP benefits, where would you go for more information?

Topic 1 - Card management

Exhibit 1: Informational Flyer

[SHOW PARTICIPANT ENTIRE INFORMATIONAL FLYER. START WITH GENERAL PROBES AND THEN PROBE ON SELECTED SECTIONS]

Here is a flyer that someone would get with their EBT card. Please take a few minutes to read this flyer and then we will talk about it. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what this is saying?

• Is there any information in the flyer that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• Was there anything that you expected to see in this flyer that was not here?

Let’s take a look at this section. [POINT TO DON’T THROW THIS CARD AWAY]

• Can you tell me in your own words what this means? [INTERVIEWER: DETERMINE WHETHER P UNDERSTANDS THAT THIS CARD WILL BE USED MONTH AFTER MONTH]

  • In your opinion, is the point of this message clear?

  • If no: What about this message is unclear? What would you change to make it clearer?

• Now let’s look at this section. [POINT TO YOUR CARD WILL NOT WORK]

• What does this first sentence mean to you? [INTERVIEWER: DETERMINE WHETHER P UNDERSTANDS WHAT A PIN IS]

  • (if needed) Can you tell me in your own words what a PIN is?

• Now let’s look at this section. [POINT TO KEEP YOUR PIN SECRET]

• What does this section mean to you?

• This section says to keep your PIN secret. What does that mean to you?

  • (if needed) What are some ways someone might keep their PIN safe?

• Are there any situations where someone might want to give their PIN to others?

  • (if yes) What are some examples of those kinds of situations?

• What would you do if someone knew your PIN and you didn’t necessarily want them to know it?

Thank you for sharing your thoughts on that document. Now let’s take a look at another one. Some of this information is similar to what we just looked at. Thank you for your patience as we talk about these materials.

Exhibit 2: EBT Brochure

[POINT TO TWO SECTIONS ABOUT “PIN” AT BOTTOM OF FIRST TWO INSIDE PAGES]

Please take a few minutes to read these two sections and then we will talk about them. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what sections are about?

• Is there any information in these sections that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part difficult to understand?

  • If yes: Which parts were difficult?

• Was there anything you expected to see in this brochure that was not here?

• [POINT TO “YOUR PIN”, 3^rd BULLET] Can you tell me in your own words what this part is about? [INTERVIEWER: DETERMINE WHETHER P ATTENDS TO THREE CHANCES AND CARD STOPS WORKING]

• [POINT TO “KEEP YOUR PIN SAFE”, 4^th BULLET] Can you tell me in your own words what this part is about?

Thank you. Now let’s take a look at the next one. [POINT TO HOW TO TAKE CARE…] Please take a moment to read this section.

• In your own words, what is this section about?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• Did you learn anything in this section you didn’t know before?

• One of these sentences mentions safe places to keep your card. What kinds of places does this make you think of?

Thank you. Now let’s take a look at the next one.

• Do you remember ever seeing these symbols before?

• What is the purpose of these symbols?

• Thinking about all the information we talked about in this flyer, is there anything you think should be included in the first flyer we looked at, the one that comes in the mail with the EBT card?

  • If yes: What do you think should be on the first flyer? Why do you say that?

Thank you. I’d now like to look another document.

Exhibit 3: Social Media Posting on Scams

[SHOW PARTICIPANT EACH SOCIAL MEDIA POST ONE AT A TIME AND HAVE PARTICIPANT REVIEW IN ITS ENTIRETY BEFORE PROBING.]

I’d now like to look at some messages that were posted on social media by the Pennsylvania SNAP program. Please take a few minutes to read it and then we will talk about it. If necessary: Please remember to think aloud as you read.

FRAUD ALERT: A warning has been issued to protect participants from possible EBT fraud attempts via text messages.

Be aware of a scam using texting to obtain your personal info. The text might say that your EBT card is locked. If you do not know if a request for information about your EBT card is real or not, contact our EBT card contractor, Conduent, at 1-888-328-7366.

Remember to never share personal information like your social security number, bank information, or benefits information with individuals or organizations that you do not know.

Visit: https://www.dhs.pa.gov/Services/Assistance/Pages/EBT.aspx for more information about EBT.

• Please tell me in your own words what you think the purpose of this section is.

• Is there any information in this message that you think is especially important?

  • If yes: What information do you think is important? Why do you say that?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

Now let’s look at another message.

• Is there any information in this message that you think is especially important?

  • (If yes) What information do you think is important? Why do you say that?

  • (if needed) [POINT TO LINE STARTING WITH “DO NOT RESPOND”] What does this sentence mean to you?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

Topic 2 - Eligible/ineligible purchases

Exhibit 4. SNAP FAQ

[START WITH GENERAL PROBES ABOUT ELIGIBLE/INELIGIBLE PURCHASES; THEN PRESENT AND HAVE PARTICIPANT REVIEW SPECIFIC FAQ BELOW FOR THIS TOPIC AND PROBE.]

Please take a moment to read this section. [POINT TO OR HIGHLIGHT FIRST PARAGRAPH, UNDER LETTERHEAD]

• What does this section mean to you?

• Did you find any part of it difficult to understand?

  • If yes: Which parts were difficult?

• How would you describe the kinds of food that you are allowed to buy with SNAP?

• How would you describe the kinds of foods and other items that you are NOT allowed to buy with SNAP?

• The second bullet refers to lunch counter items or foods to be eaten in the store. What does that mean to you?

  • What are some examples of those kinds of foods?

• In your opinion, do you feel like you have a good understanding of what you are allowed to buy with SNAP? Why do you say that?

• Is there any other information you would want to see on a webpage like this?

Thank you for sharing your thoughts about this document. Let’s look at the next one.

Topic 3 – Trafficking

Exhibit 5. SNAP Application (page 14)

[SHOW PARTICIPANT ENTIRE PAGE.]

Please take a moment to look over this page.

• Please tell me in your own words what this page is about. [INTERVIEWER: DETERMINE WHETHER P FINDS THE “SNAP” ROW AND UNDERSTANDS THE “IF THIS HAPPENS” AND “GOOD CAUSE” COLUMNS.]

• In your opinion, is anything about this page confusing or unclear?

  • If yes: Which parts were not clear?

    • Do you have any suggestions for making this page clearer or easier to read?

• I’d now like you to read a couple of sections on this page. [POINT PARTICIPANT TO “SNAP” SECTION ON THE SECOND ROW]. Let’s start with this section. Please take a few minutes to read this section.

• [POINT TO THE INFORMATION ON THE SNAP ROW IN THE SECOND COLUMN] What does the information in this section mean to you?

  • What does the phrase, ‘If this happens without good cause’ mean to you?

    • Do you recall seeing those words when you were first looking at this page?

• [POINT TO THE INFORMATION ON THE SNAP ROW IN THE THIRD COLUMN] And how about this section, what does this information mean to you?

  • What does the phrase, ‘this may happen (penalty)’ mean to you?

    • Do you recall seeing those words when you were first looking at this page?

• [POINT TO “NOT ELIGIBLE: FIRST TIME, SECOND TIME,” ETC. IN TOP CELL IN THIRD COLUMN] What does the information in this box mean to you?

  • What does the last bullet mean to you? (Read if needed) “First time court conviction over $500 - forever.”

Now let’s take a look at one more section on this page. [POINT TO “IF YOU ARE FOUND” ROW].

• Please tell me in your own words what this section means to you.

• In your opinion, is this information clear?

  • (if no) What parts are not clear?

• What does the last line mean to you?

Exhibit 6: Excessive Card Letter

[SHOW PARTICIPANT ENTIRE EXCESSIVE CARD LETTER AND HAVE PARTICIPANT REVIEW IN ITS ENTIRETY BEFORE PROBING.]

Now let’s look at another letter. Please take a few minutes to read the letter and then we will talk about it. If necessary: Please remember to think aloud as you read.

• In your own words, can you tell me what this letter is saying? [INTERVIEWER: DETERMINE WHETHER P UNDERSTANDS THE MEANING AND CONSEQUENCES OF ASKING FOR TOO MANY REPLACEMENT CARDS]

• Are there any parts of this letter that are confusing or unclear?

  • If yes: Which parts?

• (IF NOT ALREADY DISCUSSED) This letter talks about asking for an excessive number of EBT cards. In your own words, what does that mean to you?

• What would you expect to happen next after receiving this letter?

Topic 4 - Fraud reporting

Exhibit 7. Fraud FAQs

[SHOW PARTICIPANT ENTIRE FRAUD FAQ PAGE. HAVE PARTICIPANT REVIEW SELECTED SECTIONS ONLY.]

Can you please read the section below and tell me what comes to mind? [POINT TO SELECTED FAQ]

• In your opinion, is anything in this section confusing or unclear?

  • (if yes) What do you think is not clear?

• Can you name any examples of welfare fraud?

How about this paragraph? [POINT TO SELECTED FAQ]

• In your opinion, is anything in this section confusing or unclear?

  • (if yes) What do you think is not clear?

• Have you ever thought about filing a welfare fraud report?

  • (If yes) You don’t have to tell me if you don’t want to, but did you file the complaint?

    • (If no) What made you decide not to?

How about this paragraph? [POINT TO SELECTED FAQ]

• Please tell me in your own words what this section means to you.

Those are all the documents. Thank you for sharing your thoughts with me about them.

Debriefing Questions:

Now I just have a few more questions before we are finished.

• Thinking about all the materials we have reviewed today, is there anything you would like to add that we didn’t get a chance to talk about yet?

• In the past, have you had any issues with your benefits from the Food Assistance program?

• If you find yourself having trouble with the program, what do you do at home?

  • Do you call for help? Who tends to help you?

  • Are there other things you do when you have trouble filling out forms?

• Government agencies collect data when people fill out forms for programs, such as applying for food assistance benefits. Do you think this data should be shared with other government agencies?

• Do you have any other last thoughts you’d like to share?

Thank you for your help today!

[ASK PARTICIPANT TO FILL OUT VOUCHER FORM. GIVE PARTICIPANT INCENTIVE]

Thinkaloud Practice Paragraph

Our solar system was formed about 4.5 billion years ago. It consists of our sun and eight main planets: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune. Mercury, Venus, Earth and Mars are mostly made of rock and metal and are called “terrestrial planets.” Jupiter, Saturn, Uranus and Neptune are called “gas giants” because they are mostly composed of gas.

8