Appendix G-Pretest Summary

APPENDIX G

PRETEST SUMMARY

SNAP/Extra Help

Client Survey Pretest Results

The SNAP-Extra Help Project is studying the effectiveness of SNAP pilot projects in Pennsylvania, New Mexico, and Washington that are designed to improve SNAP access among Medicare’s Extra Help population. The evaluation project includes a survey of Medicare Extra Help applicants and Medicare Savings Plan participants in these states. The survey will include both SNAP participants and nonparticipants and address their perceptions of and experiences with SNAP. FNS’ contractors conducted a pretest of the survey instrument to test its length; examine items for potential comprehension problems among respondents; evaluate the ease of use for the interviewer, and identify which items respondents may be sensitive to. This memorandum briefly describes the pretest procedures used and presents results and changes to the instrument based on this experience.

A. Pretest Procedures

The pretest interviews were conducted with individuals in Pennsylvania who have applied for Extra Help. At the time, Pennsylvania was furthest along in developing its initiative. The interviews were conducted by contractor staff. An advance letter was mailed to a total of 15 sample members one to two days prior to the first telephone call attempt. Interviews were completed with 8 individuals by telephone. Of these 8 respondents, all had heard of SNAP, 6 were SNAP participants, and the remaining 2 were nonparticipants.

Interviewers administered the survey by telephone using a hardcopy questionnaire. While conducting the interview, they documented comments and questions from respondents, problems with respondent comprehension, and respondent sensitivity.

B. Interview Length

Pretest respondents took an average of 26 minutes to complete the interview, which is 6 minutes longer than the target interview length. In order to ensure the average interview length is 20 minutes, the following modifications were made:

• Item SC3 was shortened.

• Items B4, B14, H8, and H9 were deleted as these items are less critical to the data collection.

C. Interview Flow

In addition to testing for length, the survey instrument was evaluated for how well it flowed, and, in particular, whether respondents felt like the interview bounced from topic to topic or whether there was a natural transition between sections. In general, the interview proceeded smoothly and the modules in the instrument formed coherent sections. The organization of the material did not require respondents to bounce back and forth in topic areas or in recall periods.

However a few transitions were added to the instrument to improve flow.

• At item SC0, before the first interview question, a definition of SNAP (or Basic Food in Washington State) was added.

• A transition statement at item B2 was added to guide respondents to think back to before they applied for SNAP.

• An opening statement was added before the transition to the topic of Medicare Savings Plan and Medicare Extra Help (item F1 for respondents in Washington State, item F7 for respondents in Pennsylvania, and item F10 for those in New Mexico).

• Items H3 and H7 about household food security ask about the respondent and other adults in the household. A condition was added so that the phrase “and other adults in your household” is only said if the respondent lives with another adult (based on the response to G6).

D. Comprehension

In general, the interaction between pretest interviewers and respondents suggested the questionnaire items were well understood by respondents. Where respondents asked clarifying questions or interviewers perceived difficulty on the part of respondents in comprehending the question, clarifying phrases were added.

• At item A1, we rephrased the question to ask respondents why they applied for SNAP rather than what changed in their life causing them to need SNAP. Based on the responses from pretest respondents, this rephrasing makes the item more closely resemble their experiences.

• At item B7, we added a clarifying phrase to make clear the question is asking only about the application and not about other documentation.

• Item B10 was revised to indicate the question is about a follow-up interview to the application. This clarifies the question for all respondents, including those who had completed the application by phone. We also added an optional probe for interviewers to use if respondents are confused about whether the question is about the application or interview.

• Wording was added to item B11 to clarify that the question is about the follow-up interview.

• At E1p, when first mentioning the EBT card, we added a definition of the card.

• We added clarifying language at E2a and E2b and now specify the question is about the SNAP application in the respondent’s state.

• We expanded F13 to clarify for respondents what we mean by completing the application. New language states “for example, submit all the necessary documentation and participate in an interview.”

• F16 was corrected so that respondents in all states will be asked this question. The question is now also broken out to ask each item in two parts – we first ask respondents if they are satisfied or dissatisfied, and then follow up with whether they are very or somewhat dis/satisfied. Pretest interviewers found they had to unfold the question to help respondents answer it – by first asking about positive or negative valence and then asking about intensity separately. We made this change in the questionnaire so that all the respondents will be asked the question in the same way.

E. Interviewer Ease of Use

Testing an instrument for interviewer ease of use is important to ensure that interviewers are able to read the questions verbatim, respond to respondent questions, and code responses correctly. Where respondents tended to ask for clarification, we revised question wording (as described above). We also added probes should interviewers need to provide additional clarification. Probes ensure that when interviewers are asked to clarify a question, they always do so in the same way. Finally, where response categories were very similar and potentially difficult for interviewers to code correctly, we combined them if possible.

• Because some respondents had applied for SNAP more than once, it was useful to reiterate that question SC5 is asking about the most recent experience. We added a probe for interviewers.

• Pretest respondents did not differentiate between some of the response categories at B1. Thus we combined options that were very similar. “Senior center” and “community center” were combined, and “while applying for other benefits” and “referred by another agency” were combined. We added a probe for interviewers to use if necessary to clarify what agency contacted the respondent about applying for SNAP.

• The probe at B8 was optional. We changed this so it will now be read by interviewers for every respondent.

• At B12, we added an optional probe for interviews indicating the question is asking about in-person visits to the SNAP office for any reason.

D. Sensitivity of Survey Items

During the pretest, interviewers made note of where they perceived some respondent sensitivity so that these sections could be addressed in interviewer training. The sections where this occurred include the series of items at E6 through E10 asking whether the respondent feels uncomfortable using SNAP, and the household food security module in section H. At no time did respondents get angry, complain, or threaten to hang up the phone. However interviewers felt empathy for respondents. During interviewer training, the contractor will address this issue so interviewers will have appropriate and standardized language to use when administering these sections.