M. Participant Survey Pretest Memo_2022 09 02

Appendix M. Participant Survey Pretest Memo

This page has been left blank for double-sided copying

To: Mehreen Ismail

From: Kim McDonald and Joyce Kim

Date: 7/22/2022

Subject: SNAP E&T RCE Participant Survey Pretest Memo

This memo summarizes our findings from pretesting the draft participant surveys for the Rapid Cycle Evaluation of Operational Improvements in Supplemental Nutrition Assistance Program Employment & Training Programs (SNAP E&T RCE). The goal of the pretest was to assess survey length, survey flow, respondents’ interpretation of questions, and completeness of response category options. Below, we describe the revisions that we implemented based on observations from pretests conducted in all four site surveys (Colorado, Connecticut, Massachusetts, and Rhode Island).

Pretest procedures and sample

Survey structure. The study team pretested four different versions of the survey with participants in each of the four SNAP E&T RCE sites where the survey will ultimately be fielded: Colorado, Connecticut, Massachusetts, and Rhode Island. The surveys share some common questions but are otherwise unique to the site-specific interventions. All four surveys have the same main sections:

• Section A: Employment

• Section B: Site specific intervention questions

• Section C: SNAP E&T participation questions

• Section D: Respondent characteristics

• Section E: Contact information

Section A and D questions are the same for all sites, except for the Connecticut survey. We did not initially pretest questions from Section A or D with Massachusetts respondents, our first set of pretests, to avoid redundancy and to reduce the pretest burden associated with the Massachusetts instrument. When pretesting these questions in other sites, we found that participants did have feedback, so we continued to include them in subsequent interviews. This approach also helped us obtain more accurate timing estimates for the full survey.

Pretest sample. Pretest participants were recruited through SNAP E&T program staff at each site between April and June 2022. Site staff reached out to current program participants to invite them to participate in a pretest interview with Mathematica. Recruiting participants through SNAP E&T site staff enabled us to target adults similar to the SNAP E&T RCE study population. Among pretest participants, about 60% were currently receiving SNAP E&T services while the other 40% were not currently receiving services. The study team pretested each survey instrument with five participants associated with each relevant site, with the exception of Rhode Island. The study team pretested the Rhode Island instrument with three respondents from Rhode Island and two participants from other sites (Connecticut and Massachusetts) due to scheduling challenges.

Pretest interviews were conducted by video using the Webex interface or by telephone. We encouraged everyone to join by Webex but worked to accommodate all participants, some of whom did not have computer access. As a result, some pretest interviews were conducted through Webex without video, and some were conducted by telephone only. To mirror self-administration of the survey by web, we sent 13 participants a PDF version of the survey in advance of the call. During the Webex interview, these respondents shared their computer screen and self-administered the questionnaire by reading each question to themselves and indicating their responses to questions on the PDF. Some respondents highlighted their responses on the PDF and others shared their responses aloud, depending on their comfort with screen sharing and technology. We offered pretest participants technology support as needed, but we asked them to hold specific survey questions until they had completed the entire survey (with the exception of Section B, described below). To mirror interviewer-administration of the survey by telephone, 7 participants were read survey questions aloud by the interviewer and asked to provide their responses verbally. Table 1 shows the breakdown of pretest administration mode by site.

Table 1. Pretest administration mode by site

	Self-Administered	Interviewer-Administered
Colorado	1	4
Connecticut	5	0
Massachusetts	5	0
Rhode Island	2	3
Total	13	7

Section B of each survey asks questions about an intervention that has not yet occurred. Given this complexity, some additional explanation was also provided to respondents ahead of this section. Specifically, the study team presented a scenario to each respondent that provided additional context and guided their responses through that section. Figure 1 provides an example of text included at the beginning of Section B for a Connecticut pretest survey. The scenario orients the respondent to the intervention design in Connecticut and provides some guidance about how to respond to certain questions. The study team adjusted the scenario for each pretest respondent to facilitate pretesting all of the different paths in Section B.

The surveys also contain several fields that will programmatically fill with site or individual specific information. For example, the SNAP E&T program name or the coach name presented in Figure 1 (Shaina), which will ultimately be dynamic fields. For pretest interviews, we filled in these fields as much as possible, except where fill-ins depended on prior survey responses. For example, in Figure 1 we refer to services from “the local food pantry” and a fictional coach, Shaina, to facilitate the interview.

Figure 1. Example context from Section B of Connecticut pretest

These next set of questions ask you to pretend that you met with your coach, Shaina, a few months ago to complete an assessment to understand where you were at with life and education-related skills. The assessment is attached to the email you received with this survey. In the assessment you indicated you were having some trouble getting access to fresh food, and so Shaina referred you to the local food pantry. A few days after you met, Shaina also sent you a text to remind you to reach out to the food pantry.

[PARTICIPANT NAME], this is Shaina, your SNAP Coach. By meeting with me to discuss resources and services you need, you have already started on the path to success at Tunxis CC. Take the next step toward meeting your goals by reaching out to the local food pantry. Contact them at 978-222-2222 or visit them at Food Pantry, 123 Main St. to start getting the support we discussed!

You reached out to the food pantry, ended up receiving some support in March and April, but not in May.

With this information in mind, please do your best to answer the questions in this section. We’ll help guide you if you’re not sure what to answer.

Following the survey, the interviewer asked a series of debriefing questions to better understand how respondents answered the survey. Interviews took between 30 and 60 minutes and pretest respondents were sent a $30 Visa gift card to thank them for their participation.

Pretest Findings

Overall, respondents felt the survey was straightforward and easy to understand. Respondents said the flow between sections made sense and felt the survey length was appropriate. Below we provide findings on respondent burden, as well as recommendations for item-specific revisions and questions to remove from the instruments.

Respondent Burden

The average survey length was between 10-15 minutes for web administration and approximately 15-20 minutes for telephone administration. While the timing varied by site and routing (as described in the prior section), overall burden of the pretested instruments exceeded our initial 10 minute estimate.

To address this, we propose dropping several questions and response options in the instruments to reduce burden. Internal survey timing estimates after the proposed pretest revisions produced an average burden of 15 minutes. We have revised the OMB estimates for the survey instrument to reflect a 15 minute burden. Recommendations on survey revisions and questions to drop are described below.

Item-Specific Revisions

Table 2 describes the proposed modifications based on pretest findings. In addition to those changes, we revised wording throughout the surveys to facilitate multimode administration. For example, we replaced “Other (SPECIFY)” response with phrases like “Somewhere else (SPECIFY)” to accommodate telephone interviewer administration. We also changed survey response options that reference “I” to consistently reference “you” (e.g., C1a. “I am currently receiving services” was edited to “You are currently receiving services”).

Table 2. Survey modifications from pretest

Site	Item(s)	Feedback	Revision
Section Aa
All	A2 – A5. Demographics (gender, race, education) (renumbered to D1 – D4)	Some respondents felt that these questions were too personal for the beginning of the survey.	Move to end of survey and added introductory text.
All	A2. What is your sex? (renumbered to D1)	At least one respondent did not feel the current set of response options was inclusive of their gender identity.	Revise wording to be more inclusive of other gender identities. Response options based off current recommendations from the Human Rights Council.1
All	A5. What is the highest degree or level of school you have completed? (renumbered to D4)	A respondent had difficulty understanding and defining what should be included in the “Vocational/Technical degree or certificate” category.	Added several common examples in this category to provide more context
CO, MA, RI	A8. Thinking about the challenges you may have had in the last year, how much did the following make it hard for you to find or keep a job? (renumbered to A3-A4)	Respondents reported that the grid was hard to process. By phone, this question took a long time to read (all response options must be read). We suspect that respondents self-administering the survey were not actually reading all response options.	Change to Yes/No response categories Drop several response options that have previously been shown to have low prevalence,2 details in Appendix A
Section B
All	B0. Introduction	One respondent felt the introduction was confusing and wordy.	Revise wording to remove repetitive fills and add spacing to make it easier to read.
CO	B1. In the last [two months], did you receive any messages encouraging you to enroll in the SNAP E&T program?	One respondent suggested that instead of splitting this up as a Yes/No to each communication type, we could list something along the lines of “missed that one.”	Change response options to “Yes, received message” and “No, did not receive message” to mirror the question stem.
CT	B1. How much do you agree or disagree with the following statements regarding the assessment you completed?	While going through different routes of the survey, the assessment did not make as much sense to control respondents who had a conversation with their coach rather than filling out a paper assessment.	Use the phrase “intake process” for control participants and include a description of the intake process to provide additional context.
CT	B9, B11. Why [haven’t you received/are you no longer receiving] services from [referral services]?	Respondents reported that these grids were hard to process. By phone, these questions took a long time to read (all response options must be read). Respondents shared they could see all of these being reasons, but usually only selected a few responses at most. We suspect that respondents self-administering the survey were not actually reading all response options.	Drop several response options that have previously been shown to have low prevalence,3 details in Appendix A.
RI	B13. Did you receive a call from an E&T staff member at DHS after you [submitted an online form/responded to the text or email] to learn more information? (renumbered to B12)	During phone interviews, this set of questions was wordy and confusing for respondents.	Simplify to clarify what point of contact we are asking about.
RI	B20. Why haven’t you received services or support from [SNAP E&T Service Provider/the provider you selected]?	Respondents reported that these grids were hard to process. By phone, these questions took a long time to read (all response options must be read). Respondents shared they could see all of these being reasons, but usually only selected a few responses at most. We suspect that respondents self-administering the survey were not actually reading all response options.	Drop several response options that have previously been shown to have low prevalence,4 details in Appendix A.
Section C
CO, MA, RI	C3. What were the main reasons you decided to receive services from the SNAP E&T program?	Several respondents had difficulty understanding and defining what was meant by “self-employment”.	Add an example to help clarify this option.
All	C4 – C5 (C3 – C4 in CT). What were the main reasons [you haven’t received/stopped receiving] SNAP E&T services?	Respondents reported that these grids were hard to process. By phone, these questions took a long time to read (all response options must be read). Respondents shared they could see all of these being reasons, but usually only selected a few responses at most. We suspect that respondents self-administering the survey were not actually reading all response options.	Drop several response options that have previously been shown to have low prevalence,5 details in Appendix A.
All	C6 (CO, RI), C7 (MA), C5 (CT). What program offerings would make the SNAP E&T program more appealing to you? (renumbered to: C6-C7 (CO, RI), C7-C8 (MA), C5-C6 (CT))	One respondent noted that “appealing” was perhaps not the best way to ask this question. Additionally, most respondents selected all potential features, particularly during phone administration, even if they did not apply to their personal circumstances. When asked about this, they noted that they said “Yes” because the feature might be a helpful program offering for someone else.	Collapse this question into categories representing the most important elements (training/course location and times, online options, career planning/job placement services, additional support services, and staff training). Adjust response options to a Likert scale instead of yes/no For respondents who have received services, collect level of satisfaction with current offerings For respondents who haven’t received services, collect information about whether the program offerings would make them more likely to enroll in services.
All	C6 (CT), C7 (CO, RI), C8 (MA). Tell us more about the program offerings or services that you feel would make you more likely to [consider/continue] participating in the SNAP E&T program. (renumbered to: C8-C9 (CO, RI), C9-C10 (MA), C7-C8 (CT))	During phone administration, the question was redundant considering the most recent question had an “Other (SPECIFY)” option for program offerings that respondents would find appealing.	Break this up into two questions so that a specify box is only available if respondents have something to share other than what was previously asked.

^a Note that several questions from Section A (previously numbered A2 – A5) were moved to newly created Section D.

In order to reduce survey burden, we suggest several questions to drop from the survey. Table 3 outlines the questions we removed from the survey instrument.

Table 3. Suggested questions to drop

Site	Item(s)	Feedback	Revision
Section A
All	Intro.	Given the addition of consent language, this introductory text was no longer necessary.	Drop this text because the content is covered in the consent at the beginning of the instrument.
Section B
All	B16 (CO), B19 (MA), B12 (RI). When would it be helpful to begin receiving information about the SNAP E&T program/?	Most respondents felt that everyone would prefer to receive this information as soon as possible, One respondent shared that “right after” is an ambiguous timeline. Another respondent shared that referencing the eligibility interview could be confusing because of recertification appointments.	We will drop this question. Originally, we were hoping to understand if a lag in the E&T notification materials might be beneficial. Site teams are not prioritizing this question and we expect that most people will select the earliest possible response option such that this question is not particularly informative.
CO	B2. Was the message of the [text message/email/postcard] related to…	Pretest respondents had a hard time responding to this question, potentially due to the fictional scenario. One respondent noted that at one point they went to the benefits website and filled out an application for multiple benefits, and then preceded to receive numerous notifications that started to blend together.	We will drop this question given that it we do not expect it to yield much insight on what messages a control sample member might be receiving.
MA	B18. [After looking at the MassHire career center website/When you received a message about SNAP E&T services], how ready were you to participate in SNAP E&T services?	Similar to B19 (MA), this question was difficult for respondents to think about. Participants had difficulty reflecting on their readiness.	We will drop the question. We expect that we will get better data from the questions about barriers to participating in SNAP E&T services.

Next Steps

We have revised the burden estimate for the participant survey to 15-minutes in all OMB documentation. This estimate is inclusive of web and telephone administration. Upon receiving FNS approval of the revised instruments and pretest changes, we will translate the survey instruments into Spanish for OMB submission.

Appendix A: Additional background

Question A8 asks about challenges that make it hard to find a new job or keep a current job. We received two main pieces of feedback about this question:

• The question is long and burdensome. Respondents self-administering the questionnaire (akin to web mode) did not appear to read all response options fully.

• Respondents had trouble responding on the three point scale (not at all, a little, a lot), specifically differentiating between challenges that affected them “a little” versus “a lot.”

This question was adapted from the SNAP Employment and Training (E&T) Characteristics Study. Table 3 shares the frequency of barriers to employment found in that study, presented in order from greatest frequency of response to lowest. Using this data, we provide recommendations for response options to drop in the SNAP E&T RCE survey question A8, presented in table 4. We also propose to provide only yes/no response options and to split the grid into two separate questions to further encourage respondents to review each response option.

Table 3. SNAP E&T Characteristics Study: Reported barriers to employment by work registrants and E&T participants

Reported barriers	Percentage of work registrants	Percentages of E&T participants
Transportation issues	32.1	35.6
Physical or mental health issues	34.7	30.3
Lack of education	27.5	27.4
Need to care for family members	19.8	13.8
Housing problems	12.7	16
Criminal record	9.5	17.2
Lack of child or dependent care	10.9	15.5
Certification/license invalid	10	11.3
English language proficiency	11.8	5.2
Could not get along with supervisor or co-workers	5.1	3.8
Other	2.7	3.9
Substance abuse	2.8	3.5
Immigration or citizenship restrictions	1.2	1

Source: Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T) Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S. Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report

Table 4. Revisions to question A8

Question	Original Item	Revision
A8. Some people have challenges that make it hard to find a new job or keep a current job. Thinking about the challenges you may have had in the last year, how much dido the following make it hard for you to find or keep a job?	a. Could not find work or lack of jobs available in the area	Keep
	b. Transportation issues or problems (for example, no car or no public transportation available, transportation costs too much, public transportation takes too much time)	Keep, move examples to an interviewer probe that is read as needed. Examples will be available for web respondents.
	c. Family responsibilities (caring for children, spouse, or parent)	Keep
	d. Do not have right clothes for a job (including uniforms)	Drop
	e. Immigration, citizenship, or documentation barriers to legal employment (for example, do not have a birth certificate or a work visa)	Drop
	f. Do not have the right schooling	Keep
	g. Do not have the right job training, skills, or experience	Drop
	h. Do not have the right job search skills or experience (for example, resume writing, interviewing, or networking)	Added based on pretest feedback. This is also one of the most important SNAP E&T services reported
	h. Have a criminal record	Drop
	i. Have physical or mental health challenges (including disability)	Keep
	j. Have challenges with alcohol or substance abuse	Drop
	k. Could not get along with supervisor or co-workers	Drop
	l. Attending school	Drop
	m. Speaking, reading, and/or writing English	Drop, but add to question A3 to help understand challenges
	n. Housing problems (for example, homelessness, unstable housing or no regular place to stay, no affordable housing)	Keep, move examples to an interviewer probe.
	o. Discrimination (for example, on the basis of gender/sex, race/ethnicity, age, religion, sexual orientation)	Drop
	p. Something else (specify)	Keep

Question C4 and C5 ask the main reasons you decided not to receive E&T services (C4) or the main reasons you stopped receiving E&T services (C5). Respondents reported that these two parallel questions were long and burdensome. Respondents self-administering the questionnaire (akin to web mode) did not appear to read all response options fully. These questions were adapted from the SNAP Employment and Training (E&T) Characteristics Study. Table 5 shares the frequency of reasons found in that study, presented in order from greatest frequency of response to lowest. Using this data, we provide recommendations for response options to drop in the SNAP E&T RCE survey questions C4 and C5, presented in table 6.

Table 5. SNAP E&T Characteristics Study: Reasons given for not participating in E&T, by individuals told to participate who did not and potential volunteers

Reasons given for not participating in E&T	Percentage of individuals told to participate who did not	Percentage of potential volunteers
Need to care for child or others	30.2	49.1
Didn’t sound useful	17.0	21.1
Transportation issues	16.6	NR
Wasn’t about something they wanted to learn	12.7	0.7
Got a joba	12.8	0.3
Other	10.0	0.7
Physical or mental health problemsa	10.1	0.0
Program at a bad time	4.3	0.0
Location not on public transportation	0.2	0.6
Too expensive to get there	0.0	0.7
Previous participation and didn’t like it	0.1	0.6
Inconvenient or unsafe location	0.0	0.0

Source: Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T) Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S. Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report

Notes: Asked only of respondents who participated in E&T in the last 12 months. Excludes missing responses, respondents who refused to answer, and responses of “don’t know.” Examples in the ‘other’ response category include lack of translation services and lack of information about the program. Respondents could select all the reasons that apply. All data presented in this table are weighted. NR= Not reported; there were too few respondents to provide a reliable estimate.

^a Category added based on coding “other” response options.

Table 6. Revisions to question C4

Question	Original Item	Revision
C4. What were the main reasons you decided not to receive services from [the SNAP E&T program/E&T PROGRAM NAME]?	1. Did not know about the program	Keep, revise to: “You lacked information about the program”
	2. Program doesn’t match my needs	Keep (captures “it didn’t sound useful” from SNAP E&T Characteristics survey)
	3. Didn’t think the program would help me to find a job	Keep (captures “it didn’t sound useful” from SNAP E&T Characteristics survey)
	4. Got a job	Keep
	5. Location was inconvenient or unsafe	Drop
	6. Transportation issues or problems (for example, no car or public transportation available, transportation costs too much, public transportation takes too much time)	Keep, move examples to an interviewer probe
	7. Started (other) school or training	Drop
	8. Child care problems	Drop (note this is now inclusive of option 11 below)
	9. Got pregnant, or had a baby	Drop
	10. Had health problems or an injury (including disability)	Keep, revise to: “Physical or mental health challenges (including disability)”
	11. Caring for family members’ physical or mental health problems	Keep, revise to: “Need to care for child or family member”
	12. Housing issues/moved	Keep
	13. Program was at a bad time	Drop
	14. Didn’t have the time	Drop
	15. No longer wanted to find employment	Drop
	16. Was in a program before and didn’t like it	Drop
	99. Other	Keep

1 https://www.thehrcfoundation.org/professional-resources/self-identification-of-lgbtq-employees

2 Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T)

Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S.

Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report

3 Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T)

Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S.

Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report

4 Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T)

Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S.

Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report

5 Rowe, Gretchen, Elizabeth Brown, and Brian Estes. SNAP Employment and Training (E&T)

Characteristics Study: Final Report. Prepared by Mathematica Policy Research for the U.S.

Department of Agriculture, Food and Nutrition Service, October 2017. https://www.mathematica.org/publications/snap-employment-and-training-e-t-characteristics-study-final-report