Appendix V. Pre-test Results Memorandum

Appendix V. Pre-test Results Memorandum

OMB No. 0584-[NEW]

Assessing SNAP Participants’ Fitness for Work

Project Officer: Eric Sean Williams

Office of Policy Support

SNAP Research and Analysis Division

Food and Nutrition Service

U.S. Department of Agriculture

1320 Braddock Place

Alexandria, VA 22314

703.305.2640

[email protected]

1. Protocols

Methods

We pretested the interview guides with Washington State between September 8^th and 17^th, 2021. We scheduled and conducted four interviews, to incorporate different potential respondents to the protocols:

• 1 State-level administrator (State/Local Admin Protocol)

• 1 Local administrator (State/Local Admin Protocol)

• 3 Public Benefit Specialists (State Eligibility Worker Protocol)

• 1 SNAP E&T Manager (SNAP E&T Protocol)

For each call, we scheduled the expected amount of time for the interview, with an additional 30 minutes to debrief and discuss feedback. Three study team members joined each call: one person who led the interview, one who took notes, and the third who also asked questions and served as a general subject matter expert related to SNAP and SNAP E&T.

For each call, we provided the consent form in advance via email. We walked through the consent to see if respondents had any feedback, though we are not using or storing their data. We then asked if they had any questions or feedback on the consent process.

Following the consent process, we conducted the interview as if it were a real interview. Due to the nature of an interview pretest, some changes that needed to be made were obvious in real time (e.g., if a respondent did not understand the language in a question). In other cases, respondents provided feedback after the conclusion of the interview.

It is important to note that questions related to provider determinations were not asked, as the final rule related to provider determinations was not in effect at the time of the interviews. In addition, we were not able to pretest the observation guide due to constraints related to COVID.

Findings

Overall, we found that the protocols worked well. Feedback from respondents was positive, including that they generally understood the questions, they felt the flow of the conversation was intuitive, they did not feel that the time needed was too burdensome, they felt the key topics were covered, and they appreciated the importance of the topic and the opportunity to have a discussion about it. Below, we outline some high-level findings and feedback that we heard from respondents. We discuss these in more detail related to the changes to the protocols in the next section.

• Clarity about type of work requirements. We found that it was critical to be clear about what types of work requirements we were asking about at any given time. Washington defaulted to discussing ABAWDs, so in order to understand the distinctions between general work requirements and ABAWD requirements, we had to be very explicit with each question.

• Flow and reducing repetition. We also found that there could be improvements to the flow of the protocol and some deduplication/consolidation of questions to reduce repetition for respondents. This was especially the case for questions related to the application and determination process.

• Identifying appropriate respondents. For the SNAP E&T protocol, we learned that there will be variation in which staff members can answer which types of questions and how the questions should build on each other.

• Timing needed. We learned that the time required for each interview may not be what we intended. We expect that the state admin protocol may take up to 90 minutes, especially as they will be the first people we talk to. Eligibility workers may take less than 90 minutes.

• Distinguishing types of good cause: One respondent noted that it would be important to be clear about good cause related to not meeting work requirements and good cause for voluntarily quitting a job.

• Incorporating client experience. One respondent noted that it would be important to incorporate how the client experiences the process into the questions.

• Unusual situations due to COVID-19. It was apparent from our conversations that respondents’ answers were heavily influenced by the conditions imposed by COVID-19. For example, with so many of the requirements currently waived, some respondents had trouble recalling some details of policies related to work requirements. In addition, questions regarding the comparison between telephone interviews and in-person interviews were more difficult for respondents to recall, since they have not been doing in-person interviews for about 18 months.

We discuss the implications of these findings further in the next section.

Overview of Changes

This section describes the types of changes we plan to make to the interview guides to address the feedback and findings above.

Changes to wording and order of questions. Throughout the protocols, we have updated the order of questions, removed questions that were covered elsewhere in the protocol to reduce duplication, and added some clarifying probes, questions, language, and guidance. Tables 1-3 show each question or section that we expect to change (using the original numbering scheme), a description of the change, and the reason for the change.

Timing of interviews: We found that the State administrator interview could take closer to 90 minutes, especially as we expect State administrators will be the first staff we speak to and will provide critical context for the rest of the interviews. Therefore, we propose to raise the estimated time and burden estimate per interview to 90 minutes. Conversely, we found that eligibility workers may take less than 90 minutes, as there is less to cover in their interview. However, recognizing that there may be group interviews for eligibility workers, we propose to maintain the 90 minute burden for eligibility workers so that we do not risk underestimating the burden for this group of respondents.

Implications for data collection training

• SNAP E&T staff guidance: Based on feedback from the SNAP E&T respondent, we will provide additional guidance in data collection training about which SNAP E&T staff to schedule with, how to get a mix of respondents that can address the questions, and which questions may be most appropriate for different staff roles.

• Use local names. As we anticipated, it was helpful for the respondents if we used the local names of “Basic Food” and “BFET” for SNAP and SNAP E&T respectively. We will include this in the guidance for the data collection training.

Changes related to public comments

In addition to changes relating to pre-test findings, we also added some questions to the instruments to respond to public comments related to potential racial disparity and bias in determining whether individuals should be exempt from work requirements due to physical or mental limitations. These are also detailed in the tables below, but fall into three broad categories:

• Use of data: We added a question on whether States review data related to race to understand possible racial disparities in providing exemptions.

• Training: We added questions and probes related to training for eligibility workers related to implicit bias, racial disparities, and cultural differences related to acknowledgement and understanding of mental illness.

• Mental health assessment: We added questions and probes asking whether States provide opportunities for mental health assessment for SNAP applicants or participants and how they consider cultural differences related to mental health issues.

2. Survey

Methods

We pre-tested the survey with policy staff at the State level in two states: Colorado (county-administered) and Utah (state-administered). We sent respondents a link to the survey, which we programmed to include two additional questions at the end of each module: one to gauge how many minutes the module took the respondent to complete, and the other to provide an opportunity for the respondent to provide their written feedback on the module. At the beginning of the survey, we added an instructions document that pre-test respondents could download and refer to while they took the survey.

We gave the pre-test respondents two weeks to complete the survey, at which point we scheduled a one-hour debrief. We held the debrief call with Colorado on August 11^th and with Utah on September 13^th. Prior to the debrief we analyzed their survey data to ensure the survey worked as designed and to uncover any issues. During each pre-test debrief, we asked respondent’s about:

• The overall user experience, whether there were any issues with the survey platform, or other technological issues

• Whether they enlisted the help of other staff to complete the survey or consult any documentation

• Whether the structure and the flow of the survey made sense to them

• Whether the wording and phrases used in the questions resonated, particularly “exemptions from general work requirements” and “exemptions from ABAWD” work requirements

• Whether the response options captured all of the relevant possibilities

• The feedback they provided within the survey

Findings

Below, we outline some high-level findings and feedback that we heard from respondents. We discuss these in more detail related to the changes to the survey in the next section.

• Time burden and staff involved. It took both pre-test respondents between 30 and 40 minutes to complete the survey in whole. In both instances, they completed the survey in one sitting and did not need to request assistance from other staff to complete the survey, although they did consult some of their written policy guidance and manuals. While the survey was not burdensome from the perspective of length, see discussion below about issues pre-test respondents had with the structure.

• Structure and flow. Currently, the survey is arranged into four modules – one on policies, one on development of policies, one on process, and one on data – each containing questions about different sets of requirements. Respondents reported that the structure of the survey caused confusion about whether and which questions pertained to general work requirements, ABAWD work requirements, E&T requirements, or good cause determinations. This meant they had to remind themselves what requirements they should be thinking about, which could result in respondents providing incorrect information. They also found answering the same set of questions twice (once for general work requirements and once for ABAWD work requirements) confusing and repetitive. Respondents found Module C, which dealt with process, the easiest to complete.

• Overreliance on “select all that apply”. Because there are many nuanced aspects to the determination process (mode of application, individual applicant circumstance, degree of staff discretion, etc.), selecting one response option for some questions was challenging, and in several instances where we allowed respondents to select all that apply, they selected every single response option. This did not provide us with a clear understanding of the process for making determinations.

Overview of Changes

This section describes the types of changes we plan to make to the survey to address the feedback and findings above. Tables 4 and 5 shows the modules and questions that we expect to change (using the module and question numbers from the original survey instrument), a description of the changes that we propose making, and the reason for the change.

Changes to module structure. Currently, the survey is arranged into four modules – one on policies, one on development of policies, one on process, and one on data – each containing questions about different sets of requirements. We propose arranging the survey by requirement instead, asking all questions about work requirements first, followed by questions about mandatory E&T, and lastly good cause determinations. Within the modules of the current version, questions are first asked for general work requirements and then asked for ABAWD work requirements. We propose instead adding a follow-up to each question about general work requirements that asks whether the responses are different for ABAWD work requirements. Only in the instances where their responses would be different would we ask that question for ABAWD work requirements. Appendix A includes a memorandum proposing these structural changes. The proposed structural changes were approved on November 16, 2021.

Changes to question order. After re-arranging the questions based on the suggestion above and based on feedback from respondents that Module C about process was easiest to complete, we believe ordering questions in a way that tries to mirror the order in which staff might go through the determination process will be a more logical survey flow for respondents. This revision also made it clear which questions could be combined to reduce repetitiveness while also capturing the necessary information.

Changes to question complexity. For questions that were challenging for respondents to select just one response, or for questions where respondents selected all available responses, we propose breaking them up into smaller questions that each address a more specific question to get more useful data.

Changes related to public comments. In addition to changes relating to pre-test findings, we also added a question to the survey to respond to public comments related to potential racial disparity and bias in determining whether individuals should be exempt from work requirements due to physical or mental limitations. This is detailed in the table below.

3. Next Steps

We propose conducting an additional pre-test of the revised survey with one of the original pre-test respondents concurrent to FNS’ review of this memo and proposed changes. The focus of this pre-test will be on ensuring that respondents clearly understand which requirement they should be thinking about for each question. We anticipate that any changes that arise from this pre-test will be to the introductory language or added instructions to respondents to make that clear, as opposed to any structural changes. The final survey instrument we submit with the final version of this memo will incorporate feedback from FNS as well as any minor changes from the additional pre-test. We do not believe an additional pre-test of the protocols is necessary and will submit a final version of the protocols that incorporate FNS’ feedback with the final version of this memo.

Table 1: Changes to Interview Protocol for State and Local SNAP Administrators

Original Question Number	Change	Rationale
0.4	Refined the question about describing the State agency.	It was not necessary to understand the structure of the entire agency in Washington that administers SNAP, so this question will get at the information we need with more quickly and efficiently.
0.7a	Added guidance for interviewers to bring a printed list of the work requirements to walk through.	It was awkward and disjointed to list all the work requirements verbally, so having a visual/physical item to go over will help with flow and clarity.
	Add more explicit guidance to clarify differences between ABAWDs and general work requirements	Respondents defaulted to describing requirements related to ABAWDs. We concluded that more clarity was needed to ensure we get information on all possible requirements.
0.7b and c	Move down to the section on SNAP application processes.	These questions would fit better as part of the discussion on application processes.
Section 0/1	Created a new section called “Process” and added questions about processes related to exemptions from Section 3 (Process) with background information on application process and recertification. Combine with section about how clients are notified. Reworded a few questions for clarity.	The respondents covered most of the information from Section 3 in the background section and it was very helpful to have this detail up front. Therefore, we moved this information earlier to ensure it is covered and sets the stage for the rest of the interview.
Section 0	Updated the questions on work requirements to step more explicitly through general work requirements, ABAWDs, and mandatory E&T.	We found that respondents defaulted to describing one type of requirement so we felt this would make it clearer what we are asking about at each stage.
Section 0	Added a question on how processes changed during COVID.	This will align with questions about how policies may have changed during COVID.
Section 1	Added additional intro language to show how the policy questions are distinct from process questions.	With the process questions up front now, it is more important to create a clear transition and distinction between the two sets of questions.
1.1	Add specific questions about work registrants, ABAWDs, and SNAP E&T (instead of one of a list of probes)	This was another change to ensure clarity about which work requirements we are asking about.
1.1	Added a probe on whether the State has policies related to providing access to mental health professionals for assessment	Added based on comments from Third Sector.
2.4	Added a sub-question asking about training related to implicit bias or racial disparities in fitness for work and cultural differences related to mental health.	Added based on comments from Third Sector.
Section 3	Focused this section only on good cause.	We moved the process questions up and renamed this section to focus only on good cause.
3.B	Add an introduction for the section on good cause.	We found that the shift to the discussion of good cause was a bit disjointed, so more introduction here would be helpful.
3.A.4	Delete.	The recertification process was already covered in the Background section, so asking this here was duplicative.
4.4	Added question related to reviewing data related to racial disparities	Added based on comments from Third Sector.
5.1	Deleted the reference to good cause in the first question.	We streamlined this question to ask about challenges developing policies and guidelines more generally.
Section 5	Added a question about client experience.	One respondent in Washington felt this topic was missing from the protocol and we agreed it could be helpful to include.

Table 2: Changes to Interview Protocol for State Eligibility Staff

Question	Change	Rationale
Section 1	Move up questions on data entry	Data entry: We moved this up because the process the workers described aligned with how they enter information into the system. We kept the section on data at the end as well, to ensure that all questions get covered, with guidance to skip if the questions have already been answered.
	Moved up questions on recertifications and good cause	Good cause: It was a little confusing for workers to discuss good cause for the first time later in the interview. To set the stage for the more detailed discussions on good cause and recertifications, we moved high-level questions on these topics to the general discussion of process.
2.1	Make separate questions for work registrants and ABAWDs, rather than asking as a sub-question, and added more guidance for interviewers.	To ensure we are clear about the requirements for different groups, we felt it was important to make these separate questions so we are clear about what they are describing at all times, rather than asking a general question and then asking about variation. Respondents were not always clear about which group they were referring to initially.
2.1, 2.2	Added probes about providing access to professional for assessment	Added based on Third Sector comments
2.4	Moved this to be a higher level in the protocol.	This should be asked of all States, not just mandatory States.
2.3	Moved 2.3 down to section related to E&T specifically.	This was repetitive as written.
2.5-8	Added some guidance for which States to ask these questions of.	We specified that the question about criteria for screening and referral should be asked of non-mandatory States, to ensure that it’s not repetitive with the earlier questions for mandatory States.
2.11-12	Combine the questions on flexibility into one – with a probe on flexibility in determining which criteria to use, as well as how to apply the criteria	These questions were repetitive to ask separately and created a disjointed conversation, so we created one question with probes on distinguishing between the two types of criteria to ensure that the discussion flows well but that we capture all information needed.
2.14	Remove this question and combine with 2.13 for one overall questions on what they do if they have questions or need help	These questions were repetitive when asked separately so we combined into one, with separate probes.
2.13	Added a sub-question related to how they consider cultural differences if uncertain about exemptions.	Added based on Third Sector comments
Section 3-4	Combine recertification and good cause into a single section, rename “Situations That May Lead to Changes to Exemptions”, and added some introductory text.	The shift to these topics was sudden and somewhat confusing for respondents. To make this clearer, we moved some high-level questions related to these topics to Background (see updates to Section 1, above), combined this into a section that we describe as focused on potential changes to exemptions, and added relevant introductory text.
	Add clarification about good cause related to voluntarily quitting a job.	Based on feedback from respondents, we also added a sub-question related to good cause for voluntarily quitting a job.
Section 5	Removed good cause training/resource questions and incorporated good cause into the initial questions	It was repetitive to ask all of these questions twice, so we included a probe on good cause related to the training and guidance questions and removed the separate questions.
5.2	Added a probe about whether training includes info on implicit bias or racial disparities or cultural differences	Added in response to comments from Third Sector
Section 6	Added some guidance that questions may have been addressed above in process section.	We found that some of these questions were addressed earlier on, but wanted to keep these questions separate as well to ensure interviewers address them.
Section 7	Added question about client experience	Based on feedback, we added a question about clients’ experience with exemptions due to physical or mental limitations

Table 3: Changes to Interview Protocol for SNAP E&T Staff

Question	Change	Rationale
Introduction	Added more information about respondents to the guidance.	The E&T respondent in Washington noted that some of her staff would not be able to answer our questions, so suggested that we include someone in a supervisory role for certain questions in addition to staff who serve clients directly.
Section 2:	Restructured to have two branches of questions related to whether respondent communicates with eligibility workers.	We found this worked better during the pretest, because there were questions we would want to ask of all respondents, in addition to some more specifically tailored for those who do communicate with eligibility workers. The update makes this clearer.
Section 2.d	Added a question on other instances that SNAP E&T staff may communicate with eligibility workers.	We learned there were other informal reasons that staff may communicate and want to be sure to capture these.
Section 4	Added guidance on what information respondents may or may not know	Given the different levels of staff that may be appropriate here, it seemed that additional guidance to make sure that interviewers understand the different types of training provided would be helpful.
6.1	Removed guidance on communication – relevant for all participants	During the pre-test, the question was relevant regardless of whether respondents communicate with eligibility workers.
6.4	Removed question about additional information.	This question was duplicative of an earlier question asked in Section 4.

Table 4: Changes to Survey Modules

Original Module	Change	Rationale
A.1, A.2., B.1., B.2., C, D	Combine these modules into one, Module A, which will include all questions about general and ABAWD work requirements.	Based on feedback from the pretest respondents, we believe that grouping the questions pertaining to general work requirements and ABAWD work requirements will help reduce confusion. This module will include all questions related to general and ABAWD work requirement policies, processes, and data, instead of separating them into different modules. Within the revised Module A, the flow of questions mirrors the flow of how staff would process an individual’s SNAP application and determine exemptions.
A.5	Reorder to be a standalone module, Module B, focused on mandatory E&T requirements.	This module worked well during the pretest, hence we did not make changes to the questions in this module. We reordered this module to be asked after all questions about general and ABAWD work requirements.
A.3., A.4., B.3	Combine these modules to form one module, Module C, focused on good cause for not meeting general or ABAWD work requirements.	Based on feedback from respondents that they think about policies and processes around good cause determination separately from work requirements, we pulled out questions related to good cause determinations into its own module.

Table 5: Changes to Survey Questions

Question	Change	Rationale
A.1, A.2, A.12	Combined these questions, revised response options, and moved down to come after questions about how States conduct eligibility interviews (A.7). Revised response options to differentiate between short- and long-term physical and mental limitations that can be considered when determining whether someone is exempted from work requirements, added pregnancy as a response option, and revised ‘Homelessness’ to be ‘Lack of stable housing’.	These questions are all about the physical or mental limitations that can be considered when determining if an applicant should be exempted from work requirements, so we combined them into one question. We revised the response options for clarity, per feedback from the respondents on how their States consider physical and mental limitations.
A.3	Changed this open-ended question into two close-ended questions, asked separately for individuals who participate in face-to-face interviews and those who do not (A.14, A.15, A.17, A.18).	We learned from both the pre-test of the protocols and from the debrief with survey respondents that when verification is required is a nuanced process, so we felt asking it separately for individuals participating in face-to-face interviews and those who don’t will provide clearer data.
A.4, A.23	Instead of asking the information required to verify each type of physical or mental limitation separately for general and ABAWD work requirements, we ask about the type of information required for those who participate in face-to-face interview and those who do not. We also revised the response options. (A.16, A.19)	We learned from both the pre-test of the protocols and from the debrief with survey respondents that the type of verification required is a nuanced process, so we felt asking it separately for individuals participating in face-to-face interviews and those who don’t will provide clearer data. We also revised the response options per feedback on what type of verification is accepted.
A.5– A.11	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response options of A.16 and A.19.
A.13, A.14, A.32, A.33	Combined these questions and expanded the response options to ask more generally about which applicants are screened for physical or mental limitations (A.2.).	Instead of having two shorter questions and repeating them for both general and ABAWD work requirements, we combined them into one question and expanded the response options to better reflect the process as described to us by pre-test respondents.
A.15, A.34, C.30, C.31	Split up A.15 to first ask about the type of training that staff involved in determining whether an individual is exempt from general or ABAWD work requirements receives (A.40), followed by what other ways staff are informed about how to determine exemptions (A.41). We no longer ask the question for ABAWD and general work requirements separately.	Pre-test respondents found A.15 and A.34 difficult to answer because they viewed training as very different from the other ways policies are communicated. They also did not think of these differently for ABAWD and general work requirements.
A.16–A.19	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option of A.41.
A.20, A.21, A.31	Similar to the original questions around general work requirements (A.1, A.2, A.12 discussed above), we combined these questions about ABAWD work requirements and added response options (A.9). We moved this new question to come after the same question for general work requirements. However, respondents are only asked A.9 if the physical and mental limitations that can be considered for ABAWD requirements are different than those considered for general work requirements (A.8).	These questions are all about the physical or mental limitations that can be considered when determining if an applicant should be exempted from work requirements, so we combined them into one question. We revised the response options for clarity, per feedback from the respondents on how their States consider physical and mental limitations. Based on feedback about the survey structure, we believe that asking respondents to answer the same question twice (once for general work requirements and once for ABAWD work requirements) only in the instances where there are differences between the two will reduce confusion and the burden for respondents.
A.22	We moved this question up and revised to assess the difference between ABAWD and general work requirements (A.20).	Based on feedback about the survey structure, we believe that asking respondents to answer the same question twice (once for general work requirements and once for ABAWD work requirements) only in the instances where there are differences between the two will reduce confusion and the burden for respondents.
A.24–A.30	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option A.16 and A.19.
A.35–A.38	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is captured in the “Other (please specify):” response option of A.41.
A.39	Moved down to the revised Module C, which is focused on good cause determinations specifically. Revised response options to differentiate between short- and long-term physical and mental limitations that can be considered when determining whether someone has good cause, added pregnancy as a response option, and revised ‘Homelessness’ to be ‘Lack of stable housing’ (C.1)	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion. We revised the response options for clarity, per feedback from the respondents on how their States consider physical and mental limitations.
A.40	Moved question down to the revised Module C, which is focused on good cause determinations specifically (C.4)	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion.
A.41, A.42	Moved questions down to the revised Module C, which is focused on good cause determinations specifically (C.7, C.8)	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion.
A.43, A.51	Moved questions down to the revised Module C, which is focused on good cause determinations specifically. Combined both questions to ask about how the State communicates policies regarding good cause determinations for general and ABAWD work requirements. Revised the matrix question to a multiple-choice question about how the State communicates policies regarding good cause more generally (C.14).	Respondents selected all available responses, so we felt we could reduce the complexity and ask it generally for policies related to good cause.
A.44-A.46	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option for C.14.
A.47	Moved down to the revised Module C, which is focused on good cause determinations specifically. Revised response options to differentiate between short- and long-term physical and mental limitations that can be considered when determining whether someone has good cause, added pregnancy as a response option, and revised ‘Homelessness’ to be ‘Lack of stable housing’ (C.3). Respondents only get this question if the physical or mental limitations that are considered good cause are different for ABAWD work requirements than general work requirements (C.2).	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion. We revised the response options for clarity, per feedback from the respondents on how their States consider physical and mental limitations. Based on feedback about the survey structure, we believe that asking respondents to answer the same question twice (once for general work requirements and once for ABAWD work requirements) only in the instances where there are differences between the two will reduce confusion and the burden for respondents.
A.48	Moved question down to the revised Module C, which is focused on good cause determinations specifically (C.6). Respondents only get this question if the length of a good cause determination is different for ABAWD and general work requirements (C.5).	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion. Based on feedback about the survey structure, we believe that asking respondents to answer the same question twice (once for general work requirements and once for ABAWD work requirements) only in the instances where there are differences between the two will reduce confusion and the burden for respondents.
A.49, A.50	Moved these questions down to the revised Module C, which is focused on good cause determinations specifically (C.9, C.10)	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion.
A.52-A.54	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option for C.14.
A.55	Moved questions down to the revised Module B, which is focused on E&T requirements specifically. (B.1)	Per feedback from respondents, we pulled out questions around E&T to Module B to reduce confusion.
A.56	Moved question down to the revised Module B, which is focused on E&T requirements specifically. Revised response options to differentiate between short- and long-term physical and mental limitations that can be considered when determining whether someone is exempted from work requirements, added pregnancy as a response option, and revised ‘Homelessness’ to be ‘Lack of stable housing’. (B.2)	Per feedback from respondents, we pulled out questions around E&T to Module B to reduce confusion. We revised the response options for clarity, per feedback from the respondents on how their States consider physical and mental limitations.
A.57	Moved questions down to the revised Module B, which is focused on E&T requirements specifically. Revised response options. (B.3.)	Per feedback from respondents, we pulled out questions around E&T to Module B to reduce confusion. We revised the response options to better reflect States’ processes.
A.58	Moved questions down to the revised Module B, which is focused on E&T requirements specifically. (B.4)	Per feedback from respondents, we pulled out questions around E&T to Module B to reduce confusion.
B.1, B.18	Moved to Module A. Revised the question wording and response options for clarity. Combined both questions to ask about how generally policies are established (A.45)	This question was difficult for respondents to answer because they immediately thought of the federal regulations. The response options were also revised for clarity, to understand whether the state legislature, state agency, or someone else, is responsible for codifying policies regarding exemptions from general or ABAWD work requirements. We combined both questions to reduce repetitiveness.
B.2, B.19	Moved to Module A (A.46)	With B.1 and B.18 combined into one question, only one follow-up question is necessary.
B.3, B.20	Moved to Module A (A47)	With B.1 and B.18 combined into one question, only one follow-up question is necessary.
B.4, B.21	Moved to Module A (A.48)	With B.1 and B.18 combined into one question, only one follow-up question is necessary.
B.5.1, B.6	We split the original question (B.5) into three separate questions and ask variation for each type of policy separately (A.11). We also ask the respondents to explain the type of variation within the same question.	The follow-up questions on the type of local variation were difficult for respondents to answer, so we split the questions out and ask for respondents to explain the type of variation within the original question.
B.5.2, B.7	We split the original question (B.5) into three separate questions, and ask about variation for each type of policy separately (A.21). We also ask the respondents to explain the type of variation within the same question.	The follow-up questions on the type of local variation were difficult for respondents to answer, so we have instead split these questions out and ask for respondents to explain the type of variation within the original question.
B.5.3, B.8	We split the original question (B.5) into three separate questions, and ask about variation for each type of policy separately (A.5). We also ask the respondents to explain the type of variation within the same question.	The follow-up questions on the type of local variation were difficult for respondents to answer, so we have instead split these questions out and ask for respondents to explain the type of variation within the original question.
B.9, B.26	Combined the questions to ask just one question about exemptions from general and ABAWD work requirements (A.49).	These questions were repetitive when asked separately so we combined into one question.
B.10, B.27	Moved to Module A and combined both questions to ask generally about who is involved in reviewing and updating policies (A.50)	We combined both questions to reduce repetitiveness.
B.11, B.28	Moved to Module A and combined both questions to ask generally about the information staff consult when updating policies (A.51)	We combined both questions to reduce repetitiveness.
B.12, B.29	Deleted	Given that we will be fielding this survey two years after the onset of the pandemic, asking about any changes made unrelated to COVID-19 will be difficult for respondents to answer.
B.13.1, B.30.1	Revised this multiple-choice question into two separate questions and combined the questions about general and ABAWD work requirements into the same question. The revised questions allow for text entry and ask whether States made any changes during COVID-19 to the physical or mental limitations that can be considered when determining exemptions from work requirements (A.13)	The revised questions answer the research questions more directly and the open-ended question allows for respondents to explain what changes were made during COVID-19.
B.13.2, B.30.2	Revised this multiple-choice question into two separate questions and combined the questions about general and ABAWD work requirements into the same question. The revised questions allow for text entry and ask whether States made any changes during COVID-19 to the information required to verify a physical or mental limitation (A.23).	The revised questions answer the research questions more directly and the open-ended question allows for respondents to explain what changes were made during COVID-19.
B.14, B.15, B.31, B.32	Deleted.	Given that we will be fielding this survey two years after the onset of the pandemic, asking about any changes made unrelated to COVID-19 will be difficult for respondents to answer.
B.16, B.17, B.33, B.34	Combined questions to ask about States’ policies on exemptions from general and ABAWD work requirements (A.52).	Per respondents’ feedback, we revised this to reduce repetitiveness.
B.22.1, B.23	Combined the response option and relevant follow-up question (A.12)	The follow-up questions on the type of local variation were difficult for respondents to answer, so we have instead split these questions out and ask for respondents to explain the type of variation within the original question.
B.22.2, B.24	Combined the response option and relevant follow-up question (A.22)	The follow-up questions on the type of local variation were difficult for respondents to answer, so we have instead split these questions out and ask for respondents to explain the type of variation within the original question.
B.22.3, B.25	Combined the response option and relevant follow-up question (A.6).	The follow-up questions on the type of local variation were difficult for respondents to answer, so we have instead split these questions out and ask for respondents to explain the type of variation within the original question.
B.35-B.38	Moved these questions down to the revised Module C, which is focused on good cause determinations specifically (C.15-C.18)	Per feedback from respondents, we pulled out questions around good cause determinations to Module C to reduce confusion.
B.39, B.40	Deleted.	Given that we will be fielding this survey two years after the onset of the pandemic, asking about any changes made unrelated to COVID-19 will be difficult for respondents to answer.
C.1-C.5	Moved up to Module A (A.1-A.4.).	To follow flow of how staff would process an individual’s SNAP application and determine exemptions.
C.6-C.8, , C.12-C.14	Combined questions and revised from open-ended to multiple-choice questions, to better understand which staff are involved in specific situations when determining whether an individual is exempt from general or ABAWD work requirements (A.24-A.37).	Respondents found it difficult to select just one response option for the original questions, since different staff are involved in the determination process at different points or due to the specific circumstances of the applicant.
C.9.1	Revised this response option to be it’s own question that comes after the questions around the physical or mental limitations that can be considered (A.10)	We moved this up to the flow of questions mirrors the flow of how staff would process an individual’s SNAP application and determine exemptions
C.15-C.20	Moved questions to Module A (A.43, A.44)	We moved this up to the flow of questions mirrors the flow of how staff would process an individual’s SNAP application and determine exemptions
C.21	Revised this question to ask when the different staff are involved in the good cause determination process. Ask it separately for general (C.11) and ABAWD work requirements (C.13), although respondents only get this question if the staff involved in the good cause determination process for ABAWD work requirements are different than general work requirements (C.12).	Respondents found it difficult to select just one response option for the original question, since different staff are involved in the good cause process at different points or due to the specific circumstances of the applicant.
C.22, C.23	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option of C.11 and C.13.
C.24	Moved questions up to the revised Module B, which is focused on E&T requirements specifically (B.5)	Per feedback from respondents, we pulled out questions around E&T to Module B to reduce confusion.
C.25, C.26	Deleted.	No longer require follow-up questions for respondents who select “Other” in response to matrix-type questions. This information is now captured in the “Other (please specify):” response option of B.5.
C.27	Moved to Module A (A.42)	We moved this up to the flow of questions mirrors the flow of how staff would process an individual’s SNAP application and determine exemptions
C.28, C.29	Deleted.	Given that we will be fielding this survey two years after the onset of the pandemic, asking about any changes made unrelated to COVID-19 will be difficult for respondents to answer.
D.1	Moved to Module A and added a response option and a follow-up question to address feedback from Third Sector’s public comment (A.38, A.39)	We moved this up to the flow of questions mirrors the flow of how staff would process an individual’s SNAP application and determine exemptions. We added a response option and a follow-up question to address feedback from Third Sector’s public comment.
D.4, D.5	Deleted	Respondents had a hard time answering this question because their answers depended on specific circumstances.

Appendix A. Memorandum of Proposed Changes to Survey

Memorandum

DATE: October 21^st, 2021

TO: Eric Williams

FROM: Mary Farrell, Kimberly Foley, Riley Webster, and Eunice Yau

SUBJECT: Overview of Proposed Changes to Survey

Findings from Pretest

Respondents reported that the structure of the survey caused confusion about whether and which questions pertained to general work requirements, ABAWD work requirements, E&T requirements, or good cause determinations This meant they had to remind themselves what requirements they should be thinking about, which could result in respondents providing incorrect information. Lastly, having to answer the same question multiple times for the different requirements, even when their answer did not vary, was burdensome.

Proposed Solution

To address these concerns, we are proposing two structural changes:

• Currently, the survey is arranged into four modules – one on policies, one on development of policies, one on process, and one on data – each containing questions about different sets of requirements. We propose arranging the survey by requirement instead, asking all questions about work requirements first, followed by questions about E&T requirements, and lastly good cause determinations.

• Within the modules of the current version, questions are first asked for general work requirements and then asked for ABAWD work requirements. We propose instead adding a follow-up to each question about general work requirements that asks whether the responses are different for ABAWD work requirements. Only in the instances where their responses would be different would we ask that question for ABAWD work requirements.

We are also proposing some smaller revisions to question order and complexity, discussed in greater detail in the forthcoming pre-test memo.

Justification

We believe grouping questions by type of requirement will help reduce confusion and yield higher quality data. We also believe that limiting the repeat of questions only in instances where the responses will differ will help reduce burden for respondents.