AA. Pretest Memorandum

Attachment AA. Pretest Memorandum

OMB No. 0584-[NEW]

Job Search as a Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) Component

January 14, 2021

Project Officer: Danielle Deemer

Office of Policy Support

Food and Nutrition Service

U.S. Department of Agriculture

1320 Braddock Place

Alexandria, VA 22314

703.305.2952

[email protected]

Memorandum

Date: April 23, 2019 (Updated May 14, 2019 per FNS feedback)

To: Danielle Deemer, FNS

From: Brian Estes, Kathy Wroblewska, and Brittany McGill, Insight Policy Research

Subject: Job Search as a SNAP E&T Component: Pretest Findings and Recommendations

This memorandum summarizes the pretest findings and recommendations for the following data collection instruments for the Job Search as a Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) Component study:

1. State SNAP director and E&T director interview protocol

2. Process-mapping protocol

3. Local SNAP office director and staff interview protocol

4. E&T provider interview protocol

5. E&T participant interview protocol

6. Administrative cost worksheet and transmittal email

The primary objective of the pretest was to ensure the instruments were clear and easy for respondents to understand. The pretest sought to:

• Identify problems related to communicating the intent or meaning of the questions.

• Determine whether respondents could provide the information requested.

• Identify problems with the introductions, instructions, or explanations.

• Assess the amount of time needed to complete the instruments and other respondent burden issues.

Section A describes the recruitment and data collection methods for the pretest. Section B provides an overview of the pretest findings, and Section C summarizes the pretest findings and associated recommendations for improvement for each data collection instrument. [5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.]

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-xxxx. This is an internal FNS document and no burden is associated with this document; it will not be sent to respondents. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the following address: U.S. Department of Agriculture, Food and Nutrition Services, Office of Policy Support, 1320 Braddock Place, Alexandria, VA 22314, ATTN: PRA (0584-xxxx*). Do not return the completed form to this address.

1. Recruitment and Data Collection Methods

FNS sent an email on February 6 to the Executive Director of the Family Investment Administration within Maryland’s Department of Human Services (DHS) to request the State’s participation in the pretest. The State confirmed its willingness to participate, after which Insight’s qualitative task lead followed up with the Director of Workforce Development via email and provided additional information by telephone about the instruments we planned to pretest.

After securing Maryland’s participation, the task lead scheduled an in-person visit to the DHS office to pretest the SNAP E&T director interview, the local office director interview, the process-mapping protocol, and the administrative cost worksheet. The Director informed us that Maryland’s local Department of Social Service (LDSS) offices had limited knowledge and involvement with SNAP E&T and would have difficulty responding to our questions. She noted that LDSS offices often had not heard of SNAP E&T and that it was unknown to what extent they had contracted with SNAP E&T providers.¹ She thought it would be best to pretest the local office protocol with a State staff member responsible for overseeing the providers DHS recently contracted with to provide SNAP E&T because that was the model Maryland was moving toward.

We asked the director to connect us with a SNAP E&T provider to pretest the E&T provider protocol. The State selected a provider with whom it had a good relationship, and we held a brief call with the chosen provider to introduce the study and describe the purpose of the pretest. We also asked the provider to connect us with three SNAP E&T job search participants to test the E&T participant protocol.

The provider facilitated introductions to one current job search participant, one former participant who had found employment, and one former participant who was planning to reengage with SNAP E&T. Insight conducted all three interviews by telephone and sent a $30 gift card to each participant following the completion of the interview as a token of appreciation.

Each interview was expected to take 30 to 90 minutes, but we allotted an additional 15–30 minutes to allow time for follow-up questions and feedback on the instruments. A research analyst took detailed notes during each interview and noted which questions respondents had difficulty answering or needed additional clarification to answer.

2. Overview of Pretest Findings

Maryland operates a voluntary SNAP E&T program with a partial waiver for able-bodied adults without dependents (ABAWDs). The State’s program was undergoing considerable change when we visited. The State was moving to a third-party provider system and had a different intake process for ABAWDs versus non-ABAWDs. ABAWDs would be informed about the SNAP E&T requirements and sent to an LDSS office where they could meet their work requirements through the office’s contracted vendor, which was not necessarily a SNAP E&T provider. Non-ABAWDs, on the other hand, often learned about SNAP E&T on their own and received services through third-party providers. These two different intake streams complicated some of the discussions and process mapping because we needed to probe to ascertain how the intake and job-search processes differed for ABAWDs versus non-ABAWDs.

All the respondents were relatively new to SNAP E&T. State staff had recently transitioned into roles overseeing Maryland’s SNAP E&T program and were working to expand the knowledge and presence of SNAP E&T within the State. Part of this effort included partnering with new providers. The provider we spoke with had operated as a SNAP E&T provider for less than 6 months. Because the State and provider staff were new to SNAP E&T, there were several questions they were unable to address. For the site visits, it will be important to the extent possible to target local offices and providers that are experienced in operating SNAP E&T.

The voluntary SNAP E&T participants we spoke with were enrolled in the program for a short duration, which may be common among study States operating voluntary E&T programs depending on the types of services offered. Interviewers may need to focus on helping voluntary participants recall the E&T components they enrolled in and probe to elicit details about each component. We added probes to the E&T participant protocol to help interviewers with this task.

The respondents understood most questions and felt they were clearly worded. However, one recurring challenge we encountered was confusion about how we defined “job search.” In every interview, we found it necessary to clarify what activities “job search” entailed. It will be important to define job search at the beginning of each interview to ensure respondents discuss both job search and job search training and limit discussion about other components. We have added clarifying language to the interview and process-mapping protocols to help interviewers focus the discussion on the components of interest.

On the whole, the protocols flowed well and stayed within the allotted time. We made minor edits to the organization of the protocols when questions were addressed under a different section than the one they originally appeared in. The E&T provider protocol ran slightly longer than expected, so we deleted questions from it that did not directly pertain to the study objectives.

3. Findings and Recommendations by Data Collection Instrument

This section summarizes the pretest results for each data collection instrument and the associated recommendations for improvements.

1. State SNAP Director and E&T Director Interview Protocol

Insight interviewed Maryland’s Director of Workforce Development on March 22.

1. Duration

The 90-minute interview, which included a review of the administrative cost worksheet, took 85 minutes to complete. No adjustments are needed to the number of questions, and interviewers should have substantial time to probe for depth throughout the interview.

2. General findings

The respondent felt the questions were clearly worded. There were a few questions she was unable to answer as a result of her lack of time in the position, but we have retained these questions in the protocol because other E&T directors may be better equipped to answer them.

3. Question-by-question findings and recommendations

Table 1 provides our findings and recommendations for specific questions. The revised State SNAP director and E&T director interview protocol, including the suggested changes mentioned in table 1, is presented in appendix A.²

Table 1. Item-Level Recommendations for the State SNAP Director and E&T Director Interview Protocol

Question Number From Draft Instrument	Findings/Observations	Recommendations
A. Introduction	Additional clarification is needed about what activities we are referring to when we talk about “job search.”	In the introduction, refer to both “job search” and “job search training” initially and “activities” thereafter. Add a question to section B clarifying the specific activities we plan to discuss, thereby setting the stage for the rest of the interview.
B.1. How would you describe the State economy? What jobs are in demand? What are the large industries in [State]?	This question proved somewhat difficult to answer, and the respondent focused mostly on the economy in Baltimore. It also took a long time to obtain an accurate response.	Delete this question because it overlaps with B.2, which directly addresses a research question. Deleting B.1 should also allow more time for probes.
C.3. According to your State plan, [fill in percentage of participants from State plan] percent of clients are expected to participate in the following job search activities in [year]: [fill in job search activities from State plan]. Is this correct? How do you estimate participation in each activity?	This question did not yield helpful information and did not directly address any research questions.	Delete this question to allow time for more useful probes.
C.4. According to your State E&T plan, your State uses the following providers for job search activities [list providers from State E&T plan]. Is this correct? [Gather any information the State agency has describing the various providers, such as participant brochures or program descriptions.] How are these providers selected and vetted?	The parent question does not address any research questions or provide helpful information about providers. It would be more productive to replace this question.	Replace the parent question with the subquestion, “How does your State select and vet providers for job search?”
C.6. [If not already addressed:] How, if at all, has your State’s approach to job search changed as a result of the 2018 Farm Bill?	This question did not flow well here.	Move this question to section D.
D.1. According to your State plan, [State] offers the following job search activities and [fill in information from State plan]. Is that still correct? Are other job search activities available to E&T participants that I did not mention? Are any of the activities I mentioned no longer available?	This question comes too late in the protocol given the lack of clarity about what is included under “job search.”	Move this question to the beginning of section B.
F.4. We discussed engagement, but I would also like to know about participant retention. How does retention in job search activities compare with other components? What are the reasons for this variation? How does retention vary based on job search activity? What activities have the highest engagement? The lowest? What are the reasons for this variation across activity?	The parent question draws a comparison between job search and other components when the focus should be on different job search activities. There is also redundancy in the subquestions.	Rephrase the parent question to ask how retention varies between each job search activity. Drop the subquestions, “How does retention vary based on job search activity?” and “What are the reasons for this variation across activity?”

1. Process-Mapping Protocol

To pretest the process-mapping protocol, the Insight team met on March 22 with the Director of Workforce Development and the Workforce Development Coordinator. Insight’s moderator outlined Maryland’s E&T intake process on a whiteboard and asked the respondents to describe each step in the process, including what happens, who is involved, the time required to complete each step, and communications between stakeholders.

1. Duration

The pretest took approximately 92 minutes to complete. No adjustments are needed to the length of the protocol.

2. General findings

The process mapping started with a general overview of each step in the intake process: screening, notification, intake, assessment, E&T participation, and reporting. After obtaining a basic summary of Maryland’s intake process, we asked pointed questions about each step in the process. The respondents indicated that the questions were clear. They appreciated that we had reviewed Maryland’s SNAP E&T plan in advance and that we were prepared to ask questions specific to their program.

Because ABAWDs are directed to LDSS offices, and non-ABAWDs must connect with a third-party provider, part of the intake process differed for each group. We explored the flows of both types of participants, which may have extended the protocol slightly. We would expect the protocol to take less time in voluntary States with a unified process for ABAWDs and non-ABAWDs, and more time in States where the intake process diverges for different populations. In these latter States, we will aim to compress the protocol as needed to ensure we can map each intake process within 90 minutes.

Because of the limited awareness of and involvement in SNAP E&T at the LDSS offices, we conducted the process mapping with State staff. Such flexibility may be necessary in the study States if we learn that local offices have limited involvement. We will identify where the process mapping should take place and whether there are multiple participant pathways (e.g., for ABAWDs versus non-ABAWDs) during our site visit planning discussions with States.

3. Question-by-question findings and recommendations

Table 2 provides our findings and recommendations for specific questions. The revised process-mapping protocol, including the suggested changes from table 2, is presented in appendix B.³

Table 2. Item-Level Recommendations for the Process-Mapping Protocol

Question Number From Draft Instrument	Findings/Observations	Recommendations
A. Introduction	Additional clarification is needed about what activities we are referring to when we talk about “job search.”	Add a note for the moderator to specify the activities included under the office’s job search and job search training components before beginning the process mapping.
C.1. (Voluntary States) How do clients learn about E&T services and activities?	In Maryland, ABAWDs and non-ABAWDs learned about SNAP E&T in different ways.	Add a question about how notification differs for ABAWDs and non-ABAWDs.
D.3.b. [Probe] Do you create individualized/ employment plans? Does every client have one of these plans? Who works with clients to develop their plans?	This probe would fit better under D.1 and could be rephrased to be more open ended.	Rephrase and move this probe to D.1.
D.6. Do clients who have recertified stay with the same provider?	This question does not address any research questions.	Delete this question to save time.
E. E&T Participation and Reporting	Participation and reporting questions address different steps in the intake process and can function better as separate sections.	Break questions about reporting into their own section.
F.1.e. [Probe] Do clients ever drop out of one SNAP E&T activity but remain in another activity? If so, what components do they drop out of? Which do they stay in?	This probe was difficult for respondents to answer and did not provide useful information.	Delete this probe because it does not directly address any research questions.
G.1. Looking back at all the steps we discussed, I want you to think about the level of dropoff at each one. Using a scale of low/medium/high, how would you rate the dropoff at each step?	This question is redundant with question F.2.	Delete this question to save time.
G.2. Can you think of anything that could be done to reduce the likelihood of dropoff at these points?	This question does not address any research questions and puts respondents on the spot.	Delete this question to save time.

3. Local SNAP Office Director and Staff Interview Protocol

To pretest the local SNAP office director and staff interview protocol, we interviewed the State’s Workforce Development Coordinator on March 22. In Maryland, LDSS offices have a limited role in the SNAP E&T program. The Workforce Development Coordinator works closely with the State E&T Director to monitor and support third-party providers and, thus, was the best available substitute to pretest the local office protocol.

1. Duration

The interview took 60 minutes to complete. No changes to the length of the protocol are needed.

2. General findings

The respondent was able to answer most of the questions in the protocol; therefore, few changes are recommended. The respondent has been in her role since December 2018, and the questions she was unable to answer were often related to her lack of time in the position. The respondent was also unable to answer some questions that were not relevant to her position at the State office. We believe local office staff will be able to answer these questions, so we have retained them in the protocol.

We added new questions to the protocol to better capture variation in E&T programs across States. For instance, ABAWDs in Maryland learn about SNAP E&T services in a different way than other participants, and clients are often asked to complete Individualized Employment Plans (IEPs).

3. Question-by-question findings and recommendations

Table 3 provides our findings and recommendations for specific questions. The revised local SNAP office director and staff interview protocol, including suggested changes from table 3, is presented in appendix C.⁴

Table 3. Item-Level Recommendations for the Local SNAP Office Director and Staff Interview Protocol

Question Number From Draft Instrument	Findings/Observations	Recommendations
How would you describe the local economy? What jobs are in demand? What are the large industries in the area? Is it hard for clients to find jobs these days, or are there lots of available jobs?	Questions about the local economy took respondents a long time to answer and are not directly relevant to the research questions.	Delete this question to save time.
B.2. How well do current job search activities match up with the types of available jobs? If you could, what would you change about the job search activities currently offered to better prepare participants to find work?	Questions about in-demand jobs took respondents a long time to answer and are not directly relevant to the research questions.	Delete this question to save time.
B.3. What is the approximate size of your local SNAP E&T population? Of those, how many participate in job search activities? Are any groups specifically targeted for job search activities? If so, which ones? What are the reasons they are targeted? Are any groups required to participate in job search activities? If so, which ones? What are the reasons they are required to participate?	We learned it is important to define job search at the outset of the interview to ensure respondents understand what job-search activities include.	Add a note to the moderator to define job search activities if the respondent did not participate in the process mapping.
C.1. Please describe what job search activities are offered. [Probe for: independent job search, resource centers, group job search, mock interviews, resume workshops] [For frontline staff ONLY, ask the following subquestions for each type of job search activity mentioned.] How long does this activity last? What is the intensity of this activity? Is it heavy or light touch? How long do clients typically stay in this activity?	We learned activities typically do not have a set start or end date, making it difficult to answer C.1.a.	Delete C.1.a and lead with C.1.c.
C.3. [If a voluntary State] How do clients learn about the E&T activities available in your State? Do they find out during screening/certification, or do they know about the programs before they apply for SNAP? [If clients know about E&T] Which activities do they know about? How did they learn of them?	This question would flow better under section D. It would be beneficial to include a question about the differences in how ABAWDs and non-ABAWDs learn about and enter SNAP E&T.	Move this question to D.1.b and include a question about the differences in how ABAWDs and non-ABAWDs learn about SNAP E&T services.
D.1.c.iii. What types of assessments do clients complete during intake? D.1.c.vii. What criteria are used to assign clients to job search?	We learned from our respondent that E&T participants complete IEPs with providers.	Ask about IEPs in each of these subquestions.

4. E&T Provider Interview Protocol

To pretest the E&T provider protocol, we interviewed the Regional Director and Program Manager at America Works of Maryland on March 26. The provider has operated as a SNAP E&T provider for less than 6 months, and the respondents had difficulty answering some questions because of their lack of experience with SNAP E&T.

1. Duration

The interview lasted 67 minutes. Whereas the presence of two staff may have lengthened the interview, the inexperience of the provider with SNAP E&T likely shortened it. We suggest eliminating several questions from the protocol to ensure there is sufficient time to cover all topics.

2. General findings

The provider struggled to understand how we defined job search activities and interpreted this term as assistance looking for work. As with the other protocols, we suggest adding language to clarify which activities we are referring to under the job search and job search training components. We also plan to use information in a State’s SNAP E&T plan to specify the activities we hope to discuss with providers when emailing them about the site visits.

Questions in the protocol were clearly worded; therefore, few changes are needed. Respondents had difficulty answering a few questions largely because of their lack of experience with SNAP E&T.

3. Question-by-question findings and recommendations

Table 4 provides our findings and recommendations for specific questions. The revised E&T provider protocol, including suggested changes from table 4, is presented in appendix D.⁵

Table 4. Item-Level Recommendations for the E&T Provider Interview Protocol

Question Number From Draft Instrument	Findings/Observations	Recommendations
A.2. How long have you been in this position?	This question provides contextual information that is not necessary for the study.	Delete this question to save time.
B.1. Please describe your organization and mission. What populations does your organization serve? Approximately what proportion of your clients are SNAP E&T participants? What other programs/participants do you serve?	The provider spent a lot of time providing background on the organization, which did not address any research questions.	Replace the parent question with B.1.a.
B.2. Please describe your organization’s role in SNAP job search activities. How long has your organization been working with SNAP E&T? Has this relationship changed over time? If yes, how?	The parent question was too open ended and led to a lengthy discussion of SNAP E&T that would be better to ease into.	Delete the parent question and incorporate the subquestions into B.1.
B.4. How would you describe the State and local economy? What are the in-demand jobs in the area? How does your program keep abreast of local labor market conditions? How do you use this information to shape the job search activities offered?	This question required a lot of time for respondents to answer. They had difficulty answering questions about the State economy and struggled to answer B.4.b and B.4.c.	Delete this question to save time.
C.1. What services does your program offer? Please describe. [Probe for: list State job search activities from State E&T plan.] Is there anything else I missed?	This question needs to clarify that we are interested in both job search and job search training activities.	Rephrase this question to stress our interest in both job search and job search training activities. Throughout the protocol, refer to specific activities under these components to hone the discussion.
C.2. What geographic area does your program serve? Do you offer the same services in each area?	This question would flow better under section B.	Move this question under question B.1.
5. Are job search services integrated with other program resources and/or activities? How? Roughly what proportion of SNAP job search participants in your program participate in other types of activities you offer? Which other activities are most common? Roughly what proportion of SNAP job search participants in your program are dually enrolled in another program (for example, WIOA)? How did your organization decide to integrate these services? What was the motivation behind the decision?	We had to clarify this question because the respondents were uncertain as to what other activities or programs we were referring.	Rephrase this question to offer examples of other activities and programs in which SNAP E&T participants may engage.
6. Do you know how many other employment and training providers there are in the area? Do the services they provide overlap with yours? [Probe for whether these providers also provide similar job search services.] Do you refer the clients you serve to any of these other organizations to provide additional E&T services you are not able to provide (co-enrollment)? Do you ever refer clients to these other organizations for their primary services because your organization is not a good fit for some SNAP E&T participants? How frequently?	This question was difficult for respondents to answer and does not directly address any research questions.	Delete this question to save time.
D.1. Please describe the flow of SNAP clients from referral to completing the program	We did not ask this question because we did not interview a frontline staff member. However, the State staff noted that providers create IEPs and use them to tailor services to each client. It would be helpful to know how IEPs are used by those who create them.	Add a question asking frontline staff to describe the IEPs and how they use them.
E.1. What types of participation and outcome data do you track for job search participants? For instance, do you track…	The provider had some difficulty understanding to what data we were referring.	Rephrase to specify that we are interested in what participation and outcome data they track for job search participants. Add a question to distinguish which data are shared with the State versus used for internal purposes.

5. E&T Participant Interview Protocol

Insight interviewed three SNAP E&T participants: one former participant on March 14, one current participant on March 18, and one former participant on March 26. All three respondents receive or have received services from America Works of Maryland.

1. Duration

Each of the three interviews took 30 minutes or less to complete. We did not shorten the interview protocols but did add some additional probes as described in the sections that follow to elicit richer data from the participants.

2. General findings

All the respondents spent only a short amount of time in job search activities, ranging from several days to 2 weeks. As a result, some questions did not yield much descriptive detail or insight. These included questions asking what the participant liked best about participating in the activity and how that had changed, challenges the participant faced, and questions about their expectations for the activities and services they would receive. This issue may be a challenge in other voluntary States where job search is more likely to be a short-term endeavor compared with States where E&T participation is mandatory. We added several questions and probes to gather deeper insight from respondents who have limited engagement with E&T.

Section A required substantial probing to identify the job search activities participants had engaged in, which naturally flowed into a discussion about those activities. Additional probes will be necessary to ensure that respondents list every SNAP E&T job search activity in which they have participated. Some activities, such as resume review, can be of short duration, and participants may not immediately remember they received this service. Because this probing yielded information that addressed questions that appeared later in the protocol, interviewers will need a good understanding of the protocol to know which questions to skip after section A.

Some of the pretest respondents were confused when we referred to their participation as “job search” or “job search activities.” They were better able to answer our questions when we specifically referred to the activities they took part in, such as mock interviews or job placement.

3. Question-by-question findings and recommendations

Table 5 provides our findings and recommendations for specific questions. The revised participant interview protocol, including suggested changes from table 5, is presented in appendix E.⁶

Table 5. Item-Level Recommendations for the E&T Participant Interview Protocol

Question Number From Draft Instrument	Findings/Observations	Recommendations
A.1. I want to learn a little more about you and your experience with [SNAP E&T program name]. Information we received from [SNAP agency] says you participated in a job search activity through [the SNAP Employment and Training program/or <insert local program name>]. By job search activity, I mean [describe job search activities offered by State and listed in State plans (e.g. assistance in writing resumes and cover letters, financial literacy classes, or career assessments)].	It is helpful in some of the questions to refer back to the name of the E&T provider. This information will not be available in the SNAP administrative data.	Add a subquestion to ask about the respondent’s E&T provider.
A.2. Please tell me a little about the job search activity in which you participate(d).	Respondents needed probing to list all the job search activities/services they received and/or in which they participated.	Add probes to this question.
A.3 Did you get to choose a job search activity, or were you told which one to do? [If given choice] What were your choices? How did you find out about each one? [Probe for: activities offered by State but not mentioned by participant] i. What made you choose the activity you were/are in? [If not given choice] Do you know of any other job search activities available to SNAP participants? [Probe for: specific activities offered by State]	One respondent was confused by the wording of this question and provided a response detailing how he was able to choose which jobs he wanted to apply to rather than choosing which services he was able to receive. It is possible that referring to “job search activities” rather than the specific activities the respondent participated in (job placement and resume assistance) was confusing to the respondent.	Revise this question to first mention the activities respondents participated in and then ask if other services were offered to them. We can then ask respondents if they were able to choose which services to participate in or if their providers chose. We have also removed references to “job search” or “job search activities” and inserted “[activity]” in the protocol to allow the interviewer to refer to the specific activities the respondent participated in, which will also lessen confusion.
A.4. What made you decide to participate in [activity]?	This question failed to elicit a detailed response from any of the respondents.	Add a probe to ask voluntary participants why they chose to work with their providers instead of look for work on their own.
A.5. Why do you think SNAP offers these services? [Probe for: helping participants find a job, gain skills]	Respondents had difficulty responding to this question. They were better able to respond when we reworded the question to ask why the provider offers services instead of SNAP.	Reword this question to ask about the respondent’s provider. [5/14/19 update: Per discussion with FNS, this question will be deleted.]
B.2. What were you hoping to get out of [activity]? Has that happened? If not, do you think it still may happen?	We learned from the E&T provider and State E&T director interviews that many providers complete IEPs with SNAP E&T participants that ask about their short- and long-term employment goals.	Revise this question to ask respondents if they remember completing such a plan with their caseworkers and then asking them about the short- and long-term employment goals discussed in the plan.
B.5. [Former only] About how long did you actually participate? Why did you stop participating? [Interviewer note: Look up closure code in SNAP administrative data to confirm.] Was there a planned end date for the activity? When you started job search were you given a specific end date? Or were you able to participate for as long as you wanted? i. [If as long as you wanted] How did you decide to stop participating?	Questions B.5.b. and B.5.c were repetitive, and the reference to “end date” did not resonate with respondents.	Consolidate and reword the two subquestions into one.
C.1. How satisfied are/were you with the job search activity? Did you get what you needed from it? [If yes] How so? [If no] Why not? What would you change about it if you could? [Probe for: activities, support services, and administrative changes] What other job search activities do you think should be available to SNAP participants?	In C.1.c, the term “job search activities” did not resonate with respondents. Some viewed everything they were doing to find a job as “job search activities,” not just the services they received from the E&T provider.	Alter the wording of the sentence to ask respondents if other services could be offered to meet their needs. Add a probe to ask about other activities that the State offers but in which the respondent did not enroll.
D.2. Are you aware of any support services available to job search participants, such as transportation assistance, or childcare, providing clothing for interviews, or [add other examples from State plan]? [If yes] Have you used any of these support services while participating in job search? i. [If yes] Which ones? ii. What kinds of support services would have been useful to you? [If no] What kinds of support services would you have used if you had known they were available to you?	During the pretest interview with the former participant, it was unclear whether the respondent understood that there were supports other than transportation assistance, which he received, and childcare, which did not apply to him because he did not have young children.	Add further examples of support services to the parent question to prompt respondents to think about what supports they have received and what support they did not receive but would have been useful to them.
E.1. What other E&T activities have/are you participated/ing in, other than [activity]? How did/does that activity compare to [activity]? Which activity was most helpful to you?	In the original question, it was not clear whether respondents were being asked about their most recent experiences with SNAP E&T or if they should also discuss their past experiences with SNAP E&T.	Reword this question to clarify that the respondent should also think about past experiences with SNAP E&T. Add a probe with example activities to prompt respondents.

6. Administrative Cost Worksheet Findings

This section presents the pretest results for the administrative cost worksheet. We will use this worksheet to collect information on the costs of administering job search services as part of the overall SNAP E&T program. The pretest was held on March 22 with Maryland’s Director of Workforce Development. We provided a copy of the cost worksheet in advance of our visit for the director to review.

1. Duration

The review of the administrative cost worksheet was part of the 90-minute State SNAP director and E&T director interview, which took 85 minutes to complete.

2. General findings

The respondent noted that most States should already have the majority of the information that we are requesting. In Maryland, much of the information would come from the State plan and the annual proposals and monthly invoices submitted by the third-party E&T providers. The respondent confirmed that Maryland would be able to separate costs by component and that it would be able to provide budgeted and actual costs for participant reimbursements by type of reimbursement because this information is provided on invoices from the third-party providers. The respondent noted that most reimbursements are for transportation expenses (e.g., bus passes). Administrative costs would include payments to third-party providers, trainings, and salaries for State staff. Other costs would include conference costs, marketing materials, and other materials.

We learned that Maryland does not track unduplicated monthly participation by month but receives that information through quarterly reports from the third-party providers. The respondent indicated that if need be, quarterly participation could be divided by three to achieve an approximate monthly participation that could be used for a cost estimate.

Because of Maryland’s unique SNAP E&T structure, local offices in some counties administer their own E&T programs that are not under the umbrella of the State’s E&T program. As such, although the respondent would be able to provide cost and participation data, both budgeted and actual, for the SNAP E&T program, she would not be able to do so for the E&T services administered by local offices. We believe this is a unique aspect of Maryland’s E&T program; should this circumstance be encountered in a study State, we will work with the State to understand the data that it is able to provide and caveat any cost information as needed.

The respondent noted that discrepancies between the State plan and FNS-778 form stemmed from the State not having reimbursed third-party providers in a timely manner in fiscal year 2018. We do not anticipate as large of a discrepancy between the FNS-778 costs and budgeted State plan costs in other States.

The respondent found the cost worksheet to be useful and requested a copy to use for the State’s own purposes in tracking program costs.

3. Specific recommendations to tables

Based on the feedback received, we recommend making the following revisions to the administrative cost worksheet. The revised document is presented in appendix F.⁷

1. In table A, we have explicitly indicated that administrative costs can include both 100 percent and 50/50 funds.

2. In the other costs section of table A, we have indicated that State staff can attach documentation of other costs rather than describing them in the text box. The pretest respondent indicated this may be easier for staff to do than to write out the other costs in the text box.

1 There was turnover within the administration of DHS’s SNAP E&T program, and the new staff were still gathering information about the current state of SNAP E&T in Maryland.

2 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.

3 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.

4 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.

5 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.

6 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.

7 5/14/19 Update: Revised instruments are included as separate attachments to the OMB package and have been removed from appendices to this memorandum for version control and to avoid redundancy.