P.2. FNS Response to NASS Comments

Appendix P.2 FNS Response to National Agricultural Statistics Service Comments

OMB No. 0584-[NEW]

Assessment of Mandatory E&T Programs

Month XX, 2018

Project Officer: Jordan Younes

Office of Policy Support

Food and Nutrition Service

U.S. Department of Agriculture

3101 Park Center Drive

Alexandria, VA 22303

703-305-2935

[email protected]

FNS Response to NASS Comments

NASS Comment	FNS Response
The process is fairly well-defined but a graphic up-front that summarizes the persons contacted and data collected would be very helpful. This study is very complex.	The study team has added a chart (in appendix B.2) that graphically shows the data sources that will be used to address each research objective and question.
The data analysis is not explained in detail for this program. However, there may be concerns that the extant data collection will not provide adequate data for assessment of E&T outcomes. I recommend providing more detail in regards to the analysis to better assess the data needed.	The study team added more detail on the analysis that will be conducted using administrative data (see page 7 and 8 of part A) to better understand the data needed. The study team is confident that the combination of site visit data and extant data sources are sufficient to adequately assess E&T program outcomes.
I recommend providing more detail in regards to the process of selection. Moreover, while the expense of incorporating data from all states may be a limiting factor, the differences in programs and practices might be better observed by incorporating more states into the evaluation set.	The study team discusses State selection in Part B of the Supporting Statement (section B.1). Due to the scope of the contract and available resources, more States cannot be added to the study.
Again an algorithm was not provided to solidify an understanding of the selection process. I recommend clearly defining the selection process that ensures the selected local offices represent: 1) rural and urban areas, 2) mandatory programs, and 3) a mix of State, for profit and not-for- profit providers.	The study team discusses State selection in Part B of the Supporting Statement (see section B.1)
I recommend compiling data from a broader base of providers.	Due to the scope of the contract and available resources, a larger number of providers cannot be added to the study.
I recommend revisiting the comments from the pre-test and evaluating avenues to ensure that as much data as possible can be collected from interviews, process mapping and data collection.	The study team will ensure complete quantitative and qualitative data collection.
I would also recommend having a strategy in place to coordinate data from all sources, i.e. what identifiers are needed to ensure that State data and E&T provider data can be merged.	The study team has planned advance calls with State agencies to discuss data requirements and needs, including the availability of E&T provider data and identifiers needed to merge data.
I recommend supplementing the topic list with a general understanding of the depth of the questions. One way to do this is to ask for pre- collected information. Each State program is different and some of these questions could elicit very lengthy answers. For example, question F.1.c. Do you produce other reports for internal program management purposes? Followed by: what kind of information is in these reports? It may be beneficial to ask for a copy of the internal report, if produced. While documents are requested prior to starting the interview process, the request is fairly open and document collection may vary. Asking for copies of interview question supporting materials will assist the State Director in identifying the types of questions that may be asked, such as the key analysis of their programs and content of their evaluations.	The study team will provide a list of topics to interview participants prior to the site visits. The study team will also request all available documentation prior to the site visit to aid in data extraction.
Interviewees answered select sets of questions. While the interview may have lasted only an hour, if all questions were answered, the time may have exceeded an hour. I recommend increasing the time.	The study team does not believe that the interview time needs to be increased. Based on pre-test findings, questions were deleted from protocols where the interview went over time. Additionally, sending a list of topics in advance and extracting any extant data prior to the interview will enable the study team to keep to the current estimates of interview length.
Process questions, policy questions, examples and observations are appropriate interview questions. Questions regarding percentages, quantities, frequency, etc. are typically only requested in formats that allow for earlier or concurrent retrieval of the data. Given the quantity of questions and limited time frame, I recommend either providing the questions in advance of the interview or requesting follow-up.	The study team will send a list of topics to interviewees prior to the site visits to ensure they can adequately prepare for the discussion. With the semi-structured interview methodology, we do not typically provide the full interview protocol as interviewees tend to prepare written responses that are limiting in the flow of the discussion and limit interviewer ability to probe certain topics.
The program evaluated by the pre-test had a very narrow focus with only one provider. If there is more than one provider or venue for a client the answers may be complex. I recommend reviewing the wording or asking the set of questions to address each the services by each provider separately to avoid confusion. For example, “E.4.a. What supports do they most commonly need?” may have a different response depending on if it is a State provider versus a for profit provider. Covering more complex programs will likely require more interview time. I recommend increasing the allotted time.	The study team will make sure to differentiate between providers during all interviews. The study team does not believe that the interview time needs to be increased. Based on pre-test findings, questions were deleted from protocols where the interview went over time. Additionally, sending a list of interview topics in advance and extracting any extant data prior to the interview will enable the study team to keep to the current interview times.
If topics are distributed pre-interview, there may be more interviewees attending to address the topics. While this is a good opportunity to collect all the information, the response burden is only accounting for one interviewee. I recommend making an adjustment in good faith to account for the potential increased burden.	The study team has increased the burden time associated with preparing for the interview and reviewing the interview topics by 30 minutes. The study team does not believe that additional individuals will attend the interviews.
I recommend changing the placement of some questions from one interview to another. For example, “G.9. Are there any parts of NOAAs that clients seem to have difficulty understanding?” might better be answered by the eligibility officer that works directly with the clients. Whereas the Local SNAP Office Eligibility Worker Interview Protocol question “E.5. Approximately how many (what percentage) of your clients who receive a NOAA comply with the requirements in time to avoid being sanctioned?” might better be answered by the local office director. My recommended list of changes in question placement is in appendix A.	The study team added question G.9 in the local director protocol to question E.3.a in the eligibility worker protocol as suggested. Question E.5 in the eligibility worker protocol provides a sense of how frequently the eligibility worker encounters clients who receive an NOAA and comply or return to SNAP after being sanctioned. These data will not be extrapolated to other eligibility workers. The study team will also calculate this information for the State as a whole from the extant administrative data.
I recommend sending the topics in advance and enough description of the topics to ensure the director has the staff support available to answer the more detailed questions. If the office is large there may be many managers of specific areas involved.	The study team will be sending a list of interview topics prior to the site visit.
In section E there are a number of questions asking percentages. These questions may be phrased as “Based on your experience, what percentage….” or “Approximately how many (what percentage)….”. Questions based on personal experience may be influenced by divisions within an office, whether it is geographic, type of client, etc. I recommend cautiously using the information. More global questions may be more difficult for an eligibility worker to obtain information. I recommend the questions be queries to managers, directors, or program sponsors.	The study team understands that all qualitative data is subjective and cannot be seen as representative of all eligibility workers in the State. The questions are intended to provide a general idea of how frequently an individual eligibility worker encounters clients who receive NOAAs or are sanctioned. We anticipate these will be rough estimates and will allow the study team to better understand the individual eligibility worker’s perspective.
I recommend sending the topics in advance and enough description of the topics to ensure the appropriate staff are available to answer the questions. If the office is large there may be specialization and some sections may not be addressed.	The study team will send a list of interview topics prior to the site visit.
The testing was limited to a very narrow band of services and to a State provider only. I recommend extending pretesting to evaluate the questions using a for- profit organization and a not-for-profit organization to thoroughly vet the range of answers. For example, questions about capacity may yield differing responses.	Due to the scope of the contract and limitations on data collection prior to OMB approval, the study team is not able to pre-test the instruments with a larger number of providers.
As noted any data requirements should be sent in advance, as these types of questions are difficult to answer without preparation. I recommend sending these questions in advance or allowing for interviewee follow-up to these questions.	The study team will send a list of topics to interviewees prior to the site visits. With the semi-structured interview methodology, we do not typically provide the full interview protocol as interviewees tend to prepare written responses that are limiting in the flow of the discussion and limit interviewer ability to probe certain topics.
If multiple providers are available in a local area, there may be some questions that could naturally not be answered. For example, a “new” referral may be confusing as the provider may only know if the referral is “new” to the specific provider. I recommend restating questions as “for your organization”.	The questions for E&T providers are directed to the specific provider being interviewed. We believe it is clear that the questions in this protocol are for the E&T provider’s organization and not E&T providers in general.
However, no materials are provided. The stated purpose of the exercise is to help identify each step in the client pathway and drop-off points in the process that may lead to sanctions. I recommend clarifying the task and its relationship to other types of data collection. The in-take observation checklist may benefit by having more detail around the process- mapping as well.	The study team added a process-mapping protocol to provide more guidance on the exercise (see appendix H).
For E&T activities the ET_EARNINGS variable needs to be explicit as monthly or quarterly earnings or the value cannot be compared.	The study team will standardize all income variables during data cleaning. The data codes will identify if the reported income is monthly or quarterly.
It is not clear that not-for-profit or for-profit organizations will track participants past the point of payment for specified service, which may be the start date of unsubsidized employment. In that case, the earnings would be unclear for following months.	The study team notes this may be a study limitation.
The ET_EARNINGS is unclear as to whether it represents all income earned by the individual or only income from the employment received via the E&T program. In some states eligibility includes SNAP participants that have a part time job but do not have the requisite number of work hours.	The study team will clarify how the States define this variable in the consultative discussion about the data request that will be held with each State.
One tabulation that you may want to do is based on whether the provider is State, not-for-profit, or for-profit. While the provider name may assist in developing that information an indicator variable would be preferable.	The study team will keep this in mind when completing the quantitative analysis. Information on the type of E&T provider and how it is organized will be obtained from the qualitative interviews.
There is no indication of total hours worked per week or hours worked per week for E&T employment. I would include these variables for perspective.	We will assess whether States can provide this information in the consultative discussions to be held with States. We will add these variables if available from the States.
I would include State as a variable.	The study team will add State identifiers during data cleaning.
Local offices may serve clients that go to various service providers (for-profit, not-for-profit, etc.) based on geography or other considerations. My understanding is that we may only have outcomes for a limited number of recipients in that case.	This is dependent on the E&T outcomes data that the study team will receive from the State agency/E&T providers.
I recommend acquiring data from at least one not- for-profit or for-profit entity to ensure that data from alternative sources can be joined to provide a complete perspective on the client case. In addition, there may be no need to have data from outside sources if all variables can be obtained from the State database. If states require providers to report information to the SNAP or SNAP E&T program, there may be limited need to request this data from E&T providers	The study team will work with State agencies to determine the best method to acquire SNAP E&T outcome data. If data are available at the State level, the study team will not need to reach out to the E&T providers. If it is necessary to request data from the E&T providers, the study team will work with the State agency to coordinate reaching out to the providers for the data request.
I recommend a thorough discussion of what data are available from the State, the source of their data, and their established criteria for data transfer files or documented descriptions of what data need to be reported prior to requesting data from providers. It may be challenging to acquire meaningful data with more diversified programs than the pilot program.	The study team will schedule consultative discussions in Fall 2018 to discuss all data requirements with State policy and data/IT staff.
I recommend accounting for higher levels of complexity that may hinder meeting reporting requirements and lower response rates.	The study team has a recommended backup State to recruit if the primary State declines to participate in the study. The study team will also work with FNS to engage the States’ respective Regional Offices in encouraging their participation in the study; the Regional Offices were sent a copy of the recruitment letter to the States.
There is no information on the type of results that will be reported, tabulated or assessed. I recommend providing a short description.	The study team added information to Part A on the type of results that will be tabulated and assessed (see pages 7 and 8 of part A.)
Local Director Survey Question E.1.a. might be queried at a State Director Survey level as providers may register at a State level (see questions C.1.b. and C.1.c. on State Director survey).	Generally speaking, we expect the local offices to address questions about the specific providers in their counties. The questions referenced in the State director protocol refer to E&T components instead of providers.
Wording differences may have been intentional or not. Please be aware that on the Eligibility Worker survey question G.1. states “practices for implementing mandatory E&T policies” relative to question C.1. on the Local Director Survey which state “implementing mandatory E&T policies.” This may create slightly different answers.	We deleted “practices for” from the eligibility worker protocol to ensure consistency.
Questions D.1.a. and D.2. on the State Director Survey seem more appropriate for the Eligibility Worker Survey.	A similar question (E.4) exists in the eligibility worker protocol. We expect the State Director to offer a macro-level view of noncompliance across the State.
Questions 5 and 8 on the Eligibility Worker Survey might be more applicable for the State Director Survey (Nested in Questions D.6. and D.7.) and the Local Director Survey. Summary statistics may be made available to them.	These questions are intended to get a sense of how frequently the eligibility worker encounters clients who receive an NOAA and comply or return to SNAP after being sanctioned. These data will not be extrapolated to other eligibility workers.
On the Eligibility Worker Survey Conciliation and Good Cause Determination are in their own separate section. On the Local Director Survey it is placed in the context of other Compliance and Sanctions Section. It seems like this may be made consistent.	The eligibility worker protocol is broken out into discrete components to better describe the intake process, which will also be covered in the process mapping exercise. Descriptions of the conciliation process and good cause determination will be triangulated across all interviews in the State.
In the State Director Survey question 6 is a little typo - “does the State track of”. Perhaps there is a missing word or two in this question.	This typo has been fixed.
In the Local Director Survey Question G.3. might either be asked of the providers themselves or rephrased as what requirements or agreements are in place. Some local directors may not be willing to speak to the detail of other organizations or may not know.	Section F of the provider protocol asks about tracking E&T participation and noncompliance.