State Agency

ATTACHMENT H

MASTER SITE VISIT PROTOCOL

OMB Control No.: 0584-xxxx

Expiration Date: 00/00/20XX

Implementation site visits: Master Protocol

Notes to site visitors

The following table is a master protocol of questions for all types of respondents that will be interviewed over multiple rounds of site visits. The table identifies generally the type of respondent that will be asked each question, but site visitors will identify the roles of each actual respondent prior to the site visit and determine which of the following questions apply (and should be asked) of each. No respondent will be asked all of these questions. In addition, any information that may have been gleaned from conversations with the grantees and partners during technical assistance should be documented and confirmed during the site visits. The questions in this master protocol apply to all pilot projects, unless otherwise noted. However, prior to the visit, each site visitor should tailor the protocol to the site to reduce burden and redundancy.

We will conduct three rounds of site visits using the questions included in Instrument ATH below. The first round will focus on the planning phase and early implementation of the pilot. The second round will examine pilot operations during its full implementation period. The third round will focus on the close-out of the pilot and discuss overall impressions, lessons learned, and challenges encountered. We will tailor the questions in the protocol by round to ask respondents about events as they happen (or to reflect on what previously happened since the prior visit) and to determine how their opinions or approaches changed over time.

Introduction

My name is [X], and I work for [NAME OF FIRM]. As you might know, we are evaluating the Supplemental Nutrition Assistance Program (SNAP) Employment and Training (E&T) Pilot Program for the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture. The SNAP E&T Pilot Program provides grants for 10 pilot sites to develop innovative SNAP E&T strategies to increase employment and earnings for SNAP work registrants, and reduce their reliance on public assistance. For the purposes of this study, we are interested in understanding how SNAP E&T pilot services are implemented and how differences in implementation affect participants’ employment, earnings, and related outcomes.

My colleagues and I are currently visiting pilot sites across all 10 study states to collect information from a wide range of stakeholders involved in the pilot. We are learning about the implementation of the pilot programs at three different stages, as well as the challenges you encountered and lessons learned during the implementation and operation of the pilot. I want to start by thanking you for taking time to speak with us today. Your perspective and insights on these issues will be very helpful to the study.

Public Burden Statement

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-xxxx. The time required to complete this information collection is estimated to average 60 minutes including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate to the Office of Policy Support, Food and Nutrition Service, USDA, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302.

I want to let you know that your participation in this interview is voluntary, and your responses will be kept private to the extent provided by law. We will not share the information you provide with any other people, including your supervisor and state-level staff. You may refuse to answer any question and may stop the interview at any time. There will not be any penalities if you refuse to participate in part or full.

We will take notes over the course of the interview so that we can remember the information we collect. We will use this information in our reports to FNS, describing the range of responses expressed by staff. The reports might list the names of organizations that contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

I expect our conversation will take about 60 minutes. First, do you have any questions for me about the project in general or what we will be discussing today?

Instrument ATH. Topics, questions, and respondents for implementation interviews

Topics	Respondent				Round
	Grantee- or state-level staff	SNAP local office staff	E&T provider staff	Community stakeholders and other partners	R1	R2	R3
Background
What is your official job title or position?							
What services does your agency or organization provide?							
What are your primary responsibilities?							
How long have you been in this position?							
What are your responsibilities related to the pilot?							
Motivation and goals
Why did [state or organization] apply for or choose to participate in the pilot? What were the perceived opportunities? What were the perceived challenges?					
Who within the state drove the effort to apply for the pilot?					
Was there widespread support of the application? If not, who had concerns about the application and why?					
What are your site-specific goals for the SNAP E&T pilot?							
What outcomes will you use to measure progress toward the pilot’s goals? How did you identify these outcomes? How are you measuring outcomes? How did you select your performance targets?							
Why was the area covered by the pilot selected? Were other areas considered?					
Why was the particular target population selected? Were other target populations considered? If so, why were they ruled out?					
Why were the particular types of services/interventions chosen? Were other types of services or interventions considered? If they were considered, why were they ruled out?					
What benefits do you expect will accrue to the government as a result of the SNAP E&T pilot program? What benefits do you expect will accrue to participants? What benefits do you expect will accrue to the state?					
Have your expectations about your anticipated goals or outcomes changed since approval of your pilot? If so, how and why?							
Pilot administrative organization, partnerships, and staffing
Administrative organization and partnerships
What is your organization’s role in the pilot? Who is leading the planning, development, and implementation of the pilot?							
What other agencies, organizations, and providers are involved? What are each of their roles? How do the involved entities or their roles differ by location within the pilot? Are all pilot activities for each of these partners funded under the pilot grant? If not, please describe the specific pilot activities and/or partners that are not funded.							
How and why were these partners selected? Do these relationships predate the pilot or are they new? Are the partners involved in both treatment and control services?							
Are these formal or informal partnerships? What kinds of partnership agreements are in place for the pilot?							
Did any partnerships fail to materialize or drop out of the pilot? Were any added over time? If so, why and when? How did it affect the pilot operation?							
How many E&T providers are involved in the pilot? What types of organizations are they? How were they selected for the pilot?							
Has collaboration between partner organizations or their roles in the pilot changed over time? How did they change and why?							
What are the main advantages to the current organization and partnering structure? Are there any disadvantages?							
Did you anticipate or would you have liked more involvement from any of the partners? Please describe.							
What are the most effective elements of the collaborations? Why?							
What aspects of the collaboration could be improved?							
What have you learned about establishing and maintaining these collaborations?							
Staffing
Which staff within your organization work on the pilot? What roles do key staff members play on treatment and control services? What roles do support staff play on treatment and control services? Who leads or coordinates cross-agency or organizational work?							
What percentage of time do the various types of staff members dedicate to the pilot in an average week? Did this vary over the course of the project (for example, planning phase, early implementation, operations, and so on)? Please describe.							
Has there been turnover among staff working on the pilot? To what degree? For what reasons? Please describe the effect on the pilot.							
Were new staff hired specifically for the pilot? Which roles? Did you have sole discretion in hiring or did the SNAP grantee or program manager have a role in hiring? What were their responsibilities? Were these responsibilities related only to the treatment group? What qualifications were required? Why were these qualifications needed? Did you find it difficult to hire appropriate staff for the pilot?							
What percentage of pilot-related activities were assigned to existing staff? What were their pilot-related responsibilities? How did those responsibilities differ for the treatment and control groups? What kinds of qualifications did you require for these positions? Were these staff able to provide the time needed for pilot activities? What percentage of their time was allocated to the pilot? Were there any challenges in using existing staff?							
Did you use any volunteers to staff the pilot? Why was this decision made? If used, did the volunteers provide the needed skills and consistency for the positions?							
What lessons have you learned about the staffing needs for these types of programs?							
Planning
Please describe the planning process, including the types of activities that occurred during this phase.					
Which organizations or staff were involved in the planning process? What were the roles of each organization? Did partner organizations lead different planning activities? Why? Who oversaw the planning across all organizations?					
What mechanisms were used during the planning process (for example, work groups, steering committees, or focus groups)?					
How did individuals and organizations communicate during the planning phase (telephone calls, in-person meetings, email, and so on)? How frequently did communication occur?					
What was the time line for planning activities? When did planning activities occur? Are planning activities still occurring? If so, what is occurring and why?					
Were the planning processes perceived as effective by your organization and other stakeholders? What worked well during the planning process?					
What challenges were encountered during the planning process? Did the pilot encounter any delays? What strategies were used to address these challenges? What would you change about the planning process if you could?					
What were the most critical decisions in the planning of the pilot? Why were these decisions critical? Which individuals and organization participated in making these decisions? Were the views of all organizations considered during the decision-making process?					
In your opinion, did some organizations have more influence in the planning process than others? Who had the most influence in the planning process? Who had the least influence in the planning process? Why did these people have the most or least influence?					
Early implementation
Please describe the process used to roll out the pilot. What types of activities occurred during the early stages of implementation? Which activities were delayed until later in the pilot’s implementation?					
Who oversaw the implementation of the pilot? What other organizations or staff played critical roles in implementing the pilot? What roles did they play?					
How did individuals and organizations communicate during the early implementation phase (telephone calls, in-person meetings, email, and so on)? How frequently did communication occur? Was this sufficient? Did the level and mode of communication vary by staff level? If so, how?					
Did staff receive training related to the pilot? Who conducted the training? What was the content? How long was each training session and how many training sessions did you participate in? Was there follow-up? Do you believe the initial level of training was adequate? If not, please describe what additional training was needed.					
What was the time line for early implementation activities? When did early implementation activities occur? To what extent was the time line for early implementation activities followed? Please explain any deviations and reasons for them.					
Which staff were responsible for oversight of the pilot roll-out? How did they monitor the implementation?					
Were the early implementation processes perceived as effective by your organization and other stakeholders?					
What challenges were encountered during the early implementation process? Did the implementation encounter any delays? What strategies were used to address these challenges? What would you change about the early implementation process if you could?					
What worked well during the early implementation process?					
Were any major pilot changes needed after assessing the early implementation? If so, what and why? How were these changes implemented? Over what period? How effective were they?					
Operations: Client flow
Please describe the flow of clients (treatment and control participants) from entry into SNAP through assignment to the pilot. If there are additional pathways into the pilot (other than SNAP applications or recertification), describe these as well. Please describe any differences that exist by pilot location or target group.							
SNAP application
Please describe [STATE’s] application process for SNAP benefits.					
Please describe the available methods for applying for SNAP benefits: for example, mail or fax, walk-in, online, CBO, and telephone. How does the process differ for each method?					
Please describe the process for accepting and processing applications. Which staff are involved in the process? How and at which stages? Are interviews generally conducted face to face or by telephone? Are applications generally complete? What is the process for completing applications when information is missing? What type of follow-up is required?					
How are clients notified about determinations and requirements? How long does it generally take to make a determination?					
Do clients generally come into the office for assistance or is most contact by telephone and mail? Does it vary by subpopulations (such as working families, the elderly, ABAWDs, and so on)?					
SNAP recertification
Please describe the recertification process for SNAP benefits.					
Please describe how clients submit recertification forms: for example, mail and fax, walk-in, online, CBO, and telephone. How does the process differ for each method?					
Please describe the process for accepting and processing recertification forms. Which staff are involved in the process? How and at what stages? Are interviews generally conducted face to face or by telephone? Are recertification forms generally complete? What is the process for completing redeterminations when information is missing? What type of follow-up is required?					
How are clients notified about determinations and requirements? How long does it generally take to make a determination?					
Do clients generally come into the office for assistance or is most contact by telephone or mail? Does it vary by subpopulations (for example, working families, the elderly, ABAWDs, and so on)?					
Are all clients in the household reassessed for SNAP E&T eligibility at each recertification?					
Pathways into the SNAP E&T pilot: New and recertifying SNAP clients
Please describe the process for assessing which clients applying or recertifying for SNAP are eligible for the SNAP E&T pilot. How are SNAP E&T eligibility determinations made? Are all clients in the household assessed for eligibility?							
How are exemptions from work registration assessed? Are there any state-specific exemptions from participation in SNAP E&T? If so, what are they? How are clients assessed for these exemptions? If clients are exempt from work registration, are they eligible for the pilot?							
About what percentage of participants are not work registrants? Does the process for these clients differ?							
If SNAP E&T is mandatory, how are clients notified of SNAP E&T requirements? How soon after SNAP determination is made are clients told about SNAP E&T requirements? What are the requirements? How are clients informed of these requirements? Do workers discuss the requirements during the interview? [Collect any documents provided to participants.]							
If SNAP E&T is voluntary, how are clients notified of SNAP E&T services? In what format (for example, verbally or written)? What are clients told? Is there any encouragement to participate? [Collect any documents provided to participants.]							
Beyond determining eligibility, what role does the SNAP eligibility staff play in pilot assessment and services?							
Where are clients told to report for services? How are they notified?							
Pathways into the SNAP E&T pilot: Other pathways
For SNAP clients who are not newly applying or recertifying, how are you targeting potentially eligible pilot participants? What criteria are used? How are clients notified?							
For those targeting clients at provider sites, how are clients targeted for the pilot? What criteria are used? How are clients notified? If they are not receiving SNAP, what is the process for applying? If they are not eligible for SNAP, what is the process for including them in the pilot?							
Determination of eligibility for the pilot/assignment to pilot
What criteria are used to determine if a client is eligible for the pilot? Who makes the determination? Please describe the determination process.							
At what point in the client flow are participants randomly assigned to the demonstration?							
Please describe the random assignment process and procedures. Who conducts the assignment? Where and how is the assignment conducted (for example, at the provider’s facility, in person, over the telephone, and so on)? Have these procedures affected the intake flow for regular SNAP E&T participants?							
What information/instruction is provided to the participant after random assignment? (How much information about the evaluation is discussed?) [Collect any documents provided to participants.]							
Did you make any changes to this process over the course of the pilot? If so, why, how, and when?							
Services
Please describe the process for obtaining SNAP E&T services.							
Where can a participant access SNAP E&T services (for example, SNAP office, DOL/AJC, TANF office, other service provider, and so on)?							
Which staff interact with participants throughout the receipt of SNAP E&T services?							
Are participants assessed before receiving services? Please describe the assessment process. How long is the assessment? In what format is the assessment conducted (test, meetings, group discussions, and so on)? What organization conducts the assessment and which types of staff are involved? What is the result of the assessment (a test score, a training plan, and so on)?							
What types of barriers to employment do participants most often face? What types of barriers do participants face in participating in SNAP E&T services?							
Do participants attend an orientation before enrolling in services? Please describe the orientation and how it might differ for treatment and control groups. Which organizations and staff provide orientation?							
What services and activities are available to participants? (List all that are available, including education and training, financial coaching, life skills, and employment.) How are services assigned? Are some services in higher demand than others? How much input do participants have in determining the services they will receive?							
How do services differ for participants in the treatment group versus the control group?							
What agency or organization delivers each type of service? How do the organizations deliver the service? For those organizations delivering similar services, how if at all do services and service delivery differ?							
Please describe how long each type of services is available to an individual participant. How long to complete.							
Can participants receive multiple services? If so, are they simultaneous or sequential? Who decides?							
On average, for how long do participants receive services, including follow-up services? What is the range? Can participants continue to receive services if they cease to receive SNAP? If so, which types of services and for how long?							
What are the challenges that participants face in engaging in program activities? What are the costs of participation that might make it difficult for them to participate?							
Support services
What support services are available to assist participants in overcoming barriers to employment and participation? How do support services differ between the treatment and control groups?							
Please describe the value of each type of support service. Are there caps on how much an individual participant can receive for each type of support service? For support services overall? What is the process for approving payments?							
Which organizations provide each type of support service? How are participants referred? How does this differ for participants in the treatment and control groups?							
What other organizations and/or services are participants commonly referred to for support services?							
Are these referral agencies pilot partners or regularly available community resources? For which referral services does the pilot provide reimbursement? To what extent do case managers follow up on the referral?							
To what extent do any pilot-funded support services provided to participants (for example, for child care or transportation) cover participants’ costs for program participation and/or employment?							
Do participants receive any post-training or post-job placement employment retention or advancement services? If so, please describe. For how long?							
Case management
Please describe the case management provided to pilot participants. Who provides case management to participants? How does this differ for participants in the control group?							
What are the requirements and expectations for how often case managers contact participants? How often must they meet in person? By telephone? How often are case managers required to follow up with participants? Does this vary over time?							
How do case managers assess participants’ service needs?							
How is case management typically provided (in-person, email, or telephone)?							
What types of case management tools or models are used? Please describe.							
How responsive are participants to case management? Do they attend scheduled meetings? Do they respond to case manager outreach in a timely manner?							
How are case management activities and participant services tracked or recorded? How does this differ for participants in the control group?							
Have any changes been made to the case management approach over the course of the pilot (needed more or less)?							
When does case management end? Please describe any additional types of service provided after case management ends and for how long.							
What type of collaboration between SNAP and various service providers exists to provide participants with cross-organization support services? How is participant-specific information provided across partner organizations?							
Recruitment and outreach
What was the pilot’s approach for recruiting participants? How were potential participants identified?						
What organizations were involved in recruitment and outreach?						
What referral processes are in place for organizations to direct participants to the pilot? Are specialists notified of newly eligible work registrants? If so, what do these specialists do with the information?						
How do outreach specialists reach clients (for example, letters, calls, or face-to-face contact)?						
Were media campaigns used to disseminate information about the pilot (mail, flyers, radio, or television)? What was the main message? How widely were these materials and this information distributed? Were they provided in multiple languages? Who was responsible for creating and distributing the materials? When did the dissemination occur?						
What type of information is provided to potential pilot participants to encourage participation? How is it delivered? What are the themes and content? [Collect any documents that are distributed.]						
How many attempts are made to reach a single client? Do you have an estimate of how many contacts a single client receives (on average)? Does the level of outreach differ for those who agree to participate and those who do not? Please describe.						
Were different outreach and recruitment approaches used for different types of households (for example, the homeless or non-English speakers)? For mandatory versus voluntary participants, such as ABAWDs?						
Did you receive any feedback from clients or the community on the recruitment and outreach approaches used? Please describe.							
Beyond what was described above, did you do anything else to try to maximize participation rates among the eligible population? If so, what?							
What kinds of strategies did your organization use to encourage participants to continue to use SNAP E&T services after they enrolled?							
Did strategies change over time? Please describe. Why were the changes needed?							
What worked particularly well related to recruitment and outreach? Which methods appeared to have the best results? Did this differ by pilot location?							
What challenges did the pilot encounter related to recruitment and outreach?							
Partner communication and oversight
How does the grantee communicate with partners and providers? Who participates? Discuss the types of and reasons for communication.						
How frequently do organizations communicate—at the manager and administrator level and at the line staff level?						
Have there been challenges in maintaining communication? How could communication be improved?							
What forms of communication have been most helpful? In what ways?							
Does the grantee have data sharing agreements (MOUs) in place with each partner and provider? If not, why not? Has that caused any issues in cooperation?						
Please describe the staffing and management plan for the pilot? Which agency or staff lead oversight? How are providers’ and partners’ activities monitored? How frequently? Are there plans for corrective action, if needed? What are they? Have any been needed?						
Operations and procedures
Is staff training for the pilot ongoing? If so, what type of training? How are training needs assessed?						
What forms of guidance do case managers receive regarding serving pilot participants? Who developed the guidance? How was it disseminated? Has guidance or policy changed over the course of the pilot period? If so, how?						
Are staff changing pilot policies and procedures over the course of the pilot? If so, please describe. Why were these changes necessary?						
How are clients sanctioned for noncompliance?						
Staff experiences and challenges
What has been the staff response to the pilot?							
Overall, how would you rate staff satisfaction with the pilot? Does it vary by type of staff or job responsibilities? Does it vary by location?							
How has the pilot affected the pressure or burdens on staff compared with the pre-pilot period? Which aspects of the pilot are most responsible for changes in staff burden?							
What is your assessment of the pilot? What aspects of the pilot had a positive effect on service delivery? What aspects had a negative effect on service delivery? What aspects of the pilot have had a positive or negative impact on your job and your level of satisfaction?							
What were the biggest challenges for staff to implement the pilot? How were these challenges resolved? How could they have been avoided?							
What aspects of the pilot worked best and were most successful to date?							
Is there anything you would add, remove, or change in the pilot?							
Do you feel the pilot is meeting the goals identified during the planning phase?							
Program measurement and performance
What processes or systems are in place to measure aspects of pilot performance (for example, recruitment, enrollment, retention, completion, certification, job entry, employment retention, employment advancement, earnings, and so on)? How is pilot performance monitored? How is the performance of each partner monitored? How do these processes differ from other performance measurement procedures? Please describe the process. How is participation reported to the SNAP office (through lists, through MIS, and so on)? Who tracks, collects, reports, and monitors the data (describe which staff participate in each)?						
How do the organizations and individuals involved in the pilot use outcome and performance data?						
To what extent has the pilot met the goals it established during the planning phase?							
What aspects of the pilot do you believe are most responsible for improved employment outcomes? What is your evidence for this assessment?							
Funding and costs
Do you expect costs across service providers to be consistent over time, or will some contractors’ costs increase or decrease over time? Why?						
To what extent are you able to adjust the contracted budgets with your different providers based on the number of participants served?						
What, if any, are the pilot’s main sources of in-kind funding? Does the pilot rely at all on volunteers? Other sources of funding?						
Is performance of your contracted service providers tied to funding? If so, how do you monitor the performance of your contracted services providers? Please describe.						
How often (for example, monthly or quarterly) do you receive and process invoices for services from your partners and service providers?						
Were the grant funds sufficient to cover the costs of implementing and operating the pilot? If not, how much additional funding is needed? How would you use those funds?							
Have state budget concerns affected the implementation of the pilot (such as budget issues, hiring freezes, furloughs, or limiting overtime)? How was implementation affected? What strategies were used to minimize the effect of these challenges?							
Implementation environment and context
How many E&T providers exist within the pilot area? What services do they provide? How many serve neither the treatment or control groups (through the pilot)? To what extent do you believe that treatment and control participants use these other services?						
Within the state, how many providers exist outside of the pilot area? What services do they provide? Which providers might offer services similar to those offered in the pilot? (For county-based interventions, how frequently do participants from pilot counties receive services from E&T providers in neighboring counties?)						
What other supports for E&T exist in the community (for example, transportation, food banks, child care, clothing closets, and so on)? Which organizations provide these services and what kinds of services do they offer?						
Are there any unique state or local factors in STATE that might contributing to the pilot’s success or challenges?							
Were there concurrent policy or political changes that might have affected the SNAP E&T program? Please describe.							
Were there unique economic factors in STATE that might have affected the SNAP E&T program? Please describe.							
What are the labor market conditions in the pilot area? What are the key industries in the area? What are the sectors with the most current job openings and which are projected to have the most job openings over the next five years?						
What are the demographics of the population targeted by the pilot? What are common barriers to employment for the target population?						
Lessons learned and sustainability
What challenges have you encountered during the implementation of the pilot? How have you worked to overcome them?							
Did technical assistance received during the pilot influence the pilot’s implementation? If so, please describe. Would additional technical assistance have been helpful? If so, please describe.							
What would you do differently if you could implement the pilot again?							
What efforts were successful during pilot implementation? What worked especially well during pilot implementation?							
What are the three most important lessons you learned while implementing and operating the pilot?							
What conditions would have to be maintained for the program to continue successfully in the future?							
What changes would have to be made to operate the pilot program successfully in the future?							
Could this program be continued with the current level of SNAP E&T funding allocated to your state? Could you leverage other funds to continue?							
How difficult would it be to roll out this program to the rest of the SNAP E&T clients you serve?							
What advice would you give other states that want to implement this program?							

Notes: ABAWD = able-bodied adult without dependents; AJC = American Job Center; CBO = community-based organization; DOL = U.S. Department of Labor; MIS = management information system; E&T = employment and training; MOU = memorandum of understanding; SNAP = Supplemental Nutrition Assistance Program; TANF = Temporary Assistance for Needy Families.

Public Burden Statement

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-xxxx. The time required to complete this information collection is estimated to average 60 minutes including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate to the Office of Policy Support, Food and Nutrition Service, USDA, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302.