The Exploring Measurement of Performance Outcomes and Work Requirements in Programs Promoting Economic Independence (EMPOWERED) Study

Attachment A: Master Discussion Guide OMB#: 0990-0421

Date of Expiration: 10/31/2020

Introduction for respondents:

• The Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services is sponsoring this study to better understand work requirements in human services programs and identify best practices in the administration of work requirements. ASPE has contracted with Mathematica Policy Research to conduct this study.

• The Exploring Measurement of Performance Outcomes and Work Requirements in Programs Promoting Economic Independence (EMPOWERED) study will includes site visits to three states administering work requirements in three programs: Temporary Assistance for Needy Families (TANF), Supplemental Nutrition Assistance Program (SNAP), and public housing programs.

• We are meeting with state and local agency staff around the state this week. The goal of today’s conversation is to learn more about your role and to how work requirements and policies operate under your program. Your perspective and insights on these issues will be very helpful to the study.

• I want to let you know that your participation in this interview is voluntary, and your responses will be kept private to the extent provided by law. We will not share the information you provide with anyone outside of the research team, including your supervisor and state or federal -level staff. You may refuse to answer any question and may stop the interview at any time. There will not be any penalties if you refuse to participate in part or full.

• We will take notes over the course of the interview so that we can remember the information we collect. We will use this information in our reports, describing the range of responses expressed by staff. The reports might list the names of organizations that contributed information, but we will not quote you or anyone by name or title. However, because of the relatively small number of organizations participating in the study, there is a possibility that a response could be correctly attributed to you.

• I expect our conversation will take about [60/90] minutes. First, do you have any questions for me about the project in general or what we will be discussing today?

Note to Interviewers:

• Prior to the visit, tailor the discussion guide to each site and respondent. Eliminate questions that will not be asked of each type of respondent. Also, note the formal names of each program in the state, any acronyms used, and the terminology appropriate for each respondent to understand the questions.

• 

  
  
  

  Public Burden Statement

  According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0421. The time required to complete this information collection is estimated to average 90 minutes per response, including the time to review instructions, complete the information collection, search existing data resources, and gather the data needed. Send comments regarding this burden estimate to the U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, 200 Independence Avenue, SW, Washington, DC 20201.

  Throughout the discussion guide, we use employment and training or (E&T) to refer generally to employment and training (education) activities and services, we are not referring to a specific program.

Attachment A: Master Discussion Guide							OMB#: 0990-0421 Date of Expiration: 10/31/2020
Question	Respondents
	State agency				Local or regional office				Employment and training providers
	Director	Policy staff	Staff overseeing work requirements	Data managers	Director	Supervisors	Eligibility workers	Case managers	Director	Staff delivering services
Background
What is your official job title or position?										
What are your primary responsibilities?										
What services under which programs are provided by [agency/organization]?									
State and local area context
What jobs are most in demand in this area (low-, middle-, and/or high-skill jobs)?										
Are there jobs or industries that have declined or left the area in recent years?										
Are the E&T providers in the area sufficient for your program needs?						
What other kinds of supports are available in the area (for example, transportation, food banks, clothing closets, housing assistance)? Are clients who are required to comply with work requirements referred to these services?										
What are the services (E&T and supportive services) most in need by the people in this local area? Are there any gaps in services?										
Work requirement policy motivation and goals
What is the mission of the program?									
How well are the work requirements policies aligned with the mission of the program?						
If there is discretion, why did the State move to mandatory work requirements? In selected areas?			
How are decisions made about making changes to work requirement policies or work supports? What are the typical drivers?			
How are decisions made about what work supports to offer in the State?			
When developing the work requirement policies and work supports, do staff consider the local economy and needs in the area? Which factors are most important to consider?									
How much coordination does the agency have with other human service programs related to developing and administering work requirements? Are resources shared or is the existing E&T infrastructure in the local areas leveraged for this program? Describe.									
How much discretion do local area staff have in determining work policies, procedures, activities to offer? How do policies vary across areas?						
What are the expectation or goals for work requirement policies?						
Are there identified outcome goals for the program? Describe.									
Target population
Are the program services targeted to certain populations?									
What geographic area does the program serve? (Any areas exempt from work requirements?)									
What kinds of barriers do clients most often face?										
Exemptions
Which types of clients are exempt from work requirements? Why were these criteria selected?		
Are there limitations on the types of client the State may exempt? Which entity imposes these limitations?		
Are there other type of recipients that it would be beneficial to exempt? Which ones? Why?										
Assessing eligibility for work requirement and referrals
What is the process for determining whether a client is exempt from work requirements (for example, screening, assessments, documentation)?							
Do clients that have work requirements under other human service programs or are participating in a work program (e.g., under WIOA) have a work requirement? Describe the policy and any exclusions. How is information about requirements for other programs learned?								
Are some exemptions more difficult to screen for than others?								
What factors may contribute to determination errors?								
How and when are clients notified of their requirements? Describe the process.							
Once a client learns he or she has a work requirement, what happens next (for example, orientations, referrals, timing)?								
Are most clients referred to an E&T provider, or are they served here?								
Describe the referral process or the next steps if services are offered on site.								
Work support services to meet work requirements
Describe the services each provider offers and what is allowable?								
How is it determined which provider to refer to your clients?								
How do staff at the [agency] and E&T providers communicate to coordinate services? How well does this approach work?								
Provider services
Describe the intake process when a client arrives (for example, Are assessments conducted? What tests are used? How are clients placed with a case manager?).										
What kinds of barriers do clients most often face when they are referred?										
What training and employment services are offered at this organization (for example, education, vocational training, work experience, job readiness training, and follow-up services)?										
How are clients assigned to a specific service?										
In which E&T services do clients most frequently participate? Why?										
Case management
Please describe the case management provided to clients (for example, who provides it, how much time is spent with clients, what is the format).										
How responsive are clients to case management (do they attend meetings/respond in a timely manner/seem to find contact useful)?										
Support services/barrier reductions
What support services are offered to [program] clients?										
What are the limits on reimbursements?										
To what extent do you think the supports offered by your office and the providers cover the costs associated with participating in the program?										
Are there additional supports that are needed by clients to help keep them engaged in services? Describe.										
Noncompliance determination and sanction policies
What must a client do to remain compliant and what is the sanction for noncompliance?										
How did the State formulate its current policies regarding compliance?			
What are the most common reasons for noncompliance?										
Do clients tend to become noncompliant initially (by never engaging in services), or do they tend to become noncompliant after engaging in services?										
Does the State track how many clients are in noncompliance, and how many are sanctioned? If so, approximately what are the percentage of clients?										
Does the State track return rates to [the program] after a sanction is imposed? If so, what is the percentage, and how quickly do they tend to return?				
How do you track how clients are meeting their work requirements?				
What types of information do providers report to the local office?										
Are clients more or less likely to be sanctioned depending on the component they are assigned to? If so, how would you explain this variation?										
Program oversight and communication
State level
To what extent are local offices and E&T providers monitored at the State level? Describe the process.			
What kind of quality assurance procedures are in place, if any, to monitor that sanctions are made correctly?			
Local level
How does the office track what happens to clients once they are referred to services or begin working?								
Does the office or providers monitor the employment trajectory or the types of jobs clients secure following their participation in E&T? Describe.										
Does the local office conduct any monitoring of the providers? Describe.								
Is the current level of monitoring effective? If not, describe why or why not.										
Communication
How is communication conducted with [providers/case managers] about a client’s status with regard to participation in E&T and compliance with requirements? How frequently?										
How does the office know if a participant reports to a [provider/case manager]? Engages in an E&T component for the requisite hours?										
What are some of the challenges with respect to communicating with [providers/case mangers] about clients? How could it be improved?										
Data tracking and outcomes
How many recipients does the program currently serve?						
How many recipients are subject to work requirements?						
Of those subject to work requirements, how many are meeting those requirements?						
What data related to work requirements does the State collect (for example, how many clients participate in work activities, how long they participate, outcome measures)						
How are data on client status, E&T activities, and outcomes obtained?						
How are these data used (for example, are they used to inform programmatic changes or allocation of resources)?			
Are measures used to define the “success” of the program? Which measures are used for this (for example, registration for work, getting a job, improvement in overall well-being)?								
Are outcomes aligned with work supports received? Are certain supports associated with better outcomes?								
What types of reporting are required for each program? Are other types of reports created for internal use? [Ask for available reports.]								
E&T provider data
What percentage of your E&T referrals ultimately participate in services?										
At what point do those who start services tend to stop participating? What leads to drop-off at this point?										
Approximately how many unique participants do you serve each year (per program)?									
How long do mandatory E&T participants generally participate in services?										
What other types of data are available for each client?										
Through what type of system are data tracked? Are those data systems linked to the program agency? How are data transferred/received to/from the agency?										
What type of reporting are required for each program? Are other types of reports created for internal use? [Ask for available reports and data.]										
Are there other types of data that would be helpful for monitoring/program management? Describe. What are the limitations to obtaining these data?										
Perceived effectiveness of programs
Do work requirements appear to be more beneficial or effective for certain subpopulations (for example, those without children, low-skilled workers, noncustodial parents)? Which ones?										
How do work requirements affect outcomes (for example, obtaining employment, improved self-sufficiency)?									
What are the largest cost drivers in operating work requirements (for example, administration, case management, services, sanctioning)?									
What are the largest cost savings in operating work requirements (for example, reduction in benefits due to employment or sanctioning, fewer clients apply, and so on)?									
Is the current staffing structure sufficient for operating the work requirement policy and work support programs? What works best? What could be improved?										
Lessons and best practices
What is most challenging about operating a mandatory program/serving mandatory participants?										
What if anything would you change about the work requirement policies/serving participants? Why?										
Are there any lessons or best practices for coordinating with other programs or agencies that operate work requirements?										
Lessons for implementing a new mandatory work requirement
What factors do agencies/States need to consider when implementing new mandatory work requirements?										
For agencies/States with programs that have no mandatory requirements, what guidance would you provide about whether they should incorporate them, and, if so, how best to do so?									