Partner Organizations

Section 811 PRA Demonstration Evaluation – Phase I

Data Collection Instruments: Process Evaluation Interview Protocol – Partner Organizations

105 Lock Street

Suite 207

Newark, NJ 07103
(973) 622-0900 (Phone)

(973) 622-0655 (Fax)

[email protected] (E-Mail)

www.bctpartners.com (Web)

Process Evaluation Interview Protocol – Partner Organizations

State: ______________________________________________________

R name: ____________________________________________________

R affiliation: _________________________________________________

R phone: ___________________________________________________

R email: ____________________________________________________

BCT AND ABT INTERVIEWERS WILL COMPLETE AS MUCH AS THIS INTERVIEW PROTOCOL AS POSSIBLE IN ADVANCE OF THE SITE VISIT FROM AVAILABLE DATA SOURCES INCLUDING THE 2012 PRA APPLICATION FOR FUNDING, QUARTERLY GRANTEE REPORTS, AND HUD ADMINISTRATIVE DATA. INTERVIEWERS WILL ONLY ASK QUESTIONS THAT ARE NOT FILLED IN PRIOR TO THE SITE VISIT OR IF THE FILLED-IN RESPONSES REQUIRE UPDATES OR CLARIFICATION. Interviews will be conducted orally and respondents will not be asked to provide written responses to any interview query.

Thank you very much for taking the time to meet with us. BCT Partners and Abt Associates have been contracted by HUD to conduct an evaluation of the Section 811 PRA programs, specifically examining how they are being implemented across the 12 grantee sites. An important part of this evaluation is to talk with partner organizations and agencies to gain their perspective on the PRA program.

The goal of the evaluation is to document the process by which the program is being implemented by grantee states, and ways in which the actual program differs from what was planned. We hope that you will be candid in the information you provide about the program. The information you provide is crucial to improving programs that provide housing and services for non-elderly people with disabilities. In addition, this information will be used to inform the implementation of additional rounds of Section 811 PRA demonstration grants.

While we will make every effort to protect your confidentiality, HUD staff will read our reports, and it is not possible to guarantee complete anonymity given the high level of HUD involvement in the PRA program effort. However, your name will not be used in reporting what we have learned during this onsite visit, and your responses will be combined with those of other grantees and their partners in other states.

Do you have any questions about the study before we begin?

I’m going to read you a list of possible roles in regard to the Section 811 PRA Program. We are going to use this list to tailor the interview to your specific role. Please indicate whether your agency or organization participates in or leads any of the following activities:

• Outreach to Potential PRA Participants

• Applicant Referral to PRA Program

• Pre-Screening (eligibility to PRA Program)

• Referral to PRA Units

• Application for Unit and Eligibility Determination

• Property Management/Administration of Rental Assistance

• Supportive Services

  • Services provided during transition to PRA units

  • Services provided after move in

• Program Oversight Committee

• Funder

• Other

Note to Site Visitors: If the question is not relevant or THE RESPONDENT COULD NOT OR CHOSE NOT TO ANSWER, please note that in the answer field. We would like everyone to complete all questions and provide all data items to the extent possible.

All respondents are asked questions IN sECTIONS A (Role of Organization in the PRA Demonstration) and G (cLOSING).

1. Overview of Partner Organization in the PRA Demonstration

[PRIOR TO THE INTERVIEW, SITE LEAD WILL PREFILL ANSWERS TO QUESTIONS BASED ON THE GRANTEE APPLICATION AND OTHER INFORMATION PROVIDED BY THE GRANTEE AND CONFIRM WHILE ONSITE.]

1. What is the organization’s role/responsibility in the Section 811 PRA program?

2. How many staff from your organization are involved in the PRA program and what roles do they play?

   1. What percentage of staff time is spent on the PRA program?

   2. How are staff positions funded?

3. What other programs does the organization operate? What other programs are targeted to (or otherwise serve) non-elderly low income persons with disabilities?

4. Is the target population of PRA the customary target population of your agency? If not, what populations does the organization normally serve?

5. Describe your partnership with the GRANTEE AGENCY for the PRA program.

1. What type of agreement do you have to participate in the PRA program (e.g., MOU, contract, informal agreement)?

2. What mechanisms are used for coordinating your activities with those of other PRA partners and for resolving issues (e.g. regular meetings, periodic reports)?

6. Does your agency partner with GRANTEE AGENCY on any other programs?

7. Describe your relationship with the LEAD MEDICAID AGENCY/ State Human Services agency(ies) for the PRA Program.

8. Does your agency partner with LEAD MEDICAID AGENCY on any other programs?

9. Why did the organization want to (or agree to) participate in this program?

10. Did your agency have any role in planning or developing the PRA demonstration in response to the 2012 NOFA? If yes, what was the role?

11. How do actual partners, roles, and relationships differ from what you expected? What changed and why?

12. What do you consider the PRA’s role to be in the state? Does it fill an existing void in terms of program type? Population served? Geographic area?

2. Outreach to Potential Program Applicants and Referral to the Program

13. Describe the outreach activities that your agency conducts to potential PRA participants.

    1. Were outreach activities integrated into existing programs or services or were new outreach programs created for the PRA program?

    2. What staff are responsible for outreach?

    3. Where are staff based and how are they organized (e.g., in teams, single individuals with geographic territories to cover, by agency location, etc.)

    4. How much of their time do these staff spend on PRA outreach, and how are their positions funded?

14. Have your organization’s outreach methods changed compared to what was proposed in the grant application? If yes, why did the methods change and what changes were made?

15. Do outreach methods differ for different target populations?

16. How many potential PRA applicants does your organization engage through outreach in a given month? To date?

17. What happens once an interested applicant has been identified? How and to whom are they referred to complete an application to the program?

18. How does your organization coordinate referrals with the GRANTEE, LEAD MEDICAID AGENCY, or other PRA partners?

    1. Is there a centralized agency or staff person to whom all applicants are referred?

    2. Is there a referral list? Who maintains it? How often is it updated?

19. To date, how many applicants has your organization referred to apply for participation in the PRA program?

20. Does your organization learn when referred individuals are determined eligible for the PRA program? If yes:

    1. How does your agency learn about this determination?

    2. How long does it normally take between a referral and eligibility determination?

21. To date, how many applicants that your organization referred have been determined eligible for the PRA program?

22. To date, how many applicants that your organization referred have been found ineligible for the PRA program? Why?

23. Consider recent PRA referrals to the program. Is there a difference in the people being referred compared to the target populations initially identified? If yes, what are some reasons for any differences in actual referrals versus targeted populations?

24. How often do potential PRA applicants identified through outreach decide not to apply to the program? What are the reasons why?

25. Have the outreach and referral activities worked as you expected? Have outreach activities resulted in enough eligible applicants to fill available PRA units?

26. What has worked well and what have been the challenges in outreach and referral? Would you consider this model successful?

Are there any operational improvements to outreach and referral methods that you would suggest?

3. PRA Program Application and Pre-Screening (Eligibility Determination for the Program)

27. Describe how applicants apply for the PRA program and the role of your organization in that process.

1. How do potential applicants receive applications?

2. How are applications submitted?

3. What agencies and staff process applications?

4. What information and documentation must an applicant provide?

28. Does your organization perform pre-screening (i.e., eligibility determination) to the PRA program? If yes, how is pre-screening (i.e. eligibility for the PRA program) conducted?

1. How are applicants determined to be an eligible member of the target population?

2. How do eligibility determinations differ by target population?

3. How are applicants determined to be eligible under the income requirements?

4. Do you consider other eligibility criteria at pre-screening in addition to income, disability and receipt of services?

5. When does pre-screening happen?

6. What pre-screening tools are used?

7. How many people have been found ineligible for the PRA program?

8. What are the reasons that people are found ineligible for the PRA program? What is the range of barriers encountered?

9. Do you have a case management process that starts with the referral and pre-screening processes so that PRA participants receive assistance from referral through lease up? If yes, describe that process.

29. Is there anything about the pre-screening eligibility determination and application processes that is different in implementation than you expected? If yes, please describe.

30. What has worked well and what have been the challenges of the pre-screening eligibility determination processes? Would you consider this model successful? Are there any operational improvements to the eligibility determination process that you would suggest?

4. Referral to PRA Units, Tenant Application and Screening/Eligibility

31. To date, how many applicants has your organization referred to apply for a PRA unit?

32. Once determined eligible for the PRA program, how does the referring agency and the applicants learn about PRA units that are available for leasing?

1. Does the STATE use a housing locator? If yes, please describe how the locator works.

2. Does the STATE use an online housing resource such as SocialServe or HousingLink? If yes, please describe how online resources are used by program staff and by applicants.

33. Describe the process of referring clients to available, appropriate PRA units.

1. What factors (e.g. unit size, accessibility features, etc.) are taken into consideration?

2. Which agencies/partners are responsible for matching the applicant's requirements to an appropriate, available unit?

3. How many applicants are referred for each available unit? How many unit choices does an applicant have under the program?

4. Is the number of eligible applicants adequate to fill available units?

5. How long does it normally take between when a participant is determined eligible for PRA funding and placement in a PRA-funded unit?

34. Once a person is found eligible for PRA assistance, are they placed on a waiting list? If not, why not. If yes:

1. What agency(ies)/property(ies) maintain the list?

2. Does the agency use SocialServe or other program for waiting list management?

3. How is priority for available units determined?

4. How many applicants are currently on the waiting list?

35. Where do clients stay while on a waiting list? Are there any bridge subsidies available to the clients you work with?

Tenant Application and Screening/Eligibility

[INTERVIEWER: THESE QUESTIONS MAY APPLY TO PROPERTY MANGER/OWNER OR ANOTHER AGENCY]

These next questions ask about how PRA participants apply to and are determined eligible for a unit within the PRA program.

36. Describe the applicant application and screening process for a unit (this may be performed at the PRA property site).

1. Who conducts the application and screening process which determines eligibility for the unit?

2. How do potential applicants apply for a specific unit or property?

3. What role does your site/organization play in processing applications?

4. What information and documentation must an applicant provide?

36. To date, how many applicants referred to a unit/property have been found eligible for PRA units?

36. To date, how many applicants that have been referred have been found ineligible for PRA units? Why?

36. What happens to applicants found ineligible? Are they referred to other programs or properties? If yes, which ones?

36. To date, how many applicants have withdrawn their application, fallen out of the application process or have chosen not to sign a lease? Why?

36. Once a participant has selected an available PRA unit, describe the process for moving the eligible applicant into the PRA unit.

36. Once the person is housed in the PRA unit, does the GRANTEE or partner do any follow up with the tenant?

    1. With property management?

    2. With the service providers?

36. Describe the process if a PRA tenant wants to/needs to transfer to another unit. Have any PRA tenants transferred from one unit to another? If yes, why did the tenant transfer units?

36. Describe the process if a PRA-assisted unit turns over.

    1. How do referring agencies learn of the vacancy?

    2. What is the applicant referral process for newly vacant units (e.g., property waiting list, centralized waiting list, other)?

    3. Is follow-up with the tenant who left the unit conducted by grantee or partner agency?

36. Has there been unit turnover so far? If yes, how many units have been vacated and for what reasons?

36. What has worked well and what have been the challenges in the processes of referring PRA applicants to units and in PRA tenant occupancy? Are there any operational improvements that you would suggest? Would you consider this model successful?

36. Is there anything about the screening eligibility determination process that is different in implementation than you expected? If yes, please describe.

36. What has worked well and what have been the challenges of the screening eligibility determination process? Would you consider this model successful?

Are there any operational improvements to the process that you would suggest?

5. Supportive Services

The following questions ask about your organization’s role in providing services to tenants in PRA units. First we will ask about services provided during a participant’s transition to a PRA unit. Then we’ll ask about services provided as part of ongoing occupancy of tenants and the service providers and funding sources for both types of services.

Transition to PRA Units

36. Does your organization provide services directly to PRA participants as they transition to PRA units?

[IF YES, ANSWER QUESTIONS BELOW]

36. What transition services are provided or coordinated by your organization for new participants as they move into PRA units?

(Probes: assistance with moving, obtaining furniture and house goods, tenant education, housekeeping skills, setting up utilities, accessibility modifications.)

1. Who provides transition services?

2. How are these services funded? (Probes: Medicaid (MFP; Home and Community Based Services; Section 1915(I) state plan amendments; Section 1115 Research and Demonstration Projects; Balancing Incentive Program); other federal resources; state or local resources.)

3. Do transition services differ based on target population?

4. Do transition services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

36. Once a participant has been matched to an available PRA unit, describe the process for moving the eligible applicant into the PRA unit.

    1. How does your organization determine service needs for PRA participants during the transition?

    2. How do PRA participants learn about services your organization provides while they move to PRA units?

    3. How do PRA participants meet security deposit requirements (if applicable)?

36. Does your organization assign PRA participants a service coordinator or case manager to help them through the transition to PRA units? If yes:

    1. How often do case managers meet or talk with participants?

    2. What assistance do case managers provide?

    3. For how long are transition services provided?

Ongoing Supportive Services

36. Does your organization provide services directly to PRA participants once they are living in PRA units?

[IF YES, ANSWER QUESTIONS BELOW]

36. After the tenant has moved in, what services are provided and coordinated by your organization?

(Probes: case management, development of individual service plans, access to HCBS waiver services, employment services.)

1. Who provides these services?

2. How are these services funded? (Probes: Medicaid (MFP; Home and Community Based Services; Section 1915(I) state plan amendments; Section 1115 Research and Demonstration Projects; Balancing Incentive Program); other federal resources; state or local resources.)

3. Do services differ based on target population?

4. Do services differ by type of disability (Probes: physical disability, mental health disability, intellectual or developmental disability.)

36. How do participants learn about services available to them from your organization after they move to PRA units?

36. How do participants access services after they have moved into PRA units?

36. Describe the process for determining a participant’s service needs. Is a service plan developed? After a service plan is developed, how often is progress checked?

36. Does your organization assign a service coordinator or case manager to work with them after they have moved into PRA units? If yes:

    1. What is the role of the case manager?

    2. How often do case managers meet or talk with participants?

    3. How often do case managers meet or talk with property management?

Services Coordination and Funding

36. Have the funding streams for services you planned to use been available, appropriate, and sufficient for the people you are serving? What is the likelihood of continued funding?

36. Are there services that are difficult to find, in short supply, or that funding streams do not cover? If yes, what are they?

36. Are there any sources of flexible “do what it takes” funding that can be used to cover gaps in what Medicaid or other programs cover?

36. Does receiving PRA rental assistance provide quicker or more flexible access to services (e.g. priority on wait list or more flexible process for enrollment)?

6. Property Managers/Property Owners/Landlords

36. What is the inventory of properties you own or manage (e.g. private market, affordable housing, low- income housing tax credit, supportive housing)? Please describe the mix.

36. What was your prior experience housing very low income people with disabilities?

36. How did you learn about the PRA program?

36. Why did you decide to apply for PRA rental assistance? How did you determine how many PRA units you could place under contract? What would make it easier to place more units under contract? What makes it difficult to commit units to the PRA program?

36. Do you have any concerns about sustaining 811 PRA units? Is there a rent threshold that would make it difficult to participate?

36. How many Rental Assistance Agreement Contracts (RACs) have you signed with the GRANTEE AGENCY? What is the total number of PRA units funded under these contracts? How many units are currently leased? Is this more or fewer units than were expected at this point?

36. Describe the process of executing the RAC with the GRANTEE AGENCY.

    1. How long did the process take?

    2. Did the process work as expected?

    3. Were there any delays in executing the RAC with the GRANTEE AGENCY? Have you had delays in receiving payment?

    4. If this was your first RAC, do you think the next one will take the same amount of time? Shorter? Longer?

[FOR RESPONDENTS WITH MORE THAN ONE PROPERTY WITH PRA-OCCUPIED UNITS, ASK FOR EACH PROPERTY.]

36. How do the PRA client characteristics compare with characteristics of other tenants in non-PRA-funded units?

Now I would like to review the characteristics of properties with occupied PRA units. I’d like to confirm some data on the property characteristics that we learned from grantee administrative data submitted to HUD. We’d like to learn about the characteristics of the overall property, not just the units that are occupied with PRA tenants.

36. What are the characteristics of the property? [confirm data in TABLE BELOW from grantee reports and iREMS]

Characteristics of Properties with PRA Units
	Property 1	Property 2	Property 3
Building Type	Check all that apply: Row/ Townhouse Walkup Garden Mid-rise High-rise Detached Semi-detached	Check all that apply: Row/ Townhouse Walkup Garden Mid-rise High-rise Detached Semi-detached	Check all that apply: Row/ Townhouse Walkup Garden Mid-rise High-rise Detached Semi-detached
Number of buildings
Number of Units (total # in property)
Number of Bedrooms for entire property	0 BR:_____ 1 BR: ____ 2 BR:_____ 3+ BR: ____	0 BR:_____ 1 BR: ____ 2 BR:_____ 3+ BR: ____	0 BR:_____ 1 BR: ____ 2 BR:_____ 3+ BR: ____
Amenities (e.g., shared space, access to services, access to transportation)
Onsite Services (e.g. onsite service provider, meals provided, onsite medical services)
Number of total Accessible Units in property	Impairment: Mobility:____ Hearing:____ Vision:____ Other: ____	Impairment: Mobility:____ Hearing:____ Vision:____ Other: ____	Impairment: Mobility:____ Hearing:____ Vision:____ Other: ____
# Units/Percent Units Restricted as Affordable to incomes (between 30-60% of AMI)
#Units/Percent Units Restricted as affordable to incomes less than 30% of AMI
# Units/Percent Set Aside for Persons with Disabilities (for all programs)

36. Describe the process by which a tenant gets placed into a PRA unit from the property owner/manager perspective.

    1. Have there been any challenges with the move in process?

    2. ? When a person is referred, how often do they move in?

    3. Are sufficient services provided to ensure that the PRA participant is a successful tenant? Explain

36. Describe the process for when a PRA unit turns over.

36. Has any PRA units turned over? If yes, where did the PRA participant move to?

36. Describe the process for when a PRA tenant wants to transfer to a different unit.

36. Have there been transfers from one PRA unit to another? If yes, why?

36. Have there been any evictions from PRA units? If yes, for what reasons?

36. Have there been any issues with filling vacant units? How long do units remain vacant before they are matched to a participant?

36. Have you received any vacancy payments?

36. Are there supportive services available for people as they move into the units? Who provides these services?

36. Are supportive services available for PRA participants after they move in? Who provides these services?

36. [If not already discussed] Does this property have a service coordinator? If yes:

    1. What are the service coordinator’s responsibilities with respect to PRA-assisted tenants?

    2. How does the service coordinator interact and coordinate with other service providers who work with PRA-assisted tenants?

36. How often do you communicate with service providers for PRA participants?

36. What is the process when there is a disagreement between the property manager and the tenant?

36. Describe the process for billing for and receiving PRA rental assistance funds from the GRANTEE AGENCY.

    1. Is the process working as expected?

    2. Have there been any administrative issues or delays in receiving PRA funds?

36. Do you enter tenant and property information into TRACS? Describe your experience with that process.

36. Are there any changes or modifications to the PRA program you would suggest as a property manager/owner?

7. Closing Questions

36. Consider the program as implemented. Is the program being implemented the way you had imagined it when you agreed to participate?

36. Is there anything you would change if you had the chance? What is working well? What is not working well?

36. How effective are the partnerships in the PRA program? Would you participate in a program like this again?

36. Do you think the program is targeting the right population based on local need?

36. Do you think the program is a success? Why or why not?

36. What advice would you give to agencies like yours in other states about participating in the PRA program?

36. What suggestions would you make to HUD or HHS to improve the PRA program?

36. Is there anything else that we haven’t talked about that you would like to mention about your experience with the PRA program?