HUD/HHS/VA CHRONIC HOMELESSNESS INITIATIVE
NATIONAL PERFORMANCE ASSESSMENT
SUMMARY
The Office of the Assistant Secretary for Planning and Evaluation/DHHS requests an extension for a national performance assessment study (OMB# 0990-0304) to monitor the implementation and effectiveness of the Collaborative Initiative to Help End Chronic Homelessness (referred to below as the Chronic Homelessness Initiative).
On January 27, 2003, a notice of funding availability (NOFA) was published in the Federal Register for the Chronic Homelessness Initiative. Coordinated by the U.S. Interagency Council on the Homeless, the initiative involves the participation of three Council members: the Department of Housing and Urban Development (HUD), the Department of Health and Human Services (HHS), and the Department of Veterans Affairs (VA). In September 2003, eleven (11) sites were selected to receive $35 million through the initiative over the next 3 years to support the Administration’s goal to end chronic homelessness by seeking to implement a collaborative and comprehensive approach to addressing the problems of homelessness among our most vulnerable citizens. The sites include: Chattanooga, TN; Chicago, IL; Columbus, OH; Denver, CO; Ft. Lauderdale, FL; Los Angeles, CA; Martinez, CA; New York, NY; Philadelphia, PA; Portland, OR; and, San Francisco, CA
The supported housing component of the intervention funded by HUD will last 3-5 years. The supportive services components of the intervention funded by SAMHSA, HRSA, and VA will last 3 years. Data collection, along with preliminary reporting of longitudinal client outcomes and systems change, for the national evaluation funded by HUD, HHS and VA will last 3 years after the start of the clinical program at each site, subject to the availability of funds).
Although the funds were released to the sites for this initiative in September of 2003, implementation has been slower than expected, in part because of delays in implementing the program and in part because of delays in obtaining IRB approval for the evaluation. The first clients at the first sites to begin service delivery were recruited into the protocol in March of 2004.
This collaboration offers housing and treatment funding, that along with existing service resources, is intended to assist persons who are chronically homeless move from the streets and emergency shelters into stable housing and receive the range of services and other support needed to promote and maintain greater self-sufficiency.
Since $35-million of federal agency funds represents a considerable public investment in local efforts to help end chronic homelessness, a national assessment of client outcomes is needed to assure a high level of accountability and to identify which models work best for which people, using the same methods for all sites.
The goal of the national outcomes performance assessment is to provide a site-by-site description of program implementation, as well as descriptive information on clients served; services received; housing quality, stability, and satisfaction; and, client outcomes in health and
functional domains. Because the focus of the program is on a comprehensive approach it is important to measure benefits in a comprehensive range of outcome domains.
The assessment includes both client and network components. The major questions pertaining to the client component of the assessment include:
1. Are there differences across sites in client characteristics screened through the CHI?
2. In what ways do clients enrolled into the project differ from those screened, but not enrolled?
3. How similar are enrolled client populations across sites?
4. What types of services are received and does service use vary over time?
5. What are the outcomes over time in the areas of housing, employment, finances, physical health, mental health, substance use, coordination of services, criminal justice, social support, religious faith, community integration, and quality of life of client and level of functioning of children in homeless families (where applicable)?
6. Are these results biased by data loss?
7. Which group/sub-groups obtain more services and have the best outcomes?
8. Which services are more important for which outcomes?
9. How will non-response bias be assessed?
The major questions addressed in the network component of the assessment include:
1. What types of housing and service models are available for chronically homeless people in this community?
2. What is the nature of the interaction between agencies in this initiative? How involved are these agencies with a larger coalition of homeless service providers?
3. Are variations in the above associated with superior client outcomes observed in the client outcomes monitoring effort?
Client data are being collected by local evaluation assistants at each of the 11 sites participating in the project. Four instruments are being used to collect client-level data: 1) screening form (completed by program homeless outreach workers once at the time of first clinical contact for all prospective participants in the initiative); 2) baseline assessment form (completed by evaluation assistants upon clients’ entry into the project among those eligible and willing to participate in the evaluation); 3) follow-up form (completed by evaluation assistants quarterly following the baseline assessment for up to 3 years); and, 4) discharge form (completed by program case mangers once at the time clients discontinue participation in the program).
These forms are completed locally through in-person and telephone interviews, and then mailed to the VA Northeast Program Evaluation Center (NEPEC), the cross-site evaluator, weekly for data entry and data analysis. Client data collection began in March 2004, around the same time that CHI treatment teams/programs became operational. As of the end of June 2006, approximately 750 CHI clients had been enrolled into the national evaluation, and nearly 3,800 CHI client follow-up assessment interviews had been completed (approximately 2/3 of CHI client 3-year follow-up data). Recruitment of CHI clients ended in April 2006.
Comparison Group Data
In response to discussions with staff of the Office of Management and Budget and Terms of Clearance for the first year of data collection for the project, an initial comparison group of 119 persons receiving referral/access to housing resources usually available in the community and/or case management and other supportive services routinely available in the community for persons who are chronically homeless was recruited at 5 of the 11 CHI sites (Chattanooga, Los Angeles, Martinez, New York & Portland). Comparison group recruitment at these 5 sites began in May 2005, after institutional review board approvals were obtained & recruitment plans developed, and continue to the present time. The same four data collection instruments have been used with both CHI client and comparison group subjects. Over 200 comparison group follow-up interviews had been completed (approximately 1/3 of initial comparison group 3-year follow-up data) through the end of June 2006.
Based on discussions with VA research staff at the remaining 6 CHI sites, it appears that it will not be feasible to recruit a comparison group at Chicago, Denver, Ft. Lauderdale & San Francisco due to barriers presented by local institutional review boards, CHI agencies and partnering homeless services providers, existing VA research evaluation assistant caseloads, and related issues. It may, however, be possible to recruit additional comparison group subjects from the 5 existing comparison group sites, and also to recruit a comparison group from Columbus and/or Philadelphia CHI sites if the national evaluation is extended by OMB for 3 years.
Extending the national evaluation for 3 years would also provide an opportunity to collect a full 3-years’ outcome data for both CHI clients (as originally intended by federal agency sponsors) and comparison group subjects (as previously recommended by OMB). If data collection ends in October 2006, 24-month follow-up assessment data for CHI treatment group would only be available for an estimated 50% of the sample. Nine-month follow-up data for comparison group subjects would be available for the first 60 subjects or so recruited into the national evaluation. No 3-year follow-up would be available for any participants, neither would any 2-year follow-up data be available for any comparison group subjects.
Network data are being collected by NEPEC staff through telephone interviews with key informants. A network definition interview was administered at the start of the project with key informants at each of the core agencies involved in the Initiative (i.e., the lead agency and the several other core agencies specified in the NOFA--housing, mental health, substance abuse, primary care and veterans). This survey was used to establish: a) which agencies are most actively involved in the CHI (including agencies other than the core agencies in the application); b) whether there is one or more broader coalition of homeless service providers in the community in which the CHI is embedded (i.e. a continuum of care, or coalition of homeless service providers); and, c) which of these coalitions are most relevant to the performance of the
CHI. Thus, its main purpose was to identify the important agencies comprising the network of Initiative service providers at each site.
Having identified the CHI network, network participation interviews are being administered by NEPEC staff to up to 9 key informants per site (1 for each member agency) of agencies most involved in delivering services to CHI clients. The first round of interviews (N=71) was administered in January 2004, a few weeks prior to the beginning of the project (baseline) at most sites. Two additional rounds of interviews were completed in January 2005 (N=69) at the end of year 1 of the project, and in January 2006 (N=64) at the end of year 2 of the project. A fourth and final round of interviews is scheduled for January 2007, at the end of year 3 of the project, at which time an estimated 60 participation interviews will be conducted. Nine dimensions of operations are being assessed through this survey: a) the structure, scope and leadership of the network; b) goals and focus; c) planning; d) management and use of information systems, e) the key participants at the time of the survey (identifying any additions or exits); f) connectedness/ integration at both the service delivery and leadership levels; g) use of strategies to improve system integration; h) the use of various practices and housing strategies, and, i) involvement of the larger homeless coalition in the Initiative.
Without an extension by OMB, it will not be possible to conduct the final set of network participation key informant interviews, which are scheduled for January 2007.
Thus, the major justifications for a 3 year extension by OMB of CHI national evaluation data collection include:
1) to provide an opportunity to collect a full 3 years of follow-up outcomes data for both CHI client and comparison group subjects;
2) to provide more time to recruit additional comparison group subjects; and,
3) to provide an opportunity for the final “year 3” round of network participation key informant interviews.
1. Circumstances of Information Collection
Research indicates that as many as 150,000 people experience chronic homelessness in this country each year. People experiencing chronic homelessness often have an addiction or suffer from a disabling physical or mental condition and are homeless for extended periods of time or experience multiple episodes of homelessness. For the most part, they get help for a short time but soon fall back to the streets and shelters. Because the needs of these persons are not comprehensively addressed, they cycle through the homeless system and consume a significant portion of available resources. In fact, research indicates that these individuals consume more than half of all homeless services. As such, there are significantly fewer resources available for 90 percent of the homeless – including families – who, with a little assistance, could often exit homelessness relatively quickly. By addressing the housing and service needs of persons who are chronically homeless, we will have more resources available to meet the needs of other homeless people. When persons who are chronically homeless have access to basic assistance like housing and treatment, they have fewer problems and are less likely to need expensive emergency interventions. The research makes it clear that one of our best hopes for ending homelessness of every sort depends on addressing chronic homelessness.
On January 27, 2003, a notice of funding availability (NOFA) was published in the Federal Register for the Collaborative Initiative to Help End Chronic Homelessness. The initiative, coordinated by the U.S. Interagency Council on the Homeless, involves the participation of three Council members: the Department of Housing and Urban Development (HUD), the Department of Health and Human Services (HHS), and the Department of Veterans Affairs (VA). The $35-million initiative supports the Administration’s goal to end chronic homelessness by seeking to implement a collaborative and comprehensive approach to addressing the problems of homelessness among our most vulnerable citizens.
This collaboration among the departments provides housing and service funding to local consortia of agencies (networks) providing housing and supportive services to chronically homeless individuals and families in their communities. Networks typically include one or more housing management organizations, a community mental health center, a substance abuse treatment facility or organization, a health center, a VA medical center or other health care facility, and two or three other entities providing services or advocacy for chronically homeless individuals in that community (e.g., a homeless shelter, the local Social Security Administration office, an agency providing vocational rehabilitation, a faith-based organization, the Mayor’s Task Force on the Homeless). Networks are expected to use Initiative funds, supplemented by funds provided by mainstream services resources, to develop more comprehensive and integrated community strategies to assist persons who are chronically homeless and homeless parents caring for minor children move from the streets and emergency shelters into stable housing and receive the range of services and other support needed to promote and maintain greater self-sufficiency.
A chronically homeless person is defined in the Notice of Funding Availability as a homeless individual or homeless parent of a minor child, “…with a disabling condition who has either been continuously homeless for a year or more OR has had at least 4 episodes of homelessness in the past 3 years (NOFA, 2003).”
Funding provided by HUD is being used to provide permanent housing (excluding new construction and major rehabilitation activities). Permanent housing is defined as housing that has no limit imposed on the length of a resident’s stay in the project. HHS/SAMHSA (Substance Abuse and Mental Health Services Administration) funds are being used to provide substance abuse treatment, mental health and related supportive services. HHS/HRSA (Health Resources and Services Administration) funds are being used to provide primary health care services through existing Section 330 Health Centers. VA funds are being used to provide additional case management services for chronically homeless veterans enrolled into the Initiative and the VA health care system.
The goals of this 3-year program for persons experiencing chronic homelessness include: 1) increase the effectiveness of integrated systems of care for chronically homeless persons by providing comprehensive services and treatment and linking them to housing; 2) create additional permanent housing for chronically homeless persons; 3) increase the use of underused mainstream resources that pay for services and treatment for chronically homeless persons (e.g., Medicaid, TANF, Food Stamps, block grants, state-funded children’s health insurance programs); 4) replicate service, treatment, and housing models known to be effective based on sound evidence; and, 5) support the development of infrastructures that sustain the housing, services, treatments, and inter-organizational partnerships beyond the 3-year Initiative.
To help determine the extent to which these five program goals are met, grantees (networks) will be required to cooperate fully with the evaluation described in this application as a condition of award.
The Notice of Funding Availability (NOFA) for the Initiative was published in the Federal Register on January 27, 2003. The application submission deadline was April 14, 2003. Approximately 100 applications were received. U.S. Inter-agency Council on Homelessness staff first reviewed the comprehensiveness section of all applications, which described the collaborative, inter-agency intervention proposed at each site. Applications receiving a passing grade on comprehensiveness of intervention were then forwarded to agency program staff at HUD, SAMHSA, HRSA and VA for rating of agency-specific sections of the application based on a point system described in the NOFA. Agency ratings were completed around August 14, 2003, and sent back to the Inter-agency Council on Homelessness for final review and selection of sites based on comprehensiveness and agency-specific ratings.
In September 2003, eleven (11) sites were selected to receive $35 million through the initiative over the next 3 years to support the Administration’s goal to end chronic homelessness by seeking to implement a collaborative and comprehensive approach to addressing the problems of homelessness among our most vulnerable citizens. The sites include: Chattanooga, TN; Chicago, IL; Columbus, OH; Denver, CO; Ft. Lauderdale, FL; Los Angeles, CA; Martinez, CA; New York, NY; Philadelphia, PA; Portland, OR; and, San Francisco, CA.
One of the major purposes of this independent, national evaluation involving both client-level and direct service-level system data is to describe and compare program characteristics at each of the sites. Most sites are still developing clinical policies and procedures, and once established these policies and procedures may change over the next 3 years. Thus, we will not know how similar or different these 11 programs are until client and network data collection are completed 3 years from now.
To the extent that significant programmatic differences or differences in client characteristics are found across sites, these differences will be statistically adjusted for using hierarchical linear modeling (HLM) and other statistical analysis techniques described in the Data Analysis section of this submission.
As is often the case in such real-world, observational studies, there will likely be a trade-off between higher internal validity associated with greater program homogeneity across sites, and higher external validity (generalizability) resulting from greater program heterogeneity across sites.
Separate local evaluations may be conducted at some of the sites by other evaluators over the course of the project, but no other quantitative national-level evaluations. An administrative qualitative review of program operations at each site will be conducted by the National Center On Family Homelessness (NCFH) as part the technical assistance provided to agencies receiving SAMHSA funds through CHI. NEPEC and the National Center are coordinating research evaluation and administrative/technical assistance efforts to avoid redundancy in the collection of network-level data.
Eleven sites have been funded to participate in the program, and are expected to place an estimated an estimated 750 chronically homeless persons into permanent housing – 675 (90%) of whom are expected to participate in the national evaluation. Client characteristics and outcomes are being measured during outreach (i.e., first meaningful clinical contact with clients in shelters or on the streets), upon entry into the program (among those who are both eligible and willing to participate in the evaluation), and quarterly thereafter for up to 3 years, and finally at the time of formal termination from the program. Assessments data are being collected by homeless outreach workers, VA research staff, and client case managers.
Collaboration among network members and comprehensiveness of approaches taken by networks towards ending chronic homelessness in their communities is being measured through interviews with key informants of member agencies at the beginning of the program, and then annually after years 1, 2 and 3 of program operation. Key informants (N=44) of the core agencies (those receiving CHI funding) were interviewed during the fall 2004 to identify up to 9 members of the network expected to be most involved in the implementation of the program. Once identified, one key informant from each of the up to 9 agencies has assessed the level of collaboration among network members and comprehensiveness of approaches taken towards ending chronic homelessness at the beginning of the project (baseline), and then at the end of each year of program operation (follow-up). Seventy-one (71) key informants were interviewed during the first round of network participation interviews (January 2004), followed by 69 and 64 at the end of years 1 (January 2005) & 2 (January 2006) of the project, respectively. Sixty (60) key informants are expected to participate in the fourth and final round of interviews, which are scheduled for January 2007.
Legislative authority for conducting this program and evaluating it include: SAMHSA: Sections 501(d)(4) (290aa), 509 (290bb-2), 520 (290bb-31), Title 42 USC of the Public Health Service Act; HRSA: Section 330 (254b), Title 42, USC, of the Public Health Service Act; HUD: Section 42, § 11301 and11303 of the USC; and VA: Section 38 USCS, § 1710, 1712, and 1722.
2. Purpose and Use of Information
To help determine whether the goals of the initiative are being met, and to help ensure accountability in local spending of the total $35-million of federal funds invested into the initiative, the three sponsoring federal agencies (HUD, HHS, and VA) are funding an independent performance outcomes evaluation coordinated by the VA Northeast Program Evaluation Center (NEPEC).
The goal of the client-level component of the national outcomes performance assessment is to provide a site-by-site description of program implementation, as well as descriptive information on clients served; services received; housing quality, stability, and satisfaction; and, client outcomes in health and functional domains. Because the focus of the program is on a comprehensive approach it is important to comprehensively measure both processes (i.e., delivery of services) and client outcomes across multiple domains.
The same baseline and follow-up data collection instruments will be used with the comparison group. However, the Client Screening Form and Client Discharge Form instruments will be administered only to CHI clients, as they were specifically designed for the CHI program and do not apply to comparison group subjects recruited from other homeless programs.
The information gathered using the four client-level evaluation forms (i.e., screening, baseline, follow-up, and discharge forms) is intended to answer the following evaluation research questions:
1. Are there differences across sites in client characteristics screened through the CHI?
2. In what ways do clients enrolled into the project differ from those screened, but not enrolled?
3. How similar are enrolled client populations across sites?
4. What types of services are received and does service use vary over time?
5. What are the outcomes over time in the areas of housing, employment, finances, physical health, mental health, substance use, coordination of services, criminal justice, social support, religious faith, community integration, and quality of life of client and level of functioning of children in homeless families (where applicable)?
6. Are these results biased by data loss?
7. Which groups/sub-groups obtain more services and have the best outcomes?
8. Which services are more important for which outcomes?
9. How will non-response bias be assessed?
Because the NOFA calls for a comprehensive approach to care in a very heterogeneous population, including clients with disabling mental health, substance abuse, and medical problems in all possible combinations, it is essential that outcomes be well-measured in each of a broad array of domains. The extensive array of measures proposed for the evaluation is necessary to document the diverse accomplishments of the program as described in the NOFA and in reference materials published on the Center for Mental Health Services website during the application period for the grant.
The four client instruments developed for use in this evaluation drew heavily from existing measures used in previous studies of homeless persons, including the ACCESS demonstration program (OMB #0930-0164). Copies of these instruments are provided in Appendix B.
The Access to Community Care and Effective Services and Supports (ACCESS) program evaluated the integration of service systems and its impact on outcomes for homeless persons with severe mental illness. The ACCESS program provided funds and technical assistance to 9 community sites to implement strategies for system change that would promote systems integration. These experimental sites, along with 9 comparison sites, also received funds to support outreach and assertive community treatment for 100 clients a year for 4 yrs at each site. Data on the implementation of system change strategies were collected from 1994 to 1998 during annual visits to the sites. Data on changes in systems integration were obtained from interviews with key informants from relevant organizations in each community. Client outcome data were obtained at program entry and 3 and 12 months later from 7,055 program participants across the 4 annual client cohorts at all sites (Randolph et. al., 2002).
Detailed findings on the final results of the ACCESS evaluation were reported in 2 articles (Morriseey et al., 2002; Rosenheck et al, 2002), and overall conclusions were reported in another article (Goldman et al, 2002). In addition to these four summary articles, approximately 15 additional articles were published from the ACCESS evaluation in a variety of journals during the period 1997-2002.
The table which follows summarizes the various dimensions of client needs, delivery of services, and client outcomes to be assessed using the four client evaluation forms. For each form and section within a given form, the sources used in developing instrument items, and a brief justification of the importance of these items to the evaluation are given.
Form / Section Source(s)
(Completed once by member of homeless outreach team for all persons screened for eligibility, prior to enrollment into the program.)
CLIENT INFORMATION ACCESS Evaluation (Randolph et al, 2002) & Notice of Funding Available
Provides basic demographic and eligibility information for all individuals screened for participation in the program.
CONTACT INFORMATION ACCESS & NOFA
Documents nature of first contact of prospective clients by program staff.
INTERVIEWER OBSERVATIONS ACCESS
Summarizes major clinical problems observed and result of first contact by program staff.
(Completed once by evaluation assistant upon entry into the program for all persons enrolled into the national evaluation.)
ELIGIBILITY INFORMATION NOFA
Documents client’s eligibility to participate in project.
DEMOGRAPHIC DATA ACCESS
Provides basic set of socio-demographic characteristics of clients entering the program.
HOUSING
Lifetime homelessness ACCESS
Measures severity of homelessness experienced during client’s lifetime.
Recent homelessness ACCESS
Important outcome measure. The proportion of clients experiencing recent episode of homelessness is expected to decrease substantially soon after entering the program, and then remain low throughout the remainder of the program due to the provision of permanent supported housing & supportive services provided by the program..
Stability New (informed by ACCESS & CMHS Housing Initiative (CMHS) (Rog, 2003)
Describes proportion of time housed during past 3 months, the various types of living arrangements secured, and reason for leaving last residence if recently moved. Since permanent supported housing is intended to provide a long-term residence, one would expect to observe, over time, a larger proportion of clients living in their own places, fewer moves, and more “positive” reasons for leaving when clients do move.
Description of own place Based on American Housing Survey (AHS, 1990)
These items provide more detailed descriptive information about the living arrangements among clients “successfully” housed in their own apartments, rooms or houses. These data will allow us to compare living arrangements of clients with those of the general population.
Choice Modified Importance of Choice Scale (Tsemberis, Under review)
Consumer choice is an important element in most conceptual models and depictions of supported housing for persons with disabilities. The items in this brief scale (with a simplified response set to reduce respondent burden) will measure the degree of choice reported by each client, which is expected to be positively associated with housing tenure.
Satisfaction RWJ Evaluation (Newman et al, 1994) & AHS
Client satisfaction with current living arrangements, both overall and with specific aspects of housing characteristics and aspects of neighborhood life, is an important proximal subjective outcome measure. Over time, we would expect client satisfaction with housing to increase and to remain higher than reported at the time of entry into the program and exiting from homelessness.
Relationship with landlord Tenant Housing and Neighborhood Survey (Kloos, 2003)
This is a potentially important factor explaining clients’ satisfaction with their housing arrangements, and more objective measures of housing tenure outcomes. We hypothesize that clients reporting more favorable relationships with their landlords will be more satisfied with their housing, and will remain in their housing longer than other clients.
EMPLOYMENT Addiction Severity Index (ASI) (McLellan, 1980)
Distal outcome of interest. Clients will presumably be better able to attain employment, and to maintain employment once attained, after their basic housing and service/treatment needs have been adequately addressed. Thus, we would expect to see employment rates increasing across sites, possibly after the first 12 to 18-months in the program.
FINANCES Quality of Life Interview (QOLI) (Lehman, 1988)
Total income and receipt of public support (e.g., SSI) are important objective measures of clients’ access to mainstream resources. For clients to maintain themselves in independent housing over the long-term, it will be necessary for them to secure some combination of employment & public support income to support themselves financially. Thus, client income and receipt of public support are factors expected to be positively associated with a wide range of client outcomes (e.g., housing tenure & satisfaction, physical and mental health, integration into the community).
PHYSICAL HEALTH
Medical problems ……………………Primary Care Assessment Tool (PCAT) (Starfield, 1998)
Primary self-report measure of physical health needs. The total number of medical problems will be used to adjust outcome analyses for differences in physical health illness severity. Also, the proportion of clients reporting having a given condition and receiving treatment for that condition during the past 3 months will be a basic indicator of quality of physical health care provided through the program.
Perceived health status SF-12 (Ware et al, 1998)
Primarily a general health severity of illness adjuster, but also an important global health outcome measure. This standardized measure will allow us both to adjust for differences in illness severity across sites, and to compare the general health status of CHI clients with those of many other population groups.
Health insurance PCAT
Increasing access to primary care is one of the explicit goals of this program. An important enabling factor for doing so is enrolling clients into some type of health insurance program (e.g., Medicaid, VA). It is expected that clients’ health care coverage will increase over time, and that increased health insurance coverage will be positively associated with both the use of health services (proximal outcome) and health status (distal outcome).
Usual source of medical care PCAT
Along with health insurance coverage, having a usual source of medical care is a useful measure of access to primary health care. Clients having a usual source of care are expected to utilize more health services, and ultimately, to demonstrate better physical health outcomes (i.e., higher scores on the SF-12). Also, reasons for not having a regular source of care will help identify barriers to primary health care.
Primary care provider (PCP) PCAT
These items identify the client’s PCP, providing a person of reference for clients to report their confidence in the following section.
Confidence in PCP. Trust in Physician Scale (Anderson & Dedrick, 1990)
The rapport and trust established between an individual and a health care provider is hypothesized to be an important factor influencing that person’s use of health care services. Thus, we expect that clients who report a high level of trust in their PCP will use more primary health care services than clients who do not have a PCP, or than clients whose level of trust in their PCP is low.
Preventive care PCAT
Another quality of primary care measure. Clinical guidelines indicate that the diagnostic procedures and tests included in this measure should be administered annually. Thus, a greater total number of procedures/tests performed within the past year, will be an indicator of higher quality of primary health care provided.
Health behaviors Health Promotion Lifestyle Profile (Walker, Sechrist & Pender, 1987)
This scale includes self-report information on 5 major public health concerns: smoking, use of alcohol, diet, exercise, and self-examination of breasts (among female clients). The level of concern regarding these public health concerns among clients is a proximal outcome of interest.
MENTAL HEALTH
Mental health problems ACCESS
Primary self-report measure of mental health needs. The total number and types of mental health problems will be used to adjust outcome analyses for differences in mental health illness severity. Also, the proportion of clients reporting having a given condition and receiving treatment for that condition during the past 3 months will be a basic indicator of quality of mental health care provided.
Symptom burden Brief Symptom Inventory (BSI) (Derrogatis & Spencer, 1982)
These items include 3 of 9 sub-scales from one of the most widely used self-report measures of psychiatric symptom distress in the field of mental health research. The 3 sub-scales were selected as those most likely associated with client outcomes, based on findings from the ACCESS project, and include psychoticism, depression, and anxiety. Symptom burden is both a mental health severity of illness adjuster, and a proximal mental health outcome.
SUBSTANCE USE ASI
The prevalence of substance abuse (alcohol and drug) among chronically homeless persons is often high, and typically among the first treatment needs addressed by homeless service providers. We would expect clients’ use of alcohol and drugs during the past 3 months to decline following entry into the program, and relapse rates to be minimized among programs effectively administering permanent housing and supportive services. Thus, substance use is an important proximal outcome of interest.
THERAPEUTIC ALLIANCE Chinman, Rosenheck & Lam (1999)
Therapeutic alliance in the area of mental health and substance abuse treatment is analogous to the PCP and Confidence in PCP measures in the area of physical health. Thus, clients having mental health and/or substance use problems who report having a strong therapeutic alliance with a behavioral health provider/treater may be expected to have better proximal and distal housing and health outcomes than clients having behavioral health needs who lack such a therapeutic relationship.
SERVICE USE New (informed by ACCESS)
Given the intensive, comprehensive approach taken to ending chronic homelessness in this program, it is necessary to measure a wide range of supportive services in the areas of physical health, mental health, substance use, and other services to assess both the intensity and comprehensiveness of the approach taken at each site. Included in this section are items on transportation to core services, and provision of case management services – two factors believed to influence use of services (i.e., clients living closer to services and those having one or more case managers being more likely to use services than other clients).
COORDINATION OF SERVICES New (informed by NOFA)
A high degree of inter-agency collaboration and integrated delivery of housing and supportive services is expected among the estimated 12 sites funded for this project. The few items included in this section of the client interview are intended to validate some of the information collected from agency administrators in the Network Participation interview. We hypothesize that clients who report receiving more coordinated services will show better proximal and distal outcomes.
LEGAL ASI
Having a “criminal record” makes it more difficult to rent an apartment and find employment. Data from ACCESS suggest that lifetime history of criminal conviction is a major predictor (negatively) of client housing outcomes. Also, recent criminal convictions and nights spent in jail are negative proximal outcomes. So these items are included both to risk adjust for baseline differences, and to measure a proximal outcome of interest.
SOCIAL SUPPORT Vaux & Athanassopulou (1987)
Social support -- defined here as the total number of family, friends, and professionals that a client could ask for help in 3 hypothetical need scenarios – represents both risk adjustment factor and a proximal outcome of interest. Clients reporting higher levels of social support (i.e., greater numbers of persons available to help if needed) are expected to show better housing and other outcomes.
CHILDHOOD Helzer (1980); Kadushin, Boulanger & Martin (1981)
Traumatic childhood experiences (e.g., being placed into a foster home, being abused, being expelled from school, abusing alcohol or drugs) have been associated with increased risk for becoming homeless as an adult. These items are included to adjust for baseline differences among clients exposed to such risk factors to differing degrees.
RELIGIOUS FAITH Plante & Boccaccini (1997); Koenig, Parkerson & Meador (1997)
In contrast to traumatic childhood experiences, religious faith is believed to be a protective factor against becoming homeless, and in sustaining recovery among those who were formerly homeless. It is also a major area of policy interest presently. Also included are some items which ask whether religious activity within the past 3 months was practiced voluntarily, or primarily to retain eligibility for housing and/or supportive services – indicating motivation for religious activity. Our hypothesis is that clients expressing religious faith voluntarily will have more sustained improvements in outcomes than other clients.
COMMUNITY INTEGRATION CMHS
A primary objective measure of overall client functioning outcome – a distal outcome of interest. We expect clients with better proximal outcomes (i.e., higher use of services, higher employment rates and incomes, higher quality physical health care, lower levels of psychiatric symptom burden) to eventually show higher levels of integration into community life than other clients.
QUALITY OF LIFE QOLI
A primary objective measure of overall client functioning outcome – a distal outcome of interest. We expect clients with better proximal outcomes (i.e., higher use of services, higher employment rates and incomes, higher quality physical health care, lower levels of psychiatric symptom burden) to eventually show higher levels of integration into community life than other clients.
CONSUMER CHOICE MacArthur Network on Involuntary Treatment (Monahan, 2004)
These items ask whether the client has received mental health services or substance abuse treatment in the past 3 months. If the person answers no, then they skip to subsequent questions. If the answer is yes, then there is a series of 5 questions which ask to what extent the client decided themselves whether to go to treatment. There then is a question about whether anyone was managing their funds, and if they report no then there are no further questions. If they report yes that their funds were managed and some funds were withheld, there is a further series of questions which deal with whether the client felt that the money manager was sincerely trying to help them, what the reasons were for withholding the funds, whether that would actually be helpful, and whether the consumer felt that the withholding of funds was related to pressuring the person to obtain mental health services and/or substance abuse treatment.
VETERAN’S SUPPLEMENT National Evaluation for HCHV Programs (Kasprow et al, 2002)
These items provide descriptive information on veteran clients, who represent approximately one-third of the homeless population. Use of VA services is also included, partly justifying the allotment of clinical FTEE by the VA to serve CHI clients who are veterans.
Note that the percentage disability ratings for medical and psychiatric problems among veteran clients are important for two reasons: disability ratings, along with income, determine eligibility for VA health care services/treatment; and, they provide overall objective measures of illness severity. While some veteran clients may not know or reliably report these ratings, the experience of NEPEC staff suggests that most veterans receiving Veterans Health Administration services are able to reasonably accurately report such ratings. Moreover, these data will not be accessible by NEPEC staff through other administrative or medical records information systems given the exclusive use of anonymous client identifiers. And workloads of local evaluation assistants will be such that verification of client self-report of disability ratings using medical records may not be feasible.
CHILD SUPPLEMENT Children’s Problems Checklist (Schinka, 1985)
An estimated 10% of CHI clients will be homeless parents – presumably mothers mostly – with primary care giving responsibility for one or more minor children. The items in this section of the client interview ask the care giving parent to assess their youngest child’s health, level of self-esteem, and functioning in school. These items are asked only in regards to the youngest child to reduce respondent burden. We hypothesize that children’s functioning will improve as improvements are made in parents’ housing and health status. NEPEC staff reviewed several measures of child problems, including the well-known Achenbach Child Behavior Checklist (CBCL), and consulted with faculty at the Yale University Child Study Center, and ultimately selected 3 sub-scales of the Children’s Problem Checklist as the most appropriate measure of problem behavior among children with chronically homeless and disabled parents. Three sub-scales from this particular measure of child functioning were chosen as being most applicable to chronically homeless families – an opinion confirmed by child development experts at the Yale Child Study Center.
INTERVIEWER OBSERVATIONS Dohrenwend (1982)
These items provide an objective assessment of psychiatric symptomatology based on interviewer observations while the client interview is being administered. Based on items used in ACCESS, the interviewer observations composite score was among the factors most associated (negatively) with client housing and other outcomes. Thus, clients showing a greater number of verbal and non-verbal symptoms at the time of the interview(s) are more likely to show poorer outcomes than other clients.
(Completed by evaluation assistant for all persons participating in the national evaluation every 3 months, for up to 3 years, following the date of the baseline interview. Note: This form is nearly the same as the Baseline Form, except for the exclusion of some historic items/sections, such as the client’s race/ethnicity & childhood experiences.)
CONTACT New (informed by ACCESS)
Documents efforts made by local evaluation staff to maintain contact with clients between interviews, in addition to indicating whether the interview was completed, and if not, the primary reason for not completing the interview.
DEMOGRAPHIC DATA (marital status & minor children items) ACCESS
HOUSING
Recent homelessness ACCESS
Stability New (informed by ACCESS & CMHS Housing Initiative (CMHS) (Rog, 2003)
Description of own place New (informed by American Housing Survey (AHS, 1990)
Choice Modified Importance of Choice Scale (Tsemberis, Under review)
Satisfaction RWJ Evaluation (Newman et al, 1994) & AHS
Relationship with landlord Tenant Housing and Neighborhood Survey (Kloos, 2003)
EMPLOYMENT Addiction Severity Index (ASI) (McLellan, 1980)
FINANCES Quality of Life Interview (QOLI) (Lehman, 1988)
PHYSICAL HEALTH
Medical problems Primary Care Assessment Tool (Starfield, 1998)
Perceived health status SF-12 (Ware et al, 1998)
Health insurance PCAT
Usual source of medical care PCAT
Primary care provider (PCP) PCAT
Confidence in PCP. Trust in Physician Scale (Anderson & Dedrick, 1990)
Preventive care PCAT
Health behaviors Health Promotion Lifestyle Profile (Walker, Sechrist & Pender, 1987)
MENTAL HEALTH
Mental health problems ACCESS
Symptom burden Brief Symptom Inventory (BSI) (Derrogatis & Spencer, 1982)
SUBSTANCE USE ASI
THERAPEUTIC ALLIANCE Chinman, Rosenheck & Lam (1999)
SERVICE USE New (informed by ACCESS)
COORDINATION OF SERVICES New (informed by NOFA)
LEGAL ASI
SOCIAL SUPPORT Vaux & Athanassopulou (1987)
RELIGIOUS FAITH Plante & Boccaccini (1997); Koenig, Parkerson & Meador (1997)
COMMUNITY INTEGRATION CMHS
QUALITY OF LIFE QOLI
CONSUMER CHOICE MacArthur Network on Involuntary Treatment (Monahan, 2004)
VETERAN’S SUPPLEMENT (VA service use items) National Evaluation for HCHV Programs
CHILD SUPPLEMENT Children’s Problems Checklist (Schinka, 1985)
INTERVIEWER OBSERVATIONS Dohrenwend (1982)
The time frame of reference for most items on the Follow-up Assessment Form is the past 30 days or past 3 months, except for: 1) last episode of homelessness (Q10-11), 2) receipt of preventive primary health care services (Q62-64), and 3) psychiatric (Q66) and 4) substance abuse (Q68) diagnoses.
The date of last homeless episode is asked (rather than ‘Were you homeless during the past 3 months?’) to collect more accurate information on this primary outcome among clients unable to be contacted for one or more quarterly follow-up interviews. For example, if a client is unable to be located for 3, 6 and 9-month follow-up interviews, but completes the 12-month interview, a comparison of the date last homeless with the date of the baseline interview (presumably administered soon after enrollment into the clinical program) would indicate whether recurring homelessness has been successfully averted during his/her first year of participating in the program.
During the past year was chosen as the reference period for access to preventive primary health care services because published protocols of such services are recommended annually (versus mental health services, many of which should be provided at least quarterly among individuals with serious mental illness). Again, by comparing client responses at baseline and at completed follow-up interview data points (however complete or sparse these data may be), we will be able to assess what proportion of clients received preventive health care services during their first year or two or three of entering clinical treatment more comprehensively than would be possible using a shorter time period of reference.
Finally, given the high prevalence of mental illness and substance abuse among the disabled, chronically homeless target population, a primary aim of the national evaluation to compare outcomes among major diagnostic groups (e.g., physically disabled, substance abusing, seriously mentally ill, dually diagnosed clients), and the lack of medical record information, we have decided to use both one’s lifetime experience and experience during the past 3 months as dual time frames of reference regarding psychiatric and substance abuse diagnoses. Doing so will allow for comparisons of lifetime prevalence estimates of major diagnostic categories of interest over time, while also providing some information on the timeliness of psychiatric and substance abuse clinical diagnostic assessments after entering the program.
(Completed once by case manager at time of discharge from the program for all persons participating in the national evaluation. Note: Discharge criteria and procedures are expected to vary considerably across sites, along with models of case management and service/treatment.)
ENTRY INTO PERMANENT HOUSING New (informed by NOFA)
Describes the pathway, and date of entry, into permanent supported housing. Inclusion of the pathway item will inform the current debate in the field regarding which of the alternative approaches yields more favorable client outcomes.
HOUSING STATUS AT LAST CLINICAL CONTACT New (informed by NOFA)
These items provide basic information regarding the type of housing clients moved to after leaving the program, which will help answer the question whether clients’ improvements in housing status gained during their participation in the program were sustained after leaving the program (i.e., whether they left and moved into independent housing (a success), a shelter or the streets (a failure)). Street address information will allow evaluation staff at NEPEC to download objective data on the quality of their neighborhood from the 2000 Census.
DISCHARGE FROM PROGRAM VA Supported Housing Program Evaluation (Kasprow, 2002)
Documents the circumstances and reasons for clients’ discharge from the program across sites.
CONTINUING CARE New (informed by NOFA)
Indicates whether arrangements for various types of follow-up care were made prior to discharge from the program – an overall indicator of quality of care provided by program staff.
An underlying principle of the Chronic Homelessness Initiative (CHI) is that a comprehensive approach to the problem of chronic homelessness will result in more systematically planned and managed systems of care, and that such systems will be more effective at helping chronically homeless persons exit from homelessness and improve their lives.
A second assumption is that the network of agencies that joined together to submit an application for the CHI may represent a subset of the larger system of care for homeless persons that antedated the CHI proposal. In characterizing the context of the CHI initiatives it will thus be most important to obtain information on the interactions and activities of the smaller network of agencies directly implementing the CHI, but also some limited information on the larger homeless service system.
The network evaluation consists of two parts: Part I-Network Definition and Part II- Network Participation. Part I was based on a review of the application for each site and discussions with key informants at each of the core agencies involved in the CHI, i.e. the lead agency, and the several other core agencies specified in the NOFA--housing, mental health, substance abuse, primary care and veterans. This survey was used to establish: a) which agencies will be most actively involved in the CHI (including agencies other than the core agencies in the application); b) whether there is one or more broader coalition of homeless service providers in the community in which the CHI will be embedded (i.e. a continuum of care, or coalition of homeless service providers); and c) which of these coalitions will be most relevant to the performance of the CHI. The survey was conducted once, shortly after the sites are identified. Its main purpose was to identify the important agencies in the CHI network at the start of the project in fall 2003.
Once the key informants in the CHI, and the most relevant overarching coalition (if there is one) were identified, in Part I, a more detailed annual survey of agencies participating in the project, Part II, is being conducted with key informants from each of the local agencies constituting the smaller CHI network to assess nine dimensions of operations: 1) the structure, scope and leadership of the network; 2) goals and focus; 3) planning; 4) management and use of information systems, 5) review of key participants at the time of the survey (identifying any additions or exits); 6) connectedness/integration at both the service delivery and leadership levels; 7) use of strategies to improve system integration; 8) the use of various practices and housing strategies, and, 9) involvement of the larger homeless coalition.
These dimensions are based on the language of the original NOFA, with its emphasis on a comprehensive and integrated approach and on recent literature on: 1) developing Continua of Care for homeless persons (Burt et al., 2002), 2) mental health system planning (Magnabosco-Bower, 2001), 3) integrating service systems (Coccozza et al., 2000; Morrissey et al., 2002), and, 4) best practices for serving homeless persons with mental illness (SAMHSA, 2003).
The Network Participation assessment expands upon methods developed for the Center for Mental Health Services’ highly successful ACCESS initiative which demonstrated clear relationships between system integration, the intensive interaction between organizations serving homeless people with serious mental illness, and exits from homelessness at 12 months (Randolph et al., 2002; Rosenheck et al.1998, 2002).
Since the completion of the ACCESS evaluation an important report has been published on the implementation of HUD’s Continuum of Care initiative, which began in 1996. This initiative emphasizes not just integration and interaction, but a specific kind of organization that focuses on comprehensive planning to end chronic homelessness (Burt et al., 2002). These new currents of thought, currents that are richly reflected in the NOFA for the current project, have led to the emphasis in the proposed data collection on documenting the structure, scope and leadership of each network; its goals and focus; its internal planning activities; and its use of management strategies and information systems. Following additional areas of emphasis in the NOFA, we also seek to document the implementation of established evidence-based practices and housing strategies (as defined by CMHS), and involvement with larger homeless coalitions.
The information in Parts I and II of the network evaluation will be collected to answer the following questions:
1. What types of housing and service models are available for chronically homeless people in this community?
2. What is the nature of the interaction between agencies in this initiative? How involved are these agencies with a larger coalition of homeless service providers?
3. Are variations in the above associated with superior client outcomes observed in the client outcomes monitoring effort?
The table which follows summarizes the various dimensions of network structure and inter-agency collaboration to be assessed using the two network evaluation forms. Network evaluation forms were developed based on the considerable experience gained throughout the ACCESS evaluation, and with extensive discussion and consultation with Martha Burt and Joseph Morrissey, two widely recognized national experts on integrated systems of care among networks of homeless service providers. For each form and section within a given form, the sources used in developing the items for that section are provided, along with a brief justification of the relevance of these items to the national evaluation.
Form / Section Source(s)
(Completed once by NEPEC evaluation staff via telephone interview with key informants of core network agencies at the beginning of the program.)
KEY AGENCIES IN CHRONIC HOMELESSNESS INITIATIVE NOFA
Identifies core agencies and partnering agencies most likely to be involved in delivering comprehensive housing and supportive services to chronically homeless clients at each site. Once identified, these agencies (up to 9 total) will constitute the CHI “network” of key service providers.
COALITION IN WHICH THE INITIATIVE IS EMBEDDED New
Describes the larger, pre-existing coalition of homeless service providers in each community from which the CHI network was formed – if applicable.
(Completed annually by NEPEC evaluation staff via telephone interview with key informants of core network agencies and partnering agencies at the beginning of the program, and following years 1, 2, 3 and 4 of program operation.)
STRUCTURE, SCOPE AND LEADERSHIP OF THE CHI NETWORK New
Describes various dimensions of the organizational structure of the network.. It is expected that the level of organization within the network will increase over time.
GOALS AND FOCUS Evaluation of HUD Continuums of Care (Burte et al, 2002)
Identifies the goals of the network, ranking from most to least important. We hypothesize that networks will shift emphasis from obtaining grant funding towards developing integrated systems of care over time.
PLANNING Evaluation of State Public Mental Health System Performance (Magnabosco-Bower, 2001)
Assesses the extent to which the network utilizes a rational planning process for identifying needs of chronically homeless individuals in its community, and providing services to meet these needs. Over time, the level of participation in formal planning activities is expected to increase.
MANAGEMENT New
Perceived effectiveness of the network in identifying and responding to the needs of chronically homeless clients in its community, and the extent to which management information systems are utilized in doing so. The use of management information systems is expected to increase as networks become more established.
PARTICIPANTS IN THE CHRONIC HOMELESSNESS INITIATIVE New
Identifies the key participants comprising the network (as defined in the Network Identification Survey). We hypothesize that the composition of agencies comprising the network will remain relatively stable over time, or possibly adding an agency or two to the network by the end of the project.
CONNECTEDNESS AND INTEGRATION Morrissey et al., 2002
Assesses the extent to which each participant in the network interacts with every other participant in the network in the areas of delivering services, leadership activities, and overall collaboration. Over time, level of participation within the network is expected to increase.
STRATEGIES TO IMPROVE SYSTEM INTEGRATION Coccoza et al., 2000
Assesses the extent to which the network, as a whole, utilizes current state-of-the-art techniques for developing integrated systems of care for homeless clients. As networks develop, we expect these techniques to be utilized to a greater extent.
USE OF BEST CLINICAL PRACTICES & HOUSING STRATEGIES SAMHSA (2003)
Like the previous section, assesses the extent to which the network utilizes current best practices in the provision of housing and delivery of supportive services to persons recovering from homelessness. Over time, the level of use of these practices is expected to increase as systems of care are established and enhanced.
LARGER HOMELESS COALITION New
Assesses the importance and level of support provided to the network by the larger coalition of homeless service providers in the community. Over time, we would expect to find increasing importance and support for the network by the larger homeless coalition, and possibly less clearly distinguished organizational structures and boundaries between the CHI network and the larger homeless coalition near the end of the project.
3. Use of Information Technology
No special information technology is deemed necessary for reducing burden. Web-based data collection was considered, and ruled out due to substantial variability expected across sites requiring skilled and trained local interviewers at the sites (client evaluation), and graduate-level interviewers knowledgeable in the area of inter-organizational theory and behavior (network evaluation) to accurately interpret and clarify responses to the wide variety of questions asked. Also, web-based data collection presents challenging technological barriers among a chronically homeless population. Finally, web-based data collection may raise ethical concerns among human investigations committee reviewers considering the electronic transmission of sensitive clinical information over the internet. Thus, paper-and-pencil questionnaires will be filled-out by trained interviewers locally and at NEPEC at mutually agreeable times, and in semi-private locations whenever possible.
Data entry is being performed by evaluation staff at NEPEC. NEPEC provides local evaluation staff with client database computer software assisting to help monitor the progress of the evaluation by identifying which client-level forms are due, for which clients, by what date.
There are no known technical or legal obstacles to reducing burden.
4. Efforts to Identify Duplication
This is the first national evaluation of client outcomes among chronically homeless individuals and parent caregivers receiving comprehensive housing and support services provided by a collaborative effort of HUD, HHS & VA. Neither the Robert Wood Johnson Foundation Program on Chronic Mental Illness nor the Access to Community Care and Effective Services and Supports (ACCESS) – the two previous national evaluations of chronically mentally ill or homeless individuals – targeted the particularly vulnerable and costly chronically homeless subgroup. And neither involved such collaboration among the three federal agencies primarily responsible for assisting and providing care for the homeless.
There are no alternative sources for information to address the client and network evaluation research questions that are central to this evaluation. The need to perform comparative and cross-site analyses requires that specialized instruments be developed for use across sites. The proposed instruments do incorporate original and adapted items from other instruments used in other studies of homelessness, housing, primary care, mental illness, and substance abuse.
5. Involvement of Small Entities
This evaluation will not have a significant impact on small entities. Respondents for the client level evaluation will be individuals who are homeless. Respondents for the network evaluation will include administrators of human service agencies. These agencies are likely to be larger entities given the demands of the complex grant application, the rigorous review process, and the nature of multi-dimensional intervention. Smaller entities are unlikely to have the experience needed to successfully apply for initiative funds or to be identified as one of the agencies most involved in the implementation of such a program.
6. Consequences if Information Collected Less Frequently
Data on clients have been collected at outreach, upon entry into the program (baseline), every 3 months thereafter for up to 3 years (follow-up), and at discharge (formal termination from the program). These intervals were selected to determine: 1) minimal descriptive characteristics of clients receiving outreach; 2) the important descriptive characteristics of clients at the time they begin treatment; 3) interim (3-month) changes in the client's housing status, progress in getting essential needs met, utilization of services and satisfaction with services; and 4) any longer term (up to 36-month) changes in status, service utilization, engagement in treatment programs, and satisfaction with services.
The experience of clinicians who work with adults who have been homeless suggests that, once a person enters intensive community-based treatment, he/she is likely to use a number of housing and supportive services and may experience some significant improvements in his/her situation. This change can be detected at three months. Experts also note, however, that sustaining an improved situation is often difficult. One of the goals for the project is to improve the durability of positive change for clients. Therefore it is important to capture data at points long enough from the initiation of intensive treatment to determine if the change endures, e.g., whether housing has remained stable, or whether the client has returned to homelessness. Experience in the ACCESS project and other national evaluations suggests that quarterly follow-up contacts with clients is helpful in maintaining contact and monitoring longer-term client outcomes.
Baseline and annual follow-up data collection evaluating both the organizational structure and collaborative process of the agencies providing permanent housing and supportive services to Initiative clients is needed to determine which contextual aspects are most helpful in serving chronically homeless individuals. Based on the experience gained in ACCESS and other projects examining the integration of services for homeless individuals, annual data collection is recommended to effectively monitor the delivery of integrated permanent housing and supportive services among partnering agencies throughout the project.
7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)
The data collection complies fully with 5 CFR 1320.5(d)(2).
8. Consultation Outside the Agency
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on July 28, 2006, Volume 71, Number 145, on page 42849. A copy of the published notice is provided in Appendix D. No comments were received during the 60-day public comment period.
Consultation outside of APSE was sought in planning the evaluation and developing evaluation instruments. An evaluation steering committee (a.k.a., the National Performance Assessment Team) was formed at the beginning of evaluation planning. Representatives from HUD, HHS/SAMHSA, HHS/HRSA, and VA comprise the committee. The table below lists the members of the evaluation steering committee:
Name |
Title |
Organization |
Mary Silveira |
Legislative Analyst |
U.S. Interagency Council on Homelessness (ICH) |
Walter Leginski |
Senior Program Analyst |
Office of the Asst.. Secretary for Planning & Evaluation, HHS |
Frances Randolph |
Chief, Homeless Programs |
Center for Mental Health Services, SAMHSA |
Laurence Rickard |
Project Officer |
Center for Mental Health Services, SAMHSA |
Melissa Rael |
Project Officer |
Center for Mental Health Services, SAMHSA |
Paul Dornan |
Senior Program Analyst |
Office of Policy Development and Research, HUD |
Mark Johnston |
Assoc. Chief |
Office of Special Needs Assistance Programs, HUD |
Marianne Nazzaro |
Program Analyst |
Office of Special Needs Assistance Programs, HUD |
Laura Hogshead |
Program Analyst |
Office of Special Needs Assistance Programs, HUD |
Jean Hochron |
Chief, Homeless Programs |
Health Care for the Homeless Program, HRSA |
Leanne Vannostrand |
Project Officer |
Health Care for the Homeless Program, HRSA |
Lara Ki |
Project Officer |
Health Care for the Homeless Program, HRSA |
James Fagan |
Project Officer |
Health Care for the Homeless Program, HRSA |
Peter Dougherty |
Director |
Homeless Programs Office, VA |
Lucia Freedman |
Program Analyst |
Homeless Programs Office, VA |
Gay Koerber |
Associate Chief Consultant |
Health Care for Homeless Veterans, VA |
Robert Rosenheck |
Director |
VA Northeast Program Evaluation Center (NEPEC) |
Alvin Mares |
Project Director |
VA Northeast Program Evaluation Center (NEPEC) |
Greg Greenberg |
Project Director |
VA Northeast Program Evaluation Center (NEPEC) |
Committee members met twice at the VA Central Office in Washington, DC, to plan the client and network components of the national evaluation, have had several conference calls over the past 2 years, and have participated in the 1st annual SAMHSA grantee meeting in Washington, DC, in July 2004.
In developing client and network evaluation forms, steering committee members also consulted with several national scholars/experts in the areas of service system integration and primary health care for homeless individuals. These consultants are identified in the table below:
Name |
Expertise |
Organization |
Martha Burt, PhD |
Service system integration |
Urban Institute |
Lillian Gelberg, MD |
Primary health care |
UCLA Center for Health Policy Research |
Joseph Morrissey, PhD |
Service system integration |
University of North Carolina at Chapel Hill |
9. Payment to Respondents
Clients receive $15 per interview for remuneration for their time and any personal expenses involved with participating in baseline and follow-up interviews. Remuneration is being provided both to affirm the dignity and significant contribution of clients in this important national evaluation study, and to maximize response rates. The $15 remuneration amount represents the standard in this field of study, and is being paid in cash to all participants following each baseline and follow-up interview through established procedures approved by the VA Agent Cashier at each site.
In most cases, local evaluation assistants have established petty-cash accounts with the VA Agent Cashier with starting balances between $200-$300. As interviews are completed and petty cash funds disbursed, receipts are issued to clients, and evaluation assistants periodically go to the VA Agent Cashier to replenish petty cash funds (i.e., exchange receipts for remuneration paid for additional cash). Procedures are being established concerning the mailing of payments or pick-up of payments by clients with whom follow-up interviews are administered over the telephone.
Key informants at core agencies funded by CHI are expected to participate fully in network interviews administered by NEPEC staff as part of their professional commitment to serving the homeless, and to fulfilling duties assigned to them by their agencies – including fully cooperating with the national evaluation of CHI. Thus, key informants at core agencies will receive no remuneration for their participation in the national evaluation.
Key informants at partnering agencies which receive no direct funding through CHI receive remuneration in the amount of $30 for their time participating in network interviews. The reasons for providing remuneration to this group of respondents are similar to those of the client group; namely, to recognize the importance of their participation in the national evaluation, and to bolster network response rates. The $30 remuneration amount is deemed reasonable compensation given that respondents will likely be mid or higher-level administrators at relatively large agencies in the community – individuals with significant managerial responsibilities and many duties/responsibilities at their agencies. When agency employment policies prohibit remunerating employees contributing to research studies, arrangements will be made to remunerate the partnering agency itself. If agency policy prohibits both employees and the agency itself from receiving remuneration funds, then no remuneration funds will be provided to such an agency.
Incentives are needed to encourage participation and enhance the quality of network-level data provided by potentially less motivated, yet important agency partners in the Initiative. The individuals most involved in writing the grant may not be those most knowledgeable about inter-agency activities/collaboration among direct service agencies. Knowledgeable individuals in direct service agencies receiving no funding through the Initiative (e.g., an administrator in the local Social Security Administration office) may be less motivated to participate in the annual Network Participation Survey than their counterparts in direct service agencies funded through the Initiative (e.g., a supervisor in the lead agency applying for Initiative funding). Thus, we are providing a $30 incentive to Network Participation survey participants of direct service agencies not receiving funding through the Initiative to encourage participation and enhance the quality of network data collected through this project.
NEPEC evaluation staff mail remuneration payments in a manner consistent with VA Connecticut Healthcare System Agency Cashier policies regarding such payments.
10. Assurance of Confidentiality
All requirements regarding the confidentiality of client records contained in 45 CFR Part 46 will be observed. The study protocol will be reviewed and approved by the Institutional Review Board for the Department of Veterans Affairs at West Haven and each site prior to the start of data collection.
Participants in the study beginning at the point of screening to entry into the program are given assurance of confidentiality through a consent form that is provided to them to read, or can be read to them (see Appendix E). Consent may be represented by the client's signature or by consent provided and witnessed by someone other than the interviewer. In addition, the interviewer will provide assurance of confidentiality at the beginning of each interview, and in response to inquiries by the client.
No identifying client information is recorded on any evaluation forms, rather anonymous client identifiers are used to protect confidentiality. Each week local VA research staff fax to NEPEC a list of anonymous client identifiers for clients enrolled into the evaluation during the previous week. All client evaluation data mailed to NEPEC for aggregation, analysis and reporting is stored in locked files. Data processing is being done by NEPEC staff or its agent. Computer data files are password-protected consistent with current Veterans Health Administration policy.
Both NEPEC and local evaluation staff are required to follow the confidentiality protocols specified in each of their locally approved Institutional Review Board (IRB) research protocols.
We considered requesting a “blanket” certificate of confidentiality for all sites through HHS, but decided not to do so for two reasons: 1) to facilitate local IRB approvals, and 2) perceived limited benefit of doing so by NEPEC staff. Blanket certificate of confidentiality approvals require local IRB approvals from all participating sites prior to review by HHS. Given the 6-month variation in time required to obtain IRB approval across sites, and extensive review and re-review processes of the 12 IRBs involved (1 per site, plus a 2nd IRB at Chicago), waiting for all local IRB final approvals to be obtained before applying for a certificate of confidentiality was not practical. However, certificates of confidentiality were obtained at sites required to do so by their local IRBs (i.e., Los Angeles, CA, and Martinez, CA).
In its 15 years of experience evaluating outcomes of thousands of persons who were homeless, NEPEC staff report never being subpoenaed or otherwise having to disclose individual client information to a third party, nor ever hearing of requests for such certification from those participating in these evaluations. Moreover, no such requests for disclosure on individuals participating in the national evaluation by a third parties has been received to-date. Therefore, limited benefit was seen by seeking certificates of confidentiality unless required by local IRBs.
The network component of the evaluation has been approved by the VA Connecticut Healthcare System and Yale University School of Medicine institutional review boards (IRBs).
11. Questions of a Sensitive Nature
The Chronic Homelessness Initiative’s focus on improving the circumstances for individuals and families who are chronically homeless requires that some sensitive information be collected as part of the national performance outcomes evaluation. The purpose for collecting client data is to determine any changes in the client's situation or status that might be attributable to the delivery of comprehensive permanent housing and supportive services or the contextual environments within which programs are operating. Questions about housing, physical and mental health status, alcohol and other drug use, HIV status and contact with the law enforcement system are necessary to assess both services that might be needed as well as to determine whether the client's situation has improved at a later time. Questions about religious faith in general have been included because religious faith is believed to be a protective factor against becoming homeless, and in sustaining recovery among those who were formerly homeless. It is also a major area of policy interest presently.
The questions used to solicit such sensitive information have all been adapted or derived from instruments that have been successfully used in previous studies with related target populations. Anecdotal reports from local evaluation assistants interviewing individuals who are homeless in other national evaluation evaluations administered by NEPEC suggest that clients experience greater discomfort at the end of the evaluation due to discontinuation of the quarterly outcome assessments than they experience as a result of any given follow-up interview. It appears that, in most cases, formerly homeless individuals and family members very much appreciate having someone (i.e., the local evaluation assistant) care enough about their well-being to maintain contact with them and to administer detailed interviews assessing their well-being over a two or three-year period of time.
Study results will not link any sensitive information to individuals and will be reported in aggregate. The identity of individual clients will be protected primarily through the use of an anonymous client identifier, rather than identifying information, on all 4 client evaluation forms mailed to NEPEC. Thus, individual identifying information will be excluded from all national data analysis files created from data collected at each of the sites participating in the national evaluation. Moreover, evaluation assistants will be instructed to keep completed client locator sheets in locked file cabinets when not being used, and to password-protect any computer file linking anonymous client identifiers with any identifying client information. All client data collection, storage, and transmittal procedures used in the national evaluation will be approved by both the VA Connecticut Healthcare System and Yale University School of Medicine Institutional Review Boards (IRBs), in addition to being approved locally by the VA facility participating in the clinical program and/or national evaluation.
12. Estimates of Annualized Burden Hours and Costs
Recruitment for Phase 1 of this project (CHI client respondents) has been completed (N=750). Thus far, 125 participants have been recruited into Phase 2 of the project (comparison group respondents), with an additional 25 of Phase 2 participants expected to be recruited, totaling 150 Phase 2 participants. The estimated annualized burden hours presented in Table 12A therefore includes 25 baseline assessments for Phase 2 respondents, but no such assessments for Phase 1 respondents. Baseline assessment interviews take approximately 1.5 hours, on average, to administer.
Follow-up assessments are being administered quarterly among all respondents (both Phase 1 and Phase 2). However, not all respondents are able to located and assessed every quarter. Over the next 3 years, the annualized follow-up rate among Phase 1 respondents is expected to be 75%, based on observed rates thus far in the evaluation. Moreover, half of Phase 1 are expected to complete the evaluation having participated in the evaluation for a full 3-years. Thus, the estimated annual number of Phase 1 respondents is 563 (750 x 75% follow-up rate), and the estimated number of annual follow-up interview assessment responses per Phase 1 respondent is 2 (4 quarters x 50% on-going participation rate). Follow-up assessment interviews take approximately 1.25 hours, on average, to administer.
The follow-up rate among Phase 2 respondents is expected to be 70%, yielding an estimated 105 Phase 2 respondents (150 x 70%). Because Phase 2 recruitment began only recently and is on-going, only 25% are expected to complete the evaluation during the 3-year extension period. And so, the estimated number of annual follow-up interview assessment responses per Phase 2 respondent is 3 (4 quarters x 75% on-going participation rate).
Clinician respondents will continue to be involved in both screening Phase 2 respondents into the evaluation, and in discharging Phase 1 respondents from the CHI program. An estimated one-fourth (25%) of persons screened by clinicians for Phase 2 of the evaluation are enrolled into the evaluation. The remaining 25 Phase 2 respondents are expected to be recruited from two sites, by one screening clinician per site, totaling 2 screening clinicians. Each clinician is expected to screen 16 persons each year, totaling 32 screenings per year resulting in an average of 8 new Phase 2 recruits per year (32 screenings x 25% recruitment rate). It takes approximately 15 minutes for clinicians to screen each person & to complete a brief screening form on each person screened.
An estimated 20% (N=150) of the 750 Phase 1 respondents will be discharged from the program throughout the 3-year extension period at a rate of 50 respondents per year. Assuming that 25 clinicians across the 11 CHI sites will be responsible for discharging Phase 1 respondents & completing discharge summary forms for each of these respondents, discharge clinicians will complete an average of 2 discharge forms per year. Discharge summary forms require about 25 minutes (0.40 hours) to complete, on average.
Finally, an estimated 20 CHI program administrators (about 2 per site, on average) will be interviewed each year over the telephone to evaluate the level of participation among agencies participating in the CHI project. The time required to administer these telephone interviews is 45 minutes, on average.
The total estimated annualized burden hours is 1,857, and is summarized in the table below:
12A Estimated Annualized Burden Hours
Type of Respondent |
Form Name |
No. of Respondents |
No. Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden Hours |
Subjects: Phase 1 (CHI clients) |
Follow-up |
563 |
2 |
1.25 |
1,408 |
Subjects: Phase 2 (Comparison group) |
Baseline |
8 |
1 |
1.5 |
12 |
Subjects: Phase 2 |
Follow-up |
105 |
3 |
1.25 |
394 |
Clinicians |
Screening |
2 |
16 |
0.25 |
8 |
Clinicians |
Discharge |
25 |
2 |
0.40 |
20 |
Administrators |
Network participation |
20 |
1 |
0.75 |
15 |
Total 1,857 hrs. |
|||||
All instruments previously approved by OMB and used during the past three years have been found to be consistent with this burden estimate.
Using the 2003 federal minimum hourly wage rate of $5.00, and median hourly wage rates in 2000 (obtained through the U.S. Department of Labor website) for social workers ($15.00) and health services managers ($29.00), the estimated annualized burden costs for respondents is $9,925, as summarized in the table below:
12B Estimated Annualized Burden Costs
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Subjects (Phases 1 & 2) |
1,814 |
$5 |
$9,070 |
Clinicians |
28 |
$15 |
$420 |
Administrators |
15 |
$29 |
$435 |
Total $9,925 |
|||
13. Estimates of Annualized Cost Burden to Respondents
No capital, startup, operation or maintenance of services costs are expected.
14. Estimates of Annualized Cost Burden to the Government
Data collection (both client and network) for the national performance outcomes evaluation will be conducted by NEPEC at an estimated annual cost of $600,000. HUD, HHS and VA will each contribute $200,000 annually towards this cost.
After CHI client and comparison group subject 3-year follow-up data collection is completed during the 3-year extension period, these data, along with completed network participation data, will be analyzed in the year following the extension period by NEPEC at an estimated cost of $80,000 to be paid for by NEPEC using other funds.
In addition, the VA will pay an estimated $1,600,000 annually to local VA facilities participating in the initiative (N=11) in support of clinical and evaluation costs, including salary, transportation, and program-related equipment expenses.
The total estimated annual cost to the government is $2.2-million for data collection.
15. Changes in Burden
The estimated average annual hour burden has decreased by a total of 1,928 hours, from the currently approved hours 3,785 to the requested amount of 1,857. This decrease is due to 1) a majority (over 60%) of 2-year follow-up data collection among CHI clients having been completed by October 2006, and 2) completion of all network identification interviews & two out of three years’ of network participation interviews having been completed. These reductions more than offset increases in hour burden associated with the recruitment & collection of 3-year follow-up data among comparison group subjects.
16. Time Schedule, Publication and Analysis Plans
The estimated time schedule for the extension period of the project is summarized below:
1. Network participation interviews completed Mar. 2007
2. 3-year follow-up interviews completed among CHI clients Jan. 2008
3. 3-year follow-up interviews completed among comparison group subjects Oct. 2009
4. Data analysis completed for final report to federal agencies et. al. June 2010
5. Publication writing begins ……………………………………… ……….July 2010
6. Final report completed & distributed……………………………………… ……….Dec. 2010
Preliminary descriptive publications. Several preliminary publications will be developed during years 2 and 3 of the project, and will address the following descriptive aspects of this initiative:
1. Client characteristics and needs in a multi-site study of chronically homeless people with disabilities.
2. Patterns and predictors of service use in a multi-site study of chronically homeless people with disabilities.
3. Dimensions and determinants of housing satisfaction in a multi-site study of chronically homeless people with disabilities.
4. Access and quality of primary health care service use in a multi-site study of chronically homeless people with disabilities.
Core outcome publications. The following primary publications will be developed after data collection has been completed, beginning in year 4 of the project:
1. The relationship of service utilization to housing and health outcomes in a multi-site study of chronically homeless people with disabilities. (This may, in fact, require several papers addressing housing and various aspects of health and community adjustment separately)
2. Evolution of homeless service networks in the Federal Joint Initiative to End Chronic Homelessness. (This study would describe the key dimensions of network operation and how they changed over the years of this initiative)
3. System operation and outcomes of chronically homeless individuals with disabilities. (This synthetic paper would present the results of joint analysis of network and client data).
Three sets of analyses will be used to evaluate the CHI program: client analyses in which individual clients will be the unit of analysis (N=750); network analyses in which the local service networks will be the unit of analysis (N=11); and, combined client-network analyses in which network data will be merged with individual client data. A brief summary of the various types of analyses planned follows, organized by analysis type (i.e., client, network or combined), and research question.
1. Are there differences across sites in client characteristics screened through the CHI?
Using data from the Client Screening Form, basic demographic and clinical characteristics of all individuals contacted will be described overall, and for each site, using simple descriptive statistics (i.e., frequencies, percentages, means, standard deviations). Analysis of Variance and Chi square tests will be used to identify differences in client characteristics across sites.
2. In what ways do clients enrolled into the project differ from those screened, but not enrolled?
Demographic and clinical characteristics of individuals screened but not enrolled into the project will be compared with those of individuals enrolled into the project using data from the Client Screening Form using chi-square and independent samples t-test statistical techniques. Logistic regression will then be used to identify characteristics that independently differentiate clients who were only screened from those actually enrolled.
3. How similar are enrolled client populations across sites?
To compare enrolled clients across sites, mean and percentage statistics for basic demographic and clinical information available from the Client Baseline Assessment Form will be compared using one-way analysis of variance (ANOVA) and chi-square statistical tests.
4. What types of services are received and does service use vary over time?
Longitudinal self-report service use data will be obtained from the Client Baseline Assessment Form and the Client Follow-up Form in the areas of housing services and subsidies, physical health and mental health services, substance abuse treatment, case management, and other supportive services. Both dichotomous and continuous measures for each type of service and for combinations of services (e.g. all mental health-related outpatient care) will be created by summing or averaging individual items within a given type of service. Descriptive statistics will then be used to summarize these dichotomous and continuous service use measures.
Repeated measures mixed models will then be used to identify changes over time in patterns and intensity of service use. In these models each assessment available from each client is included and linked to a variable representing the elapsed time since the baseline assessment. These models allow us to use all available data from each client since the random effects adjust the standard errors for the correlations of measures across subjects, due to the fact that there are multiple observations per subject (Bryk & Raudenbush, 1992). In repeated measures mixed models, time frames are categorized (e.g. 1-6 months, 7-12 months etc) and treated as a ordinal variable allowing statistical comparisons of service use across times points.
5. What are the outcomes over time in the areas of housing, employment, finances,
physical health, mental health, substance use, coordination of services, criminal justice, social support, religious faith, community integration, and quality of life of client and level of functioning of children in homeless families (where applicable)?
Similar to the analysis of use of services, both dichotomous and continuous measures for each of these outcomes will be created using data from the Client Baseline Assessment Form and from the Client Follow-up Form. As with service use variables, random effects mixed models will be used to compare the significance of changes in outcomes across time points. These analyses will allow us to determine how long it takes to achieve these outcomes and whether they are sustained.
6. Are these results biased by data loss?
It is likely that there will be some data loss over time in this population (in ACCESS outcome data were obtained from over 90% of subjects. Marginal structural models (Robins & Finkelstein, 2000) will be used to adjust the previous two sets of analyses for differential data loss among clients with different characteristics. In the first step of these models a survival analysis is used to model successful follow-up at each time point, and to use baseline characteristics to estimate the predicted probability that each interview will be conducted. Observations will then be weighted by the inverse of the predicted probability that they would be obtained so that interviews completed with people whose baseline characteristics suggest a low probability of follow up are given more weight than likely ones.
Chi-square, t-test, analysis of variance (ANOVA), and analysis of covariance (ANCOVA) statistical analyses will also be used to compare differences among those remain in the study and those who drop out or are unable to be located.
7. Which groups/sub-groups obtain more services and have the best outcomes?
To answer this question the service use and outcomes models described above (in #5 and 6) will be re-run with covariates representing measures of socio-demographic and clinical characteristics (e.g. age, race, duration of homelessness, disabling condition) and site. These models will allow us to determine which client characteristics are most strongly associated with desired clinical outcomes and which may pose barriers to successful outcomes.
Moreover, comparisons of service use patterns and participant outcomes will be compared between groups (i.e., CHI clients vs. comparison group subjects), adjusting for differences between groups at baseline (entry into the evaluation).
8. Which services are more important for which outcomes?
To answer this question the random effects mixed models of outcomes will be re-run including pertinent measures of service use as independent variables along with baseline covariates that may potentially confound the analysis because they are associated with both service use and outcomes.
9. How will non-response bias be assessed?
Non-response bias will be assessed by comparing available data on baseline characteristics of four groups: 1) those screened but who are not served by the program, 2) those served by the program but who do not wish to participate in the national evaluation, 3) those who begin participation in the national evaluation but who drop-out after enrolling, and 4) those who are enrolled and complete participation in the national evaluation. Limited baseline data obtained from the Screening Form will be used in comparing the first two groups. More extensive baseline data from the Baseline Assessment Form will be used in comparing the later two groups. Baseline differences, to the extent known, will be statistically adjusted for in multivariate analyses as described above.
10. What types of housing and service models are available for chronically homeless people in this community?
Longitudinal data (i.e., baseline and annual follow-up data) assessing the availability of various types of housing (e.g., emergency shelter, transitional housing, affordable housing without services, permanent housing with services) and the use of 16 types of evidence-based services (practices) will be collected from the Network Participation Form.
The reliability of assessments made by key informants at a given site will be measured using a kappa statistic. If the level of agreement among key informants is generally high (e.g., kappa > 0.6), then responses will be averaged across key informants to generate a single availability rating for each type of housing and service model for each site. If, however, the level of agreement is generally low, then site ratings will either be based on a given key informant believed to be most knowledgeable (e.g., the lead agency), the average rating for the 3 agencies with the largest kappa’s, the average rating among the core agencies, etc.
Once measures of availability have been created, descriptive statistics will be used to summarize the availability of housing and services to the chronically homeless at each community at the start of the project and as the program evolves over time.
11. What is the nature of the interaction between agencies in this initiative? How involved are these agencies with a larger coalition of homeless service providers?
These questions will be answered in a manner similar to the previous question. The Network Participation Form will also provide longitudinal data on both the nature and level of interaction among network agencies, and also the level of involvement of the network with any larger homelessness coalition which may exist.
Factor analysis will be used to identify measures that address common dimensions of network interaction across respondents. Factor analyses will be used to create scales (unit weighted) of items that loaded highly on each factor to measure various dimensions of inter-agency collaboration. Then kappa statistics will be used to assess the reliability of these ratings among key informants at a given site. Finally, site-level measures of inter-agency collaboration and connectedness to the broader homeless coalition will be created based on individual ratings by key informants.
Once created, these measures will be summarized for each site using descriptive statistics.
Combined Client-Network Analyses
12. Are variations in the above associated with superior client outcomes observed in the client outcomes monitoring effort?
After merging network level data with the client outcome data, another set of random regression models will be used to identify relationships between levels of system integration/organization and network operation and client outcomes. Thus we will identify measures of site characteristics such as the quality of communication, information sharing, trust and funds exchanges between sites and merge these with the client-level data set. In these models system-measures will be the independent variables of primary interest, and covariates will include potentially confounding baseline characteristics associated with both outcomes and site characteristics. Three-level hierarchical linear modeling will be used to address the clustering of data at both the individual and site level. Thus each model will include fixed effects such as independent variables reflecting inter-site cooperation, and random effects for observations within individuals for individuals nested within sites. PROC MIXED of SAS will be used for these analyses.
17. Display of Expiration Date
The expiration date for OMB approval will be displayed.
18. Exceptions to Certification Statement
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
B. Statistical Methods
1. Respondent Universe and Sampling Methods
The respondent universe for clients includes all 1) individuals entering the program (treatment group) and 2) individuals receiving less intensive package of housing and supportive services through other programs serving homeless individuals (comparison group) who voluntarily agree to participate in the national evaluation and who provide written informed consent to do so. The inclusion criteria are the same as those of the clinical programs from which participants are recruited (i.e., the CHI program and comparison group program(s) at each site). There are no exclusion criteria.
Individuals may participate in these programs, but decline participation in the national evaluation since participation in the evaluation will be voluntary. Thus far, about 90% of CHI clients have agreed to participate in the national evaluation, with the remuneration of $15 per evaluation interview being a strong incentive for doing so. An equivalent evaluation participation/enrollment rate is expected among comparison group participants.
The expected number of persons enrolled into the program and evaluation at each site as of October 2006 is 750 (86%) CHI clients (treatment group), plus 125 (14%) “usual care” comparison group subjects recruited from other homeless programs. A relatively small number of persons at each site have been served by the CHI program (N=68, on average) due to the considerable cost of providing persons who are chronically homeless with permanent housing and a comprehensive array of supportive services needed to sustain housing tenure and to promote self-sufficiency among this target population, and limited federal funds available for the program. The relatively smaller number of comparison group subjects than CHI clients is due to 1) half of the CHI sites not being able to recruit a comparison group sample due to limited research staffing for the project and/or a lack of suitable homeless programs from which to recruit comparison group subjects, and 2) limited amount of data collection time & federal resources allocated to the national evaluation.
The relatively minimal eligibility criteria for the program with no additional inclusion or exclusion criteria added by the national evaluation, and the inclusion of a comparison group, will enhance the generalizability of evaluation findings among the source population of chronically homeless individuals and parents with disabling medical and/or mental health/substance abuse conditions.
Sampling for treatment group participants among CHI clients is not appropriate in this project given 1) the limited number of permanent housing units provided through the program, 2) the heterogeneity of the target population (e.g., single individuals and parents, prolonged homelessness and repeated episodes of homelessness, seriously mental disorders and chronic medical conditions), and 3) the federal mandate to collect outcomes data for all CHI clients willing to participate in the national evaluation.
Convenience sampling among clinically similar users of homeless services, as perceived by local VA research staff, has been used primarily in selecting participants for the comparison group at each site. Probability sampling has, for the most part, not been possible due to the “walk-in” nature of homeless services and self-referral method of accessing services from such service providers. Persons utilizing services from programs used to recruit comparison group subjects are, for the most part, not scheduled to come-in in advance, and present themselves to the community health clinics or homeless shelters or soup kitchens in an unpredictable and unscheduled manner. Thus, there has been no practical way to use probability sampling for either group in this project, given the transient nature of the target population of chronically homeless individuals.
Key Informants
The respondent universe for key informants are CHI network agency staff most knowledgeable of CHI implementation and activities at each site. Key informants were selected by first defining the network of CHI agencies at each site (the primary purpose of the Network Identification Form). This was done by first reviewing funded grant applications which identified the core agencies that are receiving federal funds through CHI. Some partnering agencies have also been identified through a review of grant applications.
After reviewing the grant applications, NEPEC staff contacted core agency staff listed in the applications, and asked these core agency contact persons to identify one staff person in their agency likely to be most knowledgeable of CHI operations/activities within their agency. NEPEC staff then contacted the key informants identified through reviewing grant applications and speaking with agency contacts listed in the applications, and scheduled a time to administer the network identification interview over the phone.
While interviewing key informants of core agencies in the CHI network, NEPEC staff identified key informants at partnering agencies in the CHI network at each site, and then contacted these individuals to complete Part I of the network evaluation – the purpose of which was to identify up to 9 agencies that will be most involved in implementing the program, and the single key informant from each of these agencies, at each site. These key informants (one from each agency) have been the respondents for Part II of the network evaluation (i.e., annual network participation interviews).
Forty four (44) network identification interviews and 71 “baseline” network evaluation telephone interviews were administered by NEPEC staff between October 2003 and February 2004. Two additional rounds of interviews were completed in January 2005 (N=69) at the end of year 1 of the project, and in January 2006 (N=64) at the end of year 2 of the project. A fourth and final round of interviews is scheduled for January 2007, at the end of year 3 of the project, at which time an estimated 60 participation interviews will be conducted.
Sampling of key informants was not appropriate in this project given that key informants have been identified and selected based on their expert knowledge of the CHI program – knowledge which few agency staff have.
2. Information Collection Procedures
Client data is being collected by three sets of people: homeless outreach workers (Client Screening Form), evaluation assistants (Client Baseline Assessment and Follow-up Forms), and case managers (Client Discharge Form). Data collection procedures for each of these three sets of people are described below
Homeless outreach workers. At each site, homeless outreach workers are recruiting and screening chronically homeless individuals from one or two portals of entry into the program (e.g., a park, shelter, soup kitchen). Outreach workers document all recruitment and screening efforts for the program by completing a Client Screening Form for each significant clinical contact. They then fax completed forms to evaluation assistants for review and follow-up when indicated (i.e., individual eligible and interested in participating in evaluation).
Evaluation assistants. A full-time, GS-9 level evaluation assistant has been hired by the VA health care facility member of the local CHI network (or another agency member in the absence of a VA facility). NEPEC staff have trained evaluation assistants on the collection and transmittal of client data. This training has included: 1) the development of an evaluation instruction manual; 2) administration of mock baseline interviews to staff and practice baseline interviews with previously homeless veteran patients receiving VA services; and, 3) convening of a national training meeting held in Los Angeles, California, in March 2004. NEPEC staff continue to provide on-going training/monitoring/support for evaluation assistants through regularly scheduled conference calls, and individual telephone calls, as-needed.
Evaluation assistants review faxed Client Screening Forms as they are received, and follow-up with eligible and interested individuals in-person, at a mutually agreed upon location, date and time. If the initial meeting between the evaluation assistant and the individual is to take place in the community where personal safety is a concern, the evaluation assistant may arrange to be accompanied by the referring outreach worker or another VA staff person.
During the initial meeting with the prospective participant, the evaluation assistant introduces him/herself, provides the individual with an overview of the evaluation, asks if the individual is interested in participating in the evaluation, and if so, schedules the baseline interview for a later date in an office setting (see Appendix F). Scheduling the baseline interview in an office setting serves the purpose of requiring prospective participants to demonstrate a minimal level of interest by keeping a scheduled appointment in an agency setting, thereby screening-out individuals who may be willing to answer a few questions at the initial meeting in exchange for a $15 remuneration, but who have little or no intention of actually enrolling into the program or participating in the evaluation in the future.
During the office visit following the initial meeting, the evaluation assistant obtains informed consent (See Appendix E), administers the baseline assessment interview, completes the client locator sheet to assist in contacting the client to schedule follow-up interviews, and pays the client $15 for his/her time spent traveling to and from, and participating in, the baseline interview.
After administering baseline interviews, evaluation assistants begin contacting participants 4-6 weeks prior to the due date of the next follow-up interview. These subsequent interviews are scheduled at a time and location of mutual convenience for clients and evaluation assistants. Evaluation assistants continue attempting to locate participants until such time that either the evaluation ends, or participants are terminated from the evaluation in cases of death or rescinding of consent to participate in the evaluation.
When follow-up interviews are unable to be completed, evaluation assistants complete the first page of the Client Follow-up Form documenting attempts made to contact the participant. Otherwise, follow-up interviews are completed as close to the due date as possible (i.e., 3, 6, 9, 12…months following the date of the baseline interview).
Evaluation assistants mail all baseline and follow-up interview forms completed by themselves, and all screening forms completed by outreach workers and discharge forms completed by case managers (described below), to NEPEC by the last day of each month.
Case managers. Depending on the model of permanent housing and supportive services developed at each site, specific criteria for discharging an individual from the program were established early in the life of the program. Typically, a person is “discharged” when he/she willfully relapses into homelessness, refuses further treatment and/or is not likely to have any further significant clinical contact with program staff. At such time that individuals are discharged from the program, a case manager at one of the core agencies in the CHI network will complete a Client Discharge Form and fax the completed form to the evaluation assistant for transmittal to NEPEC.
Note that clients discharged from the program will continue to be followed-up on by evaluation assistants until they are terminated from the evaluation; namely, until death, revocation of consent, or the end of the data collection phase of the evaluation.
Detailed client data collection procedures are currently being developed with NEPEC, VA research staff, and CHI partnering agencies at each site. To the extent possible, recruitment and data collection procedures among comparison group participants will mirror those of treatment group participants. For example, the same four evaluation instruments will be used with both groups to facilitate comparisons of outcomes between groups.
The previously approved client consent form will be used to obtain consent from both groups, given that both groups are being/will be recruited to participate in a study evaluating the effectiveness of services for individuals who are homeless with disabling condition(s), and that the data collection procedures, risks, benefits, etc. of participating in the study will be the same for both groups.
However, the evaluation assistant script used to recruit comparison group participants will differ from the script used to recruit CHI clients. A script for comparison group participants is included in Appendix G.
Network data are being collected by NEPEC staff through two types of telephone interviews with key informants: one-time network identification interviews at the start of the project, and repeated network participation interviews administered soon after the network identification interviews (baseline) and annually thereafter (follow-up).
Network identification. As key informants were identified (described in previous section), NEPEC sent a cover letter of introduction along with a copy of the Network Identification Form (see Appendix I), and followed-up with 2-4 weeks later to schedule a time to complete the network identification interview over the phone (see Appendix J). A total of 44 network identification interviews were completed in the fall 2003. No further network identification interviews are anticipated.
Network participation. A similar data collection procedure was used with the network participation interview. NEPEC mailed an introductory cover letter along with a copy of the Network Participation Form to up to 9 key informants at each site (see Appendix K). Then the NEPEC staff member responsible for administering the network participation interview followed-up within 2-4 weeks to answer any questions the respondent had and schedule a date/time to administer the interview. Seventy-one (71) initial (baseline) network participation telephone interviews were administered in January 2004. Subsequent (follow-up) interviews were administered in January 2005 (N=69) and in January 2006 (N=64). A final round of follow-up interviews is scheduled for January 2007, and is expected to include 60 key informants.
Key informant participants in the previous year’s interview have been contacted in October or November of the following year to schedule the next annual follow-up interview.
3. Methods to Maximize Response Rates
Maximizing client response rates entails taking all action to assure that clients can be located quarterly to complete follow-up assessment interviews. Methods used thus far for doing so have included (and will continue to include) the following: 1) paying clients for interviews as fair compensation for their time and effort; 2) keeping in frequent contact with clients by providing comprehensive housing and supportive services; 3) interviewing clients in-vivo -- in settings they choose; and, 4) knowing locations clients have frequented in the past and having the names, addresses, and telephone numbers of friends/family who may know their whereabouts. The interview has also been kept to a minimum to make response less burdensome. In addition, the interviews can be partitioned so that the interview can be interrupted and continued at a later meeting.
In order to maximize response rates for organization representatives, interviews will be scheduled to be as responsive as possible to respondent time constraints. Pre-interview materials are expected to cut down on interview time. If respondents are not fully prepared during an interview, arrangements will be made to follow-up and collect missing data at a later time.
4. Tests of Procedures
There are no changes to any of the evaluation procedures previously approved by OMB proposed for the 3-year extension period.
5. Statistical Consultants
The following persons consulted on the design of the CHI evaluation:
Name |
Title |
Organization |
Phone |
Greg Greenberg, PhD |
Project Director |
NEPEC |
(203) 932-5711 x5585 |
James Herrell |
Senior Program Analyst |
Center for Substance Abuse Treatment (CSAT), SAMHSA |
(301) 443-2376 |
Alvin Mares, PhD |
Project Director |
NEPEC |
(203) 932-5711 x3618 |
Frances Randolph, DrPH |
Director |
Homeless Programs Branch, Center for Mental Health Services (CMHS), SAMHSA |
(301) 443-8613 |
Robert Rosenheck, MD |
Director |
NEPEC |
(203) 937-3850 |
American Housing Survey (AHS). Codebook for the American Housing Survey. Cambridge, MA: Abt Associates, Inc., 1990.
Anderson LA, Dedrick RF. Development of the Trust in Physician Scale: A measure to assess the interpersonal trust in patient-physician relationships. Psychological Reports, 67, 1091-1100, 1990
Bryk AS, Raudenbush SW. Hierarchical Linear Models. Newbury Park, CA: Sage Publications Inc. 1992.
Burt M, Pollack D, Sosland A et al.. Evaluation of Continuums of Care for Homeless People: Final Report. Washington, DC, The Urban Institute, 2002.
Chinman M, Rosenheck RA, Lam J. The development of relationships between homeless mentally ill clients and their case managers. Psychiatric Rehabilitation Journal 23(1):47-55, 1999.
Cocozza JJ. Steadman HJ. Dennis DL. Blasinsky M. Randolph FL. Johnsen M. Goldman H. Successful systems integration strategies: the access program for persons who are homeless and mentally ill. Administration & Policy in Mental Health. 27(6):395-407, 2000.
Derogatis LR, Spencer N. The brief symptom index; Administration, scoring and procedure manual, Baltimore MD, Johns Hopkins, 1982.
Dohrenwend B. Psychiatric Epidemiology Research Interview (PERI). New York: Columbia University Social Psychiatry Unit, 1982.
Goldman, H.H., Morrissey, J.P., Rosenheck, R.A., Cocozza, J., Blasinsky, M., Randolph, F. Lessons from the evaluation of the ACCESS program. Psychiatric Services , Vol 53(8), Aug 2002. pp. 967-969.
Helzer J. Methodological issues in the interpretations of the consequences of extreme situations, in Stressful Life Events and Their Contexts. Edited by Dohrenwend B, Dohrenwend B. New York, Prodist, 108-129, 1980.
Kadushin C, Boulanger G, Martin J. Long term reactions: Some causes, consequences, and naturally occurring support systems. Legacies of Vietnam; IV (House Committee Print No. 14), 1981.
Kasprow WJ, Rosenheck RA, DiLella D, et al: Health care for homeless veterans programs: The 15th annual report. West Haven, CT: Northeast Program Evaluation Center, 2002.
Kloos B. Personal communication, 2003.
Koenig H, Parkerson GR, Meador KG. Religion Index for Psychiatric Research. American Journal of Psychiatry, 153:6, 885-886, 1997.
Lehman A. A quality of life interview for the chronically mentally ill. Evaluation and Program Planning 11:51-62, 1988
Magnobosco-Bower JL. An evaluation of state public mental health system performance for adult persons with serious mental illness: Effects of state political culture and state mental health planning & implementation characteristics on state public mental health system comprehensiveness. Doctoral thesis. Columbia University, 2001.
McLellan A, Luborsky L, Woody G, et al.: An improved diagnostic evaluation instrument for substance abuse patients: the Addiction Severity Index. Journal of Nervous and Mental Disease 168:26-33, 1980
Monahan, J. Personal communication. June, 2004
Morrissey JP, Calloway M, Thakur N, Cocozza J, Steadman H, Dennis DD, Rosenheck RA, Randolph F, Johnsen M, Blasinsky M, Goldman HH. Can Communities Improve Service Systems Integration for Homeless Persons with Serious Mental Illness? Psychiatric Services 53(8):949-957, 2002.
Newman SJ, Reschovsky JD, Kaneda K, et al. The effects of independent living on persons with chronic mental illness: An assessment of the Section 8 Certificate program. Milbank Quarterly 72:171-198, 1994.
Notice of Funding Availability for the Collaborative Initiative to Help End Chronic Homelessness. Federal Register, Vol. 68, No. 17, January 27, 2003.
Plante TG, Boccaccini MT. The Santa Clara Strength of Religious Faith Questionnaire. Pastoral Psychology 45:5, 375-387, 1997.
Randolph F, Blasinsky M, Morrissey JP, et al: Overview of the ACCESS program. Psychiatric Services 53:945-948, 2002
Randolph, F., Blasinsky, M., Morrissey, J.P., Rosenheck, R.A., Cocozza, J., Goldman, H.H. Overview of the ACCESS program. Psychiatric Services , Vol 53(8), Aug 2002. pp. 945-948.
Robins JM, Finkelstein D. Correcting for noncompliance and dependent censoring in an AIDS clinical trial with inverse probability of censoring weighted (IPCW) Log-Rank tests. Biometrics 56:776-788, 2000.
Rog D (P.I.). Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) Housing Initiative. URL: http://www.aspe.hhs.gov/progsys/homeless/programs.htm (2003)
Rosenheck, R.A., Lam, J., Morrissey, J.P., Calloway, M.O., Stolar, M., Randolph, F. Service systems integration and outcomes for mentally ill homeless persons in the ACCESS program. Psychiatric Services , Vol 53(8), Aug 2002. pp. 958-966.
SAMHSA. Best practices: An overview of mental health and substance abuse services and system coordination strategies. URL: http://www.samhsa.gov/grants/content/2003/Resource%20material.pdf (2003)
Schinka JA. Children’s Problems Checklist. Odessa, FL: Psychological Assessment Resources, Inc., 1985.
Starfield B. Primary care: Balancing health needs, services, and technology. New York, NY: Oxford University Press, 1998.
Tsemberis S. Housing satisfaction. Under-review.
Vaux A, Athanassopulou M. Social support appraisals and network resources. Journal of Community Psychology 15:537-556, 1987
Walker SN, Sechrist KR, Pender NJ. The health promotion lifestyle profile: Development and psychometric characteristics. Nursing Research 36, 76-81, 1987.
Ware JE, Kosinski M, Keller SE. How to score the SF-12 physical and mental health summary scales, 3rd edition. Lincoln, Rhode Island: Quality Metric Inc. 1998
Appendices A-L (See PDF file: Instrument pdf (384k)
Appendix M. CICH evaluation and follow-up rates report (June-2006), File Name: CICH Eval Stats—2006-06 pdf (46KB).
| File Type | application/msword |
| File Title | SUMMARY |
| Author | VHACONMARESA |
| Last Modified By | DHHS |
| File Modified | 2006-10-18 |
| File Created | 2006-09-29 |