THE SOAR EVALUATION
The Supporting Statement
A. Justification
The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting approval from the Office of Management and Budget (OMB) for the following data collections to support an evaluation of the Supplemental Security Income (SSI)/Social Security Disability Income (SSDI) Outreach Access and Recovery (SOAR) initiative:
In-person interviews with state and local SOAR stakeholders;
A social network survey of state and local SOAR stakeholders;
Focus groups with participants of in-person SOAR trainings;
Evaluative materials completed by participants of in-person SOAR trainings, including pre/post training evaluation forms and a customer satisfaction survey;
Evaluative materials completed by users of web-based SOAR trainings, including pre/post training evaluation screens and an online customer satisfaction survey.
The SSI and SSDI programs provide cash assistance to individuals with qualifying disabilities. Individuals apply to receive benefits and eligibility is contingent on having a medical condition that meets SSA’s definition of disability. SOAR aims to increase access to these programs among people who are homeless and thereby improve their quality of life. The crux of the initiative is the provision of technical assistance (TA) to help states and local communities develop strategies to increase the quality of SSI/SSDI applications homeless individuals submit. The TA includes:
Strategic planning to help states develop policies and procedures that will aid those who are homeless obtain SSI/SSDI. TA helps social service providers, advocates, and state and local agencies work together to determine how to create an effective system for obtaining SSI/SSDI. Participating organizations typically include SSA, DDS, state health and mental health agencies, state homeless services coordinating councils, and local providers of homeless and mental health services. SAMHSA will collect data from these stakeholders through in-person interviews and a brief social network survey.
Training for staff who work with homeless individuals on how to support them through the SSI/SSDI application process. SOAR employs a train-the-trainer model in which states identify a few individuals to receive training on the Stepping Stones to Recovery (SSR) curriculum. These individuals then conduct in-state in-person trainings on the SSR curriculum for front-line workers—that is, case managers, social workers, and other types of staff who work directly with individuals who are homeless. The SSR curriculum emphasizes several key strategies that may help increase and expedite SSI/SSDI application approvals, such as serving as an applicant’s representative during the application process and working closely with health care providers to obtain medical documentation. In-state training participants are typically staff from homeless shelters and service organizations, mental health agencies, and health care facilities. Beginning in 2011, SSR training will also be available via the web. SAMHSA will collect data from training participants through pre- and post-training evaluation forms and a customer satisfaction survey.
Follow-up support to help communities expand their efforts and learn from the successes and challenges of other communities. Each state receives ongoing TA and monitoring of strategic action plan implementation for one to two years. The program also disseminates promising practices and maintains a website with materials and tools for use by SOAR communities.
SOAR has the potential to substantially increase the ease and speed with which individuals who are homeless can access SOAR benefits. Determining whether it is successful in this endeavor requires the collection and analysis of data on how SOAR was implemented and what outputs and outcomes result.1 Two sections of the Public Health Service (PHS) Act authorize SAMSHA to collect the requisite data: Section 505 (42 USC 290aa-4) and Title 42, Chapter 6A, Subchapter III-A Substance Abuse and Mental Health Services Administration, Part A: Organization and General Authorities.
To evaluate SOAR’s implementation and outputs, SAMHSA will collect data from the following sources in 13 local communities that will begin receiving federally-funded SOAR TA for the first time in 2010 or 2011:
Interviews with state and local SOAR stakeholders (such as state and local SOAR leaders, SSA administrators and staff, DDS administrators and staff, SSR trainers, and administrators and staff from state mental health agencies, housing and other public assistance agencies, homeless service providers, and community-based agencies);
A social network survey of state and local SOAR stakeholders;
Focus groups with participants of in-person SOAR trainings;
Evaluative materials completed by participants of in-person SOAR trainings, including pre/post training evaluation forms and a customer satisfaction survey.
Evaluative materials completed by users of web-based SOAR trainings, including pre/post training evaluation forms and a customer satisfaction survey.
SAMHSA is seeking OMB approval of all instruments developed to collect data from all of these sources.2 Instruments are included in Attachments A-J. This is a new, one-time only data collection.
This data collection is necessary for SAMHSA to thoroughly document the performance of one of its major initiatives. To date there has been only limited documentation of SOAR’s performance. An implementation study of SOAR, funded by the Assistant Secretary for Planning and Evaluation (ASPE) at the U.S. Department of Health and Human Services, was recently completed. The goals of that study were to (1) provide a comprehensive description of SOAR processes, (2) identify factors associated with successful implementation of SOAR, and (3) determine whether and how the initiative could be strengthened. The study focused on six states that had a range of experiences implementing SOAR: Massachusetts, New Jersey, Ohio, Pennsylvania, Utah, and Virginia. The primary sources of data for the study were in-depth, in-person interviews with SOAR stakeholders in each of these states and focus groups with individuals who participated in an in-state SOAR training. The study also included an analysis of data from the pre/post training evaluations and customer satisfaction survey administered at the in-person SOAR trainings. OMB approved the focus group guide under approval number 0915-0212.
The earlier implementation study found substantial variation across states and local communities in fidelity of program implementation and underscored the critical role of TA in helping states implement the program successfully. However, no quantitative data were collected on SSI/SSDI applications (i.e., the number of application submissions, the application approval rate, and application processing time) because, at the time the study was conducted, most state and local communities had not systematically collected outcome data and study resources could not support collection of these data from SSA or state DDS systems. Thus, the study could not document rigorously the association between the intensity of TA, program implementation, and program outcomes. The proposed evaluation of SOAR will build on the earlier study by collecting data on both process measures (including the intensity of TA and the nature of implementation) and outcome measures, and analyzing the association between them. It presents an opportunity for SAMHSA to demonstrate systematically the outcomes of SOAR and determine whether it is worthy of future investment.
Evaluating whether SOAR is associated with improved SSI/SSDI application outcomes for homeless individuals is well-aligned with SAMHSA’s main goals. One of the agency’s primary goals is to make it easier for individuals with mental health, substance abuse, and co-occurring disorders to be able to have a permanent residence, job, and life integrated into the community. As many as 22 percent of adults who are homeless report a serious and persistent mental illness, and 67 percent report having received a mental health diagnosis during their lifetime (U.S. Conference of Mayors, 2004; Goering et al., 2002; North et al., 2004). As many as 68 percent of homeless adults report substance abuse problems, and many have both substance abuse and mental health disorders (Goering et al., 2002). Receipt of SSI/SSDI can increase access to stable housing, vocational and social supports and medical and mental health services for individuals who are homeless. A secondary but equally important and related goal of SAMHSA is assisting in the recovery of individuals from mental illness and substance abuse disorders. Given the access to services and community supports that SSI/SSDI provides, qualifying for these benefits can be viewed as a first step on the road to recovery.
The purpose of each data collections for which SAMHSA is seeking approval is as follows:
In-person interviews with state and local SOAR stakeholders. The in-person interviews will enable SAMHSA to gather information from a broad range of agencies and staff directly involved in the SOAR initiative, as well as from other stakeholders, on how SOAR is implemented at the ground level. The data from these interviews will be used in two ways. First, the data will provide context for and aid in the interpretation of quantitative application-related outcomes (e.g., number of applications filed, application approval rates, and application processing time) from an analysis of administrative data. Second, the data will be used to create independent variables for use in multivariate analyses that will investigate which implementation factors are associated with positive outcomes. Implementation factors include, for example, strong and consistent SOAR leadership within the state and level of engagement of SSA and DDS in SOAR.
A social network survey of state and local SOAR stakeholders. This brief survey is designed to assess the strength and capacity of collaborations associated with SOAR. The results will be used to conduct a network analysis to examine whether and to what extent stakeholders interacted with one another before the implementation of SOAR and whether and how their interactions change with the implementation of SOAR. The analysis will also provide a means of examining which stakeholders are perceived as being visible and active participants in the SOAR collaborative and the extent to which some contribute more than others. The data will also be used to create independent variables for use in multivariate analyses that will investigate the extent to which communication and collaboration between SOAR stakeholders is associated with positive outcomes.
Focus groups with participants of in-person SOAR trainings. Focus groups with participants of in-person SOAR trainings will be used to address questions regarding which of SOAR’s key strategies front-line workers do and do not use in practice (as well as why and how), the challenges they have faced in implementing what they learned in training, factors that have facilitated their implementation of SOAR, and outstanding needs they have for supporting homeless SSI/SSDI applicants.
Evaluative materials completed by participants of in-person SOAR trainings, including pre/post training evaluation forms and a customer satisfaction survey. The pre/post evaluation forms provide a concrete measure of the amount of knowledge about the SSI/SSDI application process participants acquire through the SOAR training. They also provide basic background data for each participant in the in-person trainings (for example, the type of agency where the participant works, position at the agency, and experience with SSI/SSDI applications prior to training). The customer satisfaction survey provides a systematic way to examine how participants in the training rate their experience. Data from all of these evaluative materials will be used to answer the following questions: Who participated in the initial in-state trainings? How satisfied were participants with the in-state training? What did training participants find most and least helpful? How much and what did participants learn from the training?
Evaluative materials completed by users of web-based SOAR trainings, including pre/post training evaluation screens and an online customer satisfaction survey. These forms will provide for the users of the web-based SOAR training the same type of information as the evaluative materials completed by the in-person SOAR training participants. Data from web-based trainings will be compared with data from the in-person trainings to determine whether the web-based training (which is substantially less expensive to administer) has a greater or lesser impact on gains in knowledge than the in-person trainings and whether there are quantitative and qualitative differences in the perceptions of the training content and format among those who receive the in-person training versus the web-based training. A portion of the online customer satisfaction survey will also address issues unique to the web-based training environment, such as its level of accessibility and navigability.
SAMHSA will use the information from the evaluation to demonstrate the extent to which SOAR is associated with positive outcomes such as increased application approval rates and decreased application processing times, which will ultimately help SAMHSA determine whether SOAR is worthy of future investment. SAMHSA began investing in SOAR in Fiscal Year 2005 and each year since it has provided additional funding without concrete evidence of the extent to which the TA is correlated with more SSI/SSDI application approvals for people who are homeless or systematic information on whether and how the TA has contributed toward systems change. The results from this evaluation will provide SAMHSA with that concrete and systematic evidence so that it may make more informed decisions about funding in the future. The results will also suggest how funding and TA can be allocated to maximize the value of TA to states and local communities.
The evaluation also is designed to enable, through the collection and analysis of SSA administrative data, a review of SOAR for inclusion in SAMHSA’s National Registry of Evidence-based Programs and Practices (NREPP). The NREPP is intended to provide valid, reliable, and timely information for audiences interested in identifying and/or implementing evidence-based interventions. If this study demonstrates that the SOAR model qualifies, it will represent a seminal resource for providers of services to the homeless. Evidence-based interventions in the area of benefits acquisition for homeless individuals are sorely needed. It is estimated that only 15 percent of new SSI applications submitted by homeless individuals nationwide are initially approved compared with nearly 30 percent of applications submitted by others (Rosen et. al. 2001; Social Security Advisory Board 2006).
Finally, results from the SOAR evaluation can inform future planning around Government Performance and Results Act (GPRA) reporting. In 2003, former President Bush’s New Freedom Commission on Mental Health delivered its final report calling for fundamental transformation of the mental health system in this country (New Freedom Commission, 2003). SAMHSA took the lead in implementing the Commission’s recommendations by convening a Federal Executive Steering Committee and released an Action Agenda that represented an unprecedented collaboration and broad-based commitment at the Federal level to ensure Americans with mental disorders, including people who are homeless, have the opportunity to live, work, learn, and participate fully in their communities (SAMHSA, 2005).3 SAMHSA’s FY 2006 - 2011 Strategic Plan continues to reinforce the message of building resilience and facilitating recovery across areas included in SAMHSA’s Management Matrix where homelessness is a specific priority. SAMHSA and the Homeless Programs Branch within its Center for Mental Health Services (CMHS) have recognized the importance of SOAR by including a measure of SOAR performance in the GPRA reporting requirements for the Projects for Assistance in Transition from Homelessness (PATH) program. PATH provides grants to states for community-based outreach, substance abuse prevention and treatment, mental health services, case management, and a limited set of housing services for people experiencing serious mental illness and are experiencing homelessness or at risk of becoming homeless. The performance measure is “provide training for PATH providers on SSI/SSDI Outreach, Access, Recovery (SOAR) to ensure eligible homeless clients are receiving benefits.” Results from the current evaluation could support the addition of other measures related to SOAR in PATH or related GPRA reporting.
Users of web-based SOAR trainings will complete all evaluative materials (pre/post training evaluations and a customer satisfaction survey) online. This is the most practical method for obtaining the necessary data since users will already be online for purposes of the training. The screens for the pre/post training evaluations and customer satisfaction survey have not yet been developed, though the content of the evaluations and survey has been developed and is available in Attachments H-J. All screens will display of the OMB control number, expiration date and response burden statement and all web-based data collection instruments will comply with the requirements of Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities.
The other data collections for which SAMHSA is seeking approval will be conducted through face-to-face interviews, focus groups, or through self-administration of hard copy forms. The data that will be collected during face-to-face interviews and focus groups will be qualitative. Respondents will be asked about their experiences with and opinions of the SOAR initiative. Responses will vary in length and substance and obtaining high quality data will require a flexible exchange between the interviewer and the respondents that allows for further probing into or clarification of certain responses. This type of data collection does not lend itself to electronic methods. Because the social network survey consists of only five questions and the respondents will be the same individuals who participate in the in-person interviews, it is most practical and least burdensome to collect the data on hard copy immediately following the in-person interview. The survey is designed to be self-administered, though the evaluation team will be present to answer questions respondents might have about the survey. Hard copy evaluative materials for participants of in-person SOAR trainings (pre/post training evaluations and a customer satisfaction survey) are provided by the trainers for self-administration at the end of the in-person training session. Trainers are available to answer any questions respondents might have about the materials. It would not be practical or cost effective to collect these data electronically as that would entail supplying each trainer in each community included in the study with a computer on which respondents could enter data. It would also add burden for training participants. There can be as many as thirty participants in a single in-state training session and participants would have to wait to enter data on a computer on site one after the other; through self-administration of hard copy materials, participants can all record data at the same time. In addition, there is no need in any of these four data collection for electronic tracking of a sample.
SAMHSA will use information technology make the findings from the evaluation widely available to Federal, state, and local supporters of SOAR and to the public in general. SAMHSA will place evaluation reports on its website and all reports will be compliant with Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities.
The current SOAR evaluation will not duplicate any prior evaluation efforts. SAMHSA will collect data in communities that have not yet received federally-funded SOAR TA; thus no data currently exist on the implementation of SOAR or on how implementation is associated with SSI/SSDI application outcomes in these communities. While information to assess the implementation of SOAR was collected in six states in an earlier study, no systematic data are available on application outcomes with which to assess the relationship between SOAR TA and implementation and application outcomes. The collection of implementation data in a new set of communities will enable SAMHSA to examine this relationship and gain a more complete understanding of how and how well SOAR works.
One way SAMHSA is minimizing the burden on small agencies of participation in SOAR is by developing a web-based SOAR training that will decrease the expense and travel involved in participation in in-person trainings. The evaluation will similarly minimize burden by collecting data from web-based training users online. A second way SAMHSA is being sensitive to small agencies is by ensuring that in-person interviews are only requested of program administrators and staff who have direct experience with SOAR and that each interview lasts no more than one hour. As not every employee at an organization will have contact with the initiative, burden to whole organizations should be small. A third way that SAMHSA is being sensitive to small agencies is by recruiting focus group participants from multiple agencies so that no more than 2 to 3 individuals at the group are from the same agency. Focus group discussions will last no more than one and a half hours. A fourth way SAMHSA is being sensitive to small agencies is by using self-administered data collection instruments that are designed to take no more than fifteen minutes to complete.
This is a one-time only data collection effort. If data are not collected, SAMHSA will be unable to make informed decisions around future funding for SOAR and how best to direct resources for TA.
This data collection fully complies with 5 CFR 1320.5(d)(2).
SAMHSA consulted with the public about this information collection. As required by 5 CFR 1320.8(d) a 60-Day Notice was published in the Federal Register on August 17, 2010 (75 FR 50764). No comments were received.
Table 1 identifies the individuals with whom staff from SAMHSA consulted on various iterations of the data collection instruments and procedures.
Table 1. Consultation with Individuals on Data Collection
Name |
Title |
Organizational Affiliation |
Phone Number |
Lynette Araki |
Senior Health Policy Analyst |
Health Resources and Services Administration Office of Planning and Evaluation Parklawn Bldg., Room 10-49 5600 Fishers Lane Rockville, MD 20857 |
301-443-6204 |
Anne Fletcher |
Social Science Analyst |
U.S. Department of Housing and Urban Development Office of Policy Development and Research 451 7th Street, SW Room 8120 Washington, DC 20140 |
202-402-4347 |
Edwin G. Beane, Sr. |
Team Leader |
Social Security Administration Office of Program Development and Research 3-M-1 Operations Building 6401 Security Boulevard Baltimore, MD 21235
|
410-965-9866
|
John Tambornino |
Analyst |
Social Security Administration Office of Program Development and Research Office
of Retirement and Disability Policy |
202-358-6250 |
In addition, SAMHSA conducted a pilot test of the social network survey in a large, industrial city in January of 2010 with: (1) a local community SOAR leader, (2) a DDS administrator, and (3) two in-state SOAR trainers. Based on this pilot test, revisions were made to the wording of the survey questions so that (1) they would be easier to understand, and (2) they would be interpreted more consistently across respondents. SAMHSA did not conduct a pilot test of the other data collection instruments because they were all used in the earlier implementation study of SOAR. The data collection activities in that study confirmed that the information being requested in the in-person interviews, focus groups, and SOAR training evaluative materials for the current evaluation is reasonable, clearly stated in coherent unambiguous language, and collected in the least burdensome way possible.
No payment or gift will be provided to any respondents.
Before conducting any in-person interviews or focus groups, SAMHSA will obtain informed consent to participate in the study from potential respondents. Before and during the interviews and focus groups, respondents will be advised that all information will be reported in the aggregate so that no individually identifying information can be found in any reports or publicly available documents. No questions are of a personal nature, but rather, are directed toward eliciting information about how the SOAR is functioning within the respondent’s community or state. SAMHSA is not seeking review by an Institutional Review Board (IRB) as there is no testing of human subjects or personal information being sought. No identifying information will be requested on any self-administered data collection form. Moreover, a statement will be included on all self-administered data collection forms notifying respondents that the data collection is being sponsored by SAMHSA and of its purpose and that information they provide will be kept private to the extent of the law. The statement will also make it clear that participation is voluntary, and that there will be no penalties for deciding not to respond to a particular question or for declining to complete the data collection. All resultant data will be used only by members of the evaluation team and only for the purpose of the researcher described herein. Also included in the statement will be an explanation of how all survey results will be reported in aggregate.
No information of a sensitive nature is being collected.
Table 2 provides estimates of annualized burden by data collection activity and for the evaluation overall. The text in this section explains how SAMHSA arrived at these estimates. The total hour burden of data collection for the SOAR evaluation is estimated to be 661.05 and the total cost burden is estimated to be $16,237.32.
Table 2 Burden Estimates
Data collection activity |
Estimated # Respondents |
Responses per Respondent |
Total Responses |
Burden Hours per Response |
Average Hourly Wage Cost |
Total Hour Burden |
Total Hour Cost Burden |
|
In-person interviews |
195 |
1 |
195 |
1 |
$33.05 |
195 |
$6,444.75 |
|
Social network survey |
195 |
1 |
195 |
.17 |
33.05 |
33.15 |
1,095.61 |
|
Focus groups |
143 |
1 |
143 |
1.5 |
20.09 |
214.5 |
4,309.31 |
|
Subtotal |
338 |
-- |
533 |
-- |
-- |
442.65 |
11,849.67 |
|
In-person SOAR Training Evaluative Materials |
||||||||
Pre-training evaluation form |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Post-training evaluation form |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Customer satisfaction survey |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Subtotal |
195 |
-- |
585 |
-- |
-- |
99.45 |
1,997.95 |
|
Web-based SOAR Training Evaluative Materials |
||||||||
Pre-training evaluation form |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Post- training evaluation form |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Customer satisfaction survey |
195 |
1 |
195 |
.17 |
20.09 |
33.15 |
665.98 |
|
Background form |
195 |
1 |
195 |
.10 |
20.09 |
19.5 |
391.76 |
|
Subtotal |
195 |
-- |
780 |
-- |
-- |
118.95 |
2389.70 |
|
Total |
728 |
-- |
1,898 |
-- |
-- |
661.05 |
16,237.32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In-person interviews with state and local SOAR stakeholders. Based on the earlier implementation study of SOAR, SAMHSA estimates each of the in-person interviews with SOAR stakeholders will last one hour. Stakeholders will include both program administrators and staff from a variety of government agencies and community-based organizations. Using Bureau of Labor Statistics (BLS) standards, SAMHSA estimates the average hourly wage rate of program administrators and staff to be $33.05. SAMHSA anticipates an average of 15 respondents in each of the 13 communities, for a total of 120 respondents. Thus, the total hour burden for this data collection is 195 hours and the total cost burden is $6,444.75.
Social network survey of state and local SOAR stakeholders. Based on pilot testing, SAMHSA estimates that the social network survey will take program administrators and staff approximately 10 minutes (0.17 hours) to complete. The same program administrators and staff that participate in the in-person interviews will complete the survey. Thus, SAMHSA anticipates a total of 195 respondents for a total hour burden of 33.15 hours and a total cost burden of $1,095.61 (using the average hourly wage rate of $33.05).
Focus groups with participants of in-person SOAR trainings. Based on the earlier implementation study of SOAR, SAMHSA estimates that each of the focus groups will last one and a half hours. One focus group will be conducted with in-person SOAR training participants in each of the 13 communities included in the study, and SAMHSA anticipates an average of 11 participants in each group. The majority of training participants are staff from community-based agencies. Based on BLS standards, SAMHSA estimates the average hourly wage rate for this group to be $20.09. Thus, the total hour burden of this data collection is 214.5 and the total cost burden is $4,309.31.
In-person SOAR training evaluative materials. There will be three data collection forms that participants in the in-person SOAR trainings will complete: (1) a pre-training evaluation form, (2) a post-training evaluation form, and (3) a customer satisfaction survey. Slightly different versions of these forms have been used in SOAR trainings for the past several years. This experience suggests that each form will take approximately 10 minutes (0.17 hours) to complete. There is an average of 15 participants at each in-person training in each state, but only participants in one of the in-person trainings in each state will be provided with the forms to complete. Thus, SAMHSA anticipates a total of 195 respondents (15 respondents in each of 13 states included in the study). As noted above, the majority of training participants are staff from community-based agencies, so SAMHSA estimated total cost burden for this data collection using an average hourly wage rate of $20.09. The hours burden is 33.15 and the cost burden $665.98 for each of the three data collection forms. The total hour burden for this data collection is 99.45 and the total cost burden is $1,997.95.
Web-based SOAR training evaluative materials. Users of the web-based SOAR training will complete the same pre-training evaluation form and post-training evaluation form that participants in the in-person SSR trainings complete and will complete a slightly different version of the customer satisfaction survey. Thus, these data collection forms have the same estimated response time as the forms for the in-person training participants. SAMHSA anticipates that the short background form will take an average of 6 minutes (0.1 hours) to complete. As in the in-person trainings, the majority of users of the web-based training are anticipated to be staff from community-based agencies, so SAMHSA estimated total cost burden for this data collection using an average hourly wage rate of $20.09. SAMHSA anticipates that in one community in each of the 13 states will be selected to receive web-based training in lieu of in-person training and that the number of users in the web-based traning will be equivalent to the number of participants in any given in-person training (15). Thus, the total number of respondents for this type of data collection is 195 (15 respondents in each of 13 states included in the study), for a total hour burden of 118.95 and total cost burden of $2,389.70.
There are no additional costs to the respondents.
The evaluation will occur over a five year period. The cost to the government during the base year of the evaluation is calculated to be $342,670. The total cost of the evaluation is estimated to be $1,733,904.4 Thus, annualized cost to the government is $346,781 per year ($1,733,904 divided by five evaluation years).
This is a new data collection.
SAMHSA will begin to collect data in early 2011. Table 3 presents the anticipated data collection schedule by type of data collection.
Table 3. Data Collection Schedule
|
Start of Data Collection |
Completion of Data Collection |
In-person interviews |
September 2011 |
April 2012 |
Social network survey |
September 2011 |
April 2012 |
Focus groups |
September 2011 |
April 2012 |
In-person SOAR training evaluative materials |
January 2011 |
September 2011 |
Web-based SOAR training evaluative materials |
January 2011 |
December 2012 |
Six primary research questions guide the evaluation of SOAR:
To what extent does SOAR training change front-line workers’ knowledge and practice, and do differences in these outcomes exist for participants of in-person trainings versus users of the web-based training?
To what extent does SOAR increase the number of SSI/SSDI applications that are identified as being submitted by individuals who are homeless?5
To what extent does SOAR improve SSI/SSDI application outcomes for people who are homeless?
What environmental factors (such as the frequency of contacts among the network of services for the homeless) are associated with better application outcomes?
What implementation factors (e.g., strong and consistent leadership) are associated with better application outcomes?
Are improvements in SOAR application outcomes associated with benefits for state or private service delivery systems (such as cost recovery for state General Assistance expenditures, Medicaid reimbursements for uncompensated medical care, or reduced DDS/SSA application backlog)?
Answering these questions requires both qualitative and quantitative analyses. The analyses that will be conducted for each data collection are described below.
In-person interviews with state and local SOAR stakeholders and focus groups with participants of in-person SOAR trainings. The evaluation will utilize a case-study approach (as explained by Yin, 1989) to analyze the qualitative data that result from in-person interviews and focus groups. The purpose of these data collection activities will be to gather “slices of information” from a variety of sources to create a comprehensive picture of SOAR implementation. Data collectors will take notes during each in-person interview and focus group. The notes will be compiled and synthesized into site-specific internal memorandum to be used for analyses. Each memorandum will follow a uniform outline based on the key study topics. The memorandum will highlight points of disagreement and agreement among data sources (for instance, comparing responses of representatives from community-based agencies with responses from the in-state trainers or state and local SOAR leaders). Triangulating the data in this way will enable researchers to arrive at findings based on mutually confirming lines of evidence. To facilitate cross-site comparison, key findings will be summarized for each site and important characteristics by which the programs can be classified will be identified in order to describe and evaluate their processes and outcomes. Broader themes and key findings will then be distilled to form the basis for the evaluation conclusions in the final report.
Social network survey of state and local SOAR stakeholders. The social network survey data will be used to quantify the frequency of communication among stakeholders before and after SOAR. The survey will also gather respondents’ perceptions about the extent to which each SOAR stakeholder helped to facilitate the implementation of SOAR. Findings from the survey will be used to develop visual representations of communication and regard between stakeholders and how they have changed with the implementation of SOAR. Examples of these visual representations are provided in Attachment K.
In-person and web-based SOAR training evaluative materials. The pre-training evaluation form asks about some basic characteristics of training participants including the type of agency for which they work, years of employment, and average number of homeless clients they assist with SSI/SSDI applications. This information will be used to describe the characteristics of participants and compare whether characteristics differ between those who compete the in-person versus the web-based training (see Attachment K for an example of a table shell that could be used to display the results). The primary purpose of the pre-training evaluation form, however, is to assess training participants’ baseline knowledge of the SSI/SSDI programs and application process and to measure gains in knowledge after SOAR training using the post-training evaluation form. For both the in-person and web-based trainings, the number of correct responses to questions on the pre-training form and post-training form will be summarized to determine whether there is any change in knowledge. Simple descriptive tabulations (sums, means, medians, distributions) will be conducted first and then means will be compared using simple t-tests to determine the statistical significance of any difference between the average number of correct responses to the pre- and post-training questions. Item-level analyses will be conducted to determine whether there are any differences in the specific types of knowledge gained. Analyses will also determine whether mean differences in pre- and post-training scores vary according to whether the participant received the in-person or the web-based training (see Attachment K for an example of a table shell that could be used to display the results). We will also use simple t-tests to examine whether any participant characteristics (such as place of employment, years on the job, etc.) are associated with gains in knowledge.
The customer satisfaction survey will be used to assess training participants’ perceptions of whether the training was useful and clearly delivered. The in-person and web-based customer satisfaction survey contain the same first seven questions to assess whether the participant perceives having a better understanding of the SSI/SSDI application process, feels confident to implement the application procedures, and is satisfied overall with the information provided during the training. Tabulations of each item (sums, means, medians, distributions) will be conducted to describe participants’ perceptions of specific elements of the training (see Attachment K for an example of a table shell that could be used to display the results). A summary score will then be created to assess participants’ overall satisfaction with the information presented during the training. Simple t-tests will be used to examine whether any participant characteristics such as place of employment, years on the job, etc., are associated with perceptions of the training.
The second set of questions included on the customer satisfaction survey for in-person SOAR training participants asks the respondent to rate the trainer and the organization of the training. The second set of questions included on the customer satisfaction survey for users of the web-based SOAR training is conceptually similar but asks the respondent to rate the design and usability of the website. In a similar manner as describe above, for both the in-person and web-based training customer satisfaction surveys, item-level tabulations will be conducted to describe participants’ perceptions of the training and then examine summary scores (see Attachment K for examples of table shells that could be used to display the results). T-tests will be used to examine whether any participant characteristics are associated with perceptions of either the trainer and organization of the training (for in-person training) or usability and design of the website (for web-based training).
SAMHSA anticipates producing a final evaluation report in that will be made available to the public. A draft of the report will be produced by June 2014 and the final report will be available to the public around October 2014.
The expiration date for OMB approval will be displayed on all forms and protocols.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
B. Collections of Information Employing Statistical Methods
Thirteen communities in 13 different states will be the focus of this evaluation. The sample is purposive, in that the states selected for the study are those that will receive federally-funded SOAR TA for the first time beginning in 2010. The communities within each of those states in which data will be collected from in-person interviews, the social network survey, focus groups, and in-person SOAR training evaluative materials will be the SOAR pilot site in which the state has focused most of its attention. Respondents for each of the of data collection activities in these communities include the following:
In-person interviews with and social network survey of state and local SOAR stakeholders. Respondents will include: state and local SOAR leaders (1 to 3 people); SSA administrators and staff (1 to 3 people); DDS administrators and staff (1 to 3 people); in-state, in-person SOAR trainers (1 to 3 people); and administrators and staff from state mental health agencies, housing and other public assistance agencies, homeless service providers, and community-based agencies (3 to 6 people).
Focus groups with participants of in-person SOAR trainings. Focus group respondents will consist of a selection of participants in the in-person SOAR trainings in each of the 13 states included in the evaluation. SAMHSA anticipates that the majority of training participants will be staff from community-based agencies. Focus group members will be recruited from lists of training participants complied by the SOAR trainers. Recruitment efforts will focus on individuals employed by agencies in or around the local community in which the focus group is taking place. Approximately 12 to 15 focus group members will be recruited with the expectation that 10 to 12 will actually show up at the group session. Focus group members will be recruited from multiple agencies so that no more than 2 to 3 individuals at the group are from the same agency.
In-person SOAR training evaluative materials. Respondents to this data collection include each participant in the first in-person SOAR training in each of the 13 states included in the study. As there is an average of 15 participants at each in-person training in each state, SAMHSA anticipates that there will be a total of 195 respondents (15 respondents in each of 13 states) to this data collection effort.Web-based SOAR training evaluative materials. Respondents will be the universe of users in the 13 identified communities who log on to the web-based training and receive a user identification number (assigned the first time a user logs into the systemSAMHSA will identify one community in each of the new SOAR states in which only web-based training (and no in-person trainings) will be available. SAMHSA will collect data from web-based training participants in these communities only and compare the results with those from the in-person trainings in the same states. SAMHSA anticipates that the number of web-based training participants in each select community will be similar to the number of participants in any given in-person training session. Thus, SAMHSA anticipates that there will be 15 users in each of the 13 communities.
This data collection does not involve any sampling or weighting, and this is a one-time only data collection. The remainder of this section describes the procedures SAMHSA will use the collect information for the evaluation. All data collection instruments are included in Attachments A-J.
In-person interviews with and social network survey of state and local SOAR stakeholders. The first step in this data collection process will be to send a letter to each state included in the study explaining the evaluation and seeking their cooperation with it. The letter will come from Pamela Fischer, the SAMHSA Project Officer, to lend credibility to the study and further encourage cooperation (see Attachment L for a draft of the letter). Project staff will follow up with state contacts in a telephone call to describe further the information to be gathered from SOAR stakeholders during in-person interviews and a brief self-administered survey that will be completed at the end of each interview. State contacts will be asked to identify individuals who will be able to provide the required information and for information about their general schedule constraints. A schedule for the interviews that meets participants’ needs will be developed collaboratively with the state contacts. Approximately two weeks before the interviews are scheduled to take place, an information packet will be mailed to the site contacts containing the final interview schedule. The packet will also contain contact information for the project team members who will be conducting the interviews so that respondents can reach them in the event of a schedule change or other issues that may arise before the interviews. Providing the sites with adequate information ahead of time in a professional manner will help build rapport and ensure that the interviews go smoothly and that interviewees are available and responsive. Two project team members will spend two days each at each site conducting the in-person interviews and providing respondents at the end of each interview with the social network survey for self-administration.
Focus groups with participants of in-person SOAR trainings. The first step in this data collection process will be to send a recruitment letter to potential focus group participants (this letter is included in Attachment M). Project staff will follow up on this letter with a telephone call to potential participants further explaining the evaluation and requesting their participation. Approximately two weeks before the focus group, a confirmation letter (included in Attachment N) will be mailed and emailed to those who agree on the telephone to participate. A few days before the focus group, focus group invitees will be called and emailed to confirm their attendance and answer any remaining questions about the focus group. Two project staff will spend two days each at each site, and conduct the focus group during this time. The focus group will follow the topics outlined in the focus group guide (see Attachment C) and provide participants with the opportunity to share their experiences implementing SOAR in their communities.
In-person SOAR training evaluative materials. SOAR trainers will distribute the pre-training evaluation forms to training participants at the beginning of the training, before beginning to teach the SSR curriculum. The trainer will encourage participants to complete the form for the purposes of tailoring the training to their needs. The trainer will not proceed with the training until all forms are collected. This method of data collection has worked well in previous years of training and very few participants failed to complete the pre-training evaluation form. In addition, there was very little missing data for any item on the forms. Similarly, trainers will distribute the post-training evaluation forms and customer satisfaction surveys at the conclusion of the training. Participants will complete these before they receive their certificate of completion for the training. The trainer will encourage participants to complete the form and survey to demonstrate their knowledge, troubleshoot any areas of learning that may require follow-up, and to provide feedback that can improve the training. This process has worked well in previous years of training and very few participants failed to complete the post-training evaluation form or customer satisfaction survey.
Web-based SOAR training evaluative materials. For the web-based SOAR training, a computerized version of the pre-training evaluation form will appear on the screen immediately after the user completes her/his first login to the website. The user will be prompted to answer these questions only once at the first login. The website will prevent the user from initiating the training without first completing the form. Similarly, users will complete the post-training evaluation form and customer satisfaction survey as one of the final screens before the system will generate a certificate of training completion.
SAMHSA anticipates the following response rates for each data collection:
Interviews with and social network survey of state and local SOAR stakeholders. SAMHSA anticipates attempting to interview and survey an average of 15 individuals in each of 13 communities. Based on experiences conducting interviews with the same types of state and local stakeholders during the earlier implementation study of SOAR (in which the same methodical recruitment procedures proposed in this collection and described in Section B2 above were used), SAMHSA anticipates a 95 percent response rate for the interviews and self-administered survey that will follow.
Focus groups with participants of in-person SOAR trainings. SAMHSA anticipates a response rate of approximately 81 percent. Approximately 12 to 15 focus group members will be recruited to each of 13 focus groups (for an estimated total sample of 176 across all groups) with the expectation that 10 to 12 will actually show up at each group session (for 143 estimated total respondents across all groups).
In-person SOAR training evaluative materials. The in-person SOAR training evaluative materials consist of three forms: (1) the pre-evaluation training form; (2) the post-evaluation training form; and (3) the customer satisfaction survey. Based on experience from previous years of in-person SOAR trainings, SAMHSA anticipates a 90 percent response rate on each.
Web-based SOAR training evaluative materials. The web-based SOAR training evaluative materials consist of three forms: (1) the pre-evaluation training form; (2) the post-evaluation training form; and (3) the customer satisfaction survey. Anyone who wishes to access any component of the web-based training will be asked to complete a pre-evaluation training form. After entering the web-based training site, the pre-evaluation training form will be the first screen that all users encounter before initiating any training components. Because the form will be the gateway to receiving the self-initiated training, SAMHSA anticipates a 90 percent response rate on this form. Only users who complete the entire web-based training will be able to complete the post-training evaluation form and customer satisfaction survey. SAMHSA anticipates that 70 percent of users will complete the entire training and that 90 percent of them will complete the post-training evaluation form and customer satisfaction survey. Completing the post-training evaluation form and customer satisfaction survey will be the last step users take before the system generates a certificate of training completion. Thus, SAMHSA anticipates an overall 63 percent response rate on the post-training evaluation form and customer satisfaction survey.
In addition to the procedures described in Section B2 above, the following methods will maximize response rates and ensure that the necessary information is captured during the in-person interviews, social network survey, and focus groups:
Minimize individual burden. Willingness of program administrators and staff to participate in in-person interviews may hinge on the time required of them. To minimize burden, guides are designed to be “coverable” in 60 minutes. SAMHSA has developed separate discussion guides for each potential respondent type so that respondents are not asked about activities or issues that are not applicable to them. In addition, data collectors will meet with in-person interview respondents in their own offices or at a location of their choice.
Develop interview schedules that respect site constraints and pressures. The project team will identify a liaison at each site with whom to work out logistics and a schedule for the in-person interviews. The schedule will avoid interference with other activities and allow individuals to fit the time they spend with project staff into their schedules.
Maximize the convenience of focus groups. The focus groups will be scheduled during lunchtime so that they interfere as little as possible with participants’ work flow and day-to-day responsibilities. In addition, focus groups will take place at a location in the community that is easily accessible to the majority of participants (and is located within walking distance of public transportation, if possible).
Data collection procedures for the in-person SOAR training evaluative materials have worked well in previous years of training, so no steps other than those described in Section B2 above will be taken to maximize response rates. To maximize response rates for the web-based training evaluative materials, the voiceover on the web-based training will encourage participants to complete the requested data collections and provide the rationale for each.
SAMHSA conducted a pilot test of the social network survey in a large, industrial city in January of 2010 with: (1) a local community SOAR leader, (2) a DDS administrator, and (3) two in-state SOAR trainers. Based on this pilot test, revisions were made to the wording of the survey questions so that (1) they would be easier to understand, and (2) they would be interpreted more consistently across respondents. SAMHSA did not conduct a pilot test of the other data collection instruments because they were all used in the earlier implementation study of SOAR. The data collection activities in that study confirmed that the information being requested in the in-person interviews, focus groups, and SOAR training evaluative materials for the current evaluation is reasonable, clearly stated in coherent unambiguous language, and collected in the least burdensome way possible.
SAMHSA has contracted with Policy research Associates, Inc. and Mathematica Policy Research to conduct the evaluation of SOAR. Table 4 identifies the individuals at these organizations who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation (HHSS283200700036I/ HHSS28342001T)—Gretchen Stiers—and her designee who will oversee the evaluation—Pamela Fischer—will be responsible for receiving and approving all contract deliverables. Their contact information is also included in Table 4.
Table 4. Individuals Responsible for Data Collection and Analysis
Name |
Title |
Organizational Affiliation and Address |
Phone Number |
Alisa Ainbinder |
Survey Researcher |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202- 264-3444 |
Norma Altshuler |
Research Analyst |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202- 554-7517 |
Jonathan Brown |
Senior Researcher |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202-264-3446 |
Noelle Denny-Brown |
Research Analyst |
Mathematica Policy Research 955 Massachusetts Ave. Suite 801 Cambridge, MA 02139 |
617-301-8987 |
Henry Iryes |
Senior Fellow |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202- 554-7536 |
Jacqueline Kauff |
Senior Researcher |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202-484-5226 |
Emily Sama Martin |
Researcher |
Mathematica Policy Research 600 Maryland avenue, SW Suite 550 Washington, DC 20024 |
202- 484-4512 |
Pamela Robbins |
Vice President |
Policy Research Associates, Inc. 345 Delaware Avenue Delmar, New York 12054 |
518- 439-7415 ext. 233 |
Gretchen Stiers |
Branch Chief, Homeless Programs
|
SAMHSA Center for Mental Health Services One Choke Cherry Road Room 6-1033 Rockville, MD 20857 |
240-276-1844 |
Pamela Fischer |
Social Science Analyst |
SAMHSA Center for Mental Health Services One Choke Cherry Road Room 6-1033 Rockville, MD 20857 |
240-276-1901
|
LIST OF ATTACHMENTS
Attachment A Discussion Guides for Stakeholder Interviews
Attachment B Social Network Survey
Attachment C Focus Group Guide with Participants of In-person Trainings
Attachment D Focus Group Participant Information Form
Attachment E Pre-Training Evaluation Form for In-Person SOAR Training
Attachment F Post-Training Evaluation Form for In-Person SOAR Training
Attachment G Customer Satisfaction Survey for In-Person SOAR Training
Attachment H Pre-Training Evaluation Form for Web-Based SOAR Training
Attachment I Post-Training Evaluation Form for Web-Based SOAR Training
Attachment J Customer Satisfaction Survey for Web-Based SOAR Training
Attachment K Examples of Visual Representations of Resultant Data
Attachment L State Recruitment Letter
Attachment M Focus Group Recruitment Letter
Attachment N Focus Group Confirmation Letter
Attachment O Draft Memorandum of Agreement Between Mathematica Policy Research, Incorporated and Social Security Administration
1 Outputs reflect communities’ efforts to implement SOAR (for example, the number of in-state SSR trainings that are conducted and the number of individuals who attend an in-state training) and are what enable SOAR to achieve its intended outcomes (for example, improved access to SSI/SSDI benefits for and quality of life among individuals who are homeless).
2 In addition, SAMHSA will collect administrative data from SSA or state DDS systems on SSI/SSDI application submission outcomes. SAMHSA is in the process of exploring a data sharing agreement with SSA.
3 http://www.prainc.com/soar
4 The second year of the evaluation is estimated to cost $331,986; the third year is estimated to cost $440,955; the fourth year is estimated to cost $241,634; the fifth year is estimated to cost $376,659.
5 SAMHSA will answer this research question using administrative data from SSA or state DDS systems on SSI/SSDI application submission outcomes. SAMHSA is in the process of exploring a data sharing agreement with SSA.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | DATA COLLECTION FOR THE SOAR EVALUATION |
| Author | Sharon D. Clark |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |