Supporting Statement Part B - 0806

Supported Employment Demonstration (SED) Project

Supporting Material for OMB Clearance

Part B. Statistical Methods

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The Supported Employment Demonstration (SED) study population consists of individuals aged 18 to 50 who apply for disability benefits alleging a mental illness and the initial decision is a denial of benefits in the past 60 days. Individuals eligible for the study must live within the area of one of 30 community mental health centers (CMHCs), 20 urban and 10 rural, distributed across the United States that serve as demonstration sites. We screen potentially eligible individuals for competency (ability to give consent), and confirm their allegation of a mental health impairment using the Composite International Diagnostic Interview (CIDI). We exclude from study participation any individuals who are unable to provide informed consent; or who reside in a nursing home or other custodial institution. The SED project enrolls 1,000 participants in each of the three study arms for a total of 3,000 participants: 40 participants in each of three study arms for the 20 urban sites equaling an n of 2,400 urban site participants; and 20 participants in each of three arms for the 10 rural sites equaling an n of 600 rural site participants. SSA provides Westat with an initial list of denied applicants (within the past 60 days prior to the start of the project) with an alleged impairment of mental illness within a defined area for each site. Subsequently, on a monthly basis, SSA provided Westat with a list of denied applicants within these areas. Upon receipt of the files, Westat created random Identification data (IDs) for each potential participant; sanitized the data (i.e., remove Social Security Numbers); and established release groups from which recruitment will take place. The release groups included up to 25 potential participants based on their distance from the demonstration site. Westat loaded the release groups into the study’s Management Information System (MIS), and assigned Research Assistants off site in the local community (e.g., field interviewers) exclusive access for recruitment. As contact information for each denied applicant was loaded into the study’s management information system MIS, we created another special ID that ties the contact information ID with the subsequent enrollment decision, other study data, and the individual’s SSN.

2. Procedures for the Collection of Information

Recruitment and Screening

Westat first sent potential participants a study invitation package (see Attachment A. Introductory Letter and Attachment B. Study Brochure). Several days following this mailing, the Research Assistants contacted the potential participants by phone (see Attachment C. Sample Script for Initial Phone Call); if Westat was unable to reach them by phone, the Research Assistants will make an in-person visit. As part of this initial contact, the Research Assistants determined interest in work by means of a single question, “Are you interested in getting a job?” The Research Assistants then invited potential participants who expressed a desire to work to a recruitment information meeting. The recruitment information meeting served to orient potential participants about the demonstration, giving them the opportunity to weigh the merits of the demonstration while allowing Research Assistants to dispel any misconceptions and provide detailed information on the demonstration’s advantages and risks (see Attachment D. Overview and FAQs). The recruitment meetings took place in a group or individual format. The Research Assistants then asked those interested in joining the demonstration to give informed consent to participate in a screening for eligibility to confirm competency and allegation of a mental health impairment using the Composite International Diagnostic Interview (CIDI) (see Attachment E. Screening Consent, Attachment F. Competency and CIDI Screener, and Attachment G. Ineligibility Letter). Potential participants received $45 for completing the screening process, which took approximately 40 minutes. We invited those who screened positive for a mental health impairment to give informed consent to enroll in the study (see Attachment H. Study Consent).

Random Assignment to Study Arms

Once enrolled, we randomly assigned each participant to one of three study arms: (1) Full-Service Treatment (2) Basic-Service Treatment; or (3) Usual Services. (see Attachment I. Randomization Letters and Attachment J. Reloadable Card FAQs), our procedures for which we described in detail in the Supporting Statement Part A, #2. Here we also discuss the costs associated with the random assignments:

• Full-Service Treatment (n=1,000). The plan for this model includes out‑of‑pocket expenses associated with prescription behavioral health medications, with an average allotment of $1,166 per participant over the three years of their participation in the study. Individual Placement and Support (IPS) supported employment also includes a modest allotment of funds for essential work-related expenses. Integrated behavioral health care also requires financial support for services and treatment not covered by the participant’s health insurance. The allotments for these additional expenses include $130 for essential work-related expenses and $1,034 for behavioral health expenses per participant over the three years of study participation.

• Basic-Service Treatment (n=1,000). The plan for this model includes allotments of $1,034 per participant for expenses associated with behavioral treatment, and $130 per participant for essential work-related expenses over the three years of study participation.

• Usual Services (n=1,000). We do not include any allotments for this model.

Evaluation Data Collection

Data collection for the evaluation of the SED consists of the following activities: baseline in-person participant interviews; quarterly participant telephone interviews; receipt of SSA administrative record data; and collection of site-level program data. Evaluation team members will also conduct site visits involving: pre-visit environmental scans; independent fidelity assessments; key informant interviews; focus groups; and ethnographic data collection.

• Baseline Participant Interviews

Research Assistants, hired by Westat and located off site in the local community, conducted a baseline, in-person interview with each participant enrolled in the Supported Employment Demonstration via a computer-assisted personal interview (CAPI) (see Attachment K. Baseline Interview). Participants received $50 for completing the baseline interview, which took approximately 45 minutes. The baseline interview included questions on the following topics: work history; job status; health status and quality of life; demographics and income; functional limitations; health insurance coverage; and health care utilization (doctor visits, hospitalizations, etc.). During the baseline interview, the Research Assistant also asked participants to provide contact information for two additional individuals in the event that the participant cannot be located. At the end of the baseline interview, the Research Assistant provided the participant with an assignment to one of the three study arms, as well as written material and guidance about moving forward in the study (see Attachment I. Randomization Letters and Attachment J. Reloadable Card FAQs).

• Quarterly Participant Interviews

Westat Telephone Research Center (TRC) interviewers administer blind quarterly follow-up telephone interviews to participants via a computer-assisted telephone interview (CATI) (see Attachment L. Quarterly Interview). The quarterly interview assesses employment outcomes and utilization of health care and other related services. In addition, the quarterly interviews, administered at the annual follow-up time periods (Quarters 4, 8, and 12), assess health status; functioning; symptoms; medication use; presence of two or more chronic illnesses; quality of life; and recidivism. Participants receive $25 for completing each quarterly interview (at Quarters 1, 2, 3, 5, 6, 7, 9, 10, and 11), which took approximately 20 minutes, and $40 for each annual interview (at Quarters 4, 8, and 12), which takes an average of 30 minutes.

We will continue conducting quarterly follow-up interviews with all participants, even if they stop receiving treatment services before the end of the treatment period. Only in cases where participants formally end their relationship with the study and explicitly request no further contact will we no longer pursue quarterly follow-up interviews. We employ several strategies to minimize attrition and loss to follow-up for the quarterly interviews: First, in the baseline and quarterly interviews, the interviewer confirms and updates (as appropriate) the participant’s contact information, and asks the participant if they plan to move. Secondly, we flag participates who initially refuse to continue their participation in a quarterly interview for further follow-up by a CATI interviewer experienced in refusal conversion. Third, for participants we cannot locate, TRC interviewers experienced in tracing will use contact the information the participant provided in the baseline interview to obtain updated contact information. If these efforts prove unsuccessful, we may engage local Westat field staff to conduct in-person tracing efforts, starting with visiting the participant’s last known address.

SSA Administrative Data

SSA provided Westat access to administrative data on the full participant sample (N=3,000), even for those participants who exit the study or do not complete the follow-up interviews. These data includes items such as: application for disability benefits; disability decisions; award amounts; and earnings. Westat uses the administrative data SSA provides to measure benefit-related outcomes.

Site-Level Program Data

We collect program data at the site level via the MIS and site visits. These data will include items related to the provision of services to participants. Westat uses this information largely in the participation analyses and process evaluation.

Site Visits

Evaluation of the SED includes annual site visits to each of the 30 study sites, for 120, 5-day long site visits, involving the following activities: pre-visit environmental scan; independent fidelity assessments of IPS-supported employment services and other treatment components; key informant interviews; focus groups with treatment participants; and person-centered interviews for the ethnographic data collection. Table 1 displays the data collection activities; we carry out during the site visits, and the total number of respondents for each activity.

Table 1. Target Number of Respondents for Site Visit Data Collection Activities

Contract Year	Fidelity Assessment Participant Interviews	Fidelity Assessment Family Member Interviews	Key Informant Interviews	Participant Focus Groups	Ethnographic Observations	Ethnographic Person-Centered Interviews
1
2	180	90	120	600	60	180
3	180	90	120		60	180
4	180	90	120	600	60	180
5	180	90	120		60	180
6
Total Target Number of Respondents	720	360	480	1200	240	720

Pre-Visit Environmental Scan. In preparation for each site visit, we scan the local environment to understand the local context in which we embed SED services. For example, we access unemployment rates and information on general types of work available using Bureau of Labor statistics and County Business Patterns. We also explore available community, non-profit, and charitable services through online and phone searches as well as community resources manuals. Information acquired through the environmental scans provides a framework of available services to individuals with employment and mental health needs.

Independent Fidelity Assessments. In contract Years 2 through 5, we conduct annual, independent fidelity assessments of IPS supported employment services and other treatment components, i.e., the Nurse Care Coordinator role and provision of Systematic Medication Management, for the purpose of assessing treatment fidelity (see Attachment M7. IPS Fidelity Assessment Guide). There is a total of 120 independent fidelity assessments, one for each center in each implementation year. To reduce respondent burden and efficiency, we schedule these independent fidelity assessments in conjunction with those conducted by state Mental Health/Vocational Rehabilitation staff as a part of the regular monitoring and quality improvement fidelity reviews conducted for implementation. However, we score and interpret the findings from our fidelity assessments independently.

SSA selects SED participants involved as respondents in the Independent Fidelity Assessment to represent the diversity of SED participants by gender, race, ethnicity, age, employment status, and employment type. We mail, or hand a letter to respondents (see Attachment M1. IPS Fidelity Assessment Participant Invitation Letter) describing the IPS interview with the proposed date and time. We follow up with a phone call (see Attachment M2. IPS Fidelity Assessment Participant Phone Call Sample Script) to the respondent to confirm interest in attending (see Attachment M3, IPS Fidelity Assessment Participant Consent Script).

Key Informant Interviews. In contract Years 2 through 5, at each of the 30 sites we conduct semi-structured key informant interviews with the IPS specialist, the nurse care coordinator, the case manager, and the director of the facility (see Attachment N. Key Informant Interview Administrator Initial; Attachment O. Key Informant Interview Administrator Follow Up; Attachment P. Key Informant Interview Clinician Initial; and Attachment Q. Key Informant Interview Clinician Follow Up). The key informant interviews to take up to 60 minutes. Interviews cover topics such as: agency funding streams; challenges and successes of demonstration implementation; sustainability; participant outcomes; and recommendations. In Years 2 and 3, evaluation team members conduct these interviews in person; in Years 4 and 5 they will conduct these interviews by telephone. We interview up to 4 key informants at each site for a total of up to 240 in-person interviews in Years 2 and 3 and 240 telephone interviews in Years 4 and 5, or 480 key informant interviews combined.

Focus Groups. In contract Years 2 and 4, at each of the 30 sites, we conduct 2 focus groups with treatment participants: one with Full-Service Treatment participants and one with Basic-Service Treatment participants (see Attachment R3. Participant Focus Group Guide). The IPS selects potential participants to represent a diversity of participants by gender, race, ethnicity, age, employment status, and employment type. We mail, or hand a letter to participants (see Attachment R1. Participant Focus Group Invitation Letter) describing the focus group with the proposed date and time. We follow up with a phone call (see Attachment R2. Participant Focus Group Phone Call Sample Script) to the participant to confirm interest in attending. Focus group topics to take up to 60 minutes and cover: perceived quality of services delivered; preferences among types of employment and mental health services; barriers to employment; perceived availability of community employment resources; and experiences of supported employment. We aim to include at least 6 participants and no more than 10 participants in each focus group, for a total of up to 1200 participants (20 participants x 30 sites x 2) in the focus groups combined. Each participant receives $40 for attending the focus group.

Ethnographic Data Collection. In contract Years 2 through 5, we conduct ethnographic data collection consisting of observations in the natural environment, and person-centered interviews with participants and non-participants (Attachment S6. Ethnographic Observation Template). With participants’ permission, evaluation team members observe participants and non-participants in natural environments such as: workplaces; family homes; churches; community settings; treatment settings; leisure activities; and other locations important for understanding the layered contexts in which participants sought disability benefits and now seek employment. The observations contribute to a holistic understanding of the constraints, opportunities, preferences, and assumptions that affect employment and the mental health among those applying for SSA disability benefits. We do not determine the specific locations for observation until the site visit team systematically gathers information about the environment of the treatment site. We choose some locations for ethnographic observation before the visit begins, we may not discover others until after the site visit team begins the visit. We anticipate that discussions with clinicians and other workers at the treatment site, local community leaders, and study participants will yield appropriate locations for ethnographic observations. Once we have chosen a location for observations, site visit team members mail invitation letters to location’s gatekeepers and, or to study participants who agreed to be observed (see Attachment S1. Ethnographic Observation Gatekeeper Invitation Letter, and Attachment S3. Ethnographic Observation Participant Invitation Letter). The Site Visitors follow up with phone calls (see Attachment S2. Ethnographic Observation Gatekeeper Phone Call Sample Script and Attachment S4. Ethnographic Observation Participant Phone Call Sample Script) to gatekeepers and participants to confirm interest in participating. Once at the location for observation, we ask any individuals whom the Site Visitors may observe to provide verbal consent (see Attachment S5. Ethnographic Observation Ad Hoc Consent Script). For example, observing a local job fair involves seeking consent from the organizers of the fair (the gatekeepers) and any individuals with whom the site visitors may interact. Observing the daily life of a study participant requires the consent of the participants and any individuals with whom they may interact. Evaluation team members take field notes to systematically document their observations. We observe up to 8 individuals at each of the 30 sites, for a total of 240 individuals observed in Years 2 through 5 combined.

As part of the ethnographic data collection, evaluation team members also conduct person-centered interviews to better characterize participants and non‑participants of the Supported Employment Demonstration (see Attachment T5. Person-Centered Interview Initial and Attachment U. Person-Centered Interview Follow Up). As informants, we ask interviewees to narrate their work and illness histories to describe the chain of events and decisions that led to making an SSA application. As respondents, we ask interviews to explain how these experiences motivated them to apply for SSA disability benefits. At each site, the Research Assistant selects interviewees from participants in each demonstration arm (Full‑Service, Basic-Service, and Usual Services) and from non-participants. Non‑participant interviewees include: SSA disability applicants recently denied benefits who chose not to participate in the study; individuals who enrolled in the study, but not engaged with treatment services; and those who enrolled in the study but withdrew from treatment services. To the extent possible, interviewees represent diversity of age, gender, race, ethnicity, and employment status. We mail or hand a letter to potential interviewees (see Attachment T1. Participant Person‑Centered Interview Invitation Letter and Attachment T3. Non-Participant Person-Centered Interview Invitation Letter) describing the interview with proposed dates and times. If we mail the letter, the Site Visitor or the Research Assistant follows up with a phone call (see Attachment T2. Participant Person-Centered Interview Phone Call Sample Script and Attachment T4. Non-Participant Person‑Centered Interview Phone Call Sample Script) to the potential interviewee to confirm his or her interest in attending. We interview up to 3 participants (1 from each study arm) and 3 non-participants at each of the 30 sites in contract Years 2 through 5, for a total of up to 720 person-centered interviews. Each individual receives $40 for participating in the person-centered interview, which takes approximately 60 minutes.

3. Methods to Maximize Response Rates and Deal with Nonresponse

There are two types of non-response: (1) non-participation in the study and

(2) non-response to the quarterly interviews. Non-participants in the Supported Employment Demonstration are individuals who live within the area of a participating Community Mental Health Center who choose not to enroll in the study. The study approaches all eligible denied disability applicants in the project area first by sending them a letter followed by a telephone call to ascertain their interest in learning more about participating in the study. We consider those who indicate they are not interested as well as those who learn more about the study and then decide not to enroll as non-participants. We expected a few individuals contacted would be ineligible.

Social Security has a strong policy interest in knowing who elects to participate and whether or not they are different from those who choose not to participate in the SED project. Non-Participation reflects on the generalizability of the study to the defined group of denied disability applicants. The proportion of those denied applicants who decide to participate in the study represent the take-up rate. The take-up rate is a best guess at what actual participation would be if the agency were to roll out the services to all denied disability applicants. It is an imperfect measure, but can assist

policy-makers with decisions about demand for the services associated with effective treatments. Further, knowing the characteristics of participants and non-participants also contributes to a fuller understanding of demand. Social Security maintains an extensive database on their denied disability applicants. The application and disability review process requires the agency to collect applicant information. In addition to primary demographic variables, SSA collects work history, medical history, and functional limitations. Westat has access to the file information on all denied disability applicants in the defined areas of the community mental health treatment centers participating in the demonstration. Through the enrollment process, we carefully documented the decision of each individual who received a letter seeking participation in the study. The enrollment process resulted in three expected classifications of individuals deemed eligible to participate in the study to whom we send a letter, including those who are: (1) eligible participants; (2) eligible non‑participants; and (3) unable to contact. Among eligible participants with whom we made contact during the recruitment process, we documented individuals who turn out to be ineligible, and, therefore, not included in analyses of non-participation. Once enrollment ended, we conducted an analysis of the enrollment characteristics of the three groups to determine whether non-participants and those we were unable to contact were different from those who elected to participate in the study. We also addressed interview nonresponse. We implemented several strategies to maximize response rates and minimize non-response to the quarterly follow-up interviews. To ensure maximum participation throughout the study, we established a repository of contacts for each participant – the names, addresses, and telephone numbers of individuals who are familiar to the participant and who can get in touch with them if we cannot. During the baseline interview, the Research Assistant asks the respondent for the name and number of two such individuals to contact in the event that we cannot locate the respondent. Ideally, at least one of these alternate contacts should not live in the same household as the participant. Subsequently, as part of each quarterly follow-up interview, we verify the contacts previously provided by the participant. Our MIS included the data entry fields necessary for recording and updating the additional contact persons. We established a protocol whereby the baseline and quarterly interviews require the interviewer to: ask the participants if they have plans to move; reiterate the toll-free participant number; and offer a $5 cash incentive if the participants call to report their new address. Using these strategies described above, we expected to increase the probability of relocating participants who move following the baseline interview. We worked to minimize attrition and loss to follow-up for the quarterly interviews. We flag participants who initially refuse to continue their participation in a quarterly interview for further follow-up by a CATI interviewer experienced in refusal conversion.

We also conducted intensive tracing efforts for participants who are lost to follow-up because they are not locatable. TRC interviewers experienced in tracing individuals use this information as needed to obtain updated contact information for non-respondents who are not locatable. If these efforts prove unsuccessful, we may arrange for our local field staff to conduct in-person tracing efforts, starting with visiting the participants at their last known addresses. With over 50 years of experience in survey data collection, Westat has substantial knowledge of the challenges in achieving high response rates and is experienced in implementing data collection procedures to increase survey response. In addition to the hundreds of surveys Westat has successfully fielded, and our experience with the MHTS, in which we achieved an overall survey response rate of 84 percent (without incentives for the treatment group), we are confident that we can meet a target response rate of at least 80 percent for the SED project. If we do not meet this target response rate, we will conduct a non-response bias analysis.

3. Test of Procedures or Methods to be Undertaken

Westat conducted a cognitive test of new and revised questionnaire items to assess respondent understanding of the questions, as well as pretests of the interview instruments (screener, baseline, quarterly, and annual) to confirm respondent burden and assess question flow. We conducted both the cognitive test and pretests with no more than 8 denied disability applicants local to the Washington, DC metropolitan area. We used the results of the tests to refine the questionnaire items and minimize burden.

3. Statistical Agency Contact for Statistical Information

Westat staff developed the data collection and analysis plans; they are also responsible for collecting and analyzing study data.

1