Supporting Statement for Paperwork Reduction Act Submissions
Evaluation of the Supportive Services Demonstration
OMB Control # 2528-0321
B. Collections of Information Employing Statistical Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
The Supportive Services Demonstration (demonstration, SSD) evaluation has two components: a process study to document how treatment group properties implemented the Integrated Wellness in Supportive Housing (IWISH) program and how property staff, residents, and caregivers responded to it; and an impact study to measure the effect of IWISH on key outcomes related to residents’ use of healthcare services and housing stability.
HUD’s Office of Policy Development and Research contracted with Abt Associates Inc. to conduct this two-part evaluation for the initial three years of the demonstration, from October 2017-October 2020. The Continuing Appropriations Act, 2021 and Other Extensions Act and the Consolidated Appropriations Act, 2021 provided additional funds and authorization to extend the demonstration for an additional two years (2021-2023). Abt will continue to evaluate the demonstration through September 2026.
The process study will include some data collection from the original 40 IWISH properties, with a primary focus on the 36 IWISH (treatment) properties that extended their participation in the demonstration for an additional two years, through 2023. The main data sources for the process study are interviews with IWISH program staff, residents of treatment properties, and property owners and managers of treatment and active control properties. These data collection activities do not employ statistical methods. The research team will select respondents purposively. Other data sources include data collected directly by the research team with data collected by the IWISH properties, administrative data collected by the U.S. Department of Housing and Urban Development (HUD), and public use data.
The impact evaluation will analyze administrative data obtained for residents of the treatment group, an active control group, and a passive control group. The main data sources for the impact study are Medicare Fee-For-Service claims, Medicaid Fee-For-Service claims, Medicare and Medicaid encounter data, HUD administrative data, and public use data to characterize the community. These data sources are not subject to the Paperwork Reduction Act (PRA) and are therefore not part of this ICR. They will be used to estimate the impact of the IWISH model on healthcare utilization, housing exits, transfers to nursing homes and other long-term care settings, and mortality. The impact of IWISH is the difference between the average outcomes among residents at IWISH properties and the average outcomes among similar residents in the control groups, estimated using experimental and quasi-experimental analysis methods. The public use data will be used to ensure the treatment and control groups are evenly matched on community characteristics and for contextual analysis.
All the data collection in this ICR will be done by Abt Associates Inc. and its subcontractor L&M Policy Research (the “research team”). The identification of respondents for each data collection activity is described below.
Identification of Respondents for the Interviews with Resident Wellness Directors
The research team plans to interview all Resident Wellness Directors working at the 36 treatment properties when the interviews are conducted in 2023. Most properties have one Resident Wellness Director, but a few properties have more than one. The research team expects to interview up to 54 Resident Wellness Directors.
Identification of Respondents for the Interviews with Wellness Nurses
The research team plans to interview all Wellness Nurses working at the 36 IWISH properties when interviews are conducted in 2023. Most properties have one Wellness Nurse, but a few properties have more than one. The research team expects to interview up to 44 Wellness Nurses.
Identification of Respondents for the Interviews with Treatment Group Property Owners and Managers
The research team plans to interview representatives from the organizations that own and manage each of the 40 IWISH properties. While we will invite the property owner on record to participate in the interview, owners may designate the property’s onsite manager or other staff to complete the interview.
Identification of Respondents for the Interviews with Active Control Group Property Owners and Managers
The research team plans to interview representatives from all the organizations that own and/or manage the 40 active control properties. While we will invite the property owner on record to participate in the interview, the owners may designate the property’s onsite manager or other staff to complete the interview.
Identification of Properties for Resident Interviews
The research team plans to conduct up to 150 resident interviews at 10 IWISH properties. Across all the properties, the research team is seeking a diversity of perspectives. The following criteria will be used to identify the 10 IWISH properties for resident interviews:
Residents with limited English proficiency: To ensure the experiences of residents with limited English proficiency (LEP) and preferences for languages other than English are included in the IWISH evaluation, we will prioritize properties with high proportions of LEP residents for interviews. We plan to include a minimum of five properties where at least 25 percent of residents demonstrate LEP. The percent of residents with LEP is based on data reported by IWISH and property staff during the evaluation of the initial IWISH period. We will confirm information on residents’ preferred and primary languages with IWISH staff prior to scheduling interviews with residents.
Geographic location: We plan to conduct interviews in each of the six states with treatment properties in the extension period (California, Illinois, Maryland, Massachusetts, Minnesota, and New Jersey), including at least one property in each state. Treatment properties in the seventh state, South Carolina, have declined to participate in the second phase of the demonstration.
Fidelity Ratings: Properties vary in the extent to which they implement the IWISH model in a manner aligned with the model’s goals. The research team assessed property-level fidelity to the IWISH model for the initial three years of the demonstration (2017 through 2020) using data collected from site visits, interviews, and focus groups. We plan to conduct interviews at properties with a mix of fidelity ratings to capture a range of resident experiences with IWISH. We plan to include at least two properties in each category of fidelity rating (high, medium and low).
Based on the criteria above, we will identify an initial sample of at least 12 properties. We will select first on proportion of LEP residents, then by state, and finally on fidelity to the IWISH model. We will also adjust the sample to ensure properties include variety in program type (Section 202 PRAC or Project-Based Section 8 properties) and property size. In consultation with HUD and the study’s expert panel, this initial sample of properties will be further refined to identify a final list of 10 properties. We will also take into consideration the property’s willingness to accommodate resident interviews.
Identification of Respondents for Resident Interviews
We plan to conduct between 10 and 15 interviews at each of the 10 properties selected. We will use Tenant Rental Assistance Certification System (TRACS) data to identify residents who live at the 10 IWISH properties where interviews will be conducted.
We will apply the following restrictions to ensure that the residents selected for the interview have sufficient history with the property and are of the correct age to provide relevant information relevant to the process study’s research questions:
Residency: Residents must have lived at the property for at least one year prior to the interview. In other words, if the interviews were scheduled for July 2023, we would exclude any residents who moved in after July 1, 2022. This restriction is in place to ensure that respondents have experience with the IWISH model.
Age: Residents must be aged 62 or over as of the start of the demonstration extension period in October 2021. The demonstration is targeted to people aged 62 and over and this age group is the focus of the evaluation.
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Process Study. The data collection covered by this submission is intended to provide qualitative information to support a study of the implementation of the SSD. The process study will qualitatively describe how IWISH was implemented at the treatment sites and how it was experienced by program staff, residents, and owners and managers, as well as how the IWISH implementation compares to the typical service coordination provided at HUD multifamily properties serving older adults.
The research team will not draw statistical inferences from the qualitative data covered in this ICR submission; instead, the analysis of administrative data will provide the main evidence of program outcomes and impacts.
This submission does not require a statistical sampling plan, as the sampling of properties and residents is purposive. Interview respondents for the Resident Wellness Director, Wellness Nurse, and property owner and manager interviews are chosen because they have the most knowledge of IWISH at each site. These respondents will constitute the full universe of relevant individuals.
Resident interview respondents are limited in number and are not intended to constitute a representative sample of residents at IWISH properties. The number of interviews planned for this study was determined based on the total resources available for the study and the relative importance to the study’s research questions. We expect that 150 resident interviews will yield sufficient information for answering the study’s research questions and expect there would be diminishing returns on additional interviews.
Impact Study. Prior to engagement with Abt, HUD designed a cluster-randomized controlled evaluation and randomly assigned eligible properties to treatment and control groups to support the evaluation. The number of treatment and active control sites was limited to 40 based on preliminary power analyses and budgetary constraints. The random assignment was designed and conducted to facilitate an impact evaluation so causal inferences can be made.
To support the impact analysis, the research team will use linked HUD, Medicare, and Medicaid administrative data for HUD-assisted residents of the 40 treatment properties, the 40 active control properties, and the 44 passive control properties. For the initial three-year evaluation period (October 2017 – September 2020), the research team analyzed approximately five years of person-level administrative data—two years of data prior to the start of the demonstration (October 2015 – September 2017) and three years of data covering the duration of the initial demonstration period. For the continuation of the study through 2026, the research team will analyze person-level administrative data through 2025.
To estimate the impact of IWISH on resident outcomes, the research team will conduct an intent-to-treat (ITT) analysis by comparing outcomes of residents in the treatment group to those of residents in the active and passive control groups, which will be pooled into one control group.
In addition to the main impact analyses, the research team will conduct additional exploratory analysis of key outcomes to examine: (1) non-linear trends in the cumulative effect of IWISH on healthcare utilization and spending during the demonstration; (2) potential heterogeneity of the treatment effect across important subgroups of individuals; (3) and the extent to which sample attrition due to death might bias the estimated impact of IWISH on healthcare utilization and spending.
Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
The research team expects a 95 percent or better response rate from the staff, owners, and managers of IWISH properties to be interviewed. This is because the properties in the study have entered into cooperative agreements with HUD that require their participation in evaluation activities. The research team expects a lower response rate, about 80 percent, from property owner and management representatives of active control properties because these properties are not required to participate in evaluation activities.
The research team plans to conduct up to 150 interviews with residents at 10 treatment properties, with an average of 15 interviews per property. The exact number of interviews per property may vary depending on resident interest and property size. All eligible residents will be invited to participate in the interview. As the average number of residents living at each treatment property is approximately 100, the research team expects a response rate of between 10 and 15 percent across the 10 properties. This response rate is consistent with other studies of comparable populations.
The research team plans several ways to meet the expected response rate for resident interviews. Most interviews are expected to take place in-person during site visits. Prior to a planned site visit, the study team will mail a recruitment letter to all residents in the selected properties to share information about the details of the evaluation and request their participation in a resident interview. We will also ask IWISH staff to post flyers in common areas and share information about interviews with residents. Recruitment materials will include a toll-free phone number and project-specific email address for residents to contact the study team to schedule an interview during the site visit.
Residents will also be recruited in person during the site visits. On-site recruitment will vary by setting and property-specific limitations, but we plan to staff a table in resident common areas with recruitment materials to engage residents. Members of the study team will approach residents as they pass through the space and ask them if they would like to learn more about the evaluation and resident interviews, provide residents with recruitment materials, and answer questions as needed. If residents are interested in participating but unable to participate during the time of the site visit, we will conduct interviews via telephone/videoconference after the site visit if possible.
Recruitment and consent materials will be available in the three most common languages spoken by residents at the 10 interview properties. However, residents who speak other languages will still be eligible to participate if they are interested via the use of a language access card. This card will include information about the evaluation and how to contact the evaluation team to participate in an interview in the 10 most common languages of IWISH residents. If the resident’s language is not spoken by a member of the research team on the site visit, we will schedule a telephone/videoconference interview with the resident and conduct the interview using real-time interpretation services. Verbal consent will be obtained in the resident’s language in real time at the start of the interview.
We will offer residents the opportunity to bring their caregivers to the resident interview if they choose, which may make some residents more willing to participate.
All residents participating in an interview will receive a $40 gift card. 1 The incentive amount is consistent with other studies of comparable populations with similar interview lengths.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
All interviews in this data collection will request answers to identical questions from 10 or more respondents.
The research team is planning a formal pretest of the resident interview guide prior to the start of data collection. We will pretest the Resident Wellness Director, Wellness Nurse, and owner interview guides with two treatment properties and two active control properties (owner interview only) in 2022, prior to the start of the main data collection.
After pretesting interviews are complete, the interviewer team will meet to debrief. Depending on what we learn, we may propose modifications to the interview guide or the recruitment process. HUD’s Contracting Officer’s Representative (COR) will participate in the research team’s debriefing meeting and will review any proposed modifications. The research team will submit any proposed modifications to the Office of Management and Budget for review and approval.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
HUD has contracted with Abt Associates to conduct the data collection. The data collection procedures will be similar to those used in other studies conducted by Abt Associates. The HUD COR reviewed all the procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD COR, Amanda Gold (202-402-7619) or the Abt Associates Project Director Melissa Vandawalker (617-349-2611).
In addition, Abt Associates and HUD have established an expert panel to review the evaluation design, progress, and findings, to maximize the rigor of the evaluation and its value to multiple stakeholders. The following panelists have focused on the statistical aspects of the study’s design:
Partha Deb, PhD, Professor of Economics, Hunter College, City University of New York.
Kosuke Imai, PhD, Professor of Politics and Director of the Program in Statistics and Machine Learning, Princeton University.
1 If a resident chooses to bring a caregiver to one of the resident focus groups, the resident will receive a gift card but not the caregiver.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | h03483 |
| File Modified | 0000-00-00 |
| File Created | 2023-10-26 |