20200716 life cycle evaluation ORC SSB

Submitted to:
AmeriCorps
250 E. St., SW

Washington, DC 20024

Submitted by:
ICF Incorporated, LLC

9300 Lee Highway

Fairfax, VA 22031
https://www.icf.com

May 3, 2021

Supporting Statement for Paperwork
Reduction Act Submissions

Peer Recovery Coach Model Bundled Evaluation

Part B. Collection of Information
Employing Statistical Methods

The beneficiary survey will be a census of the universe of individuals served by opioid recovery coaches (treatment) and those at the same site receiving opioid use treatment but who are not served by peer recovery coaches (comparison).

B.1. Respondent Universe

The universe of projects to be included in this data collection is shown in Exhibit B-1. Since the full universe of projects will be included, no statistical sampling methods will be implemented.

Exhibit B-1. Estimated size of universe to be sampled

Population	Universe: site/individual	Sample: site/individual
Beneficiary Treatment	15/500	15/500
Beneficiary Comparison	15/500	15/500

B.2. Statistical Methodology

B.2.1. Statistical Methodology for Stratification and Sample Selection

For the beneficiary survey, we will include all individuals served by opioid recovery coaches (~500) and similar individuals receiving opioid recovery treatment but not served by opioid recovery coaches (~500) from the same 15 projects that received AmeriCorps grants in 2022-2023. These projects were identified by ICF through an extensive grant application review.

B.2.2. Estimation Procedure

We will first generate descriptive statistics for each item in beneficiary surveys. Descriptive analyses will help to answer questions about program design, implementation, and the resulting outcomes. In addition, we will also look at changes in these outcomes between baseline and 12-month follow-up.

Comparative analyses will examine the extent to which observed differences in outcomes can be attributed to the opioid peer recovery coach model. We propose taking a regression approach that examines the relationship between program status and relevant outcomes while controlling for other confounding variables. Specifically, we anticipate using two-level hierarchical linear modeling (HLM). HLM produces more accurate results than ordinary least square models by modeling the hierarchical structure in the organization (e.g., individuals nested within sites), correcting for aggregation bias, misestimated precision, and the unit of analysis problem. At level 1, individual outcomes will be predicted as a function of program status (treatment versus comparison) and other demographic and academic characteristics. At level 2, site characteristics will be used as covariates. The differences between the treatment and comparison groups will be assessed in terms of 1) p-values, or the probability that the observed differences could be due to chance, and 2) effect sizes, which measure the magnitude of the differences. Results from the comparative analyses will provide evidence of whether the opioid peer recovery coach model made a difference in institutional and individual outcomes.

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Jurisdiction

Most of the statistics produced will be descriptive. However, in cases where group differences are examined statistically, we plan to set the α at 0.05 level as a significance threshold.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No unusual problems are anticipated.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

As described previously, the beneficiary surveys will be administered for both treatment and comparison groups at the baseline and 12-month follow-up. The evaluation technical assistance outcome survey will be administered for the participants at the baseline before they attend the workshops, and at 12-month follow-up when they complete all the sessions.

B.3. Methods to Maximize Response Rates and Deal With Issues of Non-response

For beneficiary surveys, the list of treatment and comparison groups will be provided to ICF by participating grantees; the list will include names, treatment status, and emails. Perspective respondents will receive several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission.

Before each data collection, participating grantees will email a pre-notification and encourage all perspective respondents to participate in the survey. Within one week of the pre-notification, ICF will follow up with a recruitment email providing a link to the
15-minute survey URL programmed by Qualtrics. Before responding to the survey, participants will be asked to read the consent form describing the purpose of the survey. The form assures that participation is voluntary and that responses are confidential, and provides contact information for the study lead and Institutional Review Board (IRB). After providing consent, participants will proceed to answer the survey. Respondents will be given one week to respond to the survey. The research team will continue to monitor the completion. We will send two reminders on a weekly basis to non-respondents. Respondents will be sent a “thank you” for their participation.

Based on a similar bundle evaluation on opportunity youth funded by AmeriCorps, we expect that 45% of those in the treatment group responding to the baseline surveys will respond to the follow-up survey, and 25% of the comparison group will do so. A non-response bias analysis will be conducted using auxiliary data, including demographics, from the sampling frame. Any strong predictors of non-response may be candidates for building a response propensity model, which could then be incorporated into the non-response adjustment as part of the survey weights for this group.

B.4. Tests of Procedures and Methods

The survey design, data collection procedure, and analysis approaches have been reviewed and approved by the AmeriCorps Office of Research and Evaluation. In addition, as described in Section A.8., the research plan has been reviewed by an external Technical Working Group consisting of six nationally recognized experts in evaluation of national services and a Field Working Group consisting of six program leaders in AmeriCorps. Finally, the survey instrument, consent form, and recruitment letter were reviewed and approved by ICF IRB.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

• Amy Borgstrom, Associate Director of Policy, COO Immediate Office, AmeriCorps, 202-606-6930 

• Mary Hyde, Director, Office of Research and Evaluation, AmeriCorps,
  202-606-6834

• Lily Zandniapour, Research & Evaluation Manager, Office of Research and Evaluation, AmeriCorps, 202-606-6939

• Jehyra M. Asencio-Yace, Research Analyst, Office of Research and Evaluation, AmeriCorps, 202-956-9736

Contractor

• Xiaodong Zhang, Project Director, ICF, 703-251-0883

• Jing Jun, Senior Analyst, ICF, 703-225-2328