Evaluation Questions

Attachment 1:
Evaluation Questions

The evaluation questions drive all other aspects of the evaluation, including plans for data collection. The evaluation questions are organized based on the type of evaluation (process, outcome, or economic) and on the level of data needed to answer them (grantee, performance site, or patient). In addition to process, outcome, and economic questions, there are system-wide questions that integrate evaluation results from all three to present a comprehensive picture of the effects of SBIRT on the treatment system as a whole. The current evaluation questions are listed in Sections 1 through 4.

1 Process Evaluation Questions

1.1 Grantee

1. What are the theoretical (e.g., stages of change theory) or logical underpinnings of the SBIRT programs in terms of training protocols, dissemination plans, sustainability expectations, and patient protocols (i.e., screening, brief intervention [BI], brief treatment [BT], or referral to treatment [RT])?

2. How well are the programs being implemented in the field as compared to the program description in each grantee’s implementation plan and as compared to the SBIRT evidence base?

3. What are the barriers and facilitators faced by grantees related to data collection for ongoing quality assurance (QA) and accountability?

4. What are the barriers and facilitators faced by grantees related to service delivery?

1.2 Performance Site

1. Who are the EDs, trauma centers, group practices, and other health care settings providing SBIRT services?

2. Who are the practitioners within those settings delivering SBIRT services?

3. How do the work-flow processes vary across the performance sites, and what explains these variations?

4. What are the barriers and facilitators with those performance sites related to data collection for ongoing QA and accountability?

5. What are the barriers and facilitators related to service delivery within performance sites?

1.3 Patient

1. Who are the patients/clients receiving SBIRT services? How many are there? Who and where are they, in terms of demographics, drug use and treatment history, socioeconomic status, and rurality?

2. What barriers and facilitators do patients encounter related to data collection for ongoing QA and accountability?

3. What barriers and facilitators do patients encounter related to service delivery?

2 Outcome Evaluation Questions

2.1 Grantee

1. Are more and/or different types of patients obtaining treatment through the availability of SBIRT services?

2.2 Performance Site

1. To what extent is SBIRT actually performed as measured by the number of patients screened, the number intervened with, the percentage of patients screened, and the percentage of patients screening positive who are intervened with or referred to treatment?

2.3 Patient

1. What is the effectiveness of the SBIRT program on illicit drug use?

2. What is the effectiveness of the SBIRT program on other substance use (e.g., alcohol or prescription drugs)?

3 Economic Evaluation Questions

3.1 Grantee

1. What are the costs of the SBIRT program to the grantees?

2. What financing models are feasible for sustaining the SBIRT programs as implemented by the grantees?

3.2 Performance Site

1. What are the costs of SBIRT to performance sites?

2. What financing models are feasible for sustaining the SBIRT programs as implemented in the performance sites?

3.3 Patient

1. What is the effect of SBIRT on patients’ economic outcomes, such as employment and income?

4 System Questions

1. What effect does the introduction of SBIRT have on the existing treatment system?

2. What possible changes in the treatment field overall were associated with this new development?

3. To what extent did SBIRT influence the grantee’s systems change and reimbursement agenda?

4. What are the prospects for the widespread adoption and long-run sustainability of SBIRT?

5. What is the cost-effectiveness of SBIRT in improving patient outcomes as compared to other treatment models?

6. What grantee- and performance site-level characteristics moderate the effectiveness and cost-effectiveness of SBIRT for patient-level outcomes?

7. What grantee- and patient-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt SBIRT?

2