Dsp_mrt_ss_b_dsp_mrt 7.24.18

DIVISION OF STATE PROGRAMS - MANAGEMENT & REPORTING TOOL

SUPPORTING STATEMENT

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

The Division of State Programs’-Management and Reporting Tool (DSP-Tool) will use a census approach to collecting process and programmatic data along with collecting aggregated existing archival data and data from survey samples for the outcomes measures.

Using a census approach, the targeted universe is all Partnerships for Success (PFS), Strategic Prevention Framework Rx (SPF-Rx), and Prescription Drug Overdose (PDO) grant programs. For PFS, the program includes grantee Project Directors for cohorts, 2014 (N=21), 2015 (n=32), 2016 (N=2) and PFS 2018 (N=17). PDO includes Project Directors for cohorts 2016 (N=12) and 2017 (n=11). Finally, SPF-Rx includes Project Directors for cohorts 2016 (N=25). New grant programs may be added in the future to answer standard SPF questions. A census of all Project Directors from all three programs—PFS, SPF-Rx, and PDO--is necessary as these data will be used by SAMHSA to monitor each program’s performance and grantees will also use it to track their ongoing implementation and reporting requirements such as GPRA. In order to meet SAMHSA’s annual reporting requirements for GPRA and performance measures, and more frequent reporting requirements for programs within the Division of State Programs, SAMHSA must obtain data from all grantees, which supports the need for a census approach. In addition, the program grantees encompass a wide variety of organizational types and structures that are implementing a range of prevention interventions targeted to different populations and with various outcome goals.

In addition to the GPRA, data collected by grantees will be used to demonstrate how SAMHSA’s grant programs are reducing disparities in access, service use, and outcomes nationwide.  To accomplish this, SAMHSA expects grantees to utilize their data to (1) identify subpopulations (i.e., racial, ethnic, sexual/gender minority groups) vulnerable to health disparities and (2) implement strategies to decrease the differences in access, service use, and outcomes among those subpopulations.  There will be subpopulations where sample size is too small to properly sample and where a census would be necessary to obtain relevant and reliable outcome data.

B.2. Information Collection Procedures

All instruments are self-administered, web-based data collection tools completed through an online data collection system. All respondents are the grantee Project Directors or their staff. Before data collection performance monitoring begins, respondents will be provided a unique log-in to enter the data system, where they will be required to create a password. Respondent email addresses for each login will be stored within the system so that automatic alerts and notifications can be sent.

Pending Office of Management and Budget (OMB) approval, the estimates in the DSP MRT will be collected according to the schedule outlined in section A-16.

The contractor developed user manuals for accessing and navigating the MRT online data collection system and question-by-question and frequently asked question (FAQ) guides to help respondents accurately complete the instruments.

Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time on the contractor’s website.

Availability of the system is important in any data collection system, especially one employed by grantee sites around the country, including multiple time zones and pacific jurisdictions. The online system will be maintained in an available state as much as possible to allow grantees to have access for entering data, as well as to give SAMHSA, grantees, and team access to reports.

Providing a robust system that is simple and easy to use across all areas is also critically important. To achieve this, the MRT will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the online data system will have a “Help” or “Support” link located in the upper right corner, which will allow the respondent to access the following support resources:

1. Search the Knowledge Base. More comprehensive than a list of FAQs and more organized than a support forum, the Knowledge Base offers a “layered information” approach so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• system documentation,

• user guides,

• policies and procedures,

• protocols,

• training materials, and

• FAQs.

1. Contact Us. Respondents may request assistance by calling a provided toll-free number, sending an email request, or submitting a technical assistance submission form as desired. The toll-free line will be routed to an email system that is checked regularly by members of the training and technical assistance team. Staff responding to technical assistance requests will be trained in use of the system and have ready access to the full Knowledge Base. Training and technical assistance team staff will monitor all submitted tickets to ensure timely response and resolution of technical assistance requests.

B.3. Methods to Maximize Response Rates

Grantees participate in all performance monitoring data collection activities per the Terms and conditions of the grant award. The evaluation team will employ a number of strategies to help ensure grantees participate with a 100% response rate.

As described above, the development of user manuals for accessing and navigating the online data collection system and question-by-question and FAQ guides to help respondents accurately complete the MRT instruments. Grantees will also be provided training webinars to walk through the MRT online data collection system and to review data collection procedures. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

SAMHSA Project Officers will monitor the DSP-MRT and receive e-mail notifications when their grantees submit individual monitoring data. Approximately one month after a data submission deadline the contractor will provide Project Officers a list of past due instruments. SAMHSA Project Officers will then follow up with their grantees to ensure submission.

B.4. Test of Procedures

Three contracting staff completed the instrument within a testing version of the system. These staff members have experience with SPF model used in various CSAP programs

The Standard questions in the DSP- MRT are estimated to take 3 hours to complete and the program specific modules—Rx, PDO, and PFS—are estimated to take 1 hour to complete.

Each of the grantees is a former CSAP grantee; thus they will all have experience completing instruments similar in procedure (e.g., entering data into an online data system), length, and content. Additionally, the SPF performance measure data collection used lessons learned from the earlier data systems to improve data collection procedures.

B.5. Statistical Consultants

The Government and contractor team comprises several experts who will be directly involved in data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, and data analysis, including leveraged funding analysis. The table below provides details of these team members and advisors.

Statistical Consultants for the SPF-PFS Performance Measurement and Contract Staff

Name & Role in Evaluation	Title & Address	Contact Information
Government Officers
Thomas Clarke, PhD. Senior Social Science Analyst	Senior Public Health Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1025 Rockville, MD 20857	SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 16-E03Rockville, MD 20857 [email protected] Phone (240) 276-2708
Stephanie Blake M.A. Advisor	Social Science Analyst CSAP, SAMHSA 1 Choke Cherry Road, Room 4-1031 Rockville, MD 20857	SAMHSA/CSAP Performance and Technical Assistance Branch1 Choke Cherry Road, Room 4-1031Rockville, MD 20857 [email protected]

LIST OF ATTACHMENTS

Attachment 1 - Division of State Program--Management Reporting Tool

Attachment 2- SPF-Rx

Attachment 3- Section B: PDO

Attachment 3- Section C: PFS Outcomes

3