Mai_mrt_ss_b 10.11.18

Monitoring Data Collection Tools for the Minority AIDS Initiative (MAI)

Supporting Statement

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

The Substance Abuse Mental Health Services Administration’s response universe for this data collection includes all active CSAP MAI grantees funded between now and 2021. In order to ensure accountability for the spending of federal funds, SAMHSA has employed the use of these data to ensure that grantees are meeting the goals and objectives of the grant. Data are used to monitor performance throughout the grant period. The Public Health Service Act Sec. 501 [290aa] (d) (13) with respect to grant programs authorized under this title, assure that “all grants that are awarded for the provision of services are subject to performance and outcome data collections.” SAMHSA interprets these requirements to indicate the need for data to be collected on all grantees receiving federal funds in order to provide services.

B2. Information Collection Procedures

Both the quarterly progress report and participant/client level data will be submitted to SAMHSA via an online, web-based data entry system. Data collection procedures will be shared with grantees and grantees will be trained on how and when to submit data.

The Quarterly Progress Report

The Quarterly Progress Report (Attachment 1) is a modular instrument structured around the Strategic Prevention Framework (SPF). Each section or module corresponds to a SPF step with an additional section dedicated to cultural competence and efforts to address behavioral health disparities, which is an overarching principle of the framework. A section-by-section description of the instrument is provided in Exhibit 4 below.

Exhibit 4. Structure and Frequency of Quarterly Progress Reports

Section	Contents	Frequency
Health Disparities	Activities implemented during the reporting period to address behavioral health disparities in the community Specific accomplishments and barriers related to health disparities	Twice a year, as part of the second and fourth quarterly progress reports
Assessment	Synopsis of the needs assessment conducted by the grantee and key findings: High-risk groups identified Infrastructure and capacity enhancement targets Quarterly accomplishments and barriers associated with needs assessment Quarterly updates as needed Accomplishments and barriers related to planning	At least once during the needs assessment phase of the SPF, updated as needed in subsequent quarters
Capacity	Staff, advisory group, governing board, and collaborator rosters Advisory council meetings Training and technical assistance events Accomplishments and barriers related to capacity building	At least once during the capacity building phase of the SPF, updated as needed in subsequent quarters
Planning	Planned target population(s) Project goals and objectives Targeted outcome measures Planned direct-service interventions Planned indirect services (i.e. environmental strategies and information dissemination) Planned HIV/HCV testing and hepatitis vaccination activities Accomplishments and barriers associated with strategic planning	At least once during the strategic planning phase, updated as needed in subsequent quarters
Implementation	Numbers served through direct services and numbers reached through indirect services, by demographic group Grant expenditures Direct-service intervention implementation (timing, frequency, dosage, location, adaptations) Detailed list of direct services delivered Environmental strategies and information dissemination activities implemented Numbers referred outside the grantee organization for further services, by service type Outreach/recruitment activities Accomplishments and barriers associated with intervention implementation and service delivery	Quarterly during the implementation phase
Monitoring	Link to upload evaluation plan (online only) Accomplishments and barriers associated with evaluation	At least once during the evaluation phase

Before implementing any intervention, the grantees are required to enter planning information about that intervention in the Planning module. The Planning and Implementation modules are similarly structured to allow the evaluators to assess the degree to which the implementation followed the plan (i.e. fidelity to the plan). The grantees are provided with a list of the most common evidence-based direct-service interventions (Attachment 2) in order to standardize the way in which intervention names are reported across time and across grantees. Once the online system is in place, the list will be available as a dropdown menu in the Planning section. The Implementation section will provide each grantee with a dropdown menu consisting of the interventions they entered in the Planning section. This dynamically updated grantee-specific dropdown menu will also be linked to the online participant-level instruments in order to link each participant’s pre-post and follow-up data to the direct-service intervention(s) they received.

Youth and Adult Outcome Questionnaires

Two common questionnaires will be administered to direct-service program participants. The Youth Questionnaire is designed for persons aged between 12 and 17 and the Adult Questionnaire is designed for persons aged 18 and older. For all common measures, administration guides are under development to assist program sites with administering and proctoring the surveys.

The major constructs for the youth outcome questionnaire include demographics, 30-day substance use, disapproval of peer substance use, perception of risk of substance use, perception of peers’ risky behaviors, sexual behavior, sexual self-efficacy, school connectedness, and knowledge of HIV (See Attachment 1 for a copy of the Youth Questionnaire).

The major constructs for the adult questionnaire include demographics, employment, 30-day substance use, perception of risk of substance use, sexual behavior, sexual self-efficacy, availability of social/emotional support, perception of risk of unprotected sex, knowledge of HIV, and perception of peers’ risky behaviors.

Data Collection

The last column of Exhibit 4 indicates the frequency with which each section (or module) of the Quarterly Progress Report will be completed. Data collection will start with a baseline data submission preceding the onset of funded activities and will be updated every year during the grant period. The Adult and Youth questionnaires will be administered at both baseline, and follow-up to participants.

Data Management

Storage System: Over the life of this initiative, the quarterly progress report and client level instruments have been submitted by the grantees either in paper form or as extracts from an online data submission system will be filed in a secure location at SAMHSA together with other grant management and monitoring records. Digitized versions of the data will be stored in secured databases in accordance with the security guidelines defined by the Federal Information Security Management Act (FISMA) and NIST Special Publication 800-53 for Moderate information system security.

Before initiating data collection, each site is responsible for setting up a filing and storage system that will accommodate these needs in a manner that safeguards the privacy and anonymity of the participants.

Web-Based Data Entry Upload System: SAMHSA’s online data entry system will allow grantees to enter item-by-item or to upload in batch file format all completed instruments. In order to ensure data quality and minimize data cleaning efforts, all online instruments include online validation checks that issue error messages and suggestions for correcting data entry errors such as inconsistencies or out-of-range values.

B3. Methods to Maximize Response Rates

Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for project officers to identify problems and provide technical assistance. In addition, project officers monitor data collection efforts and provide technical assistance to individual grantees as necessary. Because collection of these data is a stipulation of the grants, it is anticipated that all grantees will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents; therefore, grantees cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates at exit across grantee sites have averaged around 75 percent. Due to the high-risk nature of the participants, a substantial proportion of whom have historically been transient populations without permanent residence (e.g., homeless, unemployed, or reentry individuals). Post-exit follow-up rates have been around 35%. SAMHSA and its contractors continue to provide training and technical assistance to grantees for enhancing these response rates.

B4. Tests of Procedures

CSAP and its contractor have reviewed the Youth and Adult Questionnaires as well as the quarterly progress report to clarify terminology and language and eliminate unnecessary questions.

The estimates of time to complete the newly revised instruments are based on pilot tests. The mean of the time-to-complete figures reported by each tester was used as the final burden estimate. Individual testers’ estimates were close to each other with a narrow error margin, slightly under one minute on either side.

B5. Statistical Consultants

Thomas Clarke, Ph.D.

Public Health Analyst

Performance and Technical Assistance Branch

SAMHSA/CSAP

1 Choke Cherry Road, Room 4-1031

Rockville, MD 20857

240-276-0493

ATTACHMENTS

1. MAI Quarterly Progress Reporting Tool

2. Adult Questionnaires

3. Youth Questionnaires

4. Truth Initiative Comment (SAMHSA’s response in Section 8 of Supporting Statement A)

4