Thm_omb_ss-b_12 19 13

Assessment of the Town Hall Meetings on Underage Drinking Prevention

Supporting Statement

B. Statistical Methods

B.1 Respondent Universe and Sampling Methods

For the Organizer Survey, the respondent universe consists of community-based organizations (CBOs) that are conducting biennial town hall meeting (THM) events. For the Participant Form, the THM event attendees make up the respondent universe. The sampling method for both instruments is described below.

Organizer Survey

The Organizer Survey will be collected via a census sampling of all CBOs registered to conduct a THM (N=2,220). SAMHSA has collaborated with National Prevention Network (NPN) members of each state, U.S. territories, and the District of Columbia for the past four rounds of THMs (2006, 2008, 2010, and 2012) to identify CBOs in their respective state to organize at least one biennial THM event. Along with NPN members, other entities will be invited by SAMHSA to identify CBOs to organize events in 2014 and 2016. Once the recommendations are received by SAMHSA, the CBOs then register online and confirm their participation in the THM initiative. The CBOs are provided with information about obtaining a stipend from SAMHSA to conduct a THM in their community and of the requirement to complete the web-based Organizer Survey. They will also be notified at this time that a sample of the CBOs will be selected to take part in the Participant Form data collection.

The CBOs agree to conduct at least one event in their respective communities, and it is solely up to the CBO as to whether it will conduct more than one THM, since this is often based upon resources and funding. The information collected on the Organizer Survey will be used to document the nationwide events, determine if the THMs lead to additional activities within the community that are aimed at preventing and reducing underage drinking (UAD), identify what these activities may include, and help to plan for future rounds of THMs.

Participant Form

To determine the Participant Form respondents, we will design and select a two-stage stratified random sample of CBOs (events) and participants in the THM events. The sample is designed to address the main objectives of the study. The design is focused on capturing the intent of community members to engage in the prevention and reduction of UAD as a follow up to the THM events and to obtain other feedback on effectiveness of the THM.

At the first stage, we will select a random sample of CBOs with equal probabilities. We will identify whether the CBO has an online or in-person event, and restrict the sampling frame to the latter mode of implementation (in person). At the second stage, we will select all participants from each sample CBO to simplify operations. The first-stage (CBO) sampling frame will be compiled using the event details submitted online by every CBO in the target population. The frame can then be subset to those CBOs planning in-person events, information that will also be collected in the event details. We anticipate an average of n=40 sample participants for this stage of sampling. For an anticipated response rate of approximately 75 percent, the total number of responding participants will be 30 per CBO on average.

The expected precision depends on the anticipated intracluster correlation within CBOs (the homogeneity of clusters); i.e., on how similar participants within a same CBO are along the key survey variables. As discussed in the brief precision analysis below, we anticipate substantial clustering effects. For the 150 sample CBOs, and an average 30 responding participants per sample CBO, the anticipated total number of completed surveys is 4,500. Effective sample sizes that take into account design effects (DEFFs), however, range from 1,500 (DEFF=3) to 4,500 (DEFF=1). As discussed next, the sample sizes are large enough to protect against design effects as large as 3.0 or even 4.0.

The precision calculations are focused on estimated percentages, or proportions, that apply to the key survey variables of dichotomous form. Examples include:

• Did you learn anything about UAD and its associated problems that you didn’t know before attending the Town Hall Meeting event?

• Do you plan to share any material(s) or lessons learned from the Town Hall Meeting event with others?

• Did you learn of specific ways in which you, as an individual, can help to prevent UAD?

The table below shows the standard error of estimated percentages for a range of design effects (DEFFs). The design effect is defined as the variance under the actual design divided by the variance under simple random sampling. To be conservative, we use percentages of 50 percent, where standard errors are the largest possible. The design effect reflects clustering effects that tend to inflate variances.

Standard error of estimated percentages based on a sample of 150 CBOs (4,500 participants)

Design Effect (DEFF)	Standard Error
1.0	0.7%
2.0	1.1%
3.0	1.3%
4.0	1.5%

The table shows that estimated percentages will be within +/2.5 percent for 95 percent confidence intervals even if design effects are as large as 4.0. If design effects are expected to be smaller—for example, expected to be 2.0 or less--we could consider CBO sample sizes as small as 100 which yields an effective sample size of 2,250 for DEFF=2. To be conservative, we plan to select 150 sample CBOs for the precision described in the table above.

B.2 Information Collection Procedures

The identified individual at each organizing CBO will be e-mailed, within 1 week following his or her THM event, instructions on how to access the Organizer Survey (see Attachment 5). As respondents complete each page of the survey and click Next, data entered will be automatically saved. The respondents will be allowed to return to the survey until they click Done. Once the survey is submitted (i.e., Done is clicked), CBOs will not be allowed to go back into the survey to make changes. Organizers who do not complete the Organizer Survey more than 30 days following their THM event will be sent a reminder e-mail requesting them to complete the survey by a defined date (see Attachment 6).

Sampled CBOs will be asked to obtain feedback at the conclusion of their THM from event attendees using the Participant Form. At least 2 weeks prior to the scheduled THM, CBOs will be provided copies of the Participant Form and instructions for handling the form. The instructions will include information on, but not limited to, distributing the forms to attendees; performing visual quality control checks on the forms (e.g., reviewing the form to ensure that the date and location of the THM event is recorded); and how and where to send the completed forms.

CBOs that already have a mechanism in place to collect feedback from event attendees will be given the option to incorporate questions from the Participant Form into their instrument. Organizers will be provided a preaddressed postage-paid envelope to submit Participant Forms to the study evaluators within 30 days of their event. CBOs may, at their own discretion, also submit the data electronically to a designated e-mail address. The study evaluators will perform data entry of the Participant Forms.

B.3 Methods to Maximize Response Rates

Several methods will be used to maximize response rates:

• Contacts at participating CBOs will be sent an invitation to participate e-mail (see Attachment 7), which highlights key expectations for agreeing to host THM events. One of the key expectations is that CBOs will document their THM events and planned followup activities by completing the Organizer Survey. CBOs will be informed that they may be selected to obtain feedback at the conclusion of their THM from event attendees using the Participant Form.

• Organizers of THM events will be sent an e-mail, within 1 week following their THM event, containing information on how to access the Organizer Survey. A customized code will be used to track the submission of the survey.

• Organizers who do not complete the Organizer Survey more than 30 days following their THM event will be sent a reminder e-mail requesting them to complete the survey by a defined date.

• Organizers who are unable to access the Organizer Survey online may be sent an offline version of the survey through e-mail. Organizers will be asked to return the survey by e-mail or toll-free fax to the study evaluators.

• CBOs selected to solicit feedback from event attendees will be sent a reminder e-mail or contacted by phone if Participant Forms are not returned more than 30 days following their THM event.

For the 2012 THM cycle, which utilized the data collection instruments and survey methods under this current request, the response rate for the Organizer Survey was 78 percent (1,085/1,398 registered CBOs). SAMHSA anticipates that the response rate to the Organizer Survey in the next THM cycles will be similar to that of 2012 THM cycle, more than 75 percent.

The response rate for the Participant Form was 60 percent (211/351 CBOs), with a total of 5,446 forms submitted. With the decrease in the number of CBOs sampled to take part in the Participant Form collection in the next THM cycles, SAMHSA anticipates that the response rate would increase to approximately 75 percent. Although SAMHSA envisions strong enthusiasm by the CBOs and event participants to take part in the Participant Form collection, it is projected that some CBOs may not be able to obtain feedback from all of the sampled attendees.

B.4 Tests of Procedures

The updated Organizer Survey was sent to individuals in three CBOs that organized a 2012 THM event. These individuals were asked to identify any question that they did not understand or thought they would not be able to answer. They were also asked to report the amount of time it took them to complete the instrument. Those individuals easily understood all of the questions, and no questions were identified as being difficult to answer. Based on the review of comments, no changes were made to the instrument. In light of this input and the simple, straightforward nature of the testing procedures, pretesting is not necessary.

B.5 Statistical Consultants

The following individuals provided statistical consultation in development of the surveys and data collection methodology:

Tracy Elise Farmer, M.B.A. Public Health Analyst SAMHSA/CSAP 1 Choke Cherry Road Room 4-1067 Rockville, MD 20857 Phone: (240) 276–2555	Rená A. Agee Senior Research Analyst ICF 530 Gaither Road 7th Floor Rockville, MD 20850 Phone: (301) 572–0400	Charlotte A. Ball, M.P.A. Senior Product Development Manager ICF 530 Gaither Road 6th Floor Rockville, MD 20850 Phone: (301) 407–6590
William R. Beard, Jr. Project Director ICF 530 Gaither Road 6th Floor Rockville, MD 20850 Phone: (301) 407–6544	Ronaldo Iachan, Ph.D. Statistician ICF 530 Gaither Road 7th Floor Rockville, MD 20850 Phone: (301) 572–0538

References

Centers for Disease Control and Prevention. (2012). Teen drinking and driving. CDC vital signs. From http://m.cdc.gov/en/VitalSigns/teen-drinking-and-driving (accessed May 1, 2013).

Mothers Against Drunk Driving. (2013). MADD Analysis Finds Majority of Underage Drinking Deaths Not Traffic Related. From http://www.madd.org/media-center/press-releases/2013/madd-analysis-finds-majority.html (accessed May 1, 2013).

Masten, A., Faden, V., Zucker, R., & Spear, L. (April 2008). Underage drinking: A developmental framework. Pediatrics, 121 (Supplement 4), S235–S251.

Substance Abuse and Mental Health Services Administration. (2012). Results from the 2011 National Survey on Drug Use and Health: Summary of National Findings, NSDUH Series H-44, HHS Publication No. (SMA) 12-4713. Rockville, MD: Substance Abuse and Mental Health Services Administration.

U.S. Department of Health and Human Services. (2007). The Surgeon General’s Call to Action To Prevent and Reduce Underage Drinking. Department of Health and Human Services, Office of the Surgeon General. From www.surgeongeneral.gov/library/calls/underagedrinking/calltoaction.pdf (accessed May 1, 2013).

Bureau of Labor Statistics. (18 December 2013). May 2012 Occupational Employment Statistics. Web. From http://data.bls.gov/oes (accessed December 18, 2013).

Office of Personnel Management. (18 December 2013). Salary Table 2013-DCB. Web. From http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2013/general-schedule/dcb.pdf (accessed December 18, 2013).

U.S. Department of Labor. (18 December 2013). Wage and Hour Division. Web. From http://www.dol.gov/whd/minimumwage.htm (accessed December 18, 2013).

List of Attachments

1. Organizer Survey

2. Participant Form

3. Participant Form (Spanish version)

4. ICF IRB Review Findings Form

5. Initial E-Mail to Organizers—Organizer Survey

6. Reminder E-Mail to Organizers—Organizer Survey

7. Invitation to Participate E-Mail

2