Part B OMB ICS Multisite evaluation In Community Spirit 6 7 12

Supporting Statement for the Multisite Evaluation of the In Community Spirit Program—Prevention of HIV/AIDS for Native/American Indian and Alaska Native Women Living in Rural and Frontier Indian Country

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Key Informant Interviews—Baseline. The Key Informant Interviews—Baseline (Attachment A.1) will be conducted for each cooperative agreement community. Multisite evaluation contractor liaisons will work with grantee project directors to identify 5 potential interview participants at each community through purposive sampling. Respondents will be both tribal and non-tribal and will be selected as a function of their involvement in the implementation of the In Community Spirit Program and not as a function of their ethnicity or membership of a tribal nation. If one or more of the original potential respondents is unable or unwilling to participate in the interview, the grantee project director will identify one or more additional respondents to reach a maximum of five completed interviews per grantee community. It is estimated that 30 respondents will participate in Key Informant Interviews—Baseline, and that 30 respondents will be sufficient to ensure saturation of themes in the content analysis of the qualitative data.

Key Informant Interviews—Follow-up. The Key Informant Interviews—Follow-up (Attachment A.2) will be conducted 11 months after Baseline for each grantee community. Interviews will target the same adult grantee program administrators, providers, and/or partners that participated in the Key Informant Interviews—Baseline data collection activity. No additional sampling is anticipated. However, in the event that a true follow-up interview is not possible (e.g., person is no longer a member of the agency or organization), the multisite evaluation contractor liaison will work with the grantee project director to identify an appropriate interview replacement using purpose sampling to ensure a maximum of 5 completed interviews per grantee. It is estimated that 30 respondents will participate in Key Informant Interviews—Follow-up and that five respondents per grantee community will be sufficient to ensure saturation of themes in the content analysis of results from the qualitative interviews.

Women’s HEAL Survey—Prevention Education. Respondents for the HEAL Survey—Prevention Education (Attachment B) will represent a sample of AI/AN women participating in primary prevention education efforts. The respondent universe will be all AI/AN women over the age of 18 that participate in adapted evidence-based prevention education interventions. For each community and year, at least 100 adult AI/AN women participants in prevention education activities will be recruited to take part in a baseline, post-curriculum, and 3-month follow-up survey. This scheme would result in a maximum sample of 600 women at baseline. We anticipate that at least 50% of participants will complete the survey at a 3-month follow-up.

The Prevention Education Survey was designed to examine several areas of interest including increases in knowledge about HIV and its prevention, reduction in sexual risk behaviors, decreases in stigma around condom usage, and increases in sexual healthy behaviors. For the purpose of the power analysis we have decided to focus on the percentage of women who have ever been tested for HIV, and in particular the minimum change in that proportion that it will be possible to detect at the 3-month follow up. As a reference, the percentage of women 18 and over in the United States who had ever been tested for HIV was 37.9% in the first semester of 2011¹. We anticipate this proportion to be much lower among our target population at baseline.

Table 6 shows the minimum change in the percentage of interest that will be possible to detect at a 5% significance level with 80% power, for different hypotheses regarding the baseline proportion as well as the interclass correlation (ICC). The interclass correlation accounts for the fact that observations within each community will be likely not independent. A basic one-sided test for one proportion is used for the computations, where the relevant sample is composed of women at follow up who had not had an HIV test at baseline adjusted by the maximum expected attrition and the hypothesized design effects. In sum, with the proposed sample size it will be possible to detect small to medium size effects in Cohen’s conventional terms².

Table 6. Minimum Detectable Increase at 3-months Follow Up in the Percentage of Women Who Had Ever Been Tested for HIV at a 5 % Significance Level with 80% Power

Proportion of women who had ever been tested for HIV at baseline	Inter-class correlation
	1%	5%
20%	9.2%	15.9%
40%	11.6%	18.3%

The intent of this data collection effort is not to understand the impact of the In Community Spirit Program and generalize to other, non-participating AI/AN women or to inform future Federal policies. Rather, the intent of this effort is to understand the specific key outcomes on AI/AN participants from baseline to 3 month follow up.

2. Procedures for the Collection of Information

Key Informant Interviews—Baseline. The Key Informant Interviews—Baseline (Attachment A.1) will be conducted immediately following OMB clearance for each grantee community. The evaluation team will work with the In Community Spirit grantee project director to identify program participants and partners who are directly involved with program implementation. The project director will be responsible for identifying a list of appropriate respondents and forward the appropriate contact information to the evaluation team for administration. Because it will be necessary to facilitate administration of the interview, identifying information for each respondent will be forwarded to the multisite evaluation team. However, no identifying information will be included on the data collection instruments. The multisite evaluation team will randomly select a maximum of five respondents from each respondent list and contact the individuals via telephone to introduce the study, request participation, and to schedule an appointment for administration of the interview. Each respondent, prior to administration of the Key Informant Interview—Baseline will provide verbal consent (Key Informant Interview Verbal Consent and Script – Attachment D.1). The multisite evaluation team will be responsible for administering the interview by telephone and will be trained by the multisite evaluation project director in qualitative interviewing. Interviews will be audio recorded but respondents will not be identified by name.

Key Informant Interviews—Follow-up. The Key Informant Interviews—Follow-up (Attachment A.2) will be conducted 11 months after Baseline with up to 5 adults from each grantee community who participated in the Key Informant Interviews—Baseline. These adults will be both tribal and non-tribal participants. Interviews will be conducted by phone. In the event that a Key Informant Interviews—Baseline participant is no longer available, the multisite evaluation team will work with the grantee project director to identify appropriate replacements. The multisite evaluation team will be responsible for administering the interviews and will be trained by the multisite evaluation project director in qualitative interviewing. Each participant will provide verbal consent prior to the interview (Key Informant Interviews Verbal Consent and Script–Attachment D.1).

Women’s HEAL Survey—Prevention Education. Respondents for the HEAL Survey—Prevention Education (Attachment B) will represent a sample of AI/AN women participating in primary prevention education efforts. The grantee project director, or designated project staff, will administer the survey to all AI/AN women participating in the prevention education curriculum sponsored by the grantee and written consent will be obtained (HEAL—Prevention Education Consent­_Attachment D.3). AI/AN women participating in the prevention education curriculum will be administered a baseline, post-curriculum, and 3-month follow-up survey. At entry into the program, AI/AN women will be assigned a random ID number so that grantee project staff can track and recruit participants for the follow-up survey. Up to 600 AI/AN women will participate in the Women’s HEAL Survey—Prevention Education, which represents the maximum number of women who would be recruited.

Table 7. Instrumentation, Respondents, and Periodicity

Measure	Data Source	Method	When Collected
Key Informant Interviews—Baseline	Program representatives and partners	Interview	Once immediately following OMB clearance
Key Informant Interviews—Follow-up	Program representatives and partners	Interview	Once 11 months after baseline
Women’s HEAL Survey—Prevention Education	AI/AN women participating in prevention education curriculum	Paper/pencil surveys	Baseline Post-curriculum 3-month follow-up

3. Methods to Maximize Response Rates and Deal with Nonresponse

For cooperative agreement sites, participation in the multisite evaluation is a requirement of the In Community Spirit Program. However, the multisite evaluation has taken a number of steps to minimize the burden on local programs to ensure that completion is timely. These steps include providing training and technical assistance to each grantee as well as developing a survey participant tracking system.

The multisite evaluation team will provide training and technical assistance to all grantee programs to maximize response rates for data collection activities. This will be done by providing Web-based trainings, distributing data collection protocol and procedures manuals, and providing on-going one-on-one contact with each grantee through an evaluation liaison.

Methods will be utilized to maximize response rates for the qualitative interviews (i.e., Baseline and Follow-up Key Informant Interviews) which include gaining buy-in from In Community Spirit program staff, providing flexibility in scheduling, and conducting follow-up phone calls to non-responders. Project staff will obtain contact information for potential respondents, which will result in more accurate information, thus increasing response rates. If any identified respondent for the qualitative interviews are unable to participate, the multisite evaluation team will request that the grantee project director identify replacement respondents. Respondents for the qualitative interviews will receive an incentive for their participation.

The HEAL survey data collection protocol was designed with feedback from AI/AN serving grantees as well as AI/AN women. It is anticipated that most prevention education participants will participate in the baseline and post curriculum surveys. This estimate is based on participation rates of AI/AN women participating in similar pre and post surveys that are implemented through local level (i.e., grantee level) evaluation components. An estimate of 50% response rate at follow-up was developed, in part, with feedback from a grantee conducting long-term follow-up at 3 Tribes post intervention. To maximize response rates for the HEAL Survey, all efforts have been made to minimize burden on individual respondents by limiting the number of items on each questionnaire and building in grantee data collection protocols that encourage participation as well as providing incentives. In addition to encourage participation in data collection with AI/AN populations, the contractor will work with grantee project staff and affiliated tribes to obtain local level support and to secure local level approvals (e.g., IRB, Tribal resolution, etc.).¹⁹ Additionally, a $5 incentive will be utilized to encourage respondent participation and increase response rates at each wave, and the multisite evaluation team will provide the grantee with a survey participant tracking system to track program and survey participation.

4. Tests of Procedures to be Undertaken

The multisite evaluator was contracted to assess the process and impact of the already-funded In Community Spirit Program. As such, the instruments to be used in the evaluation were customized around In Community Spirit activities (i.e., Community Awareness, Prevention Education, and Capacity Building) and goals. As new measures were developed, standard instrument development procedures including review of the literature, item development, and content review by experts in the field were used (see below).

A thorough review of the literature was conducted related to HIV prevention education and outreach and awareness activities in efforts to develop the HEAL Survey. In addition, an expert in HIV prevention and gender-responsive prevention was consulted in developing the surveys. Drafts of all instruments were developed and reviewed by multisite evaluation team members, the OWH Contracting Officer’s Technical Representative, and an expert consultant. Once reviewed and revised, the instruments underwent cognitive testing and/or pilot testing on no more than 9 respondents matching the type appropriate for the instrument, in efforts to enhance question accuracy, determine administration time, and identify barriers to survey implementation with AI/AN women. AI/AN women were utilized for the cognitive testing of HEAL Survey items.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The multisite evaluator, a contractor for OWH, has full responsibility for the development of the overall statistical design and assumes oversight responsibility for data collection and analysis. Training, technical assistance, and monitoring of data collection will be provided by the evaluator.

The individuals responsible for the design of the data collection procedures and oversight of data collection and analysis are:

Christine M. Walrath-Greene, Ph.D.

ICF Macro

40 Wall Street, 34^th Floor

New York, NY 10005

(212) 941-5555

[email protected]

Robin Davis, PhD

ICF Macro

3 Corporate Square, NE Suite 370

Atlanta, GA 30329

(404) 592-2188

[email protected]

The following individuals will serve as statistical consultants to this project:

Robert Stephens, PhD

ICF Macro

3 Corporate Square, NE Suite 370

Atlanta, GA 30329

(404) 321-3211

[email protected]

Anupa Fabian, MPA

ICF Macro

40 Wall Street, 34^th Floor

New York, NY 10005

(212) 941-5555

[email protected]

The agency staff person responsible for receiving and approving contract deliverables is:

Judith Labiner-Wolfe, PhD.

Evaluation Specialist

Office on Women’s Health

U.S. Department of Health and Human Services

200 Independence Avenue, SW

Washington, DC 20201

(202) 260-0904

[email protected]

3. References

¹ Centers for Disease Control and Prevention. (2005). HIV/AIDS among American Indians and Alaska Natives Fact sheet. Atlanta, GA: Author. Retrieved August 18, 2010 from:

http://www.cdc.gov/hiv/resources/factsheets/aian.htm

² Centers for Disease Control and Prevention. (2008). HIV surveillance report 2008. Atlanta, GA: Author. Retrieved August 18, 2010 from: http://www.cdc.gov/giv/surveillance/resources/reports/2008report/table3a.htm

³ Indian Health Services. (n.d.). National HIV/AIDS administrative work plan 2008-2011. Retrieved August 18, 2010 from http://www.ihs.gov/medicalprograms/hivaids/docs/hivaidsworkplan.pdf

⁴ Kaufman, C .E., Shelby, L., Mosure, D. J., Marrazzo, J., Wong, D., de Ravello, L., et al. (2007) Within the hidden epidemic: Sexually transmitted diseases and HIV/AIDS among American Indians and Alaska Natives. Sexually Transmitted Diseases, 34(10), 767–777.

⁵ McNaghten, A. D., Neal, J. J., Li, J., & Fleming, P. L. (2005) Epidemiologic profile of HIV and AIDS among American Indian/Alaska Natives in the USA through 2000. Ethnicity & Health, 10(1), 57–71.

⁶ Denny, C. H., Holtzman, D., & Cobb, N. (2003). Surveillance for health behaviors of American Indian and Alaska Natives. Findings from the Behavioral Risk Factor Surveillance System 1997-2000. Morbidity and Mortality Weekly Report Surveillance Summary, 52(7), 1–13.

⁷ Stevens, S. J., Estrada, A. L., & Estrada, B. D. (2000) HIV drug and sex risk behaviors among American Indian and Alaska Native drug users: Gender and site differences. American Indian Alaska Native Mental Health Research, 9(1), 33–46.

⁸ Fisher, D. G., Fenaughty, A. M., Paschane, D. M., & Cagle, H. H. (2000) Alaska Native drug users and sexually transmitted disease: Results of a five-year study. American Indian and Alaska Native Mental Health Research, 9(1), 47–57.

⁹ Morrison-Beedy, D., Carey, M. P., Lewis, B. P., & Aronowitz, T. (2001) HIV risk behavior and

psychological correlates among Native American women: An exploratory investigation. Journal of Women’s Health and Gender-Based Medicine, 10(5), 487–494.

¹⁰ Indian Health Service. (2005). HIV/AIDS program fact sheet statistics. Rockville, MD: Author. Retrieved August 18, 2010 from

http://www.ihs.gov/medicalprograms/hivaids/index.cfm?module=cdc&option=index

¹¹ Vernon, I., & Jumper-Thurman, B. (2005). The changing face of HIV/AIDS among Native populations. Journal of Psychoactive Drugs, 37(3), 247–255.

¹² Walters, K. L., Simoni, J. M., & Evans-Campbell, T. (2002). Substance use among American Indian and Alaska Natives: Incorporating culture in an “indigenist” stress-coping paradigm. Public Health Reports, 117(Suppl 1), S104–S117.

¹³ Diffusion of Effective Behavioral Interventions. (n.d.). Retrieved August 18, 2010 from

http://www.effectiveinterventions.org

¹⁴ Barlow, K., Loppie, C., Jackson, R., Akan, M., Maclean, L., & Reimer, G. (2008). Culturally competent service provision issues experienced by aboriginal people living with HIV/AIDS. Journal of Aboriginal and Indigenous Community Health, 2, 155–180.

¹⁵ Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method. New York, New York: John Wiley & Sons, Inc.

¹⁶ Buchwald, D., Mendoza-Jenkins, V., Cray, C., McGough, H., Bezdek, M, & Spicer, P. (2006). Attitudes of urban American Indians and Alaska Natives regarding participation in research. Journal of General Internal Medicine, 21(6), 648–651.

¹⁷ Bureau of Labor Statistics, U.S. Department of Labor. (2009). May 2009 National Occupational Employment and Wage Estimates. Retrieved March 27, 2011 from http://www.bls.gov/oes/current/oes_nat.htm

¹⁸ U.S. Department of Health and Human Services. (n.d.) The 2011 HHS Poverty Guidelines. Retrieved March 27, 2011 from http://aspe.hhs.gov/poverty/11poverty.shtml

¹⁹ Lavelle, B., Larsen, M.D., & Gundersen, C. (2009). Research synthesis: Strategies for surveys of American Indians. Public Opinion Quarterly, 73(2), 385–403.

²⁰ Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2^nd ed.). Hillsdale, New Jersey: Lawrence Erlbaum.

1 CDC/NCHS, Early Release of Selected Estimates Based on Data From the January–June 2011 National Health Interview Survey. Available at http://www.cdc.gov/nchs/nhis/released201112.htm#10

2 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale,NJ: Lawrence Erlbaum.

8