Evaluation_of_the_Native_Aspirations_Project_10_01_10-2 Supporting Statement A & B

A. Justification

1. Circumstances of Data Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) is requesting clearance for data collection for the cross-community evaluation of the Native Aspirations Project. In 2005, SAMHSA created the Native Aspirations Project to address the high rates of youth suicide, bullying, and school violence in American Indian (AI) and Alaska Native (AN) communities through implementation of culturally appropriate, evidence-based interventions. The Native Aspirations Project’s goals and objectives closely align with the blueprint for suicide prevention outlined in The Surgeon General’s Call to Action to Prevent Suicide and the National Strategy for Suicide Prevention (NSSP). The Native Aspirations Project includes several of the main components of the Call to Action and the NSSP including awareness (broaden public awareness of suicide as a public health problem), intervention (enhance services and programs, both population based and clinical services), and methodology (advance the science of suicide prevention and suicide surveillance). In addition, The President’s New Freedom Commission: Transforming Mental Health in America acknowledged that significant barriers in access, quality and mental health care outcomes for minorities, including AI/AN populations, remain despite efforts to improve services for culturally diverse populations. Specifically, these minority populations experience a disproportionately high burden of disability from mental health problems due to the lack of access and quality of care. The data collection for this cross-community evaluation is authorized under SEC 505 of the Public Health Service Act (SEC. 505. [290.aa-4] Data Collection).

The Native Aspirations Project has two primary objectives: (1) decrease risk factors that contribute to suicide and school violence, and (2) increase protective factors that are linked to the healthy and safe development of children and their families. SAMHSA funded Kauffman and Associates, Inc. (KAI) in 2005 to engage and support AI/AN communities in their efforts to prevent suicide and bullying among their youth. KAI supports include training and technical assistance, community mobilization planning, seed money to support initial community-based prevention activities, and sustainability planning. The objectives for participating communities are to:

• Increase Native youth and families’ awareness, knowledge and skills regarding suicide, and violence;

• Build pro-social and help-seeking behaviors of Native youth; and

• Promote Native youth-specific services (prevention and intervention programs) and the community strategic plans for the coordination of behavioral health, justice, and education systems supporting Native youth within the respective communities.

The program provides community-specific technical assistance on topics such as high-risk youth, evidence-based interventions, community readiness assessments, resource management, community planning, and preparation for community mobilization.

The cross-community evaluation has two tiers. The first tier involves community-specific evaluation activities that are tied to the key components of community plans developed in each participating community which guides program planning and local evaluation through data-driven frameworks and inquiry.

Community specific evaluation activities include to:

• Determine the experience of the Gathering of Native Americans (GONA);

• Determine community readiness for program implementation;

• Determine how accurately community plans reflect the needs and characteristics of each community, implementation activities;

• Determine how well local resources for American Indian/Alaska Native (AI/AN) youth are mobilized as a result of the program; and

• Determine the change over time in terms of community knowledge, awareness, and behavior related to violence, bullying and suicide.

The second evaluation tier involves cross community evaluation activities and service-level change data and is directly tied to the primary objectives of the Native Aspirations Project. These data collection activities include process and outcome indicators such as community-level knowledge and awareness of suicide, violence, bullying, and substance abuse; pro-social and help-seeking behaviors among Native youth; and the provision of services specific to Native youth through existing service systems. Activities include the service provider focus groups and the Community Knowledge, Awareness, and Behavior Survey (C-KABS).

While recent research recommends employing longitudinal study designs that assess impact, this evaluation utilizes a prospective longitudinal design, and does not involve a comparison study or other experimental design to focus on questions of program impact. The design for this evaluation was determined based on the program goals and objectives, as well as consideration for the unique circumstances of American Indian and Alaska Native communities. Utilization of a comparison study design would be difficult, if not impossible, given the very different cultural context and locations of participating communities, as well as the need for taking a culturally sensitive approach to evaluation in Indian country.

The limitations of this design include potential for confounding factors when assessing the effect of the program on identified outcomes in the community. However, the strength of the design in terms of the community-specific, in addition to the cross-site approach, is such that the evaluation promises to provide rich community-level data in addition to assessing changes in core outcomes over time within and across communities.

Community-Specific Data Collection Activities (Tier 1)

Evaluation Activity	Data Collection Instrument
Gathering of Native Americans (GONA) Evaluation	Document A.1: GONA Baseline Interview Document A.2: GONA Follow-up Interview Document A.3: GONA Follow-up Youth Focus Group Moderator’s Guide
Community Mobilization Plan (CMP) Evaluation	Document B.1: Community Plan Focus Group Guide Document B.2: Community Plan In-Depth Interviews Version 1 Document B.3: Community Plan In-depth Interview Version 2
Community Readiness Assessments	Document C.1: Community Readiness Assessment
Existing Data Abstraction and Submission	Document F.1: Existing Data Inventory

Cross-Community Data Collection Activities (Tier II)

Evaluation Activity	Data Collection Instrument
Service Provider Focus Groups	Document D.1: Service Provider Focus Group Guide Version 1 Document D.2: Service Provider Focus Group Guide Version 2
Community Knowledge, Awareness, and Behavior Survey	Document E.1: C-KABS Adult Document E.2: C-KABS Youth
Existing Data Abstraction and Submission	Document F.1: Existing Data Inventory

a. Background

The suicide rate among AI/AN populations is 1.5 times the national rate and the highest rate of any ethnic group in the United States (U.S. Department of Health and Human Services, 2001a). Suicide is the third leading cause of death for young people (ages 15-24) in the United States, exceeded only by unintentional injury and homicide. For AI/AN youth in this age group, suicide is the second leading cause of death and it accounts for 64% of all suicides by American Indians and Alaska Natives (Anderson & Smith, 2003). The suicide rate for Native American males, ages 15 -24, is two to three times higher than general U.S. rate (U.S. Department of Health and Human Services, 2001a). Studies have also found that Alaska Natives are more likely to commit suicide than non-Natives living in Alaska (Gessner, 1997). Overall, violent deaths, including unintentional injuries, homicide, and suicide, account for 75% of all mortality in the second decade of life for American Indians and Alaska Natives (Resnick et al., 1997). Studies and recent reports indicate that important risk factors for attempted suicide in youth, including depression, substance abuse, and aggressive or disruptive behaviors (Aoun & Gregory, 1998; Beautrais, 2000; National Institute of Mental Health, 1999; U.S. Public Health Service, 1999), occur with greater frequency in AI/AN populations (Stiffman, Striley, Brown, Limb, & Ostmann, 2003). For example, one school-based epidemiological study of Northern Plains youth aged 13 to 17 found that, compared with nonminority children from the population at large, AI adolescents are much more likely to be diagnosed with substance abuse disorders (Beals et al., 1997). The same study also found that AI adolescents were more likely to be diagnosed with attention deficit disorder, conduct disorder, and oppositional defiant disorder (Beals et al., 1997)—conditions characterized by aggressive, disruptive, or defiant behaviors, such as bullying, fighting, carrying weapons to school, and initiating school violence.

Ongoing efforts to ensure the healthy growth and development of AI/AN youth have built on growing literature related to the risk and resilience qualities of adolescents (Institute of Medicine, 2002). For AI/AN youth, “resilience” refers to a set of qualities that help to ensure that, despite stress and adversity, youth do not succumb to negative outcomes, including school failure, substance abuse, mental health problems, bullying, or juvenile delinquency (Peacock, 2002). Risk and resilience research is aimed at identifying the contextual factors related to adolescent health and well-being in multiple domains (e.g., family, peer group, school, community, culture) in an effort to equally understand both the risk factors that lead to a greater likelihood of a negative outcome in an adolescent’s health or behavior and the protective factors that mitigate against these negative outcomes.

Fostering resilience in young people is not a new practice for AI/AN people and may even help explain historic survival in these communities that have faced adversity (Strand & Peacock, 2002). Youth in these communities benefit from protective factors provided through family support, extended tribal family support, as well as school and community support (HeavyRunner & Morris, 1997). Research also has found that cultural heritage can have a protective influence on adolescents in AI/AN communities. The degree to which children are embedded in traditional culture has been found to positively affect school performance (Whitbeck, Hoyt, Stubben, & LaFromboise, 2001) and protect against drug use (Kulis, Napoli, & Marsiglia, 2002; Moran, Fleming, Somervell, & Manson, 1999). Such research is described by some as “cultural resilience research,” or research focused on “the use of traditional life to overcome the negative influences of oppression, abuse, poverty, violence and discrimination” (Strand & Peacock, 2003). It is a promising area of inquiry for further research dedicated to the improvement of service delivery and social support for AI/AN youth.

b. The Need for Evaluation

Since its launch in 2005, the Native Aspirations project has been engaged in program evaluation efforts which include the development of an initial program logic model and evaluation protocol, a customer satisfaction survey, and the collection of preliminary, site-specific evaluation data. Additional program planning activities have included interviews with stakeholders from each community to assess community readiness in each community to inform program development. As the Native Aspirations Project progressed, SAMHSA understood the need for an independent and comprehensive evaluation of the program hence the cross-community evaluation was developed. The evaluation includes the community readiness assessments previously conducted by KAI.

The Native Aspirations project will be strengthened by obtaining information about its effectiveness and efficiency which are key objectives of the Government Performance and Results Act and a SAMHSA strategic priority (U.S. Department of Health and Human Services, 2006). The cross-community evaluation focuses on the collaborative efforts, function, and outcomes of the Native Aspirations project and will be grounded in empowerment theory (Fetterman & Wandersman, 2005) and appreciative inquiry (Cooperrider, Whitney, & Stavros, 2003). The historic lack of implementation of evidence-based programs is a concern that is directly relevant to the Native Aspirations project. The effectiveness of some services has been scientifically proven, but many are not widely used in practice. The Native Aspirations cross-community evaluation provides an opportunity to gather systematic information on practices that previously have not been implemented in AI/AN communities.

Data collected through the proposed Native Aspirations cross-community evaluation will serve a variety of overarching purposes: 1) to determine the extent to which the Native Aspirations Project has been able to achieve its overall goals of assisting tribal communities in mobilizing existing social and educational resources to establish comprehensive and collaborative community-based violence, bullying, and suicide prevention plans, and providing technical assistance in implementing those plans; 2) to help communities use data to sustain efforts and apply for additional funding, and learn evaluation techniques; 3) to focus technical assistance and support; 4) to understand the outcomes of the Native Aspiration Project in communities; and 5) to ensure accountability to stakeholders, including Federal agencies and the youth and families served by the Native Aspirations project, by informing them of progress made by the project and the funded communities.

Evaluation data provides the information necessary for shaping and sustaining efforts influencing program and policy development through the systematic analysis of outcomes and aggregation of information across the program components for both community-specific and core data collection activities. This increases the understanding of overall program effectiveness. Without comprehensive evaluation information, the degree of program implementation cannot be effectively monitored, and the expected outcomes from these programs may be difficult to identify and the potential for larger scale dissemination and purposive diffusion decreases. (Dearing, 2002). At all levels of government—Federal, State, and local—decisions are being made that are dramatically changing the lives of children, adolescents, and families. To make these decisions responsibly, policymakers and other stakeholders need information such as the data and findings to be produced by this evaluation.

c. Clearance Request

This submission requests OMB clearance for 6 evaluation activities with 12 instruments for the evaluation of the Native Aspirations Project Communities which participate across Cohorts 3, 4, and 5. Specifically, Cohort 3 was engaged in FY 2008, Cohort 4 was engaged in FY 2009, and Cohort 5 was engaged in FY 2010. Data collection will be conducted with all communities in each of the Cohorts pending OMB approval; however, the date of approval may dictate what data collection activities can be conducted. Data will not be collected from Cohort 1, funded in FY 2006 or Cohort 2, funded in FY 2007.

The communities included in this evaluation are located across several high need Indian Health Service (IHS) areas. These include the following:

Cohort 3:

• Tucson Area (3 communities)

• Albuquerque Area (3 communities)

• Bemidji Area (3 communities)

Cohort 4:

• Alaska Native Villages (4 communities)

• Aberdeen Area (4 communities)

Cohort 5:

• Alaska Native Villages (3 communities)

• Navajo Tribe (4 chapters/communities)

• Albuquerque Area (1 community)

. The specific goals of the evaluation include:

• Describing and (as appropriate) determining the key independent and mediating variables affecting program outcomes;

• Investigating the interrelationship between the community plan, the development process, and implementation of observed outcomes; and

• Identifying key factors in promoting and sustaining efforts and outcomes.

The cross-community evaluation tiers and related data collection activities are described below.

Community Specific Data Collection Activities. Community specific activities (Tier 1) include both process and outcome evaluation components. The purposes of community specific Tier I activities are to better position each community with regard to program planning and local evaluation through data-driven frameworks and inquiry, and to determine the stage of readiness of each community to mobilize efforts related to suicide prevention, bullying and violence prevention. Tier I activities also include Evaluation of the GONA through in-depth interviews with adults and focus groups with youth to determine the process and outcome of the GONA on community members who attend. Evaluation staff also will conduct Community Readiness Assessments (CRAs) with individuals in each community to determine the community’s level of readiness to mobilize efforts. Responses to interviews will be scored to determine an overall readiness score for each community. Community Plan Focus Groups will be conducted with youth and adults who attended the Community Mobilization Plan (CMP) meeting to further understand the community mobilization process. These focus groups are designed to facilitate group communication around the process, and understand early implementation of the plan, and to determine organizational and community awareness and involvement with the process. Later, Community Plan In-depth Interviews will be conducted with individuals who participated in the focus groups who continued and those which discontinued their involvement with the community mobilization process to gather information on the implementation of the CMP and the outcome of the Native Aspirations Project on the community. Data abstraction and submission from existing sources will be analyzed to support the outcome component of Tier I of the cross-community evaluation. Data elements may be requested from educational systems, juvenile justice/law enforcement sources, mental health agencies, child welfare, Medicaid, and community organizations.

Cross-Community Data Collection Activities. Cross-community evaluation questions (Tier II) are directly tied to the primary objectives of the Native Aspirations Project and include the collection of community- and systems-level change measurement. These activities are designed to be analyzed across communities. Activities include Service Provider Focus Groups and the Community Knowledge, Awareness, and Behavior Survey (C-KABS) – Adult and Youth Versions to identify community-level knowledge and awareness of suicide, violence, bullying, and substance abuse; pro-social and help-seeking behaviors among Native youth; and the provision of services specific to Native youth through existing service systems. Data abstraction and submission from existing sources will be analyzed across communities to identify the outcomes of the project.

d. Addressing National Outcome Measures and GPRA Reporting

There are no NOMs/GPRA reporting requirements for this evaluation.

2. Purpose and Use of Information

Data collected as part of the cross-community evaluation will be useful to SAMHSA and its partners: other Federal agencies; tribal communities; legislators; federal administrators; the fields of suicide, bullying, violence, and substance abuse prevention; individual youth and their families; and the communities in which they live. Comprehensive information gathered from multiple communities at various levels and stages of their programmatic activity will augment the existing knowledge base.

Specifically, information gathered through the data collection will describe for tribal communities:

• The experience of community members who attended the GONA and CMP;

• How well the CMP reflects the needs and characteristics of the community;

• How well the community plans identify and mobilize local resources for Native youth;

• The change in community knowledge, awareness, and behavior around suicide, violence, bullying, and substance abuse prevention as a result of the Native Aspirations Project;

• The change in norms surrounding pro-social and help-seeking behaviors of Native youth as a result of the Native Aspirations Project;

• The change in services and service availability for Native youth as a result of the Native Aspirations Project

Description of Instruments. The Native Aspirations Cross-Community Evaluation Instruments are described in detail below.

GONA – Baseline Interview (Document A.1- 1 Version). Baseline GONA interviews will be conducted prior to the GONA in each community and will center on four values - belonging, mastery, interdependence, and generosity - and how respondents view and describe their relationships in and with the community; how people in the community deal with youth violence, bullying, substance abuse, and suicide; community members’ willingness to work with together to address these issues; community protective factors; and suggestions for how community members can work together to address these issues. The GONA baseline interviews will be conducted by telephone in year 1 of grant funding and will take approximately 20 minutes to complete. Interviews will target six adult tribal community members who plan to participate in the GONA in each community. These respondents would include any community member who lives on the reservation (GONAs typically include any community member interested in attending). These would not necessarily be service providers or those specifically involved in the planning process.

GONA – Follow-up Interview (Document A.2 - 1 Version). The GONA follow-up interviews will be conducted several weeks after the GONA in each community. Follow-up interviews will center around the four values (belonging, mastery, interdependence, and generosity); respondents’ experience during the GONA; participation in activities; views on community relationships; knowledge of the Native Aspirations Project; knowledge of risk factors for youth violence, bullying, substance abuse, and suicide; community protective factors; willingness of community members to work together and suggestions for working together; and next steps. The GONA follow-up interviews will be conducted in person and will take approximately 60 minutes to complete. Respondents include adult community members who participated in the GONA baseline interviews.

GONA – Youth Follow-up Focus Group Guide (Document A.3 - 1 Version). The GONA follow-up focus groups will be conducted several weeks after the GONA with youth who attended the GONA. The focus group moderator’s guide follows the same content as the GONA Follow-up Interviews (see above). Cross-community evaluation staff will conduct focus groups with youth in each funded community in year 1. Focus group guides contain 11 items and the session will last 2 hours. Respondents include youth who attended the GONA. Typically all community youth are invited to participate in the GONA.

Community Plan Focus Group Guide (Document B.1 - 1 Version). Respondents participating in the Community Plan Focus Groups include youth and adults who attended the Community Mobilization Plan (CMP) meeting in year 1. The guide consists of questions designed to facilitate group communication around the community mobilization planning process, assess early implementation of the plan, and organizational and community awareness and involvement. Focus group guides contain 7 items and the session will last 2 hours. The cross-community evaluation team will conduct up focus groups in year 1 of the grant for each funded community. Respondents can include service providers affiliated with the Native Aspirations project, including mental health providers, school board members, staff at Boys’ and Girls’ Clubs, or other providers who work for agencies involved in the planning and implementation of the project; as well as community leaders (e.g., tribal elders who are active in the community, youth who have been identified as leaders in the community).

Community Plan In-depth Interviews (Document B.2 & B.3 - 2 Versions). The Community Plan In-depth Interviews will be conducted in person during year 3 of the grant. The interviews will be conducted with the same individuals who participated in the CMP focus groups; however, the participants will be divided into two groups with two respective guides (see above for description of respondents). Version 1 will be conducted with participants who remained active in the community mobilization process and Version 2 will be used with respondents who discontinued their involvement with Native Aspirations project. The interviews will be used to gather information on the CMP implementation process and how implementation of the plan affected the community, organizational and community awareness, and involvement with Native Aspirations project, and the outcomes of the overall Native Aspirations program on the community. The Community Plan In-depth Interview – Version 1 will take 60 minutes to complete. The Community Plan In-depth Interview – Version 2 will take 20 minutes to complete.

Community Readiness Assessment (Document C.1 -1 Version). The CRA addresses six dimensions focused on an identified social concern (i.e., youth violence, bullying, and suicide). These dimensions include: (a) community prevention efforts, (b) community knowledge of prevention efforts, (c) leadership, (d) community climate, (e) knowledge about the problem, and (f) resources for prevention efforts. CRAs include 26 interview questions that address each of the six community readiness dimensions and findings are used to determine the level of readiness of the community to mobilize efforts for the prevention of youth violence, suicide, and bullying. During years 1 and 3, CRAs will be conducted with key informants; interviews will last 60 minutes and will be conducted by telephone or in person. Overall readiness scores will be determined and interpreted to indicate the community’s status with respect to each of these dimensions. Key informants identified include representatives various segments of the community which may include: health and medical professionals, social services, mental health and treatment services, schools, city/county/tribal government, law enforcement, clergy or spiritual community, community at large, elders, or specific high risk groups.

Service Provider Focus Group Guide (Document D.1 & D.2 - 2 Versions). The Service Provider Focus Groups are designed to facilitate conversation and information sharing with youth service providers across communities to acquire a broader understanding of provider and service availability for Native youth and how these change over time. Version 1 respondents will include agency staff such as teachers, mental health professionals, justice providers and welfare providers. Version 2 respondents will include non-agency staff, such as paraprofessional providers or “natural helpers.” However, specific provider types will be identified for each participating community as a function of their existence and number. Version 1 of the focus group guides consists of 9 items and Version 2 consists of 7 items, each with additional sub-questions/probes covering the availability of wellness and mental health services, how agencies work together, awareness of violence/suicide prevention activities, and areas for improvement. Focus groups will be conducted each of years 1 (baseline) and 3 (follow up) of the funding.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – (Document E.1 - Adult Version). The C-KABS - Adult Version is designed to gather knowledge and awareness information from adult community members related to suicide, substance abuse, violence, and bullying, and how this changes over time as a result of the Native Aspirations project. Respondents will include a random sample of adults living in the community. In addition, respondents will report on their exposure to Native Aspirations Project activities regarding the prevention of suicide, substance abuse, violence, and bullying. Other outcome constructs include the availability of services, knowledge of youth risk factors, and stigma around and attitude toward seeking services for wellness. The C-KABS - Adult Version will be administered annually for all three years of funding to Native adults from each funded community and takes approximately 45 minutes to complete.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – (Document E.2 - Youth Version.) The C-KABS Youth Version will be administered to youth participants (age 11 and older) to gather information about social norms around help-seeking behavior, pro-social behavior (e.g., traditional Indian activities) among youth, and the extent to which respondent youth have been exposed to risky behaviors (suicide, violence, substance abuse, or bullying), exposure to prevention efforts for risky behaviors, and how these change over time as a result of the Native Aspirations Project. Respondents will include a random sample of youth between the ages of 11 and 17 living in each community. The survey will also contain items about youths’ access to pathways to risky behaviors (e.g., how hard/easy is it to get drugs/alcohol), access to and awareness of/willingness to seek help for these behaviors for themselves or others, and youths’ engagement in risky and protective behaviors. The C-KABS Youth Version will be administered annually, for all three years of funding, to Native youth from each funded community and takes approximately 45 minutes to complete.

Data Abstraction and Submission from Existing Sources (Document F.1.) Data from existing sources (i.e., management information systems (MIS), administrative records, case files, etc.) will be analyzed across communities to assess program outcomes. To minimize data collection burden on community members, this activity will be tailored to key components identified in the community plan and will be developed around existing data systems and related infrastructures using the Data Abstraction and Submission Form. Cross-community technical assistance providers will assist in the identification of existing data sources and their relevance to locally planned Native Aspirations activities. Data elements may be requested from educational systems, juvenile justice/law enforcement sources, mental health agencies, child welfare, Medicaid, and community organizations (e.g., YMCA, Boys and Girls Clubs, etc.). A maximum of 10 data elements each will be requested from education and juvenile justice/law enforcement sources and a maximum of 5 data elements each will be requested from mental health, child welfare, Medicaid, and community activities. These data will be aggregated from existing data sources, some of which are attendance sheets, management information systems, etc. Communities are responsible for aggregating these data and submitting them to the Native Aspirations Cross-community Evaluation team by mail, electronic mail, or by uploading the data.

The tier-specific utility and contribution of the cross-community data collection to SAMHSA’s mission and decision making are described below:

Community Specific Data Collection Activities. As part of the community-specific evaluation, findings related to GONA and CMP activities will inform SAMHSA and other tribal communities of what types of activities are being implemented, the intended and actual utilization and effect of these activities, who is participating, and overall satisfaction. This information will assist other tribal communities in implementing GONA and CMP activities as part of the Native Aspirations Project. Data abstraction and submission will assess the outcomes of the Native Aspirations Project in each community.

Cross-Community Data Collection Activities. As part of the cross-community evaluation, specific findings related to Native youth and adult knowledge and awareness of prevention activities, youths’ risky behaviors, norms for pro-social and help-seeking behaviors will assist other tribal communities across the country in assessing the potential outcome of prevention activities in tribal communities. Information collected through the service provider focus groups will help to identify the types of services and access to services for Native youth and barriers/protective factors to accessing services and resources.

Despite extensive knowledge regarding suicide risk and protective factors little is known about how to integrate these factors and how they work in concert to evoke suicidal behavior or to prevent it (Institute of Medicine, 2002). In addition, little is known about tribal populations in terms of suicide prevention, violence and bullying. Also, the process and outcomes of activities such as the GONA and CMP has not been examined. Little is known about whether suicide, bullying, violence, and substance abuse prevention activities are reaching Native youth. Data describing youths’ and adults’ exposure to prevention activities and awareness and knowledge of youth risk factors will significantly contribute to the existing knowledge base. All of this data will serve to inform policymakers and federal representatives in their decision making around appropriations and funding as well as youth and their families in their every day quest to identify and respond to risk. The goal to prevent suicide, violence, bullying, and substance abuse by Native youth can be approached from multiple perspectives and can be assessed and documented in its utility – while simultaneously advancing the field.

SAMHSA will use the results from the cross-community evaluation to develop policies and provide information to other tribal communities regarding the development and implementation of prevention programs. SAMHSA will also refine future funding priorities for the Native Aspirations Project and its cross-community evaluation or similar programs. Finally, information from the cross-community evaluation may also help other SAMHSA programs in developing and implementing suicide, bullying, violence, and substance abuse prevention activities for AI/AN youth; designing comprehensive data collection efforts to monitor those activities; and reporting to local and Federal stakeholders. If the data is not collected, policymakers and program planners at the Federal and local levels will not have the necessary information to determine the extent to which these activities are effective and having an effect on Native at-risk youth. Without this evaluation, Federal and local officials will not know whether the Native Aspirations Project resulted in positive outcomes related to prevention and whether communities funded by the Native Aspirations Project are meeting the goals of the project.

3. Use of Information Technology

All data collection activities will be conducted by cross-site evaluation staff and local community researchers in person or by telephone with the exception of the C-KABS Adult and Youth Versions (see Documents E.1 and E.2). The C-KABS Adult and Youth Version will be administered annually by the community researcher. Because the availability of technology (e.g., computers, internet connections) varies across communities, the C-KABS Youth and Adult Versions will be designed for use as both paper and pencil and Web-based surveys. In communities where paper and pencil surveys are conducted, completed surveys will either be collected and mailed to the cross-community evaluation team for entry into a database, or directly entered through a Web-based data collection and management system by the local community researcher. The data entry method will depend on the technology available in each community. Respondents who complete the C-KABS using Web-based surveys will submit responses to the cross-community evaluation team via the Web-based data collection and management system. Identifying information will be stored separately as hard copies in locked file cabinets.

4. Efforts to Identify Duplication

The cross-community evaluation team, in developing the data collection activities for the cross-community evaluation, conducted a literature review to avoid duplication in data collection activities and the use of similar information. The team reviewed existing research studies and other Federal initiatives evaluating suicide and suicide prevention.

a. Existing Research

More information on how to prevent suicide is needed (SPAN USA, Inc., 2001; Institutes of Medicine, 2002, U.S. Public Health Service, 2001). The studies on suicide prevention activities have provided important information, but most have been conducted with specific populations and circumstances and are not generalizable to other populations (Institutes of Medicine, 2002), including tribal populations. Similarly, the lack of longitudinal and prospective studies has been a barrier to understanding and preventing suicide (Institutes of Medicine, 2002). Acknowledging the lack of information on the effectiveness of suicide prevention programs, the Institutes of Medicine’s Report, “Reducing Suicide: A National Imperative” provides several recommendations for increasing research on suicide (2002). The report recommends that Federal funding be provided for the development, testing, and expansion of suicide prevention interventions, and for longitudinal studies that focus on the medium to long-term impacts of suicide prevention activities, such as the impact on risk and protective factors and treatment and prevention. Specifically, the report recommends exploring the impact of suicide prevention programs through large nationally coordinated efforts.

Of all youth populations, American Indian/Alaska Native males have the highest suicide rates (Anderson & Smith, 2003). Despite the prevalence data, the scope of this problem is not entirely known because of the manner in which cause of death is recorded on death certificates and because of the ambiguity of homicides and accidental deaths where the person attempting suicide intentionally places himself or herself in harm’s way (U.S. Public Health Service, 1999). As with other populations, some limited information on risk and protective factors for suicide exists. In a convenience sample of urban AI boys and girls, substance abuse and violence perpetration were significantly associated with past suicide attempts (Pettingell, Bearinger, Skay et al. 2008). For girls, when the likelihood of past suicide attempts was examined in conjunction with risk and protective factors, substance abuse was determined to be a significant risk factor for a previous suicide attempt; however, this relationship decreased when protective factors were introduced (i.e., positive mood). For boys, the prevalence of violence perpetration decreased substantially when protective factors were introduced (i.e., positive mood and parent pro-social behavior norms). However, given that this research was done with urban populations, additional research should be conducted among AI/AN youth living in more rural areas and reservations.

b. Other Federal Efforts

In 2005, SAMHSA initiated the Native Aspirations Project to address the excessively high rates of youth suicide, substance abuse, bullying, and school violence in American Indian (AI) and Alaska Native (AN) communities through the implementation of culturally appropriate, evidence-based interventions. The project was authorized shortly after a school shooting and the subsequent suicide of the teenage gunman in March 2005 on the Red Lake Indian Reservation in northern Minnesota. The Native Aspirations Project has provided technical assistance and support to 17 AI/AN communities, with an additional 25 communities to be supported over at least the next 3 years. SAMHSA is also sponsoring a cross-community evaluation of the Native Aspirations Project, for which this request for clearance is being written.

In addition, SAMHSA is supporting a cross-site evaluation of the Garrett Lee Smith (GLS) Youth Suicide and Early Intervention Program across States, tribal regions, and college campuses. The Garrett Lee Smith Memorial Act (GLSMA), passed by Congress on September 9, 2004, and signed into law by President Bush on October 21, 2004, authorized the first Federal funding to support a national approach to community-based youth suicide prevention. This law was designed to mobilize efforts to support suicide prevention and early intervention while interconnecting the recommendations and findings from the Institute of Medicine, the Surgeon General’s Call to Action, and the National Strategy for Suicide Prevention. To support suicide prevention efforts across the country, the GLSMA authorized the use of funds to support states, tribal communities, colleges, and universities in their efforts to develop and implement various suicide prevention activities as part of the program. To date, 38 State/tribal grants have been awarded, with at least 5 grants awarded specifically to tribal entities or consortiums, and 56 campuses have received grants. SAMHSA mandated the evaluation of these efforts to determine the effectiveness of program activities. The GLS Suicide Prevention Program cross-site evaluation was designed to evaluate the effectiveness of suicide prevention programs by focusing on efforts to identify risk factors and increase protective factors, as well as to gain a better understanding of process-related factors. The evaluation design includes four stages of information gathering: context, product, process, and impact. Data collection activities are tailored to the programmatic activities funded as part of the GLS Suicide Prevention Program.

As a part of this effort and through an interagency agreement with SAMHSA, the Centers for Disease Control and Prevention (CDC) is supporting evaluations of evidence-based suicide prevention programs in Maine and Virginia and with the Native American Rehabilitation Association (NARA) as part of CDC’s Targeted Injury Prevention Programs. In Maine and Virginia, CDC is supporting research that documents the efficacy of a community-based cognitive therapy program for preventing suicidal behavior among suicide attempters identified in emergency departments. The focus of the intervention is to help youth develop more adaptive ways of thinking and more functional ways of responding to periods of emotional distress. These CDC evaluations will provide valuable information on the efficacy of interventions for youth displaying suicide risk factors, but the focus of the cross-site evaluation is to evaluate the effectiveness of suicide prevention programs rather than specific interventions. The strategy for NARA’s Youth Suicide Prevention Program seeks to extend the participatory process into evaluation—a best practice outlined by the First Symposium of the Work Group on American Indian Research and Program Evaluation Methodology. This objective will be accomplished by incorporating outcomes development and informal evaluation within the existing No More Fallen Feathers (NMFF) project activities, collecting useful data from existing sources, as well as designing new data collection activities to triangulate data sources to ensure reliable and comparable baseline and project-end data. Because of the huge regional scope of NMFF, activities will target four areas: the Portland Metro area and three of the nine tribal communities participating in the statewide prevention project.

SAMHSA began funding the Circles of Care program over six years ago to support Federally recognized urban Indian programs and tribal governments in their efforts to assess and design culturally competent mental health service models for AI/AN communities. Services are focused on children with serious emotional or behavioral disturbances and their families. AI/AN communities work to reduce disparities by building the infrastructure to increase the effectiveness and capacity of behavioral health systems. To date, 16 grants have been awarded to urban Indian and tribal communities for three-year periods. SAMHSA, the Indian Health Service, and the National Institute for Mental Health cooperate on this effort. Each urban Indian or tribal community conducts strategic planning to examine the existing service system. They identify strengths and needs and develop a model for the system, which is evaluated to determine the potential cost and funding resources needed to implement the model.

In 1993, SAMHSA funded the Comprehensive Community Mental Health Services for Children and Their Families Program, or Systems of Care program, to coordinate mental health services for children with serious emotional disturbances and their families. Systems of Care is a philosophy of how care should be delivered to children and youth, and their families. This approach to services realizes the importance of family, school and community, and seeks to promote the full potential of every child and youth by addressing their emotional, physical, cultural, intellectual, and social needs. Systems of care are coordinated networks of community-based organizations and public-serving agencies that strive to provide youth and family guided services and supports to children with serious mental health needs and their families. Since its inception, 126 communities, including tribal regions and United States territories, have received funding to implement Systems of Care programs in their communities. To date, 14 tribal communities have been funded in addition to various communities serving tribal populations.

CDC is currently collecting and examining data from hospital emergency departments to assess the prevalence of suicide and suicide attempts. The National Electronic Injury Surveillance System-All Injury Program tracks data on types and external causes of nonfatal injuries and poisonings treated in U.S. hospital emergency departments. With these data, CDC researchers can generate national estimates of nonfatal injuries, including those related to suicidal behavior.

SAMHSA is sponsoring an evaluation of the National Suicide Prevention Lifeline, the national crisis hotline. The purpose of the evaluation is to assess the impact of the national crisis hotline which connects callers to mental health services and to assess participation with the Lifelines networks. Although the data collection activities planned as part of this effort will provide valuable information on the effectiveness of this important service for at-risk youth, the scope of the evaluation focuses on all callers (adult and youth) to the national hotline and is specific to one intervention. The cross-site evaluation will add to the information collected as part of this effort to assess other suicide prevention strategies (i.e., gatekeeper training, suicide screening activities, etc.) and focuses on youth specifically. To strengthen communication and collaboration between Crisis Centers in the Lifeline network and the communities they serve where there are populations of indigenous people, at least 6 tribes are served by crisis center regions.

Finally, SAMHSA also funds the National Child Traumatic Stress Initiative (NCTSI). Research has shown that an appropriate intervention at the appropriate time can drastically affect whether children recover from exposure to trauma and, if so, to what extent. A comprehensive report from the National Advisory Mental Health Council Workgroup on Child and Adolescent Health Intervention Development and Deployment (2001) indicates that even when treatments are found to be effective, protocols are not widely understood by clinicians in the field and are not being mandated into practice often enough. The NCTSI is helping to bridge this gap between research and practice. The NCTSI, designed to improve services, access to care, and outcomes for children and adolescents who have experienced trauma, was authorized on October 17, 2000, under the Children’s Health Act of 2000 (Public Law 106-310). As part of this initiative, SAMHSA launched the National Traumatic Stress Network (NCTSN), a national network of grantees, or “centers,” devoted to (1) improving children’s and adolescent’s access to trauma-informed mental health treatments and services and (2) raising the standard of care. To date, at least 6 tribal centers have been funded along with several others that serve AI/AN populations. In 2004, SAMHSA augmented the original center-specific evaluation efforts by providing funding to launch a cross-site evaluation designed to assess the effectiveness of the NCTSI across the country.

5, Impact on Small Businesses

Some of the data for this evaluation will be collected from individuals involved with public agencies, such as mental health, juvenile justice, education, and child welfare agencies. While most data will be collected from community members, it is possible that some individuals may also be employed by small businesses or other small entities. But, these data collection activities will not have a significant impact on these agencies or organizations.

6. Consequences If Information Is Collected Less Frequently

Community Specific Data Collection Activities. Community-specific data collection activities include (1) Evaluation of the GONA baseline interviews with adults, follow-up interviews with adults, and follow-up focus groups with youth; (2) Community Plan Focus Groups with youth and adults; (3) Community Plan In-depth Interviews with adults who continued and those who discontinued involvement with the Community Mobilization Plan team; (4) Community Readiness Assessments; and (5) Data Abstraction and Submission. The consequences of not collecting community-specific data include not understanding the effect of Native Aspirations Project activities on communities, and not providing feedback to communities on those activities and their progress with the project. In turn, this would affect the ability of communities to use data to sustain program efforts and secure additional funding.

GONA baseline interviews will be conducted prior to the GONA in year 1 and follow-up interviews and focus groups will be conducted after the GONA but prior to the CMP meeting (approximately 4 weeks later) in year 1. Each activity is scheduled only once in year 1; thus, consequences of collecting these data less frequently include the potential of losing baseline information on how respondents view and describe their relationships in and with the community; how people in the community deal with youth violence, bullying, substance abuse, and suicide; community members’ willingness to work together to address these issues; and community protective factors and follow-up data that capture the process of the GONA and the solutions/next steps for the CMP.

The Community Plan Focus Groups will be conducted in year 1, one to two days prior to the CMP meeting, and are structured to collect information around the CMP process and organizational/community awareness and involvement. Follow-up interviews are scheduled for year 3 with the same individuals who participated in the focus groups. The consequences of not conducting the focus groups would be a lack of information about how the community views the issues and its awareness of the issues prior to the CMO, as well as implementation of the plan and resulting activities conducted in the community. Consequences of not collecting the follow-up interviews would be a lack of important information concerning the penetration of the Native Aspirations Project activities and process over time, including the reasons why some respondents discontinued their participation in program activities.

CRAs are conducted in years 1 and 3 with key informants and the findings are used to determine the level of community readiness to mobilize efforts for the prevention of youth violence, suicide, and bullying. If CRAs were not conducted in year 1, no overall readiness score for communities would be identified. If CRAs were not conducted in year 3, the potential exists that change in CRA scores would not be assessed and the progress communities made from year 1 to year 3 in terms of community readiness would not be evaluated.

Data abstraction and submission from existing sources are planned two times per each year to support Tier I of the cross-community evaluation through data elements from the following: educational systems, juvenile justice/law enforcement, mental health agencies, child welfare, Medicaid, and community organizations. Collecting this information every 6 months is necessary to evaluate the outcomes of the Native Aspirations Project on the community on an ongoing basis.

Cross-Community Collection Activities. Core data collection activities included in Tier II of the cross-community evaluation include: (1) Service Provider Focus Groups (agency staff and non-agency staff), and (2) C-KABS Adult and Youth Versions. The Service Provider Focus Groups are scheduled for years 1 and 3 and are intended to acquire a broader understanding of provider and service availability for Native youth and how this changes as a result of the Native Aspirations Project. The consequences of not conducting these focus groups in year 1 include not gaining an understanding of service provision and accessibility for Native youth in their communities at a baseline time point. If follow-up focus groups are not conducted, it will be impossible to determine what changes, if any, occurred in the service provision network and the effect of those changes on the youth who receive services.

The C-KABS Adult and Youth Versions will be collected annually. The adult version is designed to gather information on changes in knowledge and awareness of community members related to suicide, substance abuse, violence, and bullying, as well as community members’ exposure to Native Aspirations Project activities. Additionally, adults will report on the availability of services, knowledge of youth risk/protective factors, and stigma around/attitudes toward seeking mental health assistance and how these change over time. The C-KABS Youth Version is designed to gather information about existing social norms around help-seeking behavior, pro-social behavior, exposure to prevention efforts for risky behavior through Native Aspirations, exposure to/engagement in risky behavior, access to and awareness of where to seek help, and willingness to seek help. The consequences of collecting data from the C-KABS Adult and Youth Versions less frequently include not capturing change over time in outcomes related to attitudes, experiences and knowledge.

If core data collection activities were not collected, the cross-community evaluation team’s ability to assess change over time, gather information on Native youth services and service providers, and the effect of these would be greatly impacted, if not made impossible. Additionally, the opportunity to assess program efforts and change over time across communities would not be possible.

7. Consistency with the Guidelines of 5 CFR 1320.5(d)(2)

The data collection fully complies with the requirements of 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

a. Federal Register Notice

SAMHSA published a notice in the Federal Register, Vol. 74, page 31958 on July 6, 2009 soliciting public comment. SAMHSA did not receive any comments on the planned data collection.

b. Consultation Outside the Agency

Individuals outside SAMHSA have consulted on the design, instrumentation, and statistical aspects of the evaluation. Representatives from the Native Aspirations Project contractor, Kaufman & Associates, Inc. (KAI), were consulted in 2008 with respect to the design of the cross-community evaluation and the integration of the evaluation with the Native Aspirations Project. In addition, a Tribal Evaluation Workgroup (TEW) was established in 2008 to provide input and guidance on designing and implementing the cross-community evaluation. TEW members included Christine Walrath; Kara Riehman; Robin King; Candace Fleming; Audrey Lucero; Brenda Bruun; Herminia Frias; Victoria Le Fromboise; Susie Amundson; Gloria Guillory; Davis Ja; Wendi Seibold; Cynthia Hansen; Danyelle Mannix; Estelle Bowman; Sherl Shanks; Lester R. Johnson, III; Hendy Ballot, Sr.; Lisa Wexler; Rose (Clark) Weahkee; Rob Clairemont; Yeshinebet Tulu; Claudette McLeod; JoAnn Kaufmann; and Yvette Joseph.

Consultation with the TEW began in 2008 and will be ongoing throughout the evaluation period. Representatives on the TEW include leaders in the field of suicide prevention, mental health, health services research, health systems that serve youth and families in AI/AN communities; evaluation experts; and representatives from already-funded Native Aspirations communities. The first meeting of the TEW occurred on February 21 – 22, 2008 and was attended by cross-community evaluation staff, SAMHSA staff, TEW members, and the Native Aspirations Project contractor, KAI. These consultations had several purposes: (1) to ensure continued coordination of related activities, especially at the Federal level; (2) to ensure the rigor of the evaluation design, the proper implementation of the design, and the technical soundness of study results; (3) to verify the relevance and accessibility of the data to be collected; and (4) to minimize respondent burden.

On May 13 – 14, 2008, members from the cross-community evaluation team and KAI attended training at SAMHSA offices in Rockville, MD. The training provided an opportunity for the Native Aspirations and cross-community teams to meet with SAMHSA staff and to be trained by experts in aspects of concept mapping, an activity that will be conducted with each community after the community identifies its goals, objectives, and activities as part of their community plan. The meeting also provided an opportunity to receive additional input from KAI staff into the evaluation design and components.

Members of the Native Aspirations Project team from KAI and the Center for Applied Studies in American Ethnicity (CASAE) at the University of Colorado were asked to review each of the evaluation instruments. Specifically, reviewers from KAI included Susie Amundson, Gloria Guillory, Ila McKay, Cookie Rose, and Connie Goodluck O’Mara. CASAE reviewers included Pamela Jumper Thurman, Barbara Plested, and Irene Vernon. Robin King, a member of the cross-community evaluation team, pilot-tested instruments.

9. Payment to Respondents

The incentive structure for the C-KABS Adult and Youth Versions will be determined with individual communities. A maximum of $200 per community will be allocated for C-KABS as incentives. The evaluation team will work with each community to determine the appropriate structure for provision of these incentives. For example, the incentive for each community can be allocated to support youth activities (e.g., field trip, materials for the youth center, etc.). Alternatively, communities may prefer individual $1 incentives up front.

Remuneration is a standard practice in longitudinal studies in efforts to maintain participation in the study. Recontacting survey respondents for follow-up interviews is difficult given the lapse in time between the original survey and the follow-up interview. Compounding the difficulty is circumstances when respondents are not directly affiliated with the programs being evaluated. Therefore, given the hard-to-reach nature of these populations, an incentive will be provided for cross-community evaluation data collection activities that involve follow-up activities. Key informants who consent to participate in the Community Plan In-depth Interviews will be provided a $20 incentive. Respondents to the GONA Follow-up Interviews also will receive $20 in appreciation of their time. An incentive for these respondents is particularly deemed appropriate because these respondents are community members.

Focus group participants will receive a $20 incentive in appreciation of their time. These data collection activities include GONA – Youth Follow-up Focus Groups, Community Plan Focus Groups, and Service Provider Focus Groups.

No remuneration is planned for respondents to the GONA Baseline Interviews and the Community Readiness Assessments because these are considered primary Native Aspiration Project activities.

10. Assurance of Confidentiality

A Web-based data collection and management system may be designed to facilitate data entry and management for the cross-community evaluation and may be used to administer Web-based surveys (i.e., C-KABS) and to enter data at the community level. Descriptive information will be collected from respondents to cross-community evaluation data collection activities, but no identifying information will be entered or stored into the web-based data collection and management system. Identifying information will be requested in order to facilitate GONA baseline interviews, follow-up interviews, and youth focus groups; Community Plan Focus Groups and In-depth Interviews; Service Provider Focus Groups; and Community Readiness Assessments. Identifying information will not be stored with responses and specific procedures to protect the privacy of respondents are described below for each data collection activity.

To further protect participants, ICF Macro will obtain a Federal Certificate of Confidentiality , authorized by Section 301(d) of the Public Health Service Act when SAMHSA receives OMB approval. This certificate provides additional protections of the data from civil and criminal subpoena.

GONA Baseline Interviews, Follow-up Interviews and Focus Groups. GONA interviews and focus groups will be audio recorded and transcribed. Recordings will be destroyed after transcription. Responses to the GONA Baseline and Follow-up Interviews and the GONA Follow-up Focus Groups will be entered into a qualitative software database. Identifying information will be collected from respondents to schedule and facilitate interviews and focus groups, but the identifying information will not be linked in any way to GONA instrument responses. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the qualitative database. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked file cabinets to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Interviewers and moderators will be extensively trained and will be responsible for entering data into the qualitative database. Incentives for respondents to GONA Focus Groups and Interviews will be distributed in person by the evaluation contractor.

Community Plan Focus Groups and Follow-up Interviews. Identifying information will also be obtained from participants to the Community Plan Focus Groups and Follow-up In-depth Interviews to schedule and facilitate focus groups and interviews. However, no identifying information will be entered or stored in the qualitative database, nor will be linked to responses. Community Plan interviews and focus groups will be audio recorded and transcribed and recordings will be destroyed after transcription. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the qualitative database. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked file cabinets to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Interviewers and moderators will be extensively trained and will be responsible for entering responses into the qualitative database. Incentives for respondents to Community Plan Focus Groups and Interviews will be distributed in person by the evaluation contractor.

Service Provider Focus Groups. Identifying information will also be obtained from participants to the Service Provider Focus Groups to facilitate and schedule focus groups. This information will include agency/organization affiliation. However, no identifying information will be entered or stored in the qualitative database, nor will it be linked to responses. Focus groups will be audio recorded and transcribed and recordings will be destroyed after transcription. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the qualitative database. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Moderators will be extensively trained and will be responsible for entering data into the qualitative database. Incentives for respondents to Community Plan Focus Groups and Interviews will be distributed in person.

Community Knowledge, Awareness and Behavior Survey – Adult Version. Identifying information for respondents to the C-KABS Adult Version will be necessary in order to facilitate administration of the C-KABs. The cross-community evaluation team intends to use tribal registration records or community directories to pull a random sample of participants. However, identifying information will be limited to names, addresses and telephone numbers in order to contact non-responders and distribute incentives, but will not be stored with survey responses. To ensure privacy, no identifying information will be entered in the Web-based data collection and management system or stored with hard-copy surveys and, therefore, no identifying information will be associated with individual responses. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the database. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Survey administrators (i.e., community researchers) will be extensively trained and will be responsible for sending completed surveys to the cross-community evaluation team or entering data into the Web-based data collection system, as appropriate.

Community Knowledge, Awareness, and Behavior Survey – Youth Version. Identifying information for respondents to the C-KABS Youth Version will be necessary in order to facilitate administration. The cross-community evaluation team intends to use Tribal or school registration records to pull a random sample of participants. However, identifying information will be limited to names, addresses and telephone numbers in order to contact non-responders and distribute incentives, but will not be stored with survey responses. To ensure privacy, no identifying information will be entered in the Web-based data collection and management system or stored with hard-copy surveys so that no identifying information will be associated with individual responses. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the database. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Survey administrators will be extensively trained and will be responsible for sending completed surveys to the cross-community evaluation team or entering data into the Web-based data collection system, as appropriate.

Community Readiness Assessments. Responses to the Community Readiness Assessments will be entered into a qualitative database. Identifying information will be collected from respondents in order to schedule and facilitate interviews, but the identifying information will not be entered into the database, nor will it be linked in any way to responses. Contact data and identification numbers will be kept in a password-protected Microsoft Access tracking database separate from the qualitative database. Interviews will be audio recorded and transcribed and recordings will be destroyed after transcription. Other procedures for assuring the privacy of respondents will include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms that include identifying information, assigning unique code numbers to each participant to ensure anonymity, and implementing guidelines pertaining to data submission and dissemination. Interviewers will be extensively trained and will be responsible for entering information into the qualitative software.

Existing Data Abstraction and Submission. Local agency staff will abstract and submit data on requested elements to the cross-community evaluation team. All data will be submitted in aggregate form through electronic mail or a Web-based data collection and management system. Data will be submitted in aggregate form and no identifying information will be submitted.

11. Questions of a Sensitive Nature

Survey, interview, and focus group instruments include questions that may be considered sensitive. These questions collect information about youth risk behaviors, suicide, mental health, substance abuse, bullying, violence, exposure to risk factors, and family circumstances. These questions are central to SAMHSA’s goal of learning about the implementation and reach of the Native Aspirations Project. Names collected as part of the consent process will be kept separate from responses as stated above. All data will be managed and stored in the manner described above and therefore will be unavailable to anyone but authorized project staff. Written and verbal consent forms and youth assent forms (see Documents A.4, A.5, A.6, A.7, B.4, B.5, B.6, B.7, B.8, B.9, B.10, B.11, B.12, C.2, C.3, D.3, D.4, E.3, E.4, E.5) explicitly advise potential respondents and participants about the sensitive nature and content of the data collection protocol as well as the voluntary nature of all data collection activities. Unanticipated or negative consequences will be reported immediately to local and Macro International Institutional Review Boards. The principal investigator and project director will also consult with appropriate clinical professionals and immediately determine if a participant presents a risk to themselves or others and make appropriate referrals.

12. Estimates of Annualized Hour Burden

Data collection for the cross-community evaluation in each of the Native communities engaged in FY 2008 (i.e., 9 communities) will begin in the first quarter of FY 2009 and continue through FY 2012, covering a 3-year data collection period. Data collection is expected to begin in December 2009 and continue through September 2012. Data collection for each of the tribal communities engaged in FY 2009 (i.e., 8 communities) will commence upon receipt of their funding and when local regulatory approvals are in place. The start date for data collection with FY 2009 communities is expected by the end of the first quarter of FY 2009 and will continue through FY 2012. This covers a 3-year data collection period for these communities. Data collection for communities engaged in FY 2010 (i.e., 8 communities) will commence upon their agreement to participate and when local regulatory approvals are in place. Data collection is expected to begin at the end of the first quarter of FY 2010 and will continue through FY 2012. Approval for data collection for the final year of funding for FY 2010 communities will be submitted in an OMB renewal package. Table 1 includes the burden associated with cross-community evaluation data collection activities and the associated costs.

All measures included in Table 1 were developed for the cross-community evaluation of the Native Aspirations Program. As such, the cross-community evaluation team piloted each measure with less than 10 respondents to determine burden estimates. The cost was calculated based on the hourly wage rates for appropriate wage rate categories using data collected as part of the National Compensation Survey (BLS, 2004).

Table 1: Annualized Estimate of Respondent Burden

Type of respondent	Instrument	Number of respondents	Number of responses per respondent	Hours per response per respondent	Total Burden hours*	Hourly wage rate ($)	Total Cost ($)
Community Member Attending GONA	GONA Baseline Interviews	50	1	0.33	17	$10.1911	$168.14
Community Member Attending GONA	GONA Follow-up Interviews	75	1	1.0	75	$10.19	$764.25
Youth Community Member Attending GONA	GONA Youth Follow-up Focus Groups	150	1	2.0	300	$6.5522	$1,965.60
Community Member Participating in Plan Development (Project Staff and Community Leaders)	Community Plan Focus Groups	225	1	2.0	450	$10.19	$4,585.50
Community Member Participating in Plan Development (Project Staff and Community Leaders	Community Plan In-depth Interviews – V. 1	51	1	1.0	51	$10.19	$519.69
Community Member Participating in Plan Development (Project Staff and Community Leaders)	Community Plan In-depth Interviews – V. 2	51	1	0.33	17	$10.19	$171.50
Agency Provider	Service Provider Focus Groups – V. 1	252	1	2.0	504	$20.6733	$10,417.68
Non-Agency Provider	Service Provider Focus Groups – V. 2	126	1	2.0	252	$13.6844	$3,448.37
Adult Community Member	C-KABS Adult Version	2,234	1	0.75	1676	$10.19	$1,7073.35
Yoht Community Member	C-KABS Youth Version	2,234	1	0.75	1,676	$6.55	$10,974.53
Key Informant – Community Member (Representing Agencies, Organization in Community)	Community Readiness Assessment	84	1	1.0	84	$10.19	$855.96
Agency Staff	Data Abstraction and Submission	156	2.0	6.0	1,872	$15.005	$28,080.00
Total		5,688			6,974		$7,9024.57

*Rounded to the nearest whole number

Table 2: Annualized Estimate of Respondent Burden

Respondents	Number of Respondents	Number of Responses Per Respondent	Total Responses	Total Annualized Burden hours
GONA Evaluation Instruments
Key Informant GONA Participant - Adult	125	1	125	92
Key Informant GONA Participant – Youth	150	1	150	300
Community Plan Evaluation Instruments
Key Informant Community Plan	327	1	327	518
Service Provider Evaluation Instruments
Service Providers	378	1	378	756
Community Survey Instruments
Community Member, Survey – Adult	2,234	1	2,234	1,676
Community Member, Survey – Youth	2,234	1	2,234	1, 676
Community Readiness Instruments
Key Informant, Community Readiness	84	1	84	84
Data Gathering Instruments
Agency Staff	156	2.0	312	1,875
Total	5,688		5844	6,977

13. Estimates of Annualized Cost Burden to Respondents

There are capital or start-up costs associated with the evaluation for the participating communities. There will be some burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal.

Other costs related to this effort, such as the cost of shipping completed questionnaires and consent-to-contact forms is cost to the Federal government as part of the evaluator’s contract for the cross-community evaluation. The evaluator will support a community researcher at $200 for three days a year to assist with cross-community activities. The funding for this is included in the evaluator’s contract so cost burden will not be imposed on the community.

14. Estimates of Annualized Cost to the Government

CMHS has planned and allocated resources for the efficient and effective management, processing and use of the collected information in a manner that will enhance, where appropriate, it’s utility to agencies and the public. Including the Federal contribution to the Native Aspirations Project contractor, the contract with the cross-community evaluator, and government staff to oversee the evaluation, the annualized cost to the government is estimated at $759,975. These costs are described below.

The cross-community evaluation contract has been awarded to Macro International Inc. for evaluation of the 25 communities participating in the Native Aspirations Project across Cohorts 3, 4 and 5. The current cross-community evaluation contract with SAMHSA provides $2,744,875 across five years, including three years of data collection in each community. The estimated average annual cost of the contract will be $548,975. Included in these costs are the expenses related to developing and monitoring the cross-community evaluation including, but not limited to, the following activities: development of the design and instrument package, provision of technical assistance to sites, travel to sites and relevant meetings, and data analysis and dissemination activities. In addition, these funds will support part-time community researcher positions to assist in the evaluation, and will cover other data collection costs in the funded communities.

The Native Aspirations Project contractor receives an estimated $145,000 per year through its contract with SAMHSA to help support evaluation activities. Additionally, it is estimated that CMHS will allocate 75 percent of a full-time equivalent each year for government oversight of the evaluation. Assuming an annual salary of $88,000, these government costs will be $66,000 per year.

15. Changes in Burden

This is a new project.

16. Time Schedule, Publication, and Analysis Plan

a. Time Schedule

The time schedule for implementing the cross-community evaluation is summarized in Table 2 below.

Table 2: Data Collection and Analysis Time Schedule

Begin data collection for 9 Tribal communities engaged in FY 2008 for the Native Aspirations Project	1 month after OMB approval
Begin data collection for 8 Tribal communities funded in FY 2009 for the Native Aspirations Project	1 month after OMB approval
Begin data collection for 8 Tribal communities engaged in FY 2010 for the Native Aspirations Project	1 year after OMB approval
Data collection completed for 9 Tribal communities engaged in FY 2008	24 months after OMB approval
Data collection completed for 8 Tribal communities engaged in FY 2009	24 months after OMB approval
Data collection completed for 8 Tribal communities engaged in FY2010	36 months after OMB approval
Validate data	Ongoing
Analyze data	Ongoing
Produce monthly reports	Monthly beginning in the first year of the contract
Produce interim semi-annual reports	Every 6 months beginning in the first year of the contract
Produce final dissemination report	24 and 36 months after OMB approval and end of contract

b. Publication Plans

The cross-community evaluation requires semi-annual reports beginning in the first year of the contract regarding the status of data collection and summarizing the results of the evaluation. After OMB approval is obtained, the cross-community evaluation team will analyze data collected and prepare semi-annual reports to summarize key findings. A final report on the results of the cross-community evaluation will be produced by the cross-site evaluation team at the end of the five-year contract period.

Because of the importance of the cross-community evaluation to the fields of suicide prevention, violence prevention, substance abuse prevention, and to tribal communities, in collaboration with SAMHSA and the government project officer, we will publish the results of the cross-community evaluation in relevant professional journals to inform the research community as well as the decision making of policymakers and program administrators. Up to 5 publications are planned, and will likely be submitted in the final year of the cross-community evaluation when maximum data has been accumulated. Possible publications include a manuscript providing an overview of the Native Aspirations Project Cross-Community Evaluation and key findings, as well as manuscripts reporting results from the GONA instruments, Community Plan data collection activities, service provider focus groups, and C-KABS. All publications will be submitted to the Task Order Officer (TOO) in draft form for review and approval prior to submission to the selected journal.

Examples of journals that will be considered as vehicles for publication include the following:

• American Journal of Public Health

• American Psychologist

• American Journal of Diseases of Children

• Child Development

• Evaluation Review

• Evaluation Quarterly

• Journal of the American Academy of Child and Adolescent Psychology

• Journal of Applied Development Psychology

• Journal of Child and Family Studies

• Journal of Clinical Child and Adolescent Psychology

• Journal of Consulting and Clinical Psychology

• Journal of Health and Social Behavior

• Journal of Mental Health Administration

• Psychological Reports

• Social Services Review

• Suicide and Life Threatening Behavior

c. Analysis

GONA Baseline Interviews, Follow-up Interviews, and Focus Groups with Youth. Key informants will provide qualitative data before and after the GONA on the process of the GONA, their experiences during the GONA; participation in GONA activities; views on community relationships; knowledge of the Native Aspirations Project; knowledge of risk factors for youth violence, bullying, substance abuse, and suicide; community protective factors; willingness of community members to work together; and next steps. Qualitative data will be entered into a qualitative database (e.g., using ATLAS.ti software) to allow for thematic analyses within and across communities. Descriptive statistics will be used to summarize information on community members who participated in the GONA. In addition, GONA data collection activities and Community Plan activities, collectively, will be analyzed to determine how the GONA impacted the ability of the community to come together to develop and implement the community plan.

Community Plan Focus Groups and Follow-up In-depth Interviews. The Community Plan Focus Group and Follow-up In-depth Interviews will provide qualitative data which help understand the process and outcomes of the Community Mobilization Plan and its implementation. Focus group questions were designed to facilitate group communication around the community mobilization planning process; understand the early implementation and perceived impact of the plan, and organizational and community awareness and involvement. Follow-up interviews will be used to gather information on the CMP implementation process, organizational and community awareness and involvement with Native Aspirations, and the perceived impact of the Native Aspirations program on the community. Qualitative data will be entered into a qualitative database (e.g., using ATLAS.ti software) to allow for thematic analyses within and across communities. Descriptive statistics will be used to summarize information on community members who participated in these activities.

Service Provider Focus Groups. The Service Provider Focus Groups acquire a broader understanding of provider and service availability for Native youth and how this changes after implementation of the Native Aspirations Project. Participants will include agency staff such as teachers, mental health professionals, justice providers and welfare providers and non-agency staff such as paraprofessional providers or “natural helpers.” Qualitative information collected will address resources and supports within the community to support the well-being of Native youth, how these supports and resources work together (e.g., multiple agency involvement and information sharing), activities and leaders in the community that promote wellness/wholeness, the cultural competence of available resources and supports, and barriers/protective factors to youth access to services. In addition, the longitudinal aspect of the service provider focus groups will help to determine what changes, if any, occurred in the service provision network and the effect of those changes on the youth access to services. Data will be entered into a qualitative database (e.g., using ATLAS.ti software) to allow for thematic analyses within and across communities. Descriptive statistics will be used to summarize information on types of services providers who participated in these activities.

Community Knowledge, Awareness, and Behavior Survey – Adult and Youth Versions. The C-KABS Adult and Youth Versions will collect information on exposure to Native Aspirations Project activities regarding the prevention of suicide, substance abuse, violence, and bullying, and knowledge, awareness and behavior related to these issues. Constructs measured include the availability of services, knowledge of youth risk factors, and stigma around and attitude toward seeking services for wellness. Descriptive statistics will be used to summarize information on adults and youth surveyed at each wave of data collection. For single item measures, examination of the distributions is sufficient. For items that will be part of summative scales, descriptive statistics will be calculated for the scale scores as well, based on the original conceptual groupings. Further, the psychometric properties of the scales will be assessed. First, reliability coefficients will be calculated (i.e., Cronbach’s alpha for continuous variables and KR-20 for nominal variables) for the scales based on the original item groupings. In addition, analysis will include examination of the scales’ factor structures to assess the extent to which initial conceptual groupings of items are supported by statistical item reduction techniques. To accomplish this, we will utilize exploratory factor analysis (EFA) and other scale development techniques to explore the scale properties. Items will be combined into scales based on EFA results. Then these will be tested for internal consistency using Cronbach’s alpha or KR-20, as appropriate. Item-total correlations will be calculated to examine the extent to which each item contributes to the total scale score. These results will be compared with results from the same calculations based on the original conceptual groupings of items to determine the best item clustering for scale construction. In addition, within each community, differences between the responses of the groups of respondents at each wave will be compared using two-sample t-tests and chi-square analyses.

Further, selected dichotomous items will be used as indicators in a latent class analysis (LCA) to create sub-groups of respondents based on their pattern of responses to the items. LCA attempts to categorize different patterns of responses into a small number of mutually exclusive classes of respondents, with each class having a distinct probability of endorsing each item. LCA also offers the opportunity to explore the effects of covariates on class membership as well as the relationship between classes and outcomes. This analysis will enable us to group communities according to similar characteristics and compare outcomes.

Community Readiness Assessments. The CRAs assesses six dimensions across a community’s identified concern (e.g., youth suicide, bullying): community prevention efforts, community knowledge of prevention efforts, leadership, community climate, knowledge about the problem, and resources for prevention efforts. Overall scores representing the stage of readiness of each community to mobilize its resources and efforts to prevent youth suicide, bullying, substance abuse, and violence will be determined and interpreted.

Existing Data Abstraction and Submission. Data elements may be requested from educational systems, juvenile justice/law enforcement sources, mental health agencies, child welfare, Medicaid, and community organizations. Data will be requested at the aggregate level and descriptive, bivariate, and multivariate statistics will be employed.

Table 3: Evaluation Questions, Data Sources, and Analysis Techniques

Evaluation Questions	Data Sources	Data Analysis
Community-specific Evaluation Questions
What were individuals’ experiences of the GONA and how did this affect them? What impact did the GONA have on the community? What effect did the GONA have on the development of the community plans? How do the community plans reflect the needs and characteristics of the community? How do different stakeholders perceive the key components of the community plan? How well do the community plans identify and mobilize local resources for AI/AN youth? What was the experience of developing and implementing the community plans? What was the outcome of the implementation process on the community?	GONA baseline interviews, follow-up interviews, and focus groups with youth Community plan focus groups and follow-up interviews Community readiness assessments Existing data abstraction and submission	Qualitative analysis Descriptive analysis Bivariate analysis Multivariate analysis
Cross-Community Evaluation Questions
What are community knowledge, awareness, and behavior around suicide, violence, bullying, and substance abuse prevention, and how have they changed as result of Native Aspirations project? What are norms surrounding pro-social and help-seeking behaviors of Native youth and how have they changed as a result of the NA project? How would delivery of services to Native youth be characterized? How has this changed? How are services specific to Native youth conceptualized and delivered?	Service provider focus groups C-KABS adult and youth versions	Qualitative analysis Descriptive analysis Bivariate analysis Multivariate analysis Exploratory factor analysis

17. Display of Expiration Date

All data collection instruments will display the expiration date of OMB approval.

18. Exceptions to the Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

B. Statistical Methods

1. Respondent Universe and Sampling Methods

GONA Baseline Interviews. The GONA Baseline Interviews (Document A.1) will be conducted during year 1 of the evaluation, for each community in Cohorts 3, 4, and 5. Interviews will target six adult tribal community members who plan to participate in the GONA in each community. Cross-community evaluation liaisons will collaborate with KAI; together they will communicate with the community researcher to identify a maximum of six community members at each site to participate in the GONA Baseline Interviews. Respondents will be tribal community members, and purposive sampling will be used to identify respondents for the GONA Baseline. If one or more of the original respondents is unable to participate in the interview, the community researcher will identify one or more additional respondents to reach a maximum of six completed interviews. It is estimated that up to 50 respondents will participate in GONA Baseline Interviews across three years of data collection and that interviewing a maximum of six respondents per community in year 1 will be sufficient to ensure saturation of themes in the analysis of results from the qualitative interviews.

GONA Follow-up Interviews Adults. The GONA Follow-up Interviews (Document A.2) will be conducted in the first year of the evaluation with up to nine adults from each community who participated in the GONA. These adults will be tribal community members. Interviews will be conducted in person during site visits held by cross-community evaluation liaisons. Community members selected for the interviews will be identified by the community researcher through purposive sampling. If one or more of the original respondents is not able to participate in the interview, the community researcher will identify one or more additional respondents to obtain a maximum of 9 completed interviews. It is estimated that up to 75 respondents will participate in GONA Follow-up Interviews across three years of data collection and that interviewing a maximum of nine respondents per community in year 1 will be sufficient to ensure saturation of themes in the analysis of results.

GONA Follow-up Focus Groups with Youth. Two GONA follow-up focus groups (Document A.3) with a maximum of nine Native youth participants will be held in each community from Cohort 3, 4, and 5 in follow-up to the GONA. The focus groups will be held in year 1 of the evaluation. Youth will represent a random selection of youth who participated in the GONA and will be selected by the community researcher, who will employ purposive sampling methods to select youth for participation. Across communities, youth will be recruited through their caregivers; this may include sending letters to caregivers whose youth attended the GONA. It is estimated that up to 225 respondents will participate in GONA Youth Follow-up Focus Groups across three years of data collection and that conducting two focus groups with a maximum of nine respondents per community will be sufficient to ensure saturation of themes in the analysis of results.

Community Plan Focus Groups. Participants of the Community Plan Focus Groups (Document B.1) will include Native adults and youth who attended the Community Mobilization Plan meeting. Participants will be selected through purposive sampling methods by the community researcher in collaboration with the cross-community evaluation team and KAI. Three focus groups with a maximum of nine participants in each will be conducted in each community for Cohorts 3, 4 and 5 in year 1. It is estimated that up to 225 respondents will participate in Community Plan Focus Groups across three years of data collection and that conducting two focus groups with a maximum of nine respondents per community will be sufficient to ensure saturation of themes in the analysis of results.

Community Plan In-depth Interviews. In-depth interviews with Native adults and youth will be conducted in the third year of the evaluation to understand the longer-term implementation process of the CMP and respondents will be identified through purposive sampling. These in-person interviews will be conducted in year 3 of the evaluation with some of the same respondents that participated in the Community Plan Focus Groups. There are two versions of the Community Plan In-depth Interview; one version will be used with participants who remained active in the community mobilization process and the other version will be used with respondents who discontinued their involvement in the community mobilization process. Version 1 (Document B.2) will be conducted with up to nine participants in each community for a total of 153 respondents across three years of data collection. Version 2 (Document B.3) also will be conducted with up to nine participants in each community for a total of 153 respondents. The cross-community evaluation team estimates that this will be sufficient to ensure saturation of themes in the analysis of results.

Service Provider Focus Group. Two respondent types will be selected for the Service Provider Focus Groups: (1) agency staff, including teachers, mental health professionals, juvenile justice providers, and welfare providers; and (2) non-agency staff, including paraprofessionals and natural helpers. The respondent universe will include all providers in each of these categories; specifically, providers who work with tribal populations within the community are eligible respondents. The recruitment strategies used to identify and invite respondents to participate in the focus group will be developed locally, and therefore vary, by community. To obtain a representative sample, the agencies/organizations that employ service providers serving the identified adult and youth population will be identified by the community researcher. The cross-community evaluation team will develop a sampling strategy that will be employed. The respondent pool will be stratified by agency/organization, to ensure representation from the various child-serving organizations. The community researcher will identify the agencies/organizations from which to sample and will identify individuals within these agencies/organizations to invite to participate. Cross-community evaluation staff will hold up to 3 focus groups in each community across Cohorts 3, 4, and 5 with a maximum of nine participants in each, in years 1 and 3 of the grant. Two focus groups will be conducted with agency providers (Document D.1) and one focus group will be conducted with non-agency providers (Document D.2). If one or more of the respondents is unable to participate in the interview, the community researcher will identify one or more additional respondents representing the same provider type to participate. It is estimated that up to 252 respondents will participate in the Service Provider Focus Groups for Agency Staff and that a maximum of 126 non-agency providers will participate in the Service Provider Focus Groups – Non-agency Staff Version across three years of data collection. The cross-community evaluation team estimates that this will be sufficient to ensure saturation of themes in the analysis of results.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – Adult Version. Respondents for the C-KABS-Adult Version (Document E.1) will represent a sample of the adult tribal population in each Native Aspirations cohort 3, 4, and 5 community. The community researcher will administer the survey annually in all three years of the grant. A sampling plan to obtain 100 Native adult respondents aged 18 years and over in each community will be developed by the cross-community evaluation team. Community researchers will be responsible for either pulling the sample or identifying someone in the community (or tribal council) to pull the sample. We anticipate response rates of 80%, therefore oversampling will be required. Due to the homogenous nature of the population, stratified samples are not necessary. When analyzing all the communities at each wave of administration a sample size of 100 per community will have a maximum margin of error for a proportion of ±2.0% with 95% confidence. Furthermore, a sample of 100 independent respondents per community at each administration will have at least 80% power to detect differences as small as 4.0% between administrations at 5% significance level for a two-sided alternative hypothesis. When analyzing individual communities, on the other hand, this sample size will have a maximum margin of error of ±9.8% and will detect differences of 19.7% between administrations with the same power and at the same significance level.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – Youth Version. Respondents for the C-KABS Youth Version (Document E.2) will represent a sample of tribal youth community members. The survey will be administered annually in all three years of the evaluation and 100 youth per community, per administration will be identified. Anticipated response rates are 80%, therefore oversampling will be required. The respondent universe will be all youth between the ages of 11 and 18. The community researcher will be responsible for either pulling the sample or identifying someone in the community (or tribal council) to pull the sample. As with the C-KABS Adult Version, each community will develop its own strategy for identifying lists of potential respondents. These strategies may include recruiting from local public schools, Tribal registries, or community organizations. As with the C-KABS Adult Version, stratified samples are not necessary. When analyzing all the communities at each wave of administration a sample size of 100 per community will have a maximum margin of error for a proportion of ±2.0% with 95% confidence. Furthermore, a sample of 100 independent respondents per community at each administration will have at least 80% power to detect differences as small as 4.0% between administrations at 5% significance level for a two-sided alternative hypothesis. When analyzing individual communities, on the other hand, this sample size will have a maximum margin of error of ±9.8% and will detect differences of 19.7% between administrations with the same power and at the same significance level.

Community Readiness Assessments. Six key informants for the Community Readiness Assessments (Document C.1) will be interviewed in years 1 and 3 of the evaluation for a total of 252 key informants. The respondent universe includes community members participating in the Native Aspirations Project and purposive sampling methods will be employed. Up to six key informants will be identified and randomly selected by the community researcher for participation in the interviews. No sampling strategy will be employed.

Data Abstraction and Submission from Existing Sources. Data will come from education, juvenile justice/law enforcement, mental health, child welfare, Medicaid, and community organizations. Data abstraction will occur twice a year for each year of the evaluation. The community researcher will identify appropriate agencies and organizations, based on the scope of information to be collected and the presence of agencies and organizations in the community, as resources from which to collect desired data elements. Up to 175 agency staff will be involved in the data abstraction and submission process. The respondent will include agency staff from the agencies/organizations listed above. No sampling method will be used.

2. Information Collection Procedures

GONA Baseline Interviews. The GONA Baseline Interviews (Document A.1) will be conducted in year 1 of the evaluation, for each of the 25 communities funded in cohorts 3, 4, and 5. The community researcher will work with the cross-community evaluation team and KAI and to identify respondents from the community who plan to participate in the GONA in each community (See Section 2 for description of respondents). The community researcher will be responsible for identifying a list of appropriate respondents and forwarding the appropriate contact information to the cross-community evaluation team for administration. Because it will be necessary to facilitate administration of the interview, identifying information for each respondent will be forwarded to the cross-site evaluation team. However, no identifying information will be included on the data collection instrument. The cross-community evaluation team will randomly select a maximum of six respondents from each respondent list and contact the individuals via telephone to introduce the study, request participation and to schedule an appointment for administration of the interview. Each respondent, prior to administration of the GONA Baseline Interviews, will provide verbal consent (GONA Evaluation – Baseline Consent Form, Phone Script and Verbal Consent Form - Document A.4). The cross-community evaluation team will be responsible for administering the interview by telephone and will be trained by the cross-community evaluation project director or deputy project director in qualitative interviewing. Interviews will be audio recorded but respondents will not be identified by name.

GONA Follow-up Interviews Adults. The GONA Follow-up Interviews (Document A.2) will be conducted in the first year of the evaluation with up to 9 adults from each of the 25 funded communities who participated in the GONA (See Section 2 for description of respondents). These adults will be tribal community members. Interviews will be conducted in person during site visits by cross-community evaluation liaisons. The community researcher will work in collaboration with the cross-community evaluation team and KAI to identify a list of appropriate respondents and the community researcher will schedule interviews. The cross-community evaluation team and community researcher will be responsible for administering the interviews and will be trained by the cross-community evaluation project director or deputy project director in qualitative interviewing. Each participant will provide written consent prior to the interview (GONA Evaluation – Follow-up Interview Consent Form - Document A.5).

GONA Follow-up Focus Groups with Youth. The cross-community evaluation team will conduct two focus groups with a maximum of 9 youth participants in each community from cohort 3, 4, and 5 in follow-up to the GONA (Document A.3; See Section 2 for description of respondents). The focus groups will be held in year 1 of the evaluation. The community researcher will work in collaboration with KAI and the cross-community evaluation team to identify a random selection of appropriate youth who participated in the GONA and to schedule the focus groups. Across communities, youth will be recruited through their caregivers; this may include sending letters to caregivers whose youth are enrolled in the public school system or registered as Tribal members or recruiting youth and their parents during the GONA. The cross-community evaluation team will obtain consent from caregivers for youth to participate (GONA Follow-Up Youth Focus Group Caregiver Consent - Document A.6) and youth assent to participate (GONA Follow-Up Youth Focus Group Youth Assent - Document A.7). The cross-community evaluation team and community researcher will be responsible for conducting the focus groups and will be trained in focus group moderation by the cross-community evaluation project director or deputy project director. Focus groups will be audio recorded but respondents will use first or alternate names only.

Community Plan Focus Groups. Participants of the Community Plan Focus Groups will include Native adults and youth who attended the Community Mobilization Plan meeting (Document B.1). Participants will be selected in collaboration with the community researcher, KAI, and the cross-community evaluation team (See Section 2 for description of respondents). Three focus groups with a maximum of 9 participants in each of the 25 funded communities will be conducted in year 1. The community researcher will be responsible for scheduling focus groups and the cross-community evaluation team and community researcher will conduct the focus groups. The cross-community evaluation team members will be trained in focus group moderation. Focus groups will be audio recorded but respondents will use first or alternate names only. Consent to participate will be obtained from adult participants (Community Plan Focus Group Consent – Document B.4) and youths’ caregivers (Community Plan Focus Group Caregiver Consent – Document B.5) and youth will assent to participate (Community Plan Focus Group Youth Assent -Document B.6).

Community Plan In-depth Interviews. Community Plan In-depth Interviews (see Documents B.2, B.3) will be conducted in the third year of the evaluation with each funded community to assess the long term implementation process of the CMP. Respondents include the same respondents that participated in the Community Plan Focus Groups (see Section 2 for description of respondents). There are two versions of the Community Plan In-depth Interview; one version will be used with participants who remained active in the community mobilization process and the other version will be used with respondents who discontinued their involvement in the community mobilization process. The community researcher will be responsible for identifying up to 9 appropriate respondents for each version of the instrument and scheduling interviews during a site visit by the cross-community evaluation team. The cross-community evaluation team and community researcher will conduct these interviews and will be trained by the cross-community evaluation project director or deputy project director in qualitative interviewing. Adult participants for both versions will be required to provide written consent prior to participation (Community Plan In-Depth Interview V.1 Consent – Document B.7 & Community Plan In-Depth Interview V.2 Consent – Document B.8) and youth participants will provide written caregiver consent (signed by a parent or guardian) and youth assent (Community Plan Interview V.1 & V.2 Caregiver Consent -Documents, B.9, B.10 and Community Plan Interview V.1 & V.2 Youth Assent – Documents B.11, B.12).

Service Provider Focus Groups. Two respondent types will be targeted for the Service Provider Focus Groups: (1) agency staff, including teachers, mental health professionals, juvenile justice providers, and welfare providers; and (2) non-agency staff, including paraprofessionals and natural helpers (Documents D.1, D.2). The recruitment strategies used to identify and invite respondents to participate in the focus group will be developed locally, and therefore vary, by community. The community researcher will be responsible for identifying and recruiting up to 18 participants for two focus groups with agency providers and up to 9 participants for one focus group with non-agency providers in each of the 25 funded communities in years 1 and 3. Cross-community evaluation team members will obtain written consent prior to focus group participation (Service Provider Focus Group V.1 Consent - Documents D.3 and Service Provider Focus Group V.2 Consent - Document D.4). To obtain a representative sample, the agencies/organizations that employ service providers serving the targeted adult and youth population will be identified by the community researcher. The cross-community evaluation team will develop a sampling strategy that will be employed by the community researcher to identify appropriate respondents. The community researcher will schedule focus groups during the site visit by the cross-community evaluation team and the cross-community evaluation team and community researcher will conduct the focus groups. Focus groups will be audio recorded but respondents will use first or alternate names only.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – Adult Version. Respondents for the C-KABS Adult Version (Document E.1) will represent a sample of adult tribal community members in each Native Aspirations Cohort 3, 4, and 5 community. The community researcher will administer the survey annually in all three years of the grant and written consent will be obtained (C-KABS Adult Consent - Document E.3). A sampling plan to obtain 100 Native adult respondents in each community will be developed by the cross-community evaluation team, in collaboration with KAI and the community researcher. The community researcher will be responsible for identifying strategies to pull these samples and pulling samples or identifying someone in the community to pull the samples. These strategies may include recruiting adults from tribal registries, community directories, or community organizations. Up to 6700 adults will participate in the C-KABS Adult Version across three years of data collection.

Community Knowledge, Awareness, and Behavior Survey (C-KABS) – Youth Version. Respondents for the C-KABS Youth Version (Document E.2) will represent a random sample of tribal youth community members. Youths’ caregivers will provide consent for youth to participate and youth will assent to participate (C-KABS Youth Caregiver Consent -Document E.4 and C-KABS Youth Assent – Document E.5). The survey will be administered annually in all three years of the evaluation across Cohorts 3, 4, and 5 and 100 youth per community, per administration will be targeted. As with the C-KABS Adult Version, each community researcher will develop a strategy for identifying lists of potential respondents. These strategies may include recruiting from local public schools, other community organizations, or tribal registries. Up to 6700 youth will participate in the C-KABS Youth Version across three years of data collection.

Community Readiness Assessments. Six key informants from each Cohort 3, 4, and 5 community will be interviewed in years 1 and 3 of the evaluation for the Community Readiness Assessments (Document C.1; See Section 2 for description of respondents). Respondents in year 1 also will participate in the GONA Baseline Interviews in year 1; thus, in year 1, respondents will be identified using the GONA Baseline Interview identification and recruitment methods. The cross-community evaluation team will be responsible for administering the interview by telephone and will be trained by the cross-community evaluation project director or deputy project director in qualitative interviewing. Interviews will be audio recorded but respondents will not be identified by name. In year 3, the community researcher will collaborate with the cross-community evaluation team and KAI to identify up to six key informants for in-person, follow-up interviews during a site visit by the cross-community evaluation team in year 3. Team members will administer interviews and obtain consents (Community Readiness Assessment Verbal Consent - Document C.2 and Community Readiness Assessment Written Consent – Document C.3).

Data Abstraction and Submission from Existing Sources. Data will come from education, juvenile justice/law enforcement, mental health, child welfare, Medicaid, and community organizations. Data abstraction will occur twice a year for each year of the evaluation. The community researcher will identify appropriate agencies and organization from which to collect desired data elements and will collaborate with the cross-community evaluation team to identify appropriate respondents. Up to 175 agency/organization respondents will be involved in the data abstraction and submission process across three years of data collection. Respondents will submit data elements to the cross-community evaluation team through a Web-based data collection system.

Table 4: Instrumentation, Respondents, and Periodicity

Measure	Data Source(s)	Method	When Collected
Community-specific Data Collection Instruments
GONA Baseline Interviews	Community members who will attend the GONA	Interview	Once in year 1
GONA Follow-up Interviews	Community members who attended the GONA	Interview	Once in year 1
GONA Follow-up Focus Groups with Youth	Youth who attended the GONA	Focus Groups	Once in year 1
Community Plan Focus Groups	Native youth and adults who participated in the CMP	Focus Groups	Once in year 1
Community Plan In-depth Interviews	Native youth and adults who participated in the CMP	Interviews	Once in year 3
MIS Data Abstraction and Submission	Agency/organization staff	Web-based data submission	Biannually for 3 years
Community Readiness Assessments	Key community informants	Interviews	Once in years 1 and 3
Cross-Community Evaluation Data Collection Instruments
Service Provider Focus Groups	Agency and non-agency providers	Focus Groups	Once in years 1 and 3
C-KABS Adult Version	Native adult community members	Paper/pencil survey	Annually
C-KABS Youth Version	Native youth community members	Paper/pencil survey	Annually

3. Methods to Maximize Response Rates

The cross-community evaluation team has taken a number of steps to minimize the burden on tribal communities to ensure that completion is timely. These steps include developing a web-based data collection system, and providing technical assistance to each grantee.

To maximize response rates specifically for the C-KABS, all efforts have been made to minimize the burden on individual respondents by limiting the number of items on each questionnaire and building in functions to facilitate ease in data entry into a Web-based data collection and management system that may be developed for communities that can support that technology. Additionally, a lottery incentive will be provided for adults and youth who complete the C-KABS. For the adult survey, the community coordinator will work with the cross-community evaluation team to schedule an event where the C-KABS Adult Version will be completed. Respondents invited to the event who do not attend will be contacted via telephone or mail to increase response rates. For the C-KABS Youth Version, youth will be identified through school or tribal registries or community organizations and caregivers will provide consent for youth to participate. The cross-community evaluation team will collaborate with the community researcher to obtain permission to conduct the survey in a school-based or other large setting. Non-responders will be contacted through their caregivers by telephone or mail in an attempt to increase response rates.

Methods that will be used to maximize response rates for the qualitative interviews and focus groups (i.e., GONA Follow-up Interviews, GONA Follow-up Focus Groups with Youth, Community Plan Focus Groups, Community Plan In-depth Interviews, and Service Provider Focus Groups) include obtaining buy-in from tribal council members and key stakeholders from the Native Aspirations Project, providing flexibility in scheduling, and conducting follow-up phone calls to non-responders. In addition, the community researcher will help obtain contact information for respondents, which will result in more accurate information, thus increasing response rates. If any identified respondents for the qualitative interviews are unable to participate, the cross-community evaluation team will request that the community researcher identify replacement respondents. In addition, respondents to these activities will receive an incentive for their participation.

4. Tests of Procedures

The cross-community evaluator was contracted to assess the process and outcomes of the already-funded Native Aspirations Project. As such, the instruments to be used in the evaluation were customized around Native Aspirations Project activities (e.g., GONA, CMP) and goals to evaluate the project, as well as to identify youth risk factors for suicide, violence, and bullying. As new measures were developed, standard instrument development procedures including review of the literature, item development, and content review by experts in the field were used (see below). All instruments underwent cognitive or pilot testing, or expert review. These procedures were used to enhance question accuracy, assure cultural competency, and determine administration times.

First, representatives from the Native Aspirations Project contractor, KAI, collaborated with the evaluation team to inform the instrument development process to assess the process and outcomes of Native Aspirations Project activities, including the GONA, CMP, and Community Readiness Assessments. Native experts and key staff from Native Aspirations Projects were consulted in developing the GONA, community plan, and service provider data collection activities. Second, a thorough review of the literature was conducted related to risk factors for suicide, violence, and bullying in tribal communities and the prevention of these issues to develop the C-KABS Adult and Youth Versions. Third, drafts of the instruments were developed and reviewed by cross-community evaluation team members, representatives from SAMHSA, Native content experts in the field of tribal youth risk factors, and Tribal Evaluation Workgroup members (see section 8.b for a description of the TEW). Fourth, the revised instruments underwent cognitive testing or pilot testing on no more than nine respondents matching the type and age appropriate for the instrument, in efforts to enhance question accuracy and determine administration time.

5. Statistical Consultants

The cross-community evaluator, a contractor for SAMHSA, has full responsibility for the development of the overall statistical design, and assumes oversight responsibility for data collection and analysis. Training, technical assistance, and monitoring of data collection will be provided by the evaluator. The individual responsible for overseeing data collection and analysis are:

Christine M. Walrath-Greene, Ph.D.

Macro International Inc.

116 John Street, Fl. 8

New York, NY 10038

(212) 941-5555

Kara Riehman, Ph.D.

Macro International Inc.

3 Corporate Square, Suite 370

Atlanta, GA 30329

(404) 321-3211

The following individuals will serve as statistical consultants to this project:

Christine M. Walrath-Greene, Ph.D.

Macro International Inc.

116 John Street, Fl. 8

New York, NY 10038

(212) 941-5555

Kara Riehman, Ph.D.

Macro International Inc.

3 Corporate Square, Suite 370

Atlanta, GA 30329

(404) 321-3211

Tesfayi Gebreselassie, Ph.D.

Macro International Inc.

3 Corporate Square, Suite 370

Atlanta, GA 30329

(404) 321-3211

Carrie Petrucci, Ph.D.

15720 Ventura Blvd, Penthouse

Encino, California 91436-2929

(818) 990-8301

The agency staff person responsible for receiving and approving contract deliverables is:

Michelle Carnes, Ph.D.

Suicide Prevention Branch

Center for Mental Health Services

Substance Abuse and Mental Health Services

1 Choke Cherry Road

Room 6-1132

Rockville, MD 20857

(240) 276-1873

C. References

Anderson, R. N., & Smith, B. L. (2003). Deaths: Leading causes for 2001. National Vital Statistics Reports, 52(9), 1–85. Retrieved June 10, 2007, from http://www.cdc.gov/nchs/data/ nvsr/nvsr52/nvsr52_09.pdf

Aoun, S. L., & Gregory, R. J. (1998). Mental disorders of Eskimos who were seen at a community mental health center in western Alaska. Psychiatric Services, 49, 1485–1487.

Beals, J., Piasecki, J., Nelson, S., Jones, M., Keane, E., Dauphinais, P., et al. (1997). Psychiatric disorder among American Indian adolescents: Prevalence in Northern plains youth. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 1252–1259.

Beautrais, A. L. (2000). Risk factors for suicide and attempted suicide among young people. Australian and New Zealand Journal of Psychiatry, 34(3), 420–436.

Cooperrider, D. L., Whitney, D., & Stavros, J. M. (2003). Appreciative inquiry handbook. Bedford Heights, OH: Lakeshore Publishers.

Dearing, J. W. (2002, April). Diffusion theory and societal betterment. Paper presented at the conference, Forty Years of Diffusion of Innovation: Its Utility and Value in Public Health, Washington, DC.

Fetterman, D. M., & Wandersman, A. (2005). Empowerment evaluation principles in practice. New York: The Guilford Press.

Gessner, B. D. (1997). Temporal trends and geographic patterns of teen suicide in Alaska, 1979–1993. Suicide and Life Threatening Behavior, 27, 264–273.

HeavyRunner, I., & Morris, J. S. (1997). Traditional native culture and resilience. Research Practice, 5(1), 1–6.

Indian Health Service. (1997). 1997 trends in Indian health. Rockville, MD: Author.

Institute of Medicine. (2002). Reducing suicide: A national imperative. Washington, DC: Author.

Kulis, S., Napoli, M., & Marsiglia, F. F. (2002). Ethnic pride, biculturalism and drug use norms of urban American Indian adolescents. Social Work Research, 26, 101–112.

Moran, J. R., Fleming, C. M., Somervell, P., & Manson, S. M. (1999). Measuring bicultural ethnic identity among American Indian adolescents: A factor analysis study. Journal of Adolescent Research, 14, 405–426.

National Institute of Mental Health. (1999). Frequently asked questions about suicide. Retrieved June 10, 2007 from http://www.nimh.nih.gov/SuicidePrevention/ suicidefaq.cfm.

New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final report (DHHS Pub. No. SMA-03-3832). Rockville, MD: U.S. Department of Health and Human Services.

Peacock, T. (2002). The perceptions and experiences of American Indian high school graduates and dropouts. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.

Pettingell, S. L., Bearinger, L. H., Skay, C. L., Resnick M. D., Potthoff, S. J., & Eichhorn, J. (2008). Protecting urban American Indian young people from suicide. American Journal of Health and Behavior, 23(5), 465-476.

Resnick, M. D., Bearman, P. S., Blum, R. W., Bauman, K. E., Harris, K. M., Jones, J., et al. (1997). Protecting adolescents from harm: Finding from the National Longitudinal Study on Adolescent Health. Journal of the American Medical Association, 278, 823–832.

Stiffman, A. R., Striley, C., Brown, E., Limb, G., & Ostmann, E. (2003). American Indian youth: Who Southwestern urban and reservation youth turn to for help with mental health or addictions. Journal of Child and Family Studies, 12(3), 319–333.

Strand, J. A., & Peacock, T. D. (2002). Nurturing resilience and school success in American Indian and Alaska Native students. Charleston, WV: ERIC Clearinghouse on Rural Education and Small Schools. Retrieved June 10, 2007, from http://www.ericdigests.org/ 2003-4/native-students.html (ERIC Document Reproduction Service No. ED741488)

Strand, J. A., & Peacock, T. D. (2003). Resource guide: Cultural resilience. The Tribal College Journal, 14(4), 28–31.

U. S. Department of Health and Human Services. (1999). Mental health: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; and National Institutes of Health, National Institute of Mental Health.

U.S. Department of Health and Human Services. (2001a). Mental health: Culture, race, and ethnicity—A supplement to Mental health: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.

U.S. Department of Health and Human Services. (2001b). Youth violence: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; and National Institutes of Health, National Institute of Mental Health.

U.S. Public Health Service. (1999). The Surgeon General’s call to action to prevent suicide. Washington, DC: U.S. Department of Health and Human Services. Retrieved June 10, 2007, from http://www.surgeongeneral.gov/library/calltoaction/default.htm

U.S. Public Health Service. (2001). National strategy for suicide prevention: Goals and objectives for action. Rockville, MD: U.S. Department of Health and Human Services. Retrieved June 10, 2007, from http://www.mentalhealth.samhsa.gov/suicideprevention/ strategy.asp

Whitbeck, L. B., Hoyt, D. R., Stubben, J. D., & LaFromboise, T. (2001). Traditional culture and academic success among American Indian children in the upper Midwest. Journal of American Indian Education, 40(2), 48–60.

List of Attachments

Attachment 1 –Key Informant GONA Participant - Adult Instruments

• Document A.1: GONA Baseline Interview

• Document A.2: GONA Follow-up Interview

Attachment 2 –Key Informant GONA Participant – Adult Supporting Documents

• Document A.4: GONA Baseline Interview – Phone Script and Verbal Consent Form

• Document A.5: GONA Follow-up Interview Consent Form

Attachment 3 – Key informant GONA Participant – Youth Instruments

• Document A.3: GONA Youth Follow-up Focus Group Guide

Attachment 4 – Key informant GONA Participant – Youth Supporting Documents

• Document A.6: GONA Youth Focus Groups Caregiver Consent

• Document A.7: GONA Youth Focus Groups Youth Assent

Attachment 5 – Key Informant Community Plan Instruments

• Document B.1: Community Plan Focus Group Guide

• Document B.2 Community Plan In-Depth Interviews Version 1

• Document B.3 Community Plan In-Depth Interview Version 2

Attachment 6 – Key Informant Community Plan Supporting Documents

• Document B.4: Community Plan Focus Group Adult Consent

• Document B.7: Community Plan In-Depth Interviews Version 1 Adult Consent

• Document B.8: Community Plan In-Depth Interviews Version 2 Adult Consent

• Document B.5: Community Plan Youth Focus Groups Caregiver Consent

• Document B.6: Community Plan Focus Groups Youth Assent

• Document B.9: Community Plan Interview Version 1 Caregiver Consent

• Document B10: Community Plan Interview Version 1 Youth Assent

• Document B.11: Community Plan Interview Version 2 Caregiver Consent

• Document B.12 Community Plan Interview Version 2 Youth Assent

Attachment 7 – Service Provider Instruments

• Document D.1: Service Provider Focus Group Guide Version 1

• Document D.2: Service Provider Focus Group Guide Version 2

Attachment 8 – Service Provider Supporting Documents

• Document D.3: Service Provider Focus Group Version 1 Consent

• Document D.4: Service Provider Focus Group Version 2 Consent

Attachment 9 – Community Member, Survey – Adult Instruments

• Document E.1: C-KABS Adult

Attachment 10 – Community Member, Survey – Adult Supporting Docs

• Document E.3: C-KABS Adult Consent

Attachment 11 – Community Member, Survey – Youth Instruments

• Document E.2: C-KABS Youth

Attachment 12 – Community Member, Survey – Youth Supporting Docs

• Document E.4: C-KABS Youth Caregiver Consent

• Document E.5: C-KABS Youth Assent

Attachment 13 – Key Informant, Community Readiness Instruments

• Document C.1: Community Readiness Assessment

Attachment 14 – Key informant, Community Readiness Supporting Docs:

• Document C.2: Community Readiness Assessment Phone Script and Verbal Consent

• Document C.3: Community Readiness Assessment Written Consent

Attachment 15 – Agency Staff Instruments

• Document F.1: Existing Data Inventory

No supporting docs for Agency Staff Instruments.

1 Due to high rates of unemployment and poverty in Indian country, figure is based on 2008 poverty guidelines for family of four ($21,200/2080)

2 Hourly minimum wage beginning in July 2008

3 Bureau of Labor Statistics, U.S. Department of Labor, average salary of $43,000/2080 based on mental health counselor, social worker, correctional officer, and middle school teacher salaries, June 2008

4 Bureau of Labor Statistics, U.S. Department of Labor, average annual salary of social and human service assistants, June 2008

5 Bureau of Labor Statistics, U.S. Department of Labor, average annual salary for office and administrative support, June 2008

- 45 -