Evaluation of the Administration for Community Living’s (ACL) American Indian, Alaska Natives, and Native Hawaiian Programs (OAA Title VI) – NEW
Administration for Community Living/Administration on Aging
June 9, 2017
Project Officer
Kristen Hudgins, PhD, Social Scientist
Office of Performance and Evaluation
Center for Policy and Evaluation
Administration for Community Living
Mary E Switzer Building
330 C Street, SW, Rm 1229A
Washington DC 20201
Telephone: 202-795-7732
Email: [email protected]
In May 2017, ACL’s contractor will release a funding opportunity announcement (FOA) to Title VI Programs grantees seeking applications from grantees interested in participating in the Evaluation of the Title VI Programs. Up to 30 grantees will be selected to participate. According to the 2017 Title VI Program Performance Report (PPR), 271 grantees were providing nutrition and supportive services and 236 grantees (87%) also offered caregiver services. All efforts will be made to select grantees representative of the universe of grantees.
The Evaluation of the Title VI Programs will consist of six data collection activities: (1) program staff interviews, (2) program staff focus group, (3) tribal elders focus group, (4) tribal elder interview, (5) caregiver focus group, and (6) caregiver survey. Exhibit 5 displays the expected number of respondents to each data collection activity.
Exhibit 5. Number of Respondents by Data Collection Activity
Instrument |
Participating Grantees |
Respondents per Grantee |
Total No. Respondents |
Title VI Program Staff Interviews |
30 |
2 |
60 |
Title VI Program Staff Focus Groups |
30 |
2 |
60 |
Title VI Tribal Elder Focus Groups |
30 |
10 |
300 |
Title VI Tribal Elder Interviews |
30 |
2 |
60 |
Title VI Tribal Caregiver Focus Groups |
26 |
10 |
260 |
Title VI Tribal Caregiver Survey |
26 |
11.35 |
295 |
Program Staff Interviews: approximately 2 program staff (one program director, one evaluation assistant) from each of the 30 grantees will participate in interviews via telephone in Year 1. A total of 60 individuals will participate. The contractor will work with each grantee to identify and recruit program staff participants via convenience sampling.
Program Staff Focus Groups: approximately 2 program staff (one program director, one evaluation assistant) from each of the 30 grantees will participate in focus groups conducted at the Title VI Training and Technical Assistance Conference (Older Indians Conference) in Year 3. A total of 60 individuals will participate. Approximately 8 focus groups with 8participants each will be conducted. The contractor will work with each grantee to identify and recruit program staff participants via convenience sampling.
Tribal Elders Focus Groups: an average of 10 elders from each of the 30 grantees are expected to participate in the tribal elder focus groups, for a total of 300 participants. A community researcher will work with each grantee to determine the best methods for identifying and recruiting participants via convenience sampling.
Tribal Elder Interviews: an average of 2 elders from each of the 30 grantees are expected to participate in the tribal elder interviews, for a total of 60 participants. A community researcher will work with each grantee to determine the best methods for identifying and recruiting participants via convenience sampling.
Tribal Caregiver Focus Groups: an average of 10 caregivers from each of the 26 grantees providing caregiver services are expected to participate in the tribal caregiver focus groups for a total of 260 participants. A community researcher will work with each grantee to determine the best methods for identifying and recruiting participants via convenience sampling.
Tribal Caregiver Survey: an average of approximately 12 caregivers from each of the 26 grantees providing caregiver services are expected to participate in the caregiver survey, for a total of 295 respondents. Findings from the 2015 Title VI PPR indicated that the average number of caregivers that attended a caregiver training session per grantee was 23; however, this number ranged from 0 to 1,000 based on the size of the grantee. Thus, it is expected that larger grantees will have more survey respondents than smaller grantees. The contractor or a community researcher will work with each grantee to determine the best methods for identifying and recruiting participants via consensus and convenience sampling.
The contractor’s approach to data collection is grounded in the PAR framework, which aims to improve health and reduce health inequities through involving the people who take actions to improve their own health (Baum, MacDougall, & Smith, 2006). As such, the evaluation utilizes three different approaches to meet the needs of grantees: tribal-led, mixed, and external.
Grantees participating in the tribal-led approach will take primary responsibility for administering data collection activities locally and for gathering secondary data with guidance from the contractor via a community researcher.
Under the mixed approach, the contractor will provide more hands-on support, directly assisting the grantees with data collection where needed while also increasing local capacity for evaluation.
Grantees with limited evaluation capacity will use the external approach, under which the contractor will provide more substantive support and lead primary data collection.
Across approaches, the contractor will collaborate with grantees to ensure successful implementation of data collection plans in a manner that builds upon their capacity for evaluation without being overly burdensome. For example, the contractor will provide in-person training at each grantee on data collection goals; methods, instruments, and protocols; and roles and responsibilities for grantees and the contractor, as well as techniques to facilitate implementation of the data collection plan.
Procedures
Data collection procedures for the Evaluation of the Title VI Programs activities are described below.
Tribal Program Staff Interviews: the contractor will conduct the tribal program staff interviews with up to 2 respondents participating in each interview from each of the 30 grantees via telephone. The contractor will work with individual grantees to identify and recruit appropriate participants.
Tribal Program Staff Focus Groups: the contractor will conduct the tribal program staff focus groups with staff from each of the 30 grantees at the Older Indians Conference in Year 3 of the evaluation. The contractor will work with individual grantees to identify and recruit appropriate participants, as well as to determine a location for the focus groups during the conference. Two staff from each grantee will participate and each focus group will have an average of 8 participants.
Tribal Elder Focus Groups: the contractor will work with individual grantees to determine the best approach for conducting the tribal elder focus groups (e.g., tribal- or contractor-led). Grantees will identify appropriate strategies for identifying and recruiting elders to participate, as well as determine the location of the focus groups, in deference to local customs and traditions. An average of 10 elders from each of the 30 grantees will participate. The contractor or local community researcher will conduct the focus groups with 8-10 participants each.
Tribal Elder Interviews: the contractor will work with individual grantees to determine the best approach for conducting the tribal elder interviews (e.g., tribal- or contractor-led). Grantees will identify appropriate strategies for identifying and recruiting elders to participate in deference to local customs and traditions. An average of 2 elders from each of the 30 grantees will participate. Depending on the evaluation approach used, the contractor or a local community researcher will conduct the interviews.
Tribal Caregiver Focus Groups: the contractor will work with individual grantees to determine the best approach for conducting the tribal caregiver focus groups (e.g., tribal- or contractor-led). Grantees will identify the most appropriate strategies for identifying and recruiting elders to participate, as well as determining the location of the focus groups, in deference to local customs and traditions. An average of 10 caregivers from each grantee will participate. Depending on the evaluation approach used, the contractor or a local community researcher will conduct the focus groups with 8-10 participants each.
Caregiver Survey: the caregiver survey will be administered to caregivers of elders and/or grandchildren once in paper version during Year 3 of the evaluation. Approximately 26 evaluation grantees implementing caregiver support services as part of their Title VI Programs will participate. The contractor or a community researcher will work with each grantee to determine the best method for recruiting respondents, as well as venues for administering the survey, in deference to local customs and traditions.
20.1 Statistical Methodology for Stratification and Sample Selection
Selecting Evaluation Grantees
All Title VI Program grantees will be encouraged to apply for funding to participate in the evaluation through the FOA. However, a defined selection criteria will be used to ensure the final sample of Title VI evaluation grantees mirrors characteristics most important and reflective of all Title VI Program grantees. Examples of characteristics used to ensure representation include the number of participants enrolled in nutrition and/or supportive services (grantee size), region (geography), type of funding (sole source vs. blended), and capacity to conduct the evaluation. Title VI Program applicants were asked about their interest in participating in the Title VI evaluation, and of 269 applicants, 120 indicated that they were very or somewhat likely to apply to take part in the evaluation. Based on the current makeup of the Title VI grantees, it is anticipated that the grantees selected to participate in the evaluation will fall into two categories.
Four grantees that offer Title VI nutrition and supportive services to tribal elders, but do not provide caregiver services
Twenty-six grantees of various size that provide nutrition and supportive services to tribal elders, as well as caregiver services
Caregiver Survey Sampling
Based on the PPR, Title VI grantees have an average of 23 caregivers per site (range, 0-1000). To achieve an overall sample size of 295 completed surveys, it is likely that the census of caregivers will be recruited from small grantees (e.g., < 25 caregivers). The contractor will also work with larger grantees to recruit a sufficient number of respondents using convenience sampling. The sampling strategy to be employed at the grantee-level will be determined after evaluation grantees have been selected through the FOA process and their characteristics are known. All efforts will be made to achieve a representative sample of survey respondents.
20.2 Estimation Procedures
An important consideration for selecting evaluation grantees will be to ensure sufficient number of caregivers within the final sample to allow for the generation of robust estimates to answer the evaluation questions related to the caregiver survey. Sampling for focus groups does not lend itself to the same sample size concerns for generalizability.
A power analysis was conducted to inform the sample size for the caregiver survey. Data from the AoA National Family Caregiver Support Program’s (NFCSP) FY 2010 national survey of caregivers of elder clients indicated that 77% of caregivers reported that services enabled them to provide care longer than otherwise would have been possible had they not received services. We relied on this estimate to conduct the power calculation. To achieve the desired precision, a sample size of 295 completed surveys produces a 2-sided 95% confidence interval with a width equal to 0.10. We assume a response rate to the Title VI Evaluation Caregiver Survey of 50%. Thus, the total recruitment for the survey is 590 to achieve 295 completed surveys.
Findings from the caregiver survey will be reported at the aggregate-level to ensure the privacy of survey respondents. Our calculations indicate that a sample size of approximately 295 caregiver respondents will allow us to derive an accurate estimate of the aggregate-level parameters, as well as conduct bivariate analyses of program implementation to outcome. Data will be used to generate frequencies, means, and other descriptive analyses. We will also perform statistical tests (t-test, chi-square, and regression) to assess the relationship between grantee characteristics and Title VI process measures (from the implementation study) to outcomes of the Title VI Programs. We consider a response rate of 50% a reasonable estimate because the survey takes a short amount of time to complete (20 minutes based on a pretest conducted with Steering Committee and ACL cultural expert), respondents will be recruited through a PAR framework, and respondents will complete a paper version of the survey for their convenience. Response rates for paper surveys tend to be relatively low in general (30-40%). So, we consider a 50% response rate to be quite good and more than adequate for the analyses that need to be conducted. However if a lower than expected response rate results for the caregivers, we will conduct non-response bias tests to determine if any bias resulted from the lower response rate and make adjustments to results with the use of weight adjustments and/or response imputation if needed.
20.3 Unusual problems requiring specialized sampling procedures
There are no unusual problems requiring specialized sampling procedures.
20.4 Use of periodic (less frequent data collection)
The Tribal Program Staff Interviews and Tribal Program Staff Focus Groups will use the same basic questions to assess the nature, context, implementation, and management of Title VI Programs; document challenges and barriers to program implementation; and gather detailed information about the funding of activities. The interviews will be conducted in Year 1 and the focus groups will be conducted in Year 3 to allow for an assessment of change over time. By conducting data with this frequency, we will avoid overburdening participants.
There are no other periodic data collection cycles associated with this study.
Since the start of the evaluation planning process, a key task of the project has been to ensure internal and external support to maximize participation in the evaluation. To gain buy-in from grantee staff, the contractor will give a presentation on the Evaluation of the Title VI Programs at the Title VI Annual Conference and engage the Title VI Resource Centers during regular grantee calls and ad-hoc teleconferences and webinars to increase understanding of the evaluation among grantees. To reach elders and caregivers, the contractor will coordinate with grantee staff and/or the community researcher to communicate the importance of evaluation data collection efforts via informational sessions at congregate meals and other gatherings. Furthermore, the contractor is implementing the PAR approach, which allows for grantees and community researchers to determine the best strategy to recruit, administer, and follow-up with non-respondents.
Tribal Program Staff Interviews and Focus Groups: methods to maximize response rates for the interviews and focus groups include obtaining buy-in from key program stakeholders, conducting the interviews via telephone so respondents can participate at a location of their choosing, providing flexibility around scheduling the interviews, and conducting the focus groups at the Title VI Annual Conference and/or Evaluation Workgroup Meeting where respondents will be in attendance. The contractor will work with grantees to identify, recruit, and schedule focus group participants prior to the conference.
Tribal Elder Focus Groups and Tribal Caregiver Focus Groups: for the tribal elder focus groups and the tribal caregiver focus groups, community researchers and/or grantee staff will determine the best method for recruiting participants in deference to local customs and traditions. To encourage participation, grantees may offer a complimentary meal or gift of food or coffee. Grantees will determine the best venue for conducting the focus groups to meet the needs of the target audiences (i.e., elders and caregivers).
Tribal Elder Interviews: community researchers and/or grantee staff will determine the best method for recruiting participants in deference to local customs and traditions. To encourage participation, grantees may offer a complimentary meal or gift of food or coffee. Interviews will be conducted via phone.
Caregiver Survey: the contractor will work with the grantees to optimize response rates for the caregiver survey, as well as engage the Steering Committee and Title VI Resource Centers to understand how best to communicate with and recruit caregivers for the survey (including whether holding informational sessions would encourage participation, any issues that may impact the ability to collect data from sampled caregivers, and any other considerations that may impact the response rates). Following these conversations, the contractor will develop plans for supporting each evaluation grantee through recruitment and administration of the survey in a manner that is grounded in the PAR approach. Upon request, the contractor will develop flyers and other promotional materials about the survey, as well as develop communications to participants that describe the study, its importance, and how the information will be used. As an example, some grantees may choose to recruit caregivers and distribute the survey for them to complete at their leisure during congregate meals. If requested, the contractor also will work with community researchers to develop scripts for reminder telephone calls to prospective participants who have not responded to the survey and to identify convenient venues for group survey administration. Additionally, the contractor will work with the community researcher to identify relevant settings, such as congregate meals and other gatherings, where the surveys can be easily administered in groups. As an example, some grantees may choose to invite caregivers interested in taking the survey to a complimentary congregate meal where they can complete it.
The interview, focus group guides, and caregiver survey protocols have undergone three reviews: (1) an internal review conducted by the contractor’s Institutional Review Board; (2) a review by ACL staff and their cultural expert; and (3) pilot testing and cognitive interviewing with Steering Committee members, who represent Federal, State, local partners, and Title VI grantees. Modifications to the content, structure, and length of the focus group guides and caregiver survey were made based on feedback received.
Pilot testing of the survey was conducted to determine the burden placed on respondents and cognitive interviewing was conducted to test the cultural appropriateness, clarity of instructions, and comprehensibility of items. Survey development was an iterative process with engagement and input from ACL program staff and cultural competence experts to ensure cultural appropriateness and relevance to the needs of the Title VI Programs and communities.
The information for this study is being collected by the contractor, ICF, on behalf of ACL/AoA. With ACL/AoA’s oversight, the contractor is responsible for the evaluation design, instrument development, data collection, analysis, and report preparation. The contractor team includes doctorate and master level professionals in public health, health services research, statistics, economics, and education. The ACL/AoA project officers include:
Susan Jenkins, PhD
Email: [email protected]
Phone: 202-795-7369
Kristen Hudgins, PhD
Email: [email protected]
Phone: 202-795-7732
Administration on Aging. (2015). A profile of older Americans: 2015. Retrieved from http://www.aoa.acl.gov/Aging_Statistics/Profile/2015/docs/2015-Profile.pdf
Baum, F., MacDougall, C., & Smith, D. (2006). Participatory action research. Journal Of Epidemiology And Community Health, 60(10), 854-857.
U.S. Bureau of Labor Statistics. (2015). National Compensation Survey. Office of Compensation and Working Conditions. Washington, D.C. Retrieved from https://www.bls.gov/ncs/
Centers for Disease Control and Prevention. (2013). The state of aging and health in America 2013. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/features/agingandhealth/state_of_aging_and_health_in_america_2013.pdf
Centers for Disease Control and Prevention. (2008). Racial/ethnic disparities in self-rated health status among adults with and without disabilities—United States, 2004–2006. Morbidity and Mortality Weekly Report, 57, 1069–1073.
Goins, R. T., Moss, M., Buchwald, D., & Guralnik, J. M. (2007). Disability among older American Indians and Alaska Natives: An analysis of the 2000 Census public use microdata sample. The Gerontologist, 47, 690–696.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Windows User |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |