National Family Caregiver Support Program (NFCSP) Process Evaluation
Supporting Statement
Table of Contents
B.1. Respondent Universe and Sampling Methods
B.1.1. Universe of Potential Respondents
B.1.3. Sample Development and Selection of AAAs and LSPs
B.2. Procedures for the Collection of Information
B.2.4. Data Collection Methods
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
B4. Tests of procedures or methods to be undertaken
B5. Individuals Consulted on Statistical Aspect and Individuals Collecting and/or Analyzing Data
Attachment 1-1. 60 Day Federal Register Notice
Attachment 2: Technical Advisory Group Members
Attachment 3.1. LSP Survey IRB Documentation
Attachment 3.2. AAA Survey IRB Documentation
Part B of the Justification for this information collection activity, the Evaluation of Older Americans Act Title III-E National Family Caregiver Support Program, addresses the five points outlined in Part B of the OMB guidelines.
B.1. Respondent Universe and Sampling Methods
The universe of potential respondents includes all State Units on Aging (SUAs) and Area Agencies on Aging (AAAs)1. This will ensure that this first process evaluation of the National Family Caregiver Support Program (NFCSP) will be based on a comprehensive set of results for analysis of the similarities and differences across all states and localities. Based on the AAA survey results, a phased, stratified sample of 200 AAAs will be selected, stratified based on organizational size and geography. The additional criterion for selection will be that the AAA provides, at a minimum, respite and/or caregiver training/education services; the AAA may provide other services as well. Each of the selected 200 AAAs will be asked for a list of local providers of respite and/or caregiver training/education, from which five local providers will be randomly selected for the local provider survey. More information is provided in Section B.1.1-B.1.3.
State Unit on Aging (SUA) Survey
The web-based SUA survey will be administered to the universe of SUAs (N=56) providing NFCSP services and is available for review at: http://stageaoa.acl.gov/AoARoot/Program_Results/docs/Surveys/NFCSP_SUA_Survey.pdf.
ACL will send SUA Director a letter providing an overview of the study, followed by an e-mail notification of the survey. The Directors will be encouraged to share the survey solicitation with the most appropriate person(s) in the agency for completion. We will use web-based format and we will also prepare a PDF version of the survey for individuals who do not have the capacity to complete a web-based survey or who prefer a paper version. The research team will employ various methods to maximize survey participation, which could include telephone follow-up, postcard follow-up, email follow-up, etc.
Area Agencies on Aging (AAA) Survey
The AAA survey will be administered to the universe of 618 AAAs providing NFCSP services. All 618 of the AAAs will receive a single in-depth survey. The AAA survey is available for review at: http://stageaoa.acl.gov/AoARoot/Program_Results/docs/Surveys/NFCSP_AAA_Survey.pdf.
This approach will produce the maximum number of responses from AAAs, which will provide greater ability to compare results across different groupings of AAAs.
The comprehensive AAA survey will also be offered in a PDF version for respondents preferring to complete a paper version of the survey. The survey will solicit information about program models, operational procedures, role of consumer direction/self-directed support, policies for prioritization and cost containment, the role of care management, and activities undertaken in service delivery and quality assurance.
LSP Surveys
The LSP survey will be administered as a web-based survey by UCHC Center on Aging, with an optional PDF version. A total of 1,000 LSPs will be sampled. The LSP survey is available for review at: http://stageaoa.acl.gov/AoARoot/Program_Results/docs/Surveys/NFCSP_LSP_Survey.pdf
The rest of Section B.1 covers the sampling approach for the LSP surveys:
Universe of Potential Respondents
Stratified Sampling
Sample Development and Selection of AAAs and LSPs
B.1.1. Universe of Potential Respondents
At the local service provider (LSP)2 level, the universe of potential respondents includes LSP organizations delivering the key NFCSP-funded services of respite and/or caregiver training/education. Together, these two types of NFCSP services account for more than half of the program funding3 and serve almost 200,000 clients annually. In Fiscal Year 2011, there were 11,309 total providers across all types of Title III-E services. Of the total providers nationally, as many as 85 percent provided either respite (n=8,368) or caregiver training/education and (n= 1,247). Thus, the results of the LSP survey will be generalizable to other programs providing these services and represent a substantial majority of all LSPs.
The sample of the LSPs will be based on a stratified sample of all AAAs that both complete the AAA survey and provide either respite and/or caregiver training/education (from AAA survey question #19e and #19b, respectively). From the AAA survey, we will draw a sample of 200 AAAs based on their responses to the provider module on the AAA survey. We will stratify the sample of AAAs by the following:
Size of AAA budget (small, medium, and large from AAA survey question#100);
Population density of region served by the AAA (from AAA survey question #4)
Rural (Predominantly rural; predominantly remote or frontier)
Urban/suburban (Predominantly urban; predominantly suburban, mix of urban and rural; mix of suburban and rural; mix of urban, suburban and rural)
B.1.3. Sample Development and Selection of AAAs and LSPs
The sampling frame for selecting AAAs for the LSP survey will be the AAAs completing the AAA surveys which confirm that the AAAs provides either respite and/or caregiver training/education services. Based on an expected response rate of 80 percent, there will be completed surveys from 494 AAAs. Of these AAAs completing the AAA survey, a conservative estimate is that at least 80 percent of these 494 AAAs will confirm that they provide either respite and/or caregiver training/education services. Thus, we estimate a sampling frame of 396 AAAs. We will use a sample of 200 AAAs to serve as Primary Sampling Units (PSUs) for selecting samples to collect LSP data.
The initial sample of 200 AAAs will be selected as a stratified random sample. As indicated in Section B.1.2, the sample will be stratified to ensure that the variety of AAAs by a measure of size (MOS) defined as total operating budget and population density will be adequately represented.
We will solicit each of the 200 selected AAAs for a list of LSPs with contracts to provide respite and/or caregiver training/education. We will randomly select 5 LSPs from each AAA-provided list for a sample of 1,000 LSPs.
We will follow up with non-responders first by email, and then by telephone contact.
Considerations for the LSP sampling frame include:
The LSPs included on the AAAs’ contact lists must provide respite and/or training and education among their core services;
Both AAAs that provide direct services and their contracted service providers will be included in the lists to be randomized for survey completion;
LSPs will be asked to provide information about clients served, client assessments, interventions, financing, and use of consumer satisfaction assessments;
The sample will represent providers of services to the major caregiver population group which is caregivers of adults 60 years and older.
Of Phased Approach to LSP Sampling
The AAA survey will be in the field for 6 months. Drawing a sample of 200 AAAs from the AAA survey respondents (to get the LSP lists) will be done in three phases:
Phase 1: After the first 2 months, a proportionate sample of AAAs will be drawn from the group of AAA respondents, to get lists of LSPs.
Phase 2: After the 4th month, an additional sample of LSPs will be drawn from the group of new AAA respondents.
Phase 3: After the 6th month, the remaining sample of LSPs will be drawn from those respondents.
For each phase, we will sample a number of AAAs proportional to the responses received. From Scripps prior experience surveying the AAAs, we expect that this will be around 120 AAAs in Phase 1, 40 AAAs in Phase 2, and 40 AAAs in Phase 3.
To enhance the likelihood of a high participation rate in the process evaluation surveys at each organizational level, we have developed a contact/recruitment strategy that includes advance mailings from the NFCSP funding agency ACL to the SUA grantees and AAA subgrantees. As well, written promotional materials have been prepared, and we plan to prepare online videos to introduce the survey to SUAs, AAAs, and LSPs. Finally, we will obtain, where possible, endorsements from other relevant organizations such as the National Association of States United for Aging and Disabilities (NASUAD). The outline of the NFCSP process evaluation outreach is provided in the “NFCSP Data Collection Activities” under Study Recruitment Materials at: http://stageaoa.acl.gov/AoARoot/Program_Results/Program_survey.aspx. The documentation details the advance contacts, invitations to participate in the survey, and follow ups to be conducted with the SUA, AAA, and LSP surveys.
B.1.5. Sample Size and Response Rates
The sample sizes for each respondent group are discussed in subsections B.1.2 through B.1.3. Exhibit 1 summarizes the planned census/sample sizes and the expected number of responses. No nonresponse is anticipated at the SUA level. For the AAAs and sample of LSPs, we expect 80 percent cooperation.
Exhibit 1 Summary of Sample Sizes
Respondent Group |
Census/ Sample Selected |
Number of Responses |
State and Territorial Units on Aging (census) |
56 |
564 |
Area Agency on Aging (census) |
618 |
494 a |
Area Agency on Aging (stratified sample) |
200 |
200 |
Local Service Provider (random sample) |
1000 |
800a |
|
|
|
aApproximate.
The investigators will use a number of proven methods to maximize participation in the SUA, AAA, and LSP surveys. These include:
Use of survey instruments with clear instructions for completion;
Limited number of open-ended questions in the survey instruments;
Flexibility about the time and administration mode (e.g., online, fax, e-mail, mail) for the SUA, AAA and LSP surveys; and,
Tracking responses and conducting follow-ups with non-respondents via email, phone, and postcard mailings.
B.2. Procedures for the Collection of Information
Procedures for the collection of information addressed below include:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
B.2.1. Statistical Methodology for Stratification and Sample Selection
This is described in subsections B.1.1 through B.1.6.
Analysis weights will be prepared for the LSP survey data file and then merged onto the data files. Most of this work will occur after the data collection period has ended, so that the weights can reflect adjustments for nonresponse. Weights are needed for each of the planned analysis levels. For this process evaluation, weights are needed only for the LSP data gathered through a stratified sample.
We will select the initial sample of AAAs and the subsample of LSPs using software that can directly compute the sampling weights. After data collection is complete and the initial weights are computed, we will conduct an analysis to assess the response patterns. The analysis will consist of univariate and bivariate cross-tabulations. Where appropriate, the analysis will also include multivariate analysis to detect patterns that interactive effects may mask.
B.2.4. Data Collection Methods
The ACL task order officer will elicit the support of the ACL Regional Offices. Before telephone contact, we will send an outreach package by Federal Express to the ACL regional contacts and make a courtesy telephone call to seek their support. After we have established contact with the 10 ACL regions, we will send the 56 SUA outreach packages by FedEx and begin recruitment calling. We will enlist the support of the SUA director and request name and contact information of the designated respondent if it is someone other than the director. If an alternate proxy respondent is identified, we will request that the SUA director give the survey materials to that person, and a survey specialist will attempt to contact that person.
A cover letter
A brochure
A survey preparation worksheet
These materials for the SUA Survey Outreach Package are described in detail in subsection B.3.1 and provided in the “NFCSP Process Evaluation Activities” documentation under the heading of Study Recruitment Materials at: http://stageaoa.acl.gov/AoARoot/Program_Results/Program_survey.aspx.
After the respondent is identified, we will contact them and urge them to complete the survey. In some instances, we will complete the survey with the respondent by telephone. Other respondents will complete the paper survey and return it to the contractor.
The AAA and LSP process surveys will be web-based. Web surveys offer maximum flexibility to respondents and minimize errors associated with data entry of hard-copy surveys, although a PDF version of each survey will be made available to any organization requesting this option. High response rates are achievable when support is available to help respondents during the field period. For this purpose, AAAs and LSPs not responding to the initial mailing will be contacted to (1) identify appropriate respondent(s), (2) provide technical assistance to complete the survey, and (3) monitor completion. Reminder emails will be sent to encourage timely submission of completed surveys.
Recruitment for the AAA data collection will be similar to that for the SUA process survey; however, it will be more reliant on electronic communication. Recruitment materials will be sent by email that includes (1) a brochure; (2) a list of frequently asked questions (FAQ) about the study’s purpose, the role of the AAAs and LSPs and information on whom to contact with questions; and (3) a letter inviting their participation in this study of the National Family Caregiver Support Program. AAAs selected for the sample of LSPs will be asked to make the first contact with the five LSPs through an email drafted by the contractor.
These materials for both the AAA Survey Outreach Package and the LSP Survey Outreach Package are described in detail in subsection B.3.1 and are provided in the “NFCSP Process Evaluation Activities” documentation under the heading of Study Recruitment Materials at: http://stageaoa.acl.gov/AoARoot/Program_Results/Program_survey.aspx
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
To maximize response rate for this study, we will develop multimode data collection systems that ensure high quality data collection while minimizing burden on respondents. Exhibit 1 summarizes the data collection mode and number of responses for each survey. We will encourage greater participation through contact and recruitment materials that are relevant to each sample group. Here, we present our strategies for maximizing response rates by survey.
The process survey will examine the strategies, activities and resources of the Title III-E organizations at each of the three levels of the Aging Network: (1) SUAs, (2) AAAs, and (3) LSPs. We will initiate the contacts at the ACL region level and proceed to the SUA level and from there to the AAAs and LSPs. At each level in the Aging Network, we will not only request endorsement for the next level but also ask the respondent to directly communicate that support to the next level in the Aging Network.
A key element in a high recruitment success rate will be the recruitment materials. Dillman (2000) showed that clear, well-written, and persuasive survey materials assist in higher response rates. Recruitment materials include a cover letter, project brochure, and survey worksheet.
Cover letter. The cover letter will explain the purpose of the NFCSP evaluation and will contain endorsements from other agencies or individuals that support the evaluation.
Brochure. The trifold brochure will contain information on the purpose and importance of the study, key components of the study, contact information for the sponsoring and contracting agencies, and responses to frequently asked questions, with a toll-free number and email address in case the recipient has additional questions.
Survey preparation worksheet. The survey preparation worksheet will list the specific types of information that should be gathered, e.g., prior annual report(s) on clients and budgets. By knowing in advance the types of information we seek, the respondents will be able to identify the best respondent for the survey (him- or herself or another staff member). This will provide time for the selected respondent to prepare for the survey and help reduce the burden on the respondents.
Instruments to be used in this study were pre-tested on small numbers of appropriate respondents (2 SUA representatives, 9 AAA representatives, and 5 LSP representatives). Time burden estimates were derived and refinements made to question wording and ordering based on this pilot testing.
In developing the study protocol, every attempt was made to replicate or adapt existing instrumentation.
The results of the three pretests are summarized below:
SUA. The feedback on pre-testing the SUA survey was generally positive. Both pre-testers preferred taking the survey online, but would like to have the paper survey on hand when completing the survey. The pre-testers suggested changes to specific questions to minimize confusion, which were incorporated into the final SUA survey.
AAA. The respondents’ opinion of the survey was positive overall. A number of changes were recommended. These included reordering questions so that more efficient skip patterns could be used, adding skip patterns or “do not apply” responses, adding definitions, clarifying terms, by adding examples or changing terminology. Most of these changes are straightforward and will assist other AAA directors when they complete the final survey. These changes were made in the survey, along with some recommendations to change specific questions and reduce the length of the survey.
After incorporating the changes to the AAA survey after the first round of pre-testing, Scripps completed a smaller pre-test for the edited survey. The individuals were from the same pool that pre-tested the survey the first time. General comments about the new version were generally positive and respondents reported that the changes “made it easier to complete” and it was “better this time.”
LSP. The majority of the survey was well-received by pre-testing participants. They felt that the information requested would be available to the person being surveyed or in collaboration with colleagues. There were a few areas where questions appeared unclear or where the order of questions could be improved. This feedback was incorporated into the survey.
The investigators chosen by ACL to conduct this study include the following individuals:
Lisa Alecxih, Senior Vice President, The Lewin Group, 703-269-5542, Project Director
Cindy Gruman, Vice President, The Lewin Group, 703-269-5506, Program Manager
Ashley Tomisek, Research Consultant, The Lewin Group, 703-269-5632, Research Assistant
Jane Straker and Suzanne Kunkel, Scripps Gerontology Center
Julie Robinson and Noreen Shugrue, UConn Health Center
In addition, Greg Link, the program manager for the National Family Caregiver Support Program in the Administration on Aging/Administration for Community Living (ACL), provided extensive expertise and helpful review in the development of the study design and instruments.
The ACL task order officer for this study is:
Alice-Lynn Ryssman
Office of Performance and Evaluation, Center for Disability and Aging Policy
Administration for Community Living, US Department of Health and Human Services
1 Massachusetts Avenue, NW, Washington, DC 20201
Telephone-202.357.3491
Fax-202.357.3549
Email: [email protected]
List of Attachments
Attachment |
Type of Documents |
Attachment Number |
1 |
Federal Register Notice of Study Responses to Federal Register Notice Comments |
1-1 1-2 |
2 |
Technical Advisory Group (TAG) members |
2 |
3 |
IRB Approval UConn – IRB Approval Form Scripps – IRB Exemption Letter |
3-1 3-2 |
1 AAAs plan, coordinate, and advocate for the development of a comprehensive service delivery system to meet the needs of older adults in a specific geographic area. They administer state and federal funds for community-based services. http://www.tjaaa.org/glossary-of-terms.aspx.
2 Area agencies normally contract with local for-profit or nonprofit or public providers (LSPs) to deliver benefits. The contract service providers nationwide, providing care under the act, are the largest single network of long-term care providers in the country. An agency may be allowed to directly provide supportive services, nutrition services, or in-home services if it can prove that it can provide these services more effectively. http://www.longtermcarelink.net/eldercare/area_agencies_on_aging.htm.
3 National and state Title III-E clients and expenditures by NFCSP service type are collected through ACL’s State Program Report (SPR) and reported through ACL’s online Aging Integrated Database (AGID) at http://www.agid.acl.gov/ .
4 A 100% response rate is based on the 100% response rate to the SUA survey administered in early 2014 as part of the Title III-C Elderly Nutrition Services Program.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |