Supporting Statement B revised per OMB guidance 12-21-12

B. Collections of Information Employing Statistical Methods
The proposed study consists of the administration of two surveys. The first is a process
evaluation survey the sample of which will be all Aging and Disability Resource Centers
(ADRC) and 20 Area Agencies on Aging (AAA). The second is an outcome evaluation
survey of older adults (60 years and older) and individuals with disability (18 years and
older with a disability) seeking services from the sample of 23 ADRCs plus the 20 AAAs
that participated in the process evaluation survey. The following sections describe the
procedures for sampling and data tabulation.
B.1. Respondent Universe and Sampling Methods
Process Evaluation
The target population for the process evaluation study is all ADRC programs funded
since 2003. The data for the evaluation will be collected through a web-based survey
administered to ADRC directors at the state- and local-levels. All ADRC programs will be
contacted and, therefore, there will be no sampling of ADRCs. Based on the reported
70% response rate for the ADRC Semi-Annual Reporting Tool administered by the Lewin
Group, and the 70% – 80% response rate for the annual Aging Services Network survey
of AAAs conducted by the National Association for Area Agencies on Aging, we expect a
70% response rate to the survey. We propose to keep track of the response rates by
AoA defined geographic region and urban/rural status and will look at the frame
characteristics of non-respondents to determine whether there is any bias in the results
of the study. In addition, the 20 AAAs selected for participation as comparison sites in
the outcome evaluation will receive the web-based, process evaluation survey. A higher
response rate is expected from these organizations since they will have already agreed
to participate in the study during outcome evaluation site recruitment. We expect a
90% response rate.
Outcome Evaluation
For the outcome evaluation, the target population is older adults and individuals with a
disability seeking services from ADRCs and AAAs during the defined study enrollment
period of approximately six months. First a probability sample of 23 ADRCs will be
selected and from the selected ADRCs a sample of older adults and a sample of
individuals with disabilities will be selected. A participant experience survey (PES) will be
administered to all the selected individuals. The results from the survey of older adults
in the sample of ADRCs will be compared to results from the survey of older adults in a
sample of AAAs. The results from the survey of individuals with disabilities in ADRCs
will be compared with the results from the survey of individuals with disabilities in a
sample of AAAs.
AAAs were selected to provide the comparison group because AAAs are established
agencies operating throughout the nation and offer services similar to those offered by
ADRCs, we selected AAAs as the most appropriate comparison group for the outcomes
study. In selecting AAAs for the comparison group, we will eliminate from the pool of
Page 1 of 11

eligible AAAs those that are located in communities that are also served by an ADRC in
order to avoid contamination by competing programs. While AAAs have traditionally
focused on providing services to older adults and ADRCs were designed to serve both
older adults and individuals with disabilities regardless of age, depending on the service
referenced between 30% and 52% of AAAs offer services to younger individuals with
disabilities (Data from the 2010 AAA Survey conducted by N4A and Scripps Gerontology
Center). Therefore, the research team has a reasonable expectation that younger
consumers with disabilities can be recruited from AAAs for the comparison group.
The sampling procedure for selecting sites and individuals is described below,
Outcome Evaluation Site Selection. For the selection of a sample of older individuals and
also for the selection of individuals with disability, a two-stage sampling design will be
adopted. At the first stage of sampling, a sample of 23 ADRCs will be selected from the
population of ADRCs. Three large statewide ADRCs were identified in the population.
These ADRCs are likely to be different from other non-statewide ADRCs with regard to
characteristics of interest in the survey. Therefore, the 3 ADRCs located in the states of
Arkansas, Minnesota and New Mexico will be included in the sample with certainty. A
probability sample of 20 ADRCs will be selected from the remaining population. For
selection of the sample, the population of ADRCs will be stratified by AoA’s 10
geographic regions. Within each region, the population of ADRCs will be further
stratified into two groups based on population densities— rural or urban. One ADRC
will be selected at random from each of the resulting 20 stratum. Table 1 shows the
distribution of ADRCs by the 10 AoA regions. The sample of ADRCs to be selected from
each stratum is also shown.
Table 1: Distribution of Population and Sample of ADRCs by Strata
(excluding ADRCs selected with certainty)
Number of
Region
Sample
ADRCs
1
38
2
2
6262
2
3
49
2
4
67
2
5
62
2
6
19
2
7
4
2
8
20
2
9
22
2
10
10
2
Total
353
20

An additional selection constraint will be used to increase the distribution of study sites
across as many States as possible. Since within each region ADRCs are stratified by rural
Page 2 of 11

and urban, it is possible that some ADRCs in a state could be rural and some ADRCs
could be urban. If we make independent selections within each stratum, then it is
possible that ADRCs from a state could be selected for both the rural and urban strata in
a region. To ensure maximum representation of states, it is desired not to select more
than one ADRC from any state. This means that if an ADRC from a specific state in a
region is selected in the urban stratum, then the ADRCs from that state in the rural
stratum will not be eligible for selection. To ensure that each selected ADRC has a
known probability of selection, the following procedure will be adopted. In each region,
a rural or urban stratum will be selected at random. An ADRC will be selected with
probability proportional to size where size is the number of consumers served by the
ADRC. When an ADRC is selected the selection of the second ADRC from the remaining
stratum in that AoA region will be conducted after excluding any remaining ADRCs in the
same state as the first ADRC selected from that region.
With regard to the selection of comparison AAAs the preferred option, which will make
the comparisons more precise, is to select AAAs from matched communities. First, a list
of all AAAs in each of the 20 strata used to select ADRCs will be developed. AAAs in
communities also served by an ADRC will be eliminated. Then the list of AAAs in each
stratum will be ranked based on the degree of match to the ADRC in that same stratum.
The match will be based on a few key characteristics such as the availability of Long
Term Supports and Services (LTSS) in the service area, Medicaid rules, availability of
Medicaid Home and Community Based Services waivers, and demographic
characteristics of the community such as age distribution and average educational
attainment. From the list of highly matched AAAs in each stratum one will be randomly
selected for participation in the study. This means that some AAAs will have a zero
probability of selection.
If the preferred option is not feasible the research team will select a probability sample
of AAAs using a sampling design similar to the one used for the selection of ADRCs. The
selected AAAs will match the selected ADRCs with regard to region and rural/urban
status. One concern with this approach is that differences detected in consumer
outcomes could be due to differences in unmatched site characteristics. If the
distributions of populations of ADRCs and AAAs by these characteristics are available,
then post-stratification weights based on the number of ADRCs and AAAs will be used as
estimates in each category to reduce any conditional bias in the estimates for the
groups.
Table 2 shows the distribution of AAAs outside of ADRC service areas by the 10 AoA
regions. The sample of AAAs to be selected from each stratum is also shown.

Page 3 of 11

Table 2: Distribution of Population and Sample of AAAs outside of ADRC service areas by
Strata
Region
1
2
3
4
5
6
7
8
9
10
Total

Population
4
22
88
56
48
35
18
37
34
26
368

Sample
2
2
2
2
2
2
2
2
2
2
20

Participant Sampling. The consumers of ADRCs and AAAs will fall into one of the
following three categories: 1) Age 60 or older and disabled; 2) Age 60 or older and not
disabled; and, 3) Age less than 60 and disabled. ADRCs serve all three groups. It is
planned to select a total sample of 3,389 individuals for this study based on power
calculation and budgetary considerations. More detail about the selection of individuals
for this study is presented below.

Older Adult Outcome Evaluation Study
As shown in Table 3, the sample size for older adult consumers1 of ADRCs or AAAs is
1,565 individuals in each group. If we were to select a simple random sample of older
adults in each group, we would be rejecting the hypothesis of no difference in population
percentages with 80% power when there is actually a difference of 5 percentage points.
This assumes a two-sided statistical test at 5% level of significance. Under the sampling
design used for sample selection, the detectable difference will be higher than 5
percentage points. If we assume a design effect of 1.5, then the detectable difference
would be 6 percentage points. We consider a difference of 6 percentage points with 80%
power to be reasonable.
The population of individuals from which the sample will be drawn includes all persons
who contact these organizations for service within six months of the starting date of the
data collection period. For the selection of older adults, we need to select an average of
1

“Consumer” refers to the individual who contacted the ADRC/AAA—i.e., either the
older adult or person with a disability OR that individual’s spouse, family member, or
other caregiver who made the contact.
Page 4 of 11

70 older adults from each participating ADRC. The actual allocation of the sample will be
determined at the time of sample selection in proportion to the estimated number of
older adults served during a time period of six months. A similar procedure will be
followed to select approximately 78 older adults from each AAA site. For the selection
of older adults from each participating ADRC and AAA, we plan to use the following
procedure. Based on the volume of consumers served by these organizations and the
target sample size, a sampling interval for the selection of persons will be determined.
Although all persons contacting the organizations during the six month study enrollment
period will be asked to participate in the study, actual participants will be selected by
the research team based on a predefined sampling interval. For example, if the volume
is 200 persons and the required sample is 50, then we plan to select every 4th person
who contacts the organization. We will adjust this procedure if we expect some nonresponse to the study. We anticipate that in the smaller sites it will be necessary to
include 100 percent of eligible respondents across the full six months to reach our target
sample size. Therefore, it is expected that the sampling interval will be applied only to
larger sites to ensure that the target sample is drawn across the same months as in the
smaller sites. However, if it appears that the target sample size will be reached sooner
than six months in the smaller sites, we will recalculate the sampling interval for the
larger sites so that there is consistency across all sites in terms of the months of study
enrollment.
.
Adults with Disabilities Outcome Evaluation Study
As shown in Table 3, the sample size for persons with a disability(s) who are consumers2
of ADRCs or AAAs is 1,261 individuals in each group. With this sample size we can
detect a difference of 6.8 percentage points in population percentages with 80% power.
Again this is based on the assumption of a design effect of 1.5.
To maintain the same age distribution in the ADRC sample of disabled persons, an
independent sample of 1,261 individuals with disabilities will be drawn from the
participating ADRCs. Specifically, we plan to select a sample of 857 persons who are less
than 60 years of age with a disability and 404 persons who are aged 60 or older with a
disability. If the proportion of older adults with disabilities in ADRCs is very different
from the one assumed we will use post-stratification weight adjustments to correct the
proportion and use these to get the overall estimates for the ADRC group. We plan to
select an average of 63 individuals with disabilities from each selected AAA. An
additional average of 43 individuals who are less than 60 years old with a disability will
be selected from each ADRC. The sample will be allocated in proportion to the number
with disabilities and the selection of individuals will be similar to the methods used for
older adults.
2

“Consumer” refers to the individual who contacted the ADRC/AAA—i.e., either the older adult or person
with a disability OR that individual’s spouse, family member, or other caregiver who made the contact.

Page 5 of 11

Table 3. Populations and Sample Sizes
Sample Detectable Difference
Population
Size
with 80% Power
Older Adults*
ADRCs
1,565
6 percentage points
AAAs
1,565
Persons with a
Disability(s)**
ADRCs
1,261
6.8 percentage points
AAAs
1,261
*Person for whom contact was made to the ADRC/AAA is aged 60 and older.
**Person for whom contact was made to the ADRC/AAA is an adult of any
age (i.e., an “older adult” or an adult younger than age 60 with a disability(s) as
defined in Question 6 of the “Client Screening Tool” (Attachment C)). Because
this group includes adults of any age, it will be possible for the same individual
to be selected for the “older adult” sample as well as for the “persons with a
disability(s)” sample.

Population-Based Estimates
For obtaining population-based estimates each selected ADRC will be assigned a
sampling weight. This weight will be used for statistical analyses. We will also examine
whether weights could be assigned to AAAs, and individuals selected within these first
stage sampling units. We may also assign sampling weights to selected individuals based
on the total population size and the sample size.
B.2. Procedures for the Collection of Information
The process evaluation data collection will be conducted according to the following
procedures:






A letter of support (LOS; see Attachment H) will be sent to all 365 local ADRC
sites and 43 state level ADRC sites. In addition, a LOS will be sent to the 20 AAA
sites that are participating as comparison sites in the outcome evaluation. The
LOS will inform participants about the study, notify them of an email (see
Attachment I) that they will receive inviting them to participate in a web-based
survey, and provide contact information at the LOS originating organization. An
outline of the survey topics will accompany the LOS. All sites will receive the LOS
from AoA.
An email that includes an introduction to the study, instructions for accessing the
web-based survey, a link to the survey, and unique password will be sent to all
process evaluation participants.
Once the web-based survey data are submitted by a participant, they will be
automatically stored in a secure database system that can easily be exported to
SAS, Stata or Excel for statistical analyses.
Page 6 of 11

The outcome evaluation data collection will be conducted according to the following
procedures:






A LOS (see Attachment J) will be sent to the 85 sites that will be selected for
recruitment. Although the target number of participating of sites is 40 (plus the
3 sites selected with certainty), we estimate a response rate of approximately 70
percent (85 recruitment calls). The LOS will inform organization directors and
staff of the study, and notify them that they will be receiving a telephone
recruitment call from research staff.
Research staff will contact selected agencies via telephone to read the
recruitment script (see Attachment K). They will continue telephone recruitment
until a sample of 20 each ADRC and AAA is obtained.
Site staff will be trained by research staff to screen consumers for study eligibility
(see Attachment C), inform them of the study and obtain agreement to
participate (see Attachment D), and to collect a minimal amount of data (see
Attachment E). Once an organization has been recruited, the procedures listed
below will be followed:
a. A consumer contacts the organization (ADRC or AAA) either in-person or by
telephone.
b. The consumer speaks with an Information and Assistance (I & A)/Information
and Referral (I &R) specialist as would normally occur.
c. Once rapport has been established, the I & A/I & R specialist administers an
eligibility screening tool (See attachment C). Some screening questions will
have been answered during the routine conversation; others will need to be
asked directly.
d. If consumer is eligible for the study the I & A/ I & R specialist describes the
study to the consumer and invites him/her to participate in a 20 minute
survey at a later date, and requests permission to share his/her contact
information with the research team.
e. If the consumer agrees to participate in the study, the I & A/ I & R Specialist
will collect a minimal number of additional data elements (See attachment E)
that will be used to contact the participant approximately one month later.
f. Approximately once per month, the I & A/ I & R Specialist will forward the
eligibility screening tool, the agreement to participate form, and the
organization data collection tool to Abt Associates either through a secure
web portal or by FedEx using the pre-paid envelope.
g. For sites with high volume in which the sampling interval applies (see B.1 for
description) the Abt team will apply the interval and select only a sample of
respondents for survey participation. However, the limited data collected by
the organization from respondents who were not selected for survey
inclusion also will be analyzed and findings reported.
h. Abt Associates will prepopulate the screening data into the CATI system.
Page 7 of 11

i.

Participants will be contacted by telephone by highly trained and
experienced survey administrators. Participants will be read an informed
consent statement (See attachment G) and their willingness to participate
will be confirmed. If the individual is willing to participate at that time the
20 minute survey will be administered (see attachment B), otherwise another
time will be arranged.

B.3. Methods to Maximize Response Rates and Deal with Non-Response
We will engage several methods to maximize response rates. At the first stage of
sampling, the organization level, response rates will be maximized through a number of
strategies. First, once a site has been selected, a letter of support will be sent to them
from AoA describing the study and asking for them to participate. Second, in-depth
training, a research manual, and technical assistance will be provided to front line staff
from all participating organizations. The research team will present a 30 to 45 minute
webinar training program that will be offered multiple times to accommodate the
varying schedules of target organization staff. The training webinars will instruct
organization staff on how to: 1)screen for eligibility, 2) describe the study, 3) invite
eligible consumers to participate in the study, 4) obtain informed consent, 5) collect a
minimal amount of pre-survey data, and 6) store and transfer the data to Abt
Associates. In addition, the research team will provide each of the 43 organizations with
a training manual containing detailed instructions on each of the six components listed
above. Finally, research staff will provide ongoing technical assistance to participating
sites in the form of a dedicated e-mail box and telephone line where participating
organizations can relay questions.
To maximize respondent response rates in the second stage of sampling, the research
team will utilize several strategies. First, eligible participants will receive a call from
research staff within one month of their contact with a participating ADRC or AAA to
improve the chances of respondent recall and sample retention. Second, the survey
was designed so that it can be administered in 20 minutes and with minimal burden.
Given that the study sample consists of older adults and persons with a disability(s), a
short, clearly written survey will increase the likelihood of response rates (feedback
provided during focus group telephone interviews with ADRC , AAA, and Centers for
Independent Living stakeholders, June 2011).
Furthermore, an interviewing strategy with the following major components will be
followed. The initial contact script has been carefully developed and refined to be
persuasive and appealing to the sample. Only thoroughly trained and experienced
interviewers, highly motivated and carefully monitored, will conduct the interviewing.
Interviewers will be trained on how to overcome initial reluctance, lack of interest or
hostility during the contact phase of the interview. The interviewing corps will include
Spanish-speaking interviewers to ensure that Spanish language is not a barrier to survey
participation. Twenty-five call attempts will be made to ring-no-answer numbers, and
interviewers will leave an approved message on answering machines.
Page 8 of 11

The CATI program will record all refusals and interview terminations in a permanent file,
including the nature, reason, time, and the interviewer. This information will be
reviewed on an ongoing basis to identify any problems with the contact script,
interviewing procedures, questionnaire items, etc. Also, the refusal rate by interviewer
will be closely monitored. Using these analyses, a “Conversion Script” will be developed.
This script will provide interviewers with responses to the most common reasons given
by persons for not wanting to participate in the survey. The responses are designed to
allay concerns expressed by the telephone contacts.
A refusal conversion plan will be implemented in which each person selected for the
sample that refuses to participate will be re-contacted approximately one-to-two weeks
following the refusal. A conversion script will be utilized in an attempt to convince the
individual to reconsider and participate in the survey. Only the most experienced and
skilled interviewers will conduct the refusal conversions. Exceptions to refusal
conversion will be allowed on an individual basis if for some reason the refusal
conversion effort is deemed inappropriate.
There will be maintenance and regular review of data in the sample reporting file,
derived from both the sample control and CATI files, so that patterns and problems in
both response rate and production rates can be detected and analyzed. Meetings will
be held with the interviewing and supervisory staff and the study management staff to
discuss problems with contact and interviewing procedures and to share methods of
successful persuasion and conversion.
Non-Response Analysis
A comparison of the characteristics of the completed and non-completed cases will be
conducted to determine whether there is any evidence of significant non-response bias
in the completed sample.
Finally, we expect there will be a small percentage of terminated interviews throughout
the course of data collection. A terminated interview is one in which the designated
respondent begins the interview (answers at least one question) but refuses to
complete the interview at some point after the first question. Differences in
characteristics can be analyzed at this stage between those who terminate the interview
and those who complete the interview.
This process of analyzing the characteristics of respondents and non-respondents should
identify whether there is any evidence of significant non-response bias related to the
characteristics of most interest for this evaluation (e.g., age and type of disability).
Demographic data are collected routinely when consumers contact the ADRC and AAA,
and health-related data will be collected during screening. If there are differences, we will
examine survey responses for a small sample of respondents to see if there are differences
in responses based on these characteristics and then try to estimate the non-response bias.
Page 9 of 11

Reasons for non-participation among consumers who initially agree to participate but
decline when contacted by the survey team are included in the introduction to the PES.
These data will be analyzed in relation to the characteristics of interest to determine nonresponse bias.
B.4. Tests of Procedures or Methods to be Undertaken
The process evaluation survey has been vetted with multiple experts in the field
including the ADRC Technical Assistant contractors and a technical expert panel
comprised of health services researchers and organization directors.
The PES has also been shared with and revised based on feedback from researchers and
experts in the field. In addition, the PES will undergo a combined cognitive (40
respondents) and pilot test (70 respondents) once OMB approval has been obtained.
We expect that any revisions that may result from the cognitive and pilot testing will
result in a survey of greater clarity and reduced burden for the respondent. We do not
expect the pilot test of the PES to alter the study design; rather, it may inform nonmaterial, non-substantive changes to the PES. If material or substantive changes to the
survey are indicated from the results of the pilot study we will resubmit the revised PES
to OMB and request an expedited review for final approval that is linked to the original
approval and will not restart the approval clock.
B.5. Individuals Consulted on Statistical Aspects and Individual Collecting and/or
Analyzing Data
K. P. Srinath, Ph.D.
Principal Associate/Scientist
Survey Sampling and Methodology
Abt Associates Inc.
4550 Montgomery Avenue
Bethesda, MA 20814
(310) 634-1836
Daver Kahvecioglu, Ph.D.
Senior Research Associate
Public Health Practice Area Lead
IMPAQ International, LLC
10420 Little Patuxent Parkway, Suite 300
Columbia, MD 21044
(443) 539-1390
Rosanna M. Bertrand, Ph.D.
Associate
Abt Associates Inc.
55 Wheeler Street
Page 10 of 11

Cambridge, MA 02138
Philippe Gwet, Ph.D.
Statistical Consultant
IMPAQ International, LLC
10420 Little Patuxent Parkway, Suite 300
Columbia, MD 21044
(301) 326-9001
John Boyle, Ph.D.
Executive Vice President
Abt SRBI, Inc.
8403 Colesville Road, Suite 820
Silver Spring, MD 20910
(301) 608-3883

Page 11 of 11