CMS-10243 - Supporting Statement B_Clean

October 11, 2019

Testing Experience and Functional Tools: Functional Assessment Standardized Items (FASI) Based on the CARE Tool

Paperwork Reduction Act Submission

Part B: Collections of Information

Employing Statistical Methods

Prepared for:

Kerry M. Lida, PhD

Centers for Medicare & Medicaid Services

Disabled and Elderly Health Programs Group

Mail Stop S2-14-26

7500 Security Boulevard

Baltimore, MD 21244-1850

Prepared by:

Barbara Gage, PhD

George Washington University

Under Subcontract to Truven Health Analytics

Revised by:

Lisa Alecxih

The Lewin Group

CMS-10243

OMB 0938-1037

TESTING EXPERIENCE AND FUNCTIONAL TOOLS:

FUNCTIONAL ASSESSMENT STANDARDIZED ITEMS (FASI) BASED ON THE CARE TOOL

PAPERWORK REDUCTION ACT SUBMISSION

Part B: Collections of Information Employing statistical methods

Federal Project Officer: Kerry M. Lida, PhD

The Lewin Group

October 11, 2019

Table of Contents

Collections of Information Employing Statistical Methods 1

1. Sampling 1

A. Potential Respondent Universe 1

B. Sampling Method 1

C. Data on the Universe of Entities in Tabular Form 1

2. Collecting Information 4

3. Maximizing response rate 5

4. Analysis of reliability and validity 5

5. Contact Information 9

List of Tables

1 Estimated program population numbers by state for experience of care survey (figures supplied by each state) 2

2 Target Sample Size of Completed Assessments per State/Program 3

3 Targeted Sample Size Invitations per State/program 3

4 Example table: IRR testing: Pain and continence at time of HCBS assessment, by population group 6

5 Impairments: Percent missing responses by program type 8

[This page intentionally left blank.]

Collections of Information Employing Statistical Methods

1. Sampling

The primary goal of the FASI project is to develop and field test a setting-agnostic, interoperable set of data elements, called “items,” (hereafter, the FASI Set) that can be used for standardized functional assessment across community-based long term services and supports (CB-LTSS) populations. The targeted populations for this project are older adults (individuals aged 65 and over), younger (aged 18 to 64) adults with physical disabilities, and adults of all ages with intellectual /developmental disabilities (I/DD), severe mental illness (SMI), or traumatic brain injury (TBI). The focus of the testing is on the reliability and validity of these items.

A. Potential Respondent Universe

We will recruit up to six states for the field test. The field test will involve assessors conducting functional assessments of Medicaid beneficiaries receiving home and community-based services (HCBS) in each of these states. For statistical analysis purposes, the clients will be broken down into the following five subpopulations:

• Older adults (aged 65 and over);

• Younger adults (aged 18 to 64) with physical disabilities; and

• Adults of all ages with:

• ID/DD

• SMI

• TBI.

B. Sampling Method

The sample will include people already enrolled in the state programs and expecting to receive an interim or follow-up visit from their case managers in a six month period. States will identify the samples based on these criteria. The samples will be stratified by the five waiver groups to select individuals separately within each of the target population groups. States will be asked for a list of their beneficiaries by population. The sampling information will be transferred to Qlarant through a secure file transfer system.

From the state lists, Qlarant will select a number of individuals three times greater than the desired sample size, to account for refusal or attrition, so that we are assured of reaching the desired sample size by subpopulation. Our total target for completed assessments is 314 assessments per program population.

C. Data on the Universe of Entities in Tabular Form

Table 1 provides the estimated program population numbers by states used for the prior rounds of data collection for an experience of care survey conducted by Truven Health Analytics under contract with CMS.

Table 1: Estimated program population numbers by state for experience of care survey (figures supplied by each state)

State	Waiver Program	Aged	Physically Disabled Only	Aged/ Disabled Combined1	ID/ DD	TBI	SMI	Total
AZ	Elderly/Physically Disabled (E/PD) 1115 Waiver	.	.	18,525	.	.	.	.
AZ	Division Developmentally Disabled (DDD) 1115 Waiver	.	.	.	23,042	.	.	.
CT	Homecare Program for Elders	10,497	958	.	.	200	200	11,855
GA	Independent Care Waiver Group (ICWP) (Aged/Disabled)	.	.	1,184/ 10,636	.	100	.	100
KY	Aged/Disabled Waiver	.	11,000	.	5,000	200	.	.
MD	Aged Disabled	.	.	4,500	.	60	.	4,560
MN	Elderly Waiver	8,000	.	.	16,000	296	3,044	27,340
Total		18,497	11,958	62,709	58,554	1,041	10,544	105,536
.	Response Rate Needed to obtain 335 respondents	1.8%	2.8%	0.5%	0.6%	32.2%	3.2%	0.3%

¹ Depending on the state, some HCBS programs may serve both aged and working age adults with disabilities. The numbers shown here represent that programmatic structure. However, for purposes of creating our samples, we will treat aged and working age adults with disabilities as two distinct groups.

Based on current knowledge of the subpopulations and the TEFT FASI items, the desired sample size should be between 126 to 335 clients per subpopulation. A sample size of 335 clients per subpopulation enables us to detect a kappa of 0.80—which is significantly different than a kappa of 0.60—with 95% confidence. A 0.60 kappa statistic is commonly interpreted as the lower boundary for “substantial” agreement. A sample size of 126 represents that lower limit. Collecting data on 314 beneficiaries per program will ensure an adequate sample size to measure reliability for each of the FASI Set.

While we expect to collect data on 314 clients in each of the five programs, the exact number of clients or types of waiver program enrollments will vary by state. We will work with the six participating states to identify the respective population and sample size from each state that will be targeted. Some states may only collect data on certain populations. Table 2 shows the expected sample size or participants from each program for each state needed to conduct the field test in the prior round of data collection. Table 3 provides the total number of clients that will be approached to participate in the study. A substantially higher number of people will be contacted (Table 3) to ensure we meet the target number of completed assessments noted in Table 2.

Table 2: Target Sample Size of Completed Assessments per State/Program

State/ Assessment Entity	Aged	PD	ID/DD	BI	SMI	TOTAL
AZ	67	171	.	.	.	238
CO	.	.	189	104	100	393
CT	88	29	.	.	113	230
GA	.	.	.	.	.	104
GA	67	12	.	.	.	79
KY	92	102	.	.	.	194
MN	.	.	125	106	101	332
TOTAL	314	314	314	314	314	1,570

Table 3: Targeted Sample Size Invitations per State/program

State/ Assessment Entity	Aged	PD	ID/DD	BI	SMI	# Pop	TOTAL
AZ	164	417	.	.	.	2	581
CO	.	.	600	256	246	3	1102
CT	215	74	.	.	278	3	567
GA	164	28	.	256	.	3	448
KY	227	251	.	.	.	2	478
MN	.	.	400	258	246	3	904
TOTAL	770	770	1,000	770	770	.	4,080

We anticipate data will be collected during a regularly scheduled case management visit, which may or may not include a re-assessment for the participating beneficiaries (i.e., the assessment may be conducted during a regularly scheduled monitoring visit). FASI assessments will only be administered to beneficiaries who consent to participate in the study. Assessors will be using the FASI items to measure four domains: 1) function (mobility, self-care), 2) instrumental activities of daily living (IADL), and 3) caregiver assistance needs.

Individuals will be selected based on having an expected annual or interim assessments occurring during a six month period. All clients falling into these parameters will be selected. The individuals will be identified by the state or the assessment agency that works with the state to conduct the current Medicaid waiver assessments. Each client will be sent a letter that describes the study and requests their participation. We will ask states to send the letter on our behalf: a letter that comes from a known state agency will obtain a better response than a letter from an unknown, out-of-state organization. We also will provide information to local assessment organizations to which people may turn with questions.

Following the mailing, assessors will schedule an appointment to conduct their regularly scheduled assessment. Clients will be asked if they wish to participate during that call. Those who do not wish to participate will be thanked and those who are willing to participate will be scheduled for their assessment. All participants will be given a consent form which will be explained by the assessor and signed by the participant or their legal guardian before beginning the assessment. Assessors will keep the signed consent forms in their locked offices.

As shown in Table 1 (see last row), with the exception of the subpopulation of people with TBI, a sample of 272 people would require participation from fewer than 4% in any population, and from fewer than 1% in several of the population groups. Thus, with the exception of people with TBI, we will easily be able to randomly select more than 272 clients per subpopulation, account for attrition, and obtain the desired 272 clients per subpopulation. We will need nearly one-third of all clients with TBI to participate, which may be difficult. We will work with the states to obtain a greater response rate, reach out especially to TBI advocacy groups, or recruit other states or organizations that can provide TBI clients for the field test.

2. Collecting Information

Some individuals may require interpreters (sign language or non-English spoken language) to be able to participate fully. Others may need or prefer that another person (family member, friend, trusted caregiver) be present during the interview. This information will be collected at the time of scheduling and will help to assure that scheduled interviews can be successfully completed. It also will help to assure the participants that their needs and preferences are being respected and, in so doing, will help to increase the response rate.

Assessments will be conducted by agencies already serving the participants. All assessors will be trained by the George Washington University team in the use of the assessment items using the same on-line, mobile training modules developed for the prior data collection. GW used adult education approaches to create four modules: one on each of the four domains in the FASI set. Assessors will be able to access the training modules from their preferred locations (at home, in the agency, wherever they have internet access.) A PowerPoint copy of the four modules is attached here. All other instructions are embedded in the data collection, or FASI tool. Before being authorized to begin data collection, each assessor will have to complete and pass each of the on-line training modules.

Data will be collected through an assessment interview with the individual participant. Other people, such as knowledgeable family members or caregivers also may provide information during the course of the assessment interview, if requested by the client or if needed because the client is unable to provide information without assistance. Information from others will be provided only with the participant’s consent.

The assessment data will be collected using the assessor’s secure, professional laptop or tablet that is used in conducting their regular state assessments. We will use an electronic PDF form for the assessors to use in the assessment process. The electronic PDFs will be blind and contain no PHI or PII; observation identification numbers will be generated as the forms are uploaded to Qlarant. Each assessor will also have a study ID and the combination of the state ID, the assessor ID, and the electronically-generated observation ID will provide a study number to each assessment, independent of any personally identifying information on the client. The information collected will be submitted electronically at the time of the assessment, or stored on the assessor’s secure device and submitted electronically at the end of the day, depending on the availability of a secure wireless signal and the assessors’ current methods for submitting their assessment data to the state. The data will be submitted to Qlarant where it will be reviewed for completeness and de-identification and then be transferred to GWU for review and construction of the analytic files. Qlarant and GWU will be reviewing the data as it is submitted for completeness.

We will conduct a one-time field test in 2020.

3. Maximizing response rate

Field test assessments will be conducted as visits to the individual’s home or other place of a person’s choosing. For example, an individual may prefer to be interviewed at a day program in which they participate in order to minimize disruption to his/her day.

States will be asked to send letters to the selected individuals, requesting their participation and informing them of the value of their participation to the state. Having contact with consumers through a known and trusted agency, rather than an unknown and out-of-state organization, will increase consumer confidence in the legitimacy of the request and willingness to participate.

Assessors will use their usual practices for scheduling assessments. Where an outside agency is conducting the assessment, follow-up calls by the assessors will be made with each respondent in advance of the assessment, reminding them of the date, time, and location of the appointment. This reminder will help to ensure that individuals and their caregivers (if expected to be present for the assessment) will be present on the scheduled day.

Scheduling of interviews will be managed to assure that adequate sample size is attained. Assessors will submit weekly counts of the number of clients contacted, scheduled, visited, and assessed to the Qlarant data collection manager. No PHI or PII will be transferred in this reporting.

4. Analysis of reliability and validity

We will collect data during a field test of the TEFT FASI Item Set to examine the validity, reliability, and completeness of responses, as well as the feasibility of implementation and data collection across a variety of Medicaid waiver programs. Data analyses will include examination of the psychometric properties of the TEFT FASI Item Set when used for different subpopulations.

The goal sample size will depend on the prevalence of the characteristics being measured in the population, but should be approximately 126 to 335 cases (Sim and Wright, 2005). This range of sample sizes will allow us to detect a kappa of 0.80—that is significantly different than a kappa of 0.60—with 95 percent confidence. A 0.60 kappa statistic is commonly interpreted as being the lower bound of what is considered to be “substantial” agreement. Subsample populations will be formed by combining populations across states. Table 4 is an example of how we might examine the kappas for FASI items overall and stratified by program type. These examples are based on the approach in the prior CARE item testing under the original OMB authorization and the subsequent 2016 data collection. If an item has more than two levels, we will examine both the simple kappa, and a weighted kappa, which allows for the possibility that the “distances” between response levels may not be equal across all levels.

Table 4: Example Table: IRR testing: Pain and continence at time of HCBS assessment, by population group

Item	Effective sample size	Kappa	Weighted kappa
Ability to walk 10 feet	.	.	.
ID/DD	.	.	.
Aged (>= 65)	.	.	.
Physically Disabled	.	.	.
TBI	.	.	.
Severe Mental Illness	.	.	.
Ability to walk 50 feet and make 2 turns	.	.	.
I/DD	.	.	.
Aged (>= 65)	.	.	.
Physically Disabled	.	.	.
TBI	.	.	.
Severe Mental Illness	.	.	.

After we have analyzed the field test data, we will prepare a Field Test Report on the results of reliability, validity, and other testing and its impact on TEFT FASI deliverables (e.g., training materials, the TEFT FASI Item Set) that will need to be revised for the second round of data collection. Both the draft and final Field Test Reports will be submitted to CMS within a timeframe still to be negotiated.

We will perform a variety of statistical tests to assess reliability and validity. The statistical tests will:

• Analyze response distribution.

• Compare responses across functional needs and caregiver assistance across each of the five samples (I/DD, Aged, PD, TBI, SMI).

• Examine the distribution of missing data by client characteristic by program type and geography. Uneven distribution of missing responses on an item either by client characteristic, program type, or geography, suggests that bias may be present. Table 5 shows an example of how we might examine response rates for FASI items by program type.

• Conduct Rasch, confirmatory factor analysis, and other internal consistency analyses to provide information on the construct validity of TEFT FASI items, their proposed scales, and their psychometric properties when used within/across study populations.

• Assess whether there are ceiling effects to be sure that the modified items cover the range of characteristics of clients.

• Evaluate key item-level components of FASI functional assessment, including response scale usage (i.e., the distribution of assistance needs on function items), an examination of the item difficulty hierarchy and how it compares to clinical expectations of item difficulty, and potential disability group differences.

• Perform analyses to assess inter-rater reliability. Inter-rater reliability will be assessed by comparing the information captured by two assessors, recording information independently, but who are both present at the same assessment. This will minimize the burden on respondents, but not requiring each respondent to go through two separate interviews.

Table 5: Impairments: Percent missing responses by program type

Item	Item Name	ID/DD (assessments n) percent missing responses	Aged (>=65) (assessments n) percent missing responses	Physically Disabled (assessments n) percent missing responses	Mental Health (assessments n) Percent missing responses	TBI (assessments n) percent missing responses
VA1A	Bladder and Bowel Management A1a. Bladder Incontinence	.	.	.	.	.
VA1B	A1b. Bowel Incontinence	.	.	.	.	.
VA2A	A2a. Bladder	.	.	.	.	.
VA2B	A2b. Bowel	.	.	.	.	.
VA3A	A3a. Bladder	.	.	.	.	.
VA3B	A3b. Bowel	.	.	.	.	.
VB1	Swallowing B1. Swallowing Disorder Signs and symptoms of possible swallowing disorder.	.	.	.	.	.
VB2	B2. Indicate the person’s usual ability to swallow.	.	.	.	.	.
VC1	Hearing, Vision, and Communication Comprehension C1. Understanding verbal content (With hearing aid or device if used)	.	.	.	.	.
VC2	C2. Expression of ideas and wants	.	.	.	.	.
VC3	C3. Ability to see in adequate light (with glasses or other visual appliances):	.	.	.	.	.
VC4	C4. Ability to hear (with hearing aid or hearing appliance if normally used):	.	.	.	.	.

5. Contact Information

Trudy Mallinson, PhD Lisa Alecxih

FASI Lead Project Director

George Washington University The Lewin Group

Email: [email protected] Email: [email protected]

Phone: (202) 994-6833 Phone: 703-362-7073