CSupportingStmtPartB508

March 23, 2016

Testing Experience and Functional Tools: Functional Assessment Standardized Items (FASI) Based on the CARE Tool

Paperwork Reduction Act Submission

Part B: Collections of Information

Employing Statistical Methods

Prepared for

Allison Lynn Weaver

Centers for Medicare & Medicaid Services

Disabled and Elderly Health Programs Group

Mail Stop C3-19-26

7500 Security Boulevard

Baltimore, MD 21244-1850

Prepared by

Barbara Gage, PhD

George Washington University

Under Subcontract to Truven Health Analytics

CMS-10243

OMB 0938-1037

TESTING EXPERIENCE AND FUNCTIONAL TOOLS: FUNCTIONAL ASSESSMENT STANDARDIZED ITEMS (FASI) BASED ON THE CARE TOOL

PAPERWORK REDUCTION ACT SUBMISSION

Part B: Collections of Information Employing statistical methods

by Barbara Gage, PhD, Co-Task Director

Federal Project Officer: Allison Lynn Weaver, MPH FAC-COR

Truven Health Analytics

March 23, 2016

CONTENTS

B. Collections of Information Employing Statistical Methods 1

1. Sampling 1

Potential Respondent Universe 1

Sampling Method 1

Data on the Universe of Entities in Tabular Form 1

2. Collecting Information 4

3. Maximizing response rate 5

4. Analysis of reliability and validity 5

5. Subsequent State Data Collection in FASI Round 2 9

6. Contact Information 9

List of Tables

1 Estimated program population numbers by state for experience of care survey (figures supplied by each state) 2

2 Target Sample Size of Completed Assessments per State/Program 3

3 Targeted Sample Size Invitations per State/program 3

4 Example table: IRR testing: Pain and continence at time of HCBS assessment, by population group 6

5 Impairments: Percent missing responses by program type 7

[This page intentionally left blank.]

2. Collections of Information Employing Statistical Methods

1. Sampling

The primary goal of the TEFT FASI project is to develop and field test a setting-agnostic, interoperable set of data elements, called “items,” (hereafter, the FASI Item Set) that can be used for standardized functional assessment across community-based long term services and supports (CB-LTSS) populations. The targeted populations for this project are older adults (individuals aged 65 and over), younger (aged 18 to 64) adults with physical disabilities, and adults of all ages with intellectual /developmental disabilities (I/DD), severe mental illness (SMI), or traumatic brain injury (TBI). The focus of the testing is on the reliability and validity of these items.

Potential Respondent Universe

Six states (Arizona, Colorado, Connecticut, Georgia, Kentucky, and Minnesota) are participating in the TEFT FASI component of the TEFT project. The field test will involve assessors conducting functional assessments of Medicaid beneficiaries receiving home and community-based services (HCBS) in each of these states. For statistical analysis purposes, the clients will be broken down into the following five subpopulations:

• Older adults (aged 65 and over);

• Younger adults (aged 18 to 64) with physical disabilities; and

• Adults of all ages with:

• ID/DD

• SMI

• TBI.

Sampling Method

The sample will include people already enrolled in the state programs and expecting to receive an interim or follow-up visit from their case managers between June and December 2016. States will identify the samples based on these criteria. The samples will be stratified by the five waiver groups to select individuals separately within each of the target population groups. States will be asked for a list of their beneficiaries by population. The sampling information will be transferred to Truven Health Analytics through a secure file transfer system already in place from the prior TEFT component on the Experience of Care survey.

From the state list, Truven Health Analytics will select a number of individuals three times greater than the desired sample size, to account for refusal or attrition, so that we are assured of reaching the desired sample size by subpopulation. Our total target for completed assessments is 314 assessments per program population.

Data on the Universe of Entities in Tabular Form

Table 1 provides the estimated program population numbers by state. These data were collected for the experience of care survey conducted by Truven Health Analytics under the earlier part of this contract with CMS.

Table 1

Estimated program population numbers by state for experience of care survey (figures supplied by each state)

State	Waiver Program	Aged	Physically Disabled Only	Aged/ Disabled Combined1	ID/ DD	TBI	SMI	Total
AZ	Elderly/Physically Disabled (E/PD) 1115 Waiver	.	.	18,525	23,042	.	.	.
AZ	Division Developmentally Disabled (DDD) 1115 Waiver	.	.	.	.	.	.	.
CT	Homecare Program for Elders	10,497	958	.	.	200	200	11,855
CT	Personal Care Assistance (which may be replaced by #3 and #4)	.	.	.	.	.	.	.
CT	Acquired Brain Injury Waiver	.	.	.	.	.	.	.
CT	Severely Mentally Ill Waiver	.	.	.	.	.	.	.
GA	Independent Care Waiver Group (ICWP) (Aged/Disabled)	.	.	1,184/ 10,636	.	100	.	100
GA	Community Care Services Program (CCSP)	.	.	.	.	.	.	.
KY	Aged/Disabled Waiver	.	11,000	.	5,000	200	.	.
KY	Michelle P. Waiver (ID/DD)	.	.	.	.	.	.	.
KY	Acquired Brain Injury Waiver	.	.	.	.	.	.	.
MD	Aged Disabled	.	.	4,500	.	60	.	4,560
MD	TBI	.	.	.	.	.	.	.
MN	Elderly Waiver	8,000	.	.	16,000	296	3,044	27,340
MN	Personal Care Assistance	.	.	.	.	.	.	.
Total	.	18,497	11,958	62,709	58,554	1,041	10,544	105,536
.	Response Rate Needed to obtain 335 respondents	1.8%	2.8%	0.5%	0.6%	32.2%	3.2%	0.3%

¹ Depending on the state, some HCBS programs may serve both aged and working age adults with disabilities. The numbers shown here represent that programmatic structure. However, for purposes of creating our samples, we will treat aged and working age adults with disabilities as two distinct groups.

Based on current knowledge of the subpopulations and the TEFT FASI items, the desired sample size should be between 126 to 335 clients per subpopulation. A sample size of 335 clients per subpopulation enables us to detect a kappa of 0.80—which is significantly different than a kappa of 0.60—with 95% confidence. A 0.60 kappa statistic is commonly interpreted as the lower boundary for “substantial” agreement. A sample size of 126 represents that lower limit. Collecting data on 314 beneficiaries per program will ensure an adequate sample size to measure reliability for each of the FASI items.

While we expect to collect data on 314 clients in each of the five programs, the exact number of clients or types of waiver program enrollments will vary by state. We have been working with the six participating FASI states to identify the respective population and sample size from each state that will be targeted. Some states are only collecting data on certain populations. Table 2 shows the expected sample size or participants from each program for each state needed to conduct the field test. Also included are the participating assessment agencies. In certain states, such as Georgia and Kentucky, more than one assessment agency will be contracting with Truven Health Analytics to reach the desired numbers of participants. Table 3 provides the total number of clients that will be approached to participate in the study. A substantially higher number of people will be contacted (Table 3) to ensure we meet the target number of completed assessments noted in Table 2.

Table 2

Target Sample Size of Completed Assessments per State/Program

State/Assessment Entity	Aged	PD	ID/DD	BI	SMI	TOTAL
AZ- Centene	67	171	.	.	.	238
CO – HCBS Strategies	.	.	189	104	100	393
CT- CCCI	88	29	.	.	113	230
GA- GMFC	.	.	.	.	.	104
GA-Legacy	67	12	.	.	.	79
KY – KIPDA	60	60	.	.	.	120
KY- Buffalo Trace	32	42	.	.	.	74
MN- Vital Research	.	.	125	106	101	332
TOTAL	314	314	314	314	314	1,570

Table 3

Targeted Sample Size Invitations per State/program

State/ Assessment Entity	Aged	PD	ID/DD	BI	SMI	# Pop	TOTAL
AZ	164	417	.	.	.	2	581
CO	.	.	600	256	246	3	1102
CT	215	74	.	.	278	3	567
GA	164	28	.	256	.	3	448
KY	227	251	.	.	.	2	478
MN	.	.	400	258	246	3	904
TOTAL	770	770	1,000	770	770	.	4,080

For all but Minnesota, the data will be collected during a regularly scheduled case management visit, which may or may not include a re-assessment for the participating beneficiaries (i.e., the assessment may be conducted during a regularly scheduled monitoring visit). FASI assessments will only be administered to beneficiaries who consent to participate in the study. Assessors will be using the FASI items to measure four domains: 1) function (mobility, self-care), 2) instrumental activities of daily living (IADL), and 3) caregiver assistance needs.

Individuals will be selected based on having an expected annual or interim assessments occurring between June and December 2016. All clients falling into these parameters will be selected. The individuals will be identified by the state or the assessment agency that works with the state to conduct the current Medicaid waiver assessments. Each client will be sent a letter that describes the study and requests their participation. We will ask states to send the letter on our behalf: a letter that comes from a known state agency will obtain a better response than a letter from an unknown, out-of-state organization. We also will provide information to local assessment organizations to which people may turn with questions.

Following the mailing, assessors will schedule an appointment to conduct their regularly scheduled assessment. Clients will be asked if they wish to participate during that call. Those who do not wish to participate will be thanked and those who are willing to participate will be scheduled for their assessment. All participants will be given a consent form which will be explained by the assessor and signed by the participant or their legal guardian before beginning the assessment. Assessors will keep the signed consent forms in their locked offices.

As shown in Table 1 (see last row), with the exception of the subpopulation of people with TBI, a sample of 272 people would require participation from fewer than 4% in any population, and from fewer than 1% in several of the population groups. Thus, with the exception of people with TBI, we will easily be able to randomly select more than 272 clients per subpopulation, account for attrition, and obtain the desired 272 clients per subpopulation. We will need nearly one-third of all clients with TBI to participate, which may be difficult. We will work with the states to obtain a greater response rate, reach out especially to TBI advocacy groups, or recruit other states or organizations that can provide TBI clients for the field test.

2. Collecting Information

Some individuals may require interpreters (sign language or non-English spoken language) to be able to participate fully. Others may need or prefer that another person (family member, friend, trusted caregiver) be present during the interview. This information will be collected at the time of scheduling and will help to assure that scheduled interviews can be successfully completed. It also will help to assure the participants that their needs and preferences are being respected and, in so doing, will help to increase the response rate.

Assessments will be conducted by agencies already serving the participants, with the exception of Minnesota where a research agency will be conducting the assessments. All assessors will be trained by the Truven Health Analytics team in the use of the assessment items. On-line, mobile training modules are being developed by the George Washington University which is a subcontractor on this contract. GW is using adult education approaches to create four modules: one on each of the four domains in the FASI item set. Assessors will be able to access the training modules from their preferred locations (at home, in the agency, wherever they have internet access.) A PowerPoint copy of the four modules is attached here. All other instructions are embedded in the data collection, or FASI tool. Before being authorized to begin data collection, each assessor will have to complete and pass each of the on-line training modules.

Data will be collected through an assessment interview with the individual participant. Other people, such as knowledgeable family members or caregivers also may provide information during the course of the assessment interview, if requested by the client or if needed because the client is unable to provide information without assistance. Information from others will be provided only with the participant’s consent.

The assessment data will be collected using the assessor’s secure, professional laptop or tablet that is used in conducting their regular state assessments. An electronic PDF form is being developed by Truven Health Analytics for the assessors to use in the assessment process. The electronic PDFs will be blind and contain no PHI or PII; observation identification numbers will be generated as the forms are uploaded to Truven Health. Each assessor will also have a study ID and the combination of the state ID, the assessor ID, and the electronically-generated observation ID will provide a study number to each assessment, independent of any personally identifying information on the client. The information collected will be submitted electronically at the time of the assessment, or stored on the assessor’s secure device and submitted electronically at the end of the day, depending on the availability of a secure wireless signal and the assessors’ current methods for submitting their assessment data to the state. The data will be submitted to Truven where it will be reviewed for completeness and de-identification and then be transferred to GWU for review and construction of the analytic files. Truven Health and GWU will be reviewing the data as it is submitted for completeness.

Truven Health Analytics is conducting a one-time field test in 2016. The states may collect a subset of these items in 2017 as part of their demonstration of the FASI item use but this data will not be collected more frequently than annual data collection as the second potential data collection is conducted by the state in the following year. The exact sample the states may use is unknown (see section 5 below).

3. Maximizing response rate

Field test assessments will be conducted as visits to the individual’s home or other place of a person’s choosing. For example, an individual may prefer to be interviewed at a day program in which they participate in order to minimize disruption to his/her day.

States will be asked to send letters to the selected individuals, requesting their participation and informing them of the value of their participation to the state. Having contact with consumers through a known and trusted agency, rather than an unknown and out-of-state organization, will increase consumer confidence in the legitimacy of the request and willingness to participate.

Assessors will use their usual practices for scheduling assessments. Where an outside agency is conducting the assessment, follow-up calls by the assessors will be made with each respondent in advance of the assessment, reminding them of the date, time, and location of the appointment. This reminder will help to ensure that individuals and their caregivers (if expected to be present for the assessment) will be present on the scheduled day.

Scheduling of interviews will be managed to assure that adequate sample size is attained. Assessors will submit weekly counts of the number of clients contacted, scheduled, visited, and assessed to the Truven data collection manager. No PHI or PII will be transferred in this reporting.

4. Analysis of reliability and validity

Truven Health Analytics will collect data during a field test of the TEFT FASI Item Set to examine the validity, reliability, and completeness of responses, as well as the feasibility of implementation and data collection across a variety of Medicaid waiver programs. Data analyses will include examination of the psychometric properties of the TEFT FASI Item Set when used for different subpopulations. Field test data collection will begin June 2016.

The goal sample size will depend on the prevalence of the characteristics being measured in the population, but should be approximately 126 to 335 cases (Sim and Wright, 2005). This range of sample sizes will allow us to detect a kappa of 0.80—that is significantly different than a kappa of 0.60—with 95 percent confidence. A 0.60 kappa statistic is commonly interpreted as being the lower bound of what is considered to be “substantial” agreement. Subsample populations will be formed by combining populations across states. Table 4 is an example of how we might examine the kappas for FASI items overall and stratified by program type. These examples are based on the approach in the prior CARE item testing under the original OMB authorization. If an item has more than two levels, we will examine both the simple kappa, and a weighted kappa, which allows for the possibility that the “distances” between response levels may not be equal across all levels.

Table 4
Example table: IRR testing: Pain and continence at time of HCBS assessment, by population group

Item	Effective sample size	Kappa	Weighted kappa
Ability to walk 10 feet	.	.	.
ID/DD	.	.	.
Aged (>= 65)	.	.	.
Physically Disabled	.	.	.
TBI	.	.	.
Severe Mental Illness	.	.	.
Ability to walk 50 feet and make 2 turns	.	.	.
I/DD	.	.	.
Aged (>= 65)	.	.	.
Physically Disabled	.	.	.
TBI	.	.	.
Severe Mental Illness	.	.	.

After we have analyzed the field test data, we will prepare a Field Test Report on the results of reliability, validity, and other testing and its impact on TEFT FASI deliverables (e.g., training materials, the TEFT FASI Item Set) that will need to be revised for the second round of data collection. Both the draft and final Field Test Reports will be submitted to CMS within a timeframe still to be negotiated.

We will perform a variety of statistical tests to assess reliability and validity. The statistical tests will:

• Analyze response distribution.

• Compare responses across functional needs and caregiver assistance across each of the five samples (I/DD, Aged, PD, TBI, SMI).

• Examine the distribution of missing data by client characteristic by program type and geography. Uneven distribution of missing responses on an item either by client characteristic, program type, or geography, suggests that bias may be present. Table 5 shows an example of how we might examine response rates for FASI items by program type.

• Conduct Rasch, confirmatory factor analysis, and other internal consistency analyses to provide information on the construct validity of TEFT FASI items, their proposed scales, and their psychometric properties when used within/across study populations.

• Assess whether there are ceiling effects to be sure that the modified items cover the range of characteristics of clients.

• Evaluate key item-level components of FASI functional assessment, including response scale usage (i.e., the distribution of assistance needs on function items), an examination of the item difficulty hierarchy and how it compares to clinical expectations of item difficulty, and potential disability group differences.

• Perform analyses to assess inter-rater reliability. Inter-rater reliability will be assessed by comparing the information captured by two assessors, recording information independently, but who are both present at the same assessment. This will minimize the burden on respondents, but not requiring each respondent to go through two separate interviews.

Table 5

Impairments: Percent missing responses by program type

Item	Item Name	ID/DD (assessments n) percent missing responses	Aged (>=65) (assessments n) percent missing responses	Physically Disabled (assessments n) percent missing responses	Mental Health (assessments n) Percent missing responses	TBI (assessments n) percent missing responses
VA1A	Bladder and Bowel Management A1a. Bladder Incontinence	.	.	.	.	.
VA1B	A1b. Bowel Incontinence	.	.	.	.	.
VA2A	A2a. Bladder	.	.	.	.	.
VA2B	A2b. Bowel	.	.	.	.	.
VA3A	A3a. Bladder	.	.	.	.	.
VA3B	A3b. Bowel	.	.	.	.	.
VB1	Swallowing B1. Swallowing Disorder Signs and symptoms of possible swallowing disorder.	.	.	.	.	.
VB2	B2. Indicate the person’s usual ability to swallow.	.	.	.	.	.
VC1	Hearing, Vision, and Communication Comprehension C1. Understanding verbal content (With hearing aid or device if used)	.	.	.	.	.
VC2	C2. Expression of ideas and wants	.	.	.	.	.
VC3	C3. Ability to see in adequate light (with glasses or other visual appliances):	.	.	.	.	.
VC4	C4. Ability to hear (with hearing aid or hearing appliance if normally used):	.	.	.	.	.

5. Subsequent State Data Collection in FASI Round 2

As part of the states’ participation in the TEFT FASI grant component, they are required to demonstrate their use of the items. During 2016, each of the 6 FASI states will be developing an approach to demonstrate item use. They have no standard requirements to use all of the FASI items, or to use them for any specific purpose. Truven, as part of the technical assistance work, has identified potential ways the states may use the standardized items, and will be working with them to determine if they may use them for any of the following purposes:

• Develop person-centered service plans

• Determine eligibility for different state programs

• Monitor quality and measure program impact

• Report across multiple programs within a state and across states, especially rebalancing initiatives

• Update systems to reflect national measurement standards

• Create exchangeable data platforms

Over the course of 2016, each of the six states will determine how they wish to use the FASI items, if at all, in their program management activities. During this time, they also will determine which populations they may include, which items they may collect, how the items will be used in their state management or reporting systems, and the reasons for their decisions. None of this is known at this time and will not affect the Truven Health Analytics data collection efforts in 2016.

6. Contact Information

Barbara Gage, PhD Beth Jackson, PhD

FASI Co-lead Project Director, FASI Co-lead

George Washington University Truven Health Analytics

Email: [email protected] email: [email protected]

Phone: 617-610-1760 Phone: 508-520-1507