Supporting Statement – Part B
Collections of Information Employing Statistical Methods
1. Description of sample selection and universe
A. Potential respondent universe.
The target universe is current Medicare beneficiaries enrolled in managed care, living in the 50 states, the Virgin Islands, and Puerto Rico. Aged and disabled Medicare beneficiaries enrolled in an MA plan with more than 500 members are eligible. Institutionalized members are eligible.
B. Sample Sizes.
Target sample sizes.
The survey protocol is designed to achieve a 70 percent response rate for the baseline survey. A five year running average (Cohort 9 to Cohort 13) response rate for the baseline survey is 63 percent. The targeted response rate for the follow-up survey is 80 percent. The initial sample size and targeted baseline and follow-up response rates are designed to achieve an analytic sample size with adequate statistical power to detect significant variation between health plans on physical and mental health outcomes.
A post hoc analysis exploring the power and minimum detectable effect associated with the HOS focused on “net effects,” or the mean change in PCS or MCS score between baseline and follow-up for enrollees in each plan relative to the mean change of enrollees in all other plans. The results indicated the HOS has adequate power to detect “small” net plan effects, namely 0.2 of a standard deviation between means. Depending on the measure (PCS or MCS) and cohort, the power was greater than 0.93 to 0.99 for half of the plans and greater than 0.81 to 0.96 for 75% of the plans. The minimum detectable effects for PCS and MCS score, at 80% power and α=.05, ranged from 1.24 to 1.62. As expected, increasing the effective sample size reduced the minimum detectable effect in all cohorts.
(ii) Baseline Survey sampling.
Due to variations in health plan population size, three sampling approaches are used.
In MCOs with Medicare populations greater than or equal to 3,000, 1,200 members are randomly selected for the Baseline Survey. To reduce burden on survey respondents, members who were sampled for and returned a completed survey the previous year are excluded from sampling.
In MCOs with Medicare populations of 1,201 to 2,999, 1,200 members are randomly selected for the Baseline Survey. To ensure a sample size of 1,200, members who were sampled for and responded with a completed survey the previous year are not excluded from sampling.
c) In MCOs with populations of 500 to 1,200 members, all members who meet the eligible population criteria are included in the sample.
MCOs with populations less than 500 members are exempt from HOS reporting.
Follow-Up Survey sampling
Eligible members include all Baseline (two years prior) respondents for whom a valid physical component summary score or mental component summary score were calculated and who remain enrolled in the same MA plan.
C. Response rates.
A total survey response rate is calculated for each sample that is the total number of complete surveys divided by all eligible members of the sample. Eligible members of the sample include the entire random sample minus members assigned a survey disposition code of ineligible. In other words, invalid beneficiaries are removed from the denominator in the response rate calculations.
The total survey response rate is calculated as follows:
Number of Complete Surveys
Entire random sample – [Ineligible for any of the following reasons: Deceased + Not enrolled in MCO + Language barrier + Bad address and nonworking and unlisted phone number or person unknown at the dialed phone number]
Survey response rates from 1998 through 2010 (the latest completed year of data collection) are illustrated in Attachment 3. Over the course of time, the refinement of the Medicare HOS measure has resulted in changes to the definition of a “completed survey”. In general, for Cohorts 1-3 Baseline, a completed survey is defined as a survey that has at least 80% of the questions answered. However, beginning with Cohort 1 Follow Up and Cohort 4 Baseline, a completed survey is defined as a survey with calculatable physical or mental health summary scores.
Given that the definition of a completed survey has evolved over time, this table should not be utilized for response rate comparisons across the cohorts. (For the Medicare HOS-Modified, a completed survey is defined as a survey that, at a minimum, had responses to all six activities of daily living measures.)
For the baseline cohorts, the reporting units represent the individual plans sampled for the survey. However for the follow up cohorts, the reporting units have been adjusted to accommodate selected plan consolidations and service area reductions at the time of performance measurement reporting (which typically occurs in the year subsequent to the collection of the follow up data).
2. Data collection procedures
This section describes the procedures used for the national survey. It includes a summary of the questionnaire content, and a general discussion of the case-mix and risk-adjustment methodology. Finally, there is a discussion of rules for allowing proxy response.
A. Years 2013 through 2015 data collection procedures.
(i) Mail phase.
The mail component of the survey uses standardized questionnaires, cover letters and postcards. Members may receive up to four survey-related items by mail. A Pre-Notification Postcard is mailed to all members of the sample one week prior to the mailing of the First Questionnaire. Two weeks later, a Reminder Postcard is mailed to all members who have not returned the First Questionnaire. Two weeks after the Reminder Postcard is sent, a Replacement Questionnaire is mailed to all members who have not returned the First Questionnaire.
Because of the challenge in corresponding with a diverse population, it is necessary to support material in multiple languages. The three most common languages encountered with the Medicare population are English, Spanish and Chinese. Both the standard and modified instruments (Attachments 4 and 6) plus associated supporting correspondence (Attachments 5 and 7) are available in each of the three identified languages.
To ensure comparability of results, the survey vendors must follow strict adherence to the established survey protocol, specifications manual and the quality assurance plan.
The Pre-notification Postcard and Letter for First Questionnaire contain English and Spanish text. The Letter for First Questionnaire is double-sided. One side of the letter contains English text and the other side contains Spanish text. The Spanish text invites Spanish-speaking members to request a Spanish version of the questionnaire by calling the survey vendor’s toll-free customer support number.
Completed questionnaires can be manually key-entered into the computer, or optically scanned. To ensure quality for key-entered data, two separate data entry specialists must independently key answers for each questionnaire. A comparison of the separate entries identifies data entry errors that need adjudication by a supervisor.
(ii) Telephone phase.
The telephone component uses a standardized CATI script and specific design specification. The survey vendor is responsible for programming the scripts and specifications into its existing CATI software. To ensure the comparability of survey results, the survey vendor cannot change the wording of survey questions, the response categories or order of the questions.
Following the mail portion of the process, survey vendors identify members who did not respond to the mail survey and members who returned a blank or incomplete mail questionnaire (a questionnaire with less than 80 percent of questions complete). These members are eligible for telephone interviews.
The survey vendor attempts to contact non-respondents by telephone so that at least six telephone calls are attempted at different times of day, on different days of the week and in different weeks.
The survey vendor establishes training programs for all personnel involved in the telephone phase of the process. It establishes quality control procedures and monitors staff performance to ensure the integrity of the telephone interviewing process. The survey vendor monitors 10 percent of CATI interviews to evaluate the quality of interviewing and provides feedback and additional training as necessary.
B. Interview content.
There are two survey instruments used by the Medicare HOS. While both are similar, there are some distinct differences. These two instruments are: Medicare HOS 2.5 and Medicare HOS 2.0-Modified (HOS 2.0-M).
While the Medicare HOS 2.5 is the primary instrument used for data collection for both baseline and follow-up cohorts beginning in 2013, considerations were made for how the Medicare HOS 2.5 is administered to vulnerable Medicare beneficiaries at greatest risk for poor health outcomes. These beneficiaries are enrolled in Program of All-Inclusive Care for the Elderly (PACE) organizations. The main goal of the Medicare HOS 2.0-M is to assess the frailty of the population in these plans in order to adjust annual Medicare payments.
PACE is a capitated benefit authorized by the Balanced Budget Act of 1997 (BBA) that features a comprehensive service delivery system and integrated Medicare and Medicaid financing. The PACE model was developed to address the needs of long-term care clients, providers, and payers. For most participants, the comprehensive service package permits them to continue living at home while receiving services rather than be institutionalized. Capitated financing allows providers to deliver all services participants need, rather than be limited to those reimbursable under the Medicare and Medicaid fee-for-service systems.
The Medicare HOS 2.0-M survey is a modified version of the Medicare Health Outcomes Survey. The instrument assesses the physical and mental health functioning of the Program members to generate information for payment adjustment. It includes 12 physical and mental health status questions, one question about memory loss interfering with daily activities, and one question about urinary incontinence. If the participant received assistance completing the questionnaire, the respondent was asked why a proxy was needed, how the proxy assisted the participant and the staff position of the proxy.
The Medicare HOS 2.5 is comprised of three major components: 1) the VR-12, as the core component; 2) questions to gather information for case-mix and risk-adjustment; and 3) questions added by CMS to gather information required by either the 1997 Balanced Budget Act or from Section 4302 of the Affordable Care Act.
(i) The VR-12.
The VR-12 consists of items categorized into 8 concepts of health. Items are scored and summarized into a physical component summary (PCS) and a mental component summary (MCS).
The taxonomy underlying the construction of the VR-12 scales (concepts) and summary measures has three levels: 1) fourteen items; 2) eight scales that aggregate one or two items each; 3) two summary measures that aggregate the eight scales.
Scales covered in the VR-12 are:
Physical Functioning (PF) consists of two questions that ask respondents to indicate the extent to which their health limits their physical activities.
Role—Physical (RP) consists of two questions that assess whether respondents’ physical health limits them in the kind of work or other usual activities they perform both in terms of time and performance.
Bodily Pain (BP) consists of one question that determines the extent to which pain interferes with the respondent’s normal activities.
General Health (GH) consists of one question that asks respondents to rate their current health status overall.
Vitality (VT) consists of one question that asks respondents to rate their well being by indicating how frequently they experience energy.
Social Functioning (SF) consists of one question that asks respondents to indicate limitations in social functioning due specifically to health.
Role—Emotional (RE) consists of two questions that assess whether emotional problems have caused respondents to accomplish less in their work or other usual activities in terms of time and performance.
Mental Health (MH) consists of two questions that ask respondents how frequently they experience feelings representing the four major mental health dimensions: anxiety; depression; loss of behavioral / emotional control; and psychological well being.
In addition, a two-item measure of Change in Health asks respondents to rate their general physical health and emotional problems now compared to their health one year ago.
(ii) Questions to gather information for case-mix and risk-adjustment.
Case-mix and risk-adjustment are essential for meaningful and valid plan-to-plan comparison of health outcomes. By including variables to control for differences in demographics and socioeconomic characteristics, chronic medical conditions, and Medicare HOS study design variables, regression techniques can be used to adjust the scales and summary measures
Demographics include questions on beneficiary age, gender, race, education, marital status, and annual household income.
Thirteen chronic medical conditions are included in the questionnaire. These conditions are: hypertension; angina pectoris or coronary artery disease; congestive heart failure; myocardial infarction or heart attack; other heart conditions, such as heart valve defects or arrhythmias; stroke; emphysema, asthma, or Chronic Obstructive Pulmonary Disease (COPD); inflammatory bowel disease, including Crohn’s disease and ulcerative colitis; arthritis of the hip or knee; arthritis of the hand or wrist; sciatica; diabetes, hyperglycemia, or glycouria; and any cancer (other than skin cancer).
Study design variables include who completed the survey, the mode of survey administration, CMS region, and the survey vendor.
(iii) Questions to gather comparative information on managed care plans and to measure health disparities.
Six Activities of Daily Living (ADLs) are included to determine self-reported difficulty with performance of daily tasks. ADLs include bathing, dressing, eating, getting in or out of chairs, walking, and using the toilet.
Four depression screening questions include levels of depression that encompass: feeling sad / blue for two weeks in the past year; depression for much of the past year; or depression for two years or more in life.
Three Healthy Days questions, which encompass the number of days in the past thirty days that physical health was not good, mental health was not good, and activities were limited due to poor physical or mental health.
HEDIS® Effectiveness of Care includes four measures on the management of urinary incontinence, physical activity in older adults, fall risk management in older adults, and osteoporosis testing in older women.
Height and weight for the calculation of body mass index (BMI)
Section 4302 of the Affordable Care Act standardized race, ethnicity, sex primary language and disability status questions.
The new items will not be used to calculate the Medicare Advantage Plan Ratings.
C. Adjustment methodologies.
(i) Case-mix adjustment.
The analytic sample for the Medicare HOS is limited to those seniors (age 65 and over) with calculatable physical component summary and mental component summary scores. Regression techniques are used to case mix adjust these measures for each beneficiary.
Models used to adjust the scales and summary measures included variables to control for differences in demographic and socioeconomic characteristics, chronic medical conditions, and study design variables. A series of three different models are used for each measure since all beneficiaries do not have completed data for all of the covariates. Only one model, the most comprehensive model possible, was used for each beneficiary and an adjusted score was calculated for each. Since the Chinese language version of the survey does not contain the marital status and income questions, only model three is used for adjustment of the scores for these respondents.
Model One: If the beneficiary has completed data for all of the covariates, then the adjusted scores are calculated using Model One which contains all variables.
Model Two: If the beneficiary has completed data for all covariates but annual household income, Model Two is used.
Model Three: If the beneficiary does not have enough completed data for Model One or Two, then Model Three is used. The variables included in Model Three, which are available for all beneficiaries sampled, are age, gender, race, mode of survey administration, CMS region, and survey vendor.
See (Attachment 9) for a detailed breakdown of covariates used in the Case Mix Adjustment of PCS and MCS measures for the 2010 Cohort 13 Baseline Data User’s Guide. This methodology will remain in practice, although minor refinements could be explored to maintain the accuracy of the measures.
(ii) Risk Adjustment.
In addition to the VR-12 Health Survey, the Medicare HOS questionnaire collects information for purposes of a standardized plan-to-plan risk adjustment. Additional items include variables for morbid conditions, activities of daily living, and socio-demographic characteristics. Risk adjustment accounts for patient-associated factors before comparing outcomes across different health plans or populations. Therefore, risk adjustment enables a fair comparison between health plans.
For each respondent, a change in functional health score will be estimated by subtracting the first (baseline) questionnaire score from the second (follow up) score. Taking into account an expected decline in health and risk-adjustment factors, change in physical and mental health status can be: better, the same as, or worse than expected.
Because outcome is defined as a change score, each respondent serves as his or her own ‘control’. In addition plan-to-plan risk adjustments are based on morbid conditions at baseline, income, household size, social support, education, race and gender. Results are aggregated across respondents for each health care plan.
Proxy rules.
While sampled members are encouraged to respond directly to the mail or telephone survey, not all elderly or disabled respondents are able to do so. In such cases, proxy responses are acceptable. The survey instrument instructs members who are unable to complete the survey to have a family member or other proxy complete the survey for them.
If a proxy completed the baseline survey, every effort is made to have the same proxy or, if possible, the sampled member, complete the follow-up survey. Having the same proxy complete both questionnaires minimizes bias. If the same proxy or the sampled member is unable to complete the follow-up survey, another proxy may be used to collect responses.
3. Methods to Maximize Response Rates
Medicare HOS is sampling a heterogeneous population that presents a unique challenge for maximizing response rates. The household survey will be approaching two groups--aged and disabled Medicare beneficiaries‑‑who have characteristics that often lead to refusals on surveys. Increasing age, poor health or poor health of a family member are prevalent reasons for refusal. On the other hand, older persons are the least mobile segment of the population and thus less likely to be lost due to failure to locate. Little is known about the disabled population in terms of survey response rates. It is anticipated that they will be most similar to the oldest old population because of their ill health.
Because this is a longitudinal survey it is essential that we maximize the response rates. In order to do so, CMS employs an extensive annual outreach effort. This includes the notification of government entities (CMS regional offices and hotline, carriers and fiscal intermediaries, and Social Security Offices). These efforts are undertaken to increase the likelihood that respondents would answer the Medicare HOS questions and remain in the survey panel by: 1) informing authoritative sources to whom survey participants are likely to turn if they suspect the legitimacy of the Medicare HOS; 2) giving interviewers resources to which they can refer to reassure respondents of the legitimacy/importance of the survey; and 3) generally making information about Medicare HOS available through senior centers, other networks to which survey participants are likely to belong and through the CMS website.
In addition to the outreach efforts, the following efforts remain in place to maintain a sense of validity and relevance among the survey participants.
An advance letter is sent to potential respondents from CMS
Interviewer training emphasizes the difficulties in communicating with the older population and ways to overcome these difficulties.
Proxy respondents are sought for survey participants unable to participate for themselves.
Mail non-respondents are re-contacted by telephone.
A toll-free number is available to answer respondent's questions.
An E-mail address and website are available at CMS to answer respondent’s questions.
A shortened version of the survey instrument is administered to the frailest portion of the sample.
“Take my name off your list” refusals are flagged for exclusion from all future Medicare HOS activity.
The following disclosure statement is included with our survey instrument:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0701. The time required to complete this information collection is estimated to average 20 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.
4. Test of Procedures or Methods to be Undertaken
At this time there are no plans to conduct field testing of the currently established procedures or methods. From time to time various parts of the questionnaire are modified or augmented to reflect changes to the Medicare program, capture information on emerging areas of interest, reduce unnecessary burden or to improve the quality of the data. If field testing becomes desirable in the future, it will be submitted for approval separately or in combination with the next main collection of information.
5. People responsible for statistical aspects of design.
Lewis Kazis, ScD William Rogers, PhD Judy Ng, PhD
Boston University 19 Rebecca Road 57 College Road, West
715
Albany Street (T-3W) Newton, MA 02465 Princeton, NJ 08540
Boston,
MA 02118 Phone: (617) 332-1814 Phone: (202) 955-5146
Phone: (617) 414-1434
National Committee for Quality Assurance, of Washington, DC conducts the Medicare Health Outcomes Survey under contract to CMS.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement – Part B |
| Author | CMS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |