Supporting Statement - Part B

Supporting Statement – Part B for Statistical Methods
1. Respondent Universe and Sampling Methods
The target population for this survey is all nursing homes in the 50 States and the District
of Columbia. The total number of nursing homes in the population is approximately
15,700. There will be two waves of data collection approximately 12 months apart. For
each a wave, it is planned to collect data from a nationally representative random sample of
2,100 nursing homes from the population.
For the selection of the sample at each wave, the population of nursing homes will be
stratified by size and CMS region. Three size groups will be created: 1) nursing homes
with 50 or fewer beds will be classified as small; 2) nursing homes with between 50 and
149 beds will be classified as medium; and 3) nursing homes with 150 or more beds will
be classified as large. The sample will be allocated to each stratum in proportion to the
number of nursing homes in that stratum. Exhibit 5 shows the distribution of the
population and sample by strata. The states that are included in each CMS region are
shown in the table
Exhibit 5: Distribution of Population and Sample of Nursing Homes by Strata
CMS Region
CT, MA, ME,
NH, RI, VT(1)
NJ, NY, PR, VI
(2)
DE, DC, MD,
PA, VA, WV
(3)
AL, FL, GA,
KY, MS, NC,
SC, TN (4)
IL, IN, MI, OH,
WI (5)
AR, LA, NM,
OK, TX (6)
IA, KS, MO, NE
(7)
CO, MT, ND
SD, UT, WY (8)
AZ, CA, HI (9)
AK, ID, OR,
WA (10)
Total

Small
Population Sample

Bed Size
Medium
Population Sample

Large
Population Sample

Total
Population Sample

138

18

677

91

162

22

977

124

81

11

430

57

494

66

1,005

128

193

26

897

120

318

42

1,408

179

246

33

2,011

269

428

57

2,685

342

556

74

2,332

312

565

76

3,453

440

141

19

1,689

226

257

34

2,087

266

334

45

1,059

142

130

17

1,523

195

173
255

23

399

53

57

8

629

80

34

1,007

135

211

28

1,473

187

95

13

331

44

34

5

460

59

2,212

296

10,832

1,449

2,656

355

15,700

2,100

8

We expect to get a response rate of 65% to 70% the data collection effort. Therefore, a
larger sample will be selected based on the expected response rate at the time of the
survey.
2. Information Collection Procedures
In each stratum, we propose to select an equal probability systematic sample of nursing
homes after sorting the list of nursing homes by for profit, non-profit, government and
LLC to provide proportional representation to the various subgroups.
For producing population-based estimates, each nursing home in the sample that is a
respondent to the survey will be assigned a sampling weight. This weight will combine a
base sampling weight and an adjustment for nonresponse. The base weight is the inverse of
the probability of selection of the responding nursing home. These weights will be used
for all statistical analyses of the data obtained from the survey.
It is of interest to compare the difference between estimates between to two waves of data
collection in addition to estimating various population parameters at each wave of data
collection. With a sample of 2,100 nursing homes at each time period, we will be able to
detect a difference of 4.4 percentage points in population percentages of a characteristic of
interest with 80% power when we do a two-sided statistical test at 5% level of
significance. For this computation, it is assumed that the population percentage at the first
wave is 50%. If we are estimating population percentages at each wave, then with a sample
of 2,100, the margin of error for the estimated percentage will be plus or minus 2.2
percentage points at 95% confidence level. Because the sample is a stratified sample, the
precision could be higher than give above and depending on the homogeneity of the
population within a stratum.
No specialized sampling procedures are required for the selection of nursing homes.
3. Methods to Maximize Response Rate
Plan for Distribution of QAPI Survey Materials
We have developed a multi-faceted “marketing” approach to the survey, which includes
utilizing CMS partners, the CMS listserv, the QIES mailboxes and hard copy materials.
As a first step, CMS will ask their sponsor/partner groups (i.e., AHCA, LeadingAge,
AMDA, etc.) to advertise the forthcoming QAPI survey via their newsletters, websites, etc.
Advertising can commence any time after December 1, 2011 when it is expected that the
OMB package will be well into the clearance process and the 60-day Federal Register
notice is posted. Partners will be asked to continue to advertise the survey even as the web
version is activated.

9

After OMB clearance is received and the web survey has been programmed, CMS will do
a listserv “blast” to announce the survey to all 15,700 nursing home providers. This
listserv blast will coincide with the mailing of the hard copy “invitation” letter to the 4,200
sampled providers. In addition to the listserv blast and the hard copy mailed version of the
invitation letter, CMS will post the invitation letter in the QIES mailboxes of the 4,200
sampled providers who are being asked to participate in the QAPI survey.
The hard copy “invitation” letter from CMS to sampled nursing home providers will be
mailed 2 days prior to activation of the internet version of the QAPI survey. The
questionnaire information sheet with sponsor logos will be included with the CMS
invitation letter.
A hard copy “reminder” post card (Appendix D) will be sent to non-respondents
approximately one week after the initiation of the web survey. Hard copy mail surveys
will be sent to all internet non-responders beginning approximately two weeks after the
activation of the web survey. The hard copy surveys will be sent in official CMS
envelopes, and the CMS “invitation” letter and information sheet will also be included in
the hard copy mail packets. The internet site will remain open/available throughout the
entire data collection period in case sampled providers decide to complete the survey
electronically rather than via hard copy.
A second hard copy mail survey (including invitation letter and information sheet) will be
mailed to all sampled providers who have not responded to the questionnaire either via
internet or hard copy. Approximately three weeks after the second hard copy survey is
mailed to non-respondents, telephone reminders will begin. Non-responders to the web
and hard copy surveys will be the target for the telephone reminders. Up to 12 attempts
will be made to reach the Administrator of non-responding providers.
Non-Response Bias Analysis
Non-response bias is a function of the non-response rate and the difference between the
responding and non-responding ALFs on the variables of interest. This is typically
represented by the following formula:
(̅ )

( )(̅

̅ )

where:
Bias ( ̅ ) = the nonresponse bias of the unadjusted respondent mean;
̅ = the unadjusted mean of the respondents in a sample of the target population;
10

̅ = the mean of the respondents in the target population;
̅ = the mean of the non-respondents in the target population;
M = the number of non-respondents in the target population; and
N = the total number in the target population
We will track response rates by region and size of nursing homes to assess whether nursing
homes in certain regions or in certain bed size groups have more difficulty in completing
the questionnaire. We will also examine possible reasons for nonresponse by comparing
the frame characteristics of respondents and non-respondents to assess whether there is any
difference between the two groups with regard to frame characteristics. This may provide
some information on the formation of weighting classes for adjusting the weights of
respondents for nonresponse. We also plan to compare the characteristics of those who are
early respondents and those responding later after reminders to identify whether late
responders are more similar to non-responders and then will use the formula above to
assess the nonresponse bias.
4. Tests of Procedures
The questionnaire was developed over a period of nearly six months, and recently
completed the pre-testing phase. There were four objectives of the cognitive interview
pretesting:
1. Identify problems with the Nursing Home Quality Improvement Questionnaire,
invitation letter from CMS, and data collection information sheet;
2. Determine reasons and collect suggestions for any identified problems
3. Determine the length of time to complete the questionnaire
4. Gain insights from respondents’ reaction to the overall questionnaire data
collection plan
A total of nine cognitive interviews were conducted with nursing facilities in Colorado.
Participants were located in a geographically preferable region, to avoid extensive travel.
The nine facilities included in the pilot test were of varying size, profit status, and chain
affiliation.
A hard copy of the CMS invitation letter, information sheet and the questionnaire were
mailed to the administrator at the pretest facilities, along with an introductory letter and
instructions for completing the questionnaire during the pretesting phase. The instructions
included explicit instructions for recording the amount of time to complete the
questionnaire.

11

The invitation letter, information sheet, and each section of the questionnaire was
discussed during the pretest interview. The respondents were asked to provide their
overall reaction to the questions when first reading them, as well as specific issues with the
individual questions. The cognitive interviewer followed-up on respondent comments,
administered probes about the questions, and worked with the respondent to assess
question clarity, response wording or task difficulty, and any other aspects of the
questionnaire. Staff from CFMC conducted the interviews onsite in the pretest facilities,
with staff from Abt Associates joining by telephone for some of the interviews.
Respondents were required to read and sign a consent form at the start of the interview
session. The consent document included a description of the project in general, what the
respondent was going to be asked to do, any risks or benefits to the respondent as a result
of participating in the pretest, and the respondent’s rights as research subjects. The consent
document also included language about confidentiality of the information being collected.
The purpose of the questionnaire pretesting was conveyed to the cognitive testing
participants at the beginning of the interview. This discussion included the following
points:
1. The questionnaire was designed to be completed by nursing home
administrators and designees. Some sections of the questionnaire may need
assistance from other staff within the nursing home.
2. The objective of the questionnaire is to collect information on current quality
improvement activities, challenges, and areas where TA will benefit nursing
homes.
3. The questionnaire is intended to be completely self-explanatory, although
contact information for any needed assistance or questions is readily available.
4. The questionnaire is intended to be appropriate for all types and sizes of nursing
facilities.
5. Individuals Consulted on Statistical Aspects of Design
Individuals consulted on the statistical aspects of the design for this questionnaire
and data collection include:
Kadaba P. Srinath, Ph.D.
Abt Associates Inc.
4550 Montgomery Avenue
Bethesda, MD 20814
(301) 634-1836
Alan J. White, Ph.D.
Abt Associates Inc.
12

5001 S. Miami Boulevard, Suite 210
Durham, NC 27703
(919) 294-7719
The data will be collected by Abt SRBI, Inc. Jamie Munjack will serve as the
Survey Director. The data will be analyzed at Abt Associates by Alan White, Ph.D.
and Brant Morefield, Ph.D. Laura Palmer from CFMC will also participate in the
data analysis. Below is the contact information for these staff:
Jamie Munjack
Abt SRBI, Inc.
185 Monmouth Pkwy, Suite B4
West Long Branch, NJ 07764
(732) 403-2502
Brant Morefield, Ph.D.
Abt Associates Inc.
5001 S. Miami Boulevard, Suite 210
Durham, NC 27703
(919) 294-7734
Laura Palmer
Director, Quality Improvement and Analytic Services
Colorado Foundation for Medical Care
23 Inverness Way East, Suite 100
Englewood, CO 80112
(303) 784-5749

13