Attachment 3 - Summary of Differences 060309

Changes to Previous Documents based on the 03/19/09 OMB conference call

General issue #1 (Research Methodology):

1. Clarify the timing of the follow-up for agencies that do not respond to the initial request to complete the Web-based survey

2. Provide a summary/outline of the Web-based survey in the initial request to complete the Web-based survey rather than the survey itself

3. Provide the hard copy of the Web-based survey as an item in one of the follow-up reminders (rather than with the initial invitation) to the agencies that do not complete the survey within the initial timeframe

Responses to General issue #1 (Research Methodology):

A. The re-contact schedule has been changed as well as the materials provided to the agencies at each time point. The new schedule is as follows:

1. Initial notification to home health agencies participating in the P4P Demonstration

Materials/Method:

1. Notification letter addressed by name to the administrator or Director of Nursing for participating home health agencies from the CMS P4P Demonstration Evaluation Project Officer (William Buczko, PhD) inviting their participation in completing the Web-based survey

2. Information sheet that 1) outlines items that will be included in the survey that can be used as a navigation aid while completing the Web-based survey, 2) provides the URL address for accessing the Web-based survey,
   3) reiterates the security protocols in place to ensure that the information that is provided will remain secure, 4) includes the date for completing the Web-based survey (the work day nearest to two weeks and three days from the date of the mailing), and 5) provides an abbreviated summary of expected follow-up contacts if the Web-based survey is not completed by the specified date.

1. First follow-up (within two working days after specified date in the initial notification)

Materials/Method:

1. Email sent to administrator or Director of Nursing with a colorful, animated reminder message about completing the Web-based survey and the new date to complete (one week from the date of the email).

1. Second follow-up (within two working days after date specified in first follow-up)

Materials/Method:

1. Letter addressed by name to administrator or Director of Nursing from the CMS contractor (University of Colorado, Denver (Anschutz Medical Center)) with a request to complete the Web-based survey, along with statistics on how many have already completed the survey and the challenge to “be counted”, and new date to complete of one week and three days.

1. Third follow-up (within two working days after the date specified in the second follow-up)

Materials/Method:

1. Letter addressed by name to administrator or Director of Nursing from the CMS contractor (University of Colorado, Denver (Anschutz Medical Center)) with a hard copy of the survey instrument. The letter will explain the agency’s option either to complete the hard copy of the survey, mail it back to the CMS contractor, and have the contractor enter their data or use the hard copy as a guide when they complete the Web-based survey themselves. The new date to complete will be one week and three days from the date of the mailing.

1. Fourth follow-up (within two working days after the date specified in the third follow-up)

Materials/Method:

1. Personal phone call to administrator or Director of Nursing from the CMS contractor (University of Colorado, Denver (Anschutz Medical Center)). The phone call will follow a script where the goal is to gather the data needed to complete the Web-based survey.

The entire period from initial contact to fourth follow-up (if needed) is approximately two calendar months.

2. As detailed in the previous section, HHAs will not receive a hard copy of the survey in the initial contact to announce the Web-based survey. The cover memo that contains directions about the Web-based survey contains an outline listing of information elements contained in the survey.

2. This is addressed in the third follow-up vs. the initial contact as in the previous version of the proposed follow-up contacts.

General issue #2 (Instrument Item Specific):

1. Remove the references to “Treatment” and “Control” groups from cover page information

2. Clarify/Specify the meaning of the word “Change” in item #4

3. Item #5 is an example of an item that presents too large a cognitive burden for the user (same comment about item #17)

4. Provide options for user for items 6a and 7a

5. Randomize (preferable) the order of the long lists of alternative for “check all that apply” items

6. Consider repositioning item 12

7. Items 14 – 17 appear to be “opinion” questions. How are you going to use these data? Are there other data available to gather this information?

8. The total number of items (<20) does not seem to be problematic, even if some of the individual items are probably too cognitively burdensome.

Response to General issue #2 (Instrument Item Specific):

1. The words “Treatment” and “Control” no longer appear on the Web surveys.

2. The original item #4 was split into two items.

3. This item was redesigned to simply the options.

4. The open-ended portion of this item was replaced with a restricted set of options.

5. The list of items was reviewed to establish commonality among items. Specific cognitive burden questions addressed the efficacy of randomizing and the burden of completing the items. After reviewing the results of this cognitive burden testing (see results described in section “h”) and recognizing the limitations of the software used to create the Web-based survey, the decision was made to not randomize the short list of items for the “Mark all that apply” questions.

6. This item was redesigned to provide better clarity to the difference between “local actions” and “corporate actions”.

7. An introduction to this group of questions was created to emphasize that these were opinion questions and that the data would be corroborated using other secondary data available through CMS.

8. We conducted cognitive burden testing as part of the revision to the previous survey instruments proposed for this study. Cognitive burden testing mirrored common practice as described in the scientific literature. The cognitive burden testing used the actual instrument as delivered via the Web site. Three senior clinicians who all had numerous years in leadership positions home health agencies participated in the cognitive burden testing. Each of the three senior clinicians also reviewed the revised survey items prior to completing the Web-based version of the instrument and submitting to cognitive burden testing on that instrument. Twenty-five specific cognitive probes were used throughout the interview/testing process, such as “Please think ‘aloud’ as you answer this question. Please tell me how you chose your answer. What did you have to think about? Do the column headings for the matrix make sense to you? Why/Why not?” The specific responses are document in the PRA package. Survey items and Web-based format were revised based on the responses to these cognitive probes. Based on the comments made during the cognitive burden testing, 3 changes to the cover memo and 11 changes to the Web based survey were made.

The alternatives for items with lists, e.g., items #8 – 11 were reviewed during the cognitive burden testing to evaluate any “fatigue” issue in selecting “all that apply” on these items. Each participant was adamant that there was no fatigue and all participants chose options from the beginning, middle, and end of the lists. The fixed order of the items grouping, e.g., grouping communications items and then business related items, were identified as easing the cognitive burden when choosing answers to these items.

Summary of Changes (06/03/09) Page 4 of 4