Supporting Statement – Part B
COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
Medicare.gov / CMS.hhs.gov Websites
As in the previous studies, this project involves surveying (intercepting) Website visitors as they are visiting the sites. Visitors to these sites will be randomly selected and then asked to respond to questions about the sites’ navigability, content, interactivity, performance, available publications, and privacy policy. Using the same method, the previous studies provided benchmark measurements on each of the above Websites’ attributes. The results of the previous studies identified where Website improvements should be made in order to have the largest impact on visitor experience. Following the initial study, CMS conducted qualitative research with representative visitors to identify specific improvements to be made. The follow-up study proposed in this submission will measure the effect of these Website improvements on enhancing visitor experiences.
The potential respondent universe will be the universe of visitors perusing selected pages of the Websites during a specified time period. This universe will be sampled on a probabilistic basis. Based on historic visitor counts, an estimated sample size designed to achieve the precision requirements of the study will be determined. Using this expected sample size along with a pre-determined “window” for data collection, a probability-based sample of visitors will be randomly selected via JavaScript from the specified web pages. Survey invitations will be set to trigger for visitors who had visited at least four different pages during their visit. Based on the initial study, the sampling rate is expected to be approximately 20 percent. That is, one in five visitors using a JavaScript enabled browser will receive the survey invitation. For non-JavaScript enable browsers, a fixed link invitation will be visible.
We are going to try to increase the response rate from the last survey administration. Based on the unique nature of this study, potential respondents are approached only once. If they refuse, there will be no attempt made to convert refusals. Specifically, no “reminders” are sent to eligible participants, and there are no “call backs” as might be the case in other surveys. Also, due to the type of study, no advance notice of the survey is possible. Thus, the expected 30 percent response rate is based only on those initially consenting to respond to the survey.
2. For the purposes of this study, the sample design is simple random sampling. There is no stratification involved. The estimation procedures are designed to produce simple, standard descriptive statistics consistent with a sampling scheme based on simple random sampling.
As with the prior study, we estimate the number of completed interviews for this study to be approximately 7,000 visitors. Accordingly, we expect estimates produced from this survey to have very small levels of sampling error and that small differences (one point or less) will be able to be detected between rounds of the survey.
We expect to be able to measure differences in index scores (e.g. average score for questions about site characteristics such as “navigability” or “content” based on a scale of 1 to 100) between two rounds of the survey for differences as small as 0.90 points (i.e. to within less than one point between rounds) using a two-tailed test at the .05 level of significance.
3. This survey is very different from other traditional surveys by its very nature. One major difference is that no attempt will be made to convert initial nonrespondents. Traditional surveys often have extensive callback efforts and/or reminder letters for converting initial refusals. However, due to the nature of intercepting Website visitors in a probabilistic manner, the initial attempt to induce response is the only contact ever made with the potential respondent. Once the respondent refuses the initial survey invitation, no other contact is made. After all, we do not want to unduly interfere with the real reason the sampled visitor is visiting the Website in the first place. In surveys of this type, the data are used to point to specific problems (or dissatisfaction) visitors have when using the either site. We think that 30% response rate is good for this type of survey.
4. No tests of procedures or methods are being proposed for this study. This study will use methods that have been demonstrated to be an effective means of surveying CMS.hhs.gov Website and the Medicare.gov Website. Please note that the Medicare.gov Website was tested in 2002, 2003and 2004 using this exact same methodology. Since the 2001 Medicare.gov research was the first time this type of web intercept survey was used to test a CMS website, a pretest of the survey methodology was conducted. Pretest results were used to refine the intercept survey prior to the 2002 Medicare.gov Website testing. The same survey was used again for Medicare.gov in 2003.
5. Please contact either of the following CMS contacts regarding the statistical and methodological aspects of the design or for agency information:
Nyna Williams, Ph.D.
Center for Medicare & Medicaid Services
CMS/OEA
7500 Security Blvd. S1-15-11
Baltimore, MD 21244-1850
(410) 786-2097
Fax: (410) 786- 2097
internet: [email protected]
Reneé Clarke
Centers for Medicare & Medicaid Services
CMS/OEA
7500 Security Blvd. S1-13-05
Baltimore, MD 21244-1850
(410) 786-0006
Fax: (410) 786- 0006
internet: [email protected]
APPENDIX A.
Thank you for visiting the
CMS.hhs.gov / Medicare.gov Website
During this period of expanding program services, the Centers for Medicare & Medicaid Services (CMS), the Federal agency that administers the Medicare program is conducting this online survey to help us understand what information you need about Medicare, where you go to find the information, and if you feel we have been effective in getting you the information you need to make informed decisions about your Medicare benefits and your medical care.
As part of our commitment to enhancing this website to meet your needs, we are asking you to take a few minutes to answer some questions. All information you provide will be held in confidence by CMS and is protected by the Privacy Act. You do not have to participate in this survey. Your help is voluntary and your decision to participate or not to participate will not affect your Medicare benefits in any way. This is a legitimate research study sponsored by the Centers for Medicare & Medicaid Services (CMS), which is part of the United States Department of Health & Human Services.
Note:
If you are currently in the middle of searching for information please minimize this window until you have completed your search.
If you have already completed this survey, please close this window.
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this Medicare.gov information collection is 0938-0900. The time required to complete this information collection is estimated to average less than 5 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, Attn: Reports Clearance Officer, 7500 Security Boulevard, Baltimore, Maryland 21244-1850.
Appendix B.
Survey Tool for Medicare.gov / CMS.hhs.gov
Thank you for participating in our Website assessment. Your feedback directly influences the enhancements we are making to the site.
Survey, a national independent research firm, conducts this study. Your individual responses are confidential.
Q1. How often do you visit the Medicare Website or CMS Website?
(Please select one answer.)
1 This is my first time
2 Less than once a month
3 About once or twice a month
4 About once a week
5 A few times a week
6 Once a day
7 A few times a day
{IF Q1=2-7 ASK Q2 OTHERWISE GO TO INTRO BEFORE Q17B}
Q2. How long have you been using the Medicare Website or CMS Website?
(Please select one answer.)
1 Less than one month
2 One month to six months
3 Six months to a year
4 One year or more
Q17B. Which best describes you?
(Please select one answer.)
1 A Medicare recipient (currently or shortly)
2 A relative or friend of a Medicare recipient (currently or shortly)
3 Social worker, nurse, health professional
4 Employee of an insurer, HMO or managed care organization
5 Researcher, librarian, student, counselor, employer, attorney
6 A member of a group representing Medicare beneficiaries
7 A Medicare Customer Service Representative (CSR)
8 Another member of the Medicare staff
9 A member of a Quality Improvement Organization (QIO) or Peer Review Organization (PRO)
10 Other (SPECIFY)
{IF Q17B=7 OR 9 THANK AND TERMINATE, IF Q1=1 CONTINUE TO INTO Q17C, OTHERWISE CONTINUE TO Q3}
Q3. Why are you visiting the Medicare or CMS Website today?
(Please select all that apply.)
1 To find information on Medicare and what Medicare covers
2 To find information on Medicare eligibility and enrollment
3 To find information on nursing homes and home health agencies
4 To find information on a replacement Medicare card
5 To find information on supplemental health insurance (Medigap Policies)
6 To find information on physicians in your area
7 To find information about prescription drug coverage
8 To compare Medicare health plans
9 Following a link from another Website
10 Other (SPECIFY)
{IF Q3=9 ASK Q3A, OTHERWISE GO TO INTRO TO Q5A}
Q3a. What site were you at when you clicked the link that brought you to the Medicare Website or CMS Website?
{PROGRAMMER NOTE: Q4 DELETED BUT HISTORICAL NUMBERING RETAINED}
We'd like you to read a few statements about your OVERALL EXPERIENCE at the Medicare Website or CMS Website, and tell us how much you agree with each statement.
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID Q5A-Q5C}
Q5A. My experiences at the Medicare Website or CMS Website are good.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q5B. The Medicare Website or CMS Website meets all of my expectations.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q5C. The Medicare Website or CMS Website is as good as most other Websites that I visit.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How likely is it that you will come back to the Medicare Website or CMS Website or recommend it to a friend?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID Q6A-Q6D}
Q6A. I will return to the Medicare Website or CMS Website.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q6B. I will visit the Medicare Website or CMS Website on a regular basis.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q6C. I would recommend the Medicare Website or CMS Website to a friend.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q6D. I will add the Medicare Website or CMS Website to my bookmarks or favorite places.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Please rate the overall content or material on the Medicare Website or CMS Website.
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q7A-Q7F}
Q7A. The information is up to date.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q7B. I understand the information that I find here.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q7C. I trust the information I find here.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q7D. The information has the amount of detail that I need.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q7E. There is information on a wide number of topics.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q7F. The information is useful.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How would you rate the performance of the Medicare Website or CMS Website?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q8A-Q8D}
Q8A. Pages are displayed quickly.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q8B. When I search for something I get results quickly.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q8C. When I click a link it takes me to the right place.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q8D. The site is always available when I visit.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How would you rate your ability to interact with the Medicare Website or CMS Website?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q9A-Q9E}
Q9A. There are helpful links to other sites.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q9B. It is easy to subscribe to the mailing lists.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q9C. It is easy to order and download publications.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q9D. It is easy for me to give suggestions or make comments about the site.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q9E. It is easy to use the tools that let me find information or compare plans and facilities (For example, Medicare Personal Plan Finder, and Participating Physician Directory.)
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How would you rate the appearance of the Medicare Website or CMS Website?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q10A-Q10E}
Q10A. The pages have the right amount of color.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q10B. It is easy to read the text.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q10C. The page design is clear.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q10D. The pages on the site look good.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q10E. The pages have enough graphics.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How would you rate getting around the Medicare Website or CMS Website?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q11A-Q11D}
Q11A. It is easy to find the information I want.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q11B. The Search feature gives me links to pages I need.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q11C. The information is clearly presented.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q11D. The links are clear and easy to understand.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
How would you rate your comfort level with the Medicare Website’s or CMS Website’s privacy and security?
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q12A-Q12D}
Q12A. I believe that they honor their privacy policy all the time.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q12B. They explain their privacy policy clearly.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q12C. They don't ask for too much personal information.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q12D. The information they collect about me is only used for specific purposes that I request.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q13. Have you looked at the Compare tools feature at the Medicare Website or CMS Website?
1 Yes, during today's visit
2 Yes, within the past month
3 Yes, but more than one month ago
4 No
5 Don’t know
{IF Q13=1-3 ASK INTRO TO Q14, OTHERWISE CONTINUE TO Q17A}
How would you rate the Compare tools feature in terms of the following:
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q14A-Q14E}
Q14A. It was easy to search for the information I wanted.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q14B. The information I got back was the information I was looking for.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q14C. The information was displayed clearly.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q14D. The information was easy to understand.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q14E. The information was valuable.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q15. Did you look at the quality or cost information on the compare tools feature?
1 Yes
2 No
3 Don’t know
{IF Q15=1 ASK INTRO TO Q16A, OTHERWISE CONTINUE TO Q17A}
How would you rate the Quality Measures for Home Health Agencies on the Home Health Compare feature, in terms of the following:
(Please read each statement and provide an answer on that line; if you are unfamiliar with that item please enter "Don’t know")
{GRID AND RANDOMIZE Q16A-Q16E}
Q16A. It was easy to search for the information I wanted.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q16B. The information was displayed clearly.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q16C. The information I got back was the information I was looking for.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q16D. The information was valuable.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q16E. The information was easy to understand.
[RECORD NUMBER 0-11]
0=Very much disagree
10=Very much agree
11=Don’t know
Q17A. What other comments do you have that will help us improve our site?
(Please type your answer in the box below.)
The last few questions are used for classification purposes. Your individual responses are kept private.
Q17C. From where are you accessing this site?
(Please select one answer.)
1 Home (yours or another)
2 Office (yours or another)
3 Library
4 Senior or Community Center
5 School
6 Medical office, clinic or facility
7 Other (SPECIFY)
Q18. How comfortable are you with the Internet?
(Please select one answer.)
1 I am very comfortable using the Internet
2 I am somewhat comfortable using the Internet
3 I am somewhat uncomfortable using the Internet
4 I am very uncomfortable using the Internet
Q19A. Which of the following equipment do you use to assist you while accessing the Internet?
(Please select all that apply.)
1 Screen reviewer or screen reader software
2 Screen enlarger
3 Speech input device
4 On-screen keyboard
5 Keyboard filters
6 Alternative input device (eye-gaze device, sip-and-puff system, etc.)
7 Other (SPECIFY)
8 None of these
Q20. What is your age?
(Please select one answer.)
1 Under 18
2 18-34 Years old
3 35-44 Years old
4 45-54 Years old
5 55-64 Years old
6 65-74 Years old
7 75-84 Years old
8 85 Years or older
9 Prefer not to say
Q21. What is the highest level of education you had the opportunity to complete?
(Please select one answer.)
1 Less than high school
2 High school diploma
3 Some college/trade school
4 College degree (bachelor level)
5 Post-baccalaureate studies or degree
6 Prefer not to say
Q22. What is the annual income of your household?
Under $20,000
$20,000 to $39,999
$40,000 to $59,999
$60,000 to $79,999
More than $80,000
Prefer not to say
_____________________________________________________________________________
Q23. What is your gender?
(Please select one answer.)
1 Female
2 Male
{IF Q1=2-7 ASK Q23 OTHERWISE THANK AND TERMINATE}
Q24. What is the primary language spoken in your home?
(Please select one answer.)
1 English
2 Spanish
3 Other (SPECIFY)
Thank you for your participation in this survey!
Your responses have been saved.
| File Type | application/msword |
| File Title | Supporting Statement – Part B |
| Author | CMS |
| Last Modified By | CMS |
| File Modified | 2007-12-20 |
| File Created | 2007-12-20 |