CMS-10294.Appendix C (8-23-10)

APPENDIX C
QIO SURVEY AND INVITATION LETTER

CMS LETTERHEAD
ADVANCE LETTER - EVALUATION SURVEY OF THE 9TH QIO SOW

[Date], 2010
«QIO Director»
«Name_of_QIO»
«Mailing_Address»
«City», «State» «Zip_Code»
Dear [Mr./Ms./Dr.] «QIO Director»:
The Centers for Medicare & Medicaid Services (CMS) would very much appreciate your
thoughtful input on your QIO’s experience with the 9th SOW. In the near future, you will be
contacted by Mathematica Policy Research, Inc (Mathematica) to participate in the 9th SOW
Evaluation Survey. The survey is being conducted as part of Mathematica’s Evaluation of the 9th
SOW Quality Improvement Program, and will include a questionnaire for you (QIO Director
Survey) and one for the leader of each theme or Patient Safety sub-theme (QIO Theme Leader
Survey).
Mathematica will soon be asking for your help in identify the most appropriate respondents
for the two surveys and will request current contact information for each, including name, e-mail
address, and telephone number. Once this information is gathered, Mathematica will send
personalized emails to each individual inviting him/her to participate in the appropriate survey.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
Your input will be valuable in assisting CMS in continuing to improve the QIO program with
each new scope of work. I urge you to participate.
Please note that Section C.4.14 of the 9th SOW contract requires each QIO to provide data
for evaluation, thus, your time in providing contact information and your time to participate in
the survey is an expense covered under the contract. If you have any questions, please feel free to
call me at [phone number]. If you have questions about the survey, please call Martha Kovac,
Mathematica’s survey director, at 609-275-2331.
Sincerely,
[Signature block]
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a
valid OMB control number. The valid OMB control number for this information collection is xxx-XXX. The time required to complete
this information collection is estimated to average 0.5 to 0.75 hours or 30 to 45 minutes per response, including the time to review
instructions, search existing data resources, and gather the data needed, and complete and review the information collection. If you have
comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security
Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

C.3

REQUEST FOR CONTACT INFORMATION FROM MATHEMATICA

[Date], 2010
Dear [Mr./Ms./Dr.[QIO Director]:
Recently, the Centers for Medicare & Medicaid Services (CMS) sent you a letter about the upcoming
9th SOW Evaluation Survey being conducted by Mathematica Policy Research, Inc. This is part of
Mathematica’s evaluation of the 9th SOW Quality Improvement Program. The 9th SOW Evaluation Survey
will include a questionnaire for you (QIO Director Survey) and one for the leader of each theme or Patient
Safety sub-theme (QIO Theme Leader Survey). We need your help in identify the most appropriate
respondents for these two surveys.
The QIO Director Survey is intended for the QIO executive with ongoing management responsibility
and knowledge of the QIO’s experience operating the program under the 9th SOW contract. If you are the
best person to respond, please record your name, email address, and telephone number on the Excel
spreadsheet attached. If there is someone else at your organization that is better able to response, please
record his/her information on the form instead.
The QIO Theme Leader Survey is intended for those individuals with responsibility and day-to-day
knowledge of the QIO’s experience with themes or Patient Safety sub-themes in which the QIO is engaged.
On the attached Excel spreadsheet, please record the name, email, and telephone number for each theme
and Patient Safety sub-theme leader, as applicable.
Please complete the attached Excel spreadsheet and email it back to Mathematica
([email protected]) or via fax (Attn: Sarah Croake) at 202-863-1763 by
xx/xx/xxxx.
After receiving your completed spreadsheet, email invitations to the applicable web surveys will be
sent to the individuals you have identified. Please be assured that your responses to the form and the
surveys will remain confidential to the extent permitted by law. All data collected for the purposes of this
study will be combined and reported in aggregate form only. Neither you nor your organization will be
identified by name in any reports or documents produced from the study findings. Only Mathematica staff
that work directly on the evaluation will have access to the name of your organization and your name. Your
input will be valuable in assisting CMS in continuing to improve the QIO program with each new scope of
work.
Please note that Section C.4.14 of the 9th SOW contract requires each QIO to provide data for
evaluation, thus, your time in providing contact information and your time to participate in the survey is an
expense covered under the contract. If you have any questions, please feel free to call me at [phone
number]. If you have questions about the survey, please call Martha Kovac, Mathematica’s survey director,
at 609-275-2331.
Sincerely,
[Signature block]
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is xxx-XXX. The time required to complete this information
collection is estimated to average 0.5 to 0.75 hours or 30 to 45 minutes per response, including the time to review instructions, search existing data
resources, and gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

C.5

QUALITY IMPROVEMENT ORGANIZATION NAME

Instructions: In the spaces provided, please first indicate the most knowledgeable respondent for the QIO Director Survey. Please
provide his/her first and last name, email, and telephone number. Then, please identify the most knowledgeable respondent for the
QIO Theme Leader Survey, for each theme at your QIO. Again, please provide the first and last name, email, and telephone number
for each person listed.

Recommended Respondent 

First Name 

Last Name 

Email Address 

Telephone 

QIO Director Survey 

X' next to each QIO theme 

Patient Safety‐Pressure Ulcers 

Patient Safety‐ Physical Restraints 

Patient Safety‐Surgical Care Improvement Project 

Patient Safety‐MRSA 

Patient Safety‐Drug Safety 

Patient Safety ‐ Nursing Homes in Need 

Prevention 

Prevention ‐ Disparities 

Care Transitions 

Chronic Kidney Disease 

QIO Theme Leader Survey 

C.7

Please be assured that your response to this form will remain confidential to the extent permitted by law. All data collected for the
purposes of this study will be combined and reported in aggregate form only. Neither you nor your organization will be identified by
name in any reports or documents produced from the study findings. Only Mathematica staff that work directly on the evaluation will
have access to the name of your organization and your name.

QIO DIRECTOR SURVEY – INVITATION EMAIL

Dear ${name},
Recently, CMS sent a letter to your Quality Improvement Organization (QIO) about an
important web-based survey we are conducting. The QIO Director Survey, which you are being
asked to complete, will gather input for the program evaluation of the 9th SOW along with the
Theme Leader Survey, which was separately sent to theme leaders within the QIO. The surveys
collect information about QIO activities, experience, environment, and suggestions for program
improvement. Information you provide will support Mathematica’s evaluation team in
understanding the trends and patterns in outcomes we will be studying, and in developing
program improvement recommendations for CMS.
You were identified by [name of CEO or CMS principal contact] as the best person to
complete the QIO Director Survey. Please note that Section C.4.14 of the 9th SOW contract
requires each QIO to provide data for evaluation. Your time to participate in the survey is an
expense covered under the contract. The survey should take less than 15 minutes to complete.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
Your input is critical in assisting CMS in its efforts to improve the QIO program with each new
scope of work.
To begin the QIO Director Survey, click the link below:
[link would be embedded in email]
We look forward to receiving your completed survey within the next few days. Thanks in
advance for your participation. If you have any questions, please do not hesitate to contact me at
[email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher,
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.9

QIO DIRECTOR SURVEY – REMINDER #1

Dear ${name},
Recently, CMS sent a letter to your Quality Improvement Organization (QIO) about an
important web-based survey we are conducting. About one week ago, we sent you an email
invitation to participate in the QIO Director Survey. This survey gathers input for the program
evaluation of the 9th SOW along with the Theme Leader Survey, which was separately sent to
theme leaders within the QIO. The surveys collect information about QIO activities, experience,
environment, and suggestions for program improvement. Information you provide will support
Mathematica’s evaluation team in understanding the trends and patterns in outcomes we will be
studying, and in developing program improvement recommendations for CMS.
We have not yet received your completed survey. Please note that Section C.4.14 of the 9th
SOW contract requires each QIO to provide data for evaluation. Your time to participate in the
survey is an expense covered under the contract. The survey should take less than 15 minutes to
complete.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
Your input is critical in assisting CMS in its efforts to improve the QIO program with each
new scope of work.
To begin the QIO Director Survey, click the link below:
[link would be embedded in email]
We look forward to receiving your completed survey within the next few days. Thanks in
advance for your participation. If you have any questions, please do not hesitate to contact me at
[email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.10

QIO DIRECTOR SURVEY – REMINDER #2

Dear ${name},
We recently we sent you an email invitation to participate in the QIO Director Survey. This
survey gathers input for the program evaluation of the 9th. The surveys collect information about
QIO activities, experience, environment, and suggestions for program improvement. Information
you provide will support Mathematica’s evaluation team in understanding the trends and patterns
in outcomes we will be studying, and in developing program improvement recommendations for
CMS.
Your input is very important. Unfortunately, we have not yet received your completed
survey. Section C.4.14 of the 9th SOW contract requires each QIO to provide data for
evaluation. Your time to participate in the survey is an expense covered under the contract. The
survey should take less than 15 minutes to complete.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
Your input is critical in assisting CMS in its efforts to improve the QIO program with each
new scope of work.
To begin the QIO Director Survey, click the link below:
[link would be embedded in email]
We look forward to receiving your completed survey within the next few days. Thanks in
advance for your participation. If you have any questions, please do not hesitate to contact me at
[email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.11

QIO DIRECTOR SURVEY – REMINDER #3

Dear ${name},
Time is running out to participate in the QIO Director Survey! This survey gathers input for
the program evaluation of the 9th. Information you provide will support Mathematica’s
evaluation team in understanding the trends and patterns in outcomes we will be studying, and in
developing program improvement recommendations for CMS.
Section C.4.14 of the 9th SOW contract requires each QIO to provide data for evaluation.
Your time to participate in the survey is an expense covered under the contract.
Please take 15 minutes to complete the survey today.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
Your input is critical in assisting CMS in its efforts to improve the QIO program with each
new scope of work.
To begin the QIO Director Survey, click the link below:
[link would be embedded in email]
If you have any questions, please do not hesitate to contact me at [email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.12

QIO THEME LEADER SURVEY – INVITATION EMAIL

Dear ${name},
Recently, CMS sent a letter to your Quality Improvement Organization (QIO) about an
important web-based survey we are conducting. The QIO Theme Leader Survey will gather input
for the program evaluation of the 9th SOW. The survey collects information about QIO
activities, experience, environment, and suggestions for program improvement specific to the
${theme} theme. Information you provide will support Mathematica’s evaluation team in
understanding the trends and patterns in outcomes we will be studying, and in developing
program improvement recommendations for CMS.
You were identified by [name of CEO or CMS principal contact] as the best person to
complete the survey for the ${theme} theme. Please note that Section C.4.14 of the 9th SOW
contract requires each QIO to provide data for evaluation. Your time to participate in the survey
is an expense covered under the contract.
The survey should take about 45 minutes to complete. Please be assured that your response
to the survey will remain confidential to the extent permitted by law. All data collected for the
purposes of this study will be combined and reported in aggregate form only. Neither you nor
your organization will be identified by name in any reports or documents produced from the
study findings. Only Mathematica staff that work directly on the evaluation will have access to
the name of your organization and your name.
As a theme leader, your input is critical in assisting CMS in its efforts to improve the QIO
program with each new scope of work.
To begin the QIO Theme Leader Survey for the ${theme} theme, click the link below:
[link would be embedded in email]
We look forward to receiving your completed survey within the next few days. Thanks in
advance for your participation. If you have any questions, please do not hesitate to contact me at
[email protected] or (202) 484-4519.
Please note, if you are the theme leader for more than one theme at your QIO, you will
receive an email invitation to complete a QIO Theme Leader Survey for each theme that you
lead. This survey is specific to the ${theme} theme.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024
C.13

QIO THEME LEADER SURVEY – REMINDER #1

Dear ${name},
Recently, CMS sent a letter to your Quality Improvement Organization (QIO) about an
important web-based survey we are conducting. About a week ago, we sent you an email
invitation to participate in the QIO Theme Leader Survey. The survey collects information about
QIO activities, experience, environment, and suggestions for program improvement specific to
the ${theme} theme. Information you provide will support Mathematica’s evaluation team in
understanding the trends and patterns in outcomes we will be studying, and in developing
program improvement recommendations for CMS.
You were identified by [name of CEO or CMS principal contact] as the best person to
complete the survey for the ${theme} theme. Section C.4.14 of the 9th SOW contract requires
each QIO to provide data for evaluation. Your time to participate in the survey is an expense
covered under the contract.
We have not yet received your completed survey for the ${theme} theme. The survey should
take about 45 minutes to complete. Please be assured that your response to the survey will
remain confidential to the extent permitted by law. All data collected for the purposes of this
study will be combined and reported in aggregate form only. Neither you nor your organization
will be identified by name in any reports or documents produced from the study findings. Only
Mathematica staff that work directly on the evaluation will have access to the name of your
organization and your name.
As a theme leader, your input is critical in assisting CMS in its efforts to improve the QIO
program with each new scope of work.
To begin the QIO Theme Leader Survey for the ${theme} theme, click the link below:
[link would be embedded in email]
We look forward to receiving your completed survey within the next few days. Thanks in
advance for your participation. If you have any questions, please do not hesitate to contact me at
[email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.14

QIO THEME LEADER SURVEY – REMINDER #2

Dear ${name},
Recently, we sent you an email invitation to participate in the QIO Theme Leader Survey.
The survey collects information about QIO activities, experience, environment, and suggestions
for program improvement specific to the ${theme} theme. Information you provide will support
Mathematica’s evaluation team in understanding the trends and patterns in outcomes we will be
studying, and in developing program improvement recommendations for CMS.
We have not yet received your completed survey for the ${theme} theme. Section C.4.14 of
the 9th SOW contract requires each QIO to provide data for evaluation. Your time to participate
in the survey is an expense covered under the contract.
Please take time to complete the survey today – it should take about 45 minutes. Please be
assured that your response to the survey will remain confidential to the extent permitted by law.
All data collected for the purposes of this study will be combined and reported in aggregate form
only. Neither you nor your organization will be identified by name in any reports or documents
produced from the study findings. Only Mathematica staff that work directly on the evaluation
will have access to the name of your organization and your name.
As a theme leader, your input is critical in assisting CMS in its efforts to improve the QIO
program with each new scope of work.
To begin the QIO Theme Leader Survey for the ${theme} theme, click the link below:
[link would be embedded in email]
Thanks in advance for your participation. If you have any questions, please do not hesitate to
contact me at [email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.15

QIO THEME LEADER SURVEY – REMINDER #3

Dear ${name},
Time is running out to participate in the QIO Theme Leader Survey! Information you
provide will support Mathematica’s evaluation team in understanding the trends and patterns in
outcomes we will be studying, and in developing program improvement recommendations for
CMS.
Section C.4.14 of the 9th SOW contract requires each QIO to provide data for evaluation.
Your time to participate in the survey is an expense covered under the contract.
Please complete the survey today.
Please be assured that your response to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
As a theme leader, your input is critical in assisting CMS in its efforts to improve the QIO
program with each new scope of work.
To begin the QIO Theme Leader Survey for the ${theme} theme, click the link below:
[link would be embedded in email]
Thanks in advance for your participation. If you have any questions, please do not hesitate to
contact me at [email protected] or (202) 484-4519.
Sincerely,
Sue Felt-Lisk
Sr. Health Researcher
Co-Principal Investigator
Mathematica Policy Research
600 Maryland Ave., SW, Suite 550
Washington, DC 20024

C.16

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Director Survey
January 5, 2010

NINTH SCOPE OF WORK QIO PROGRAM EVALUATION:
QIO Director Survey

The QIO Director Survey is the key mechanism for gathering QIO input for the program
evaluation of the 9th SOW being conducted by Mathematica Policy Research (Mathematica).
Your suggestions for program improvement will support Mathematica’s evaluation team in
developing program- and theme-level recommendations for Centers for Medicare & Medicaid
Services (CMS). The survey should take 10-20 minutes to complete, and your time is a covered
expense under your contract per Section C.4.B.13 of the 9th SOW contract, which requires
each QIO to provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Prepared by Mathematica Policy Research

C.18

1.

In Column A, for each of the areas listed, please indicate if you would recommend that CMS make a
change to improve the QIO program’s success in generating quality improvement. If you do recommend
a change, please briefly describe your recommendation in Column B.
Column A

Column B

Recommend
that CMS make
a change

In the space provided, briefly describe what
change you would recommend.

Yes

No

a. Focus of QIO contract? ..............................

1

□

0

□

b. How QIOs are evaluated? ..........................

1

□

0

□

c. Program emphasis on QIOSCs?................

1

□

0

□

d. Program emphasis on data support
functions? ...................................................

1

□

0

□

e. How QIOs are expected to work with
other providers? .........................................

1

□

0

□

How QIOs are expected to work with
other health care organizations (such as
provider associations or health plans)?......

1

□

0

□

g. Other needed change (Please specify) ........

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.19

Challenges
2.

What area of quality improvement is presenting the most important challenge for you in the 9th SOW?

Recommended Changes
3.

What improvements would you suggest making to the tools or resources made available by CMS? Please list up to
three improvements that should be a high priority.
1.

2.

3.

4.

5.

Is there anything specific you would like to be doing to improve the quality of care in this state that you feel you
cannot do under the current contract?
1

□ Yes

GO TO Q.5

0

□ No

Thank you for completing this survey. See return instructions below.

Please list up to three things you would like to do to improve the quality of care in this state that you feel you cannot
do under the current contract.
1.

2.

3.

Thank you for completing the QIO Director Survey
Please return your completed survey to the following:
Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.20

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Care Transitions
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Care Transitions

Prepared by Mathematica Policy Research

C.22

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.23

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff? Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree,
disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.24

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.25

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is
addressing...............................................................

1

□

0

□

b. Enable design of intervention with high likelihood
of success ...............................................................

1

□

0

□

c. Identify disparities related to this theme ..................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere..

1

□

0

□

n

□

e. Adequately justify the intervention to providers and
others ......................................................................

1

□

0

□

n

□

Prepared by Mathematica Policy Research

C.26

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

Medium
Value

Low
Value

Did Not
Use

a. Reports from the QIOSC that contain data
analysis .........................................................

1

□

2

□

3

□

0

□

b. Conference calls convened by the QIOSC ...

1

□

2

□

3

□

0

□

c. Tools provided by the QIOSC (regardless of
source) ..........................................................

1

□

2

□

3

□

0

□

d. Data from IFMC that the QIO analyzes itself

1

□

2

□

3

□

0

□

e. QualityNet (including MedQIC) .....................

1

□

2

□

3

□

0

□

f.

QualityNet conferences .................................

1

□

2

□

3

□

0

□

g. Conferences sponsored by other
organizations .................................................

1

□

2

□

3

□

0

□

h. Webinars or teleconferences sponsored by
other organizations........................................

1

□

2

□

3

□

0

□

i.

Key websites (other than MedQIC) ...............

1

□

2

□

3

□

0

□

j.

Personal contacts with other QIOs................

1

□

2

□

3

□

0

□

k. Personal contacts with other health care
organizations .................................................

1

□

2

□

3

□

0

□

1

□

2

□

3

□

0

□

l.

Other key information source (Please specify
below) ............................................................
___________________________________

Prepared by Mathematica Policy Research

C.27

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

13.

□
□

Yes
No

GO TO Q.14

Please identify the tools or other resource(s) and describe how they were adapted.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.28

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.29

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.30

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.31

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Care Transitions
20.

In Column A, for each of the following care transitions activities, indicate if it is a major
or minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality with regard to this theme. If an activity is “N/A” in Column A, then skip
Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Encouraging and training

on the use of the CARE
instrument ...........................

1

□

2

□

n

□

1

□

2

□

3

□

b. Use of a Transitions
Coach ..................................

1

□

2

□

n

□

1

□

2

□

3

□

Prepared by Mathematica Policy Research

C.32

SECTION V: STAFFING
21.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

| YEARS

| YEARS

Professional level:
1
2
3
4
5

22.

Associate’s degree

b.

|
e.

Some college

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify) ___________________________________________

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| STAFF

Prepared by Mathematica Policy Research

C.33

23.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Care Transitions. For each, please indicate his/her highest
level of education attainment, field of study, and years of relevant QI experience, and
years of experience working with the types of providers or organizations relevant to this
theme.

PERSON #1
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc]
Doctoral degree [EdD, PhD]
Other (please specify): __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

|

| YEARS

| YEARS

PERSON #2
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

|

| YEARS

| YEARS

Prepared by Mathematica Policy Research

C.34

24.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
25.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

26.

Regulatory oversight
Actively engaged with others (such as the QIO and/or
other provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.________________________________________________________________
2.________________________________________________________________

Prepared by Mathematica Policy Research

C.35

27.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 26.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

The association tends to work with a different set of providers than
the QIO..............................................................................................

1

□

0

□

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................

1

□

0

□

h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.36

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 26. If only one organization was listed at
Question 26, go to Question 29.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g. quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

f.

The association tends to work with a different set of providers than
the QIO..............................................................................................

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................
h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................
Role of Large Provider Organizations
The following questions apply to the state level.
29.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

30.

□
□

Yes
No

GO TO Q.33

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1. __________________________________________________________________
2. __________________________________________________________________

Prepared by Mathematica Policy Research

C.37

31.

To what extent does the headquarters of the first organization listed in Question 30 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

32.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 30
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 30, then go to Question 33.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
33.

Please list up to three other external organizations whose efforts are proving important
to achieving improvements on this theme. (Please spell out the full name of the
organization.)
1. __________________________________________________________________
2. __________________________________________________________________
3. __________________________________________________________________

Prepared by Mathematica Policy Research

C.38

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
34.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
agree

Agree

Disagree

Strongly
disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.39

If you responded “agree” or “strongly agree” to Question 34, item g, go to Question 35.
Otherwise, go to Question 36.
35.

What types of providers lack motivation to improve on this theme?
1
2
3
4

36.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
37.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.40

If you responded “agree” or “strongly agree” to Question 37, item c, go to Question 38.
Otherwise, go to Question 39.
38.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
39.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline. .............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines. .........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines. .........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.41

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
40.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY

Standardized announcement to eligible providers from the QIO

3

□
□
□

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)
__________________________________________________

1
2

QIO initiated personalized contact with providers
Provider associations or other organizations outside the QIO
delivered announcement

Prepared by Mathematica Policy Research

C.42

41.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 42.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.43

NA

NA

42.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those
who asked questions but ultimately decided not to participate.
1
2
3

43.

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

44.

□
□
□

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough providers

GO TO Q.45

Please tell us what types of providers were the most difficult to persuade to participate?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

45.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

46.

Yes
No

GO TO Q.47

How many providers lobbied the QIO for participation?
1
2
3

47.

□
□
□
□
□

1 to 2
3 to 4
5 or more

What is the total number of participating providers at present?
|

|

|

| NUMBER

Prepared by Mathematica Policy Research

C.44

48.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

49.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL
50.

100%

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.51

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Prepared by Mathematica Policy Research

C.45

Thank you for
completing this
survey

51.

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.46

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Chronic Kidney Disease
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.

If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Chronic Kidney Disease

Prepared by Mathematica Policy Research

C.48

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme. For
each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.49

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff. Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree, disagree,
or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.50

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel .........................
(specify role below):

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.51

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is
addressing.....................................................

1

□

0

□

b. Enable design of intervention with high
likelihood of success .....................................

1

□

0

□

c. Identify disparities related to this theme ........

1

□

0

□

n

□

d. Identify what interventions are working
elsewhere ......................................................

1

□

0

□

n

□

e. Adequately justify the intervention to
providers and others .....................................

1

□

0

□

n

□

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

a. Reports from the QIOSC that contain data
analysis .........................................................

1

b. Conference calls convened by the QIOSC ...

1

c. Tools provided by the QIOSC (regardless of
source) ..........................................................

1

d. Data from IFMC that the QIO analyzes itself

1

e. QualityNet (including MedQIC) .....................

1

f.

QualityNet conferences .................................

1

□
□
□
□

g. Conferences sponsored by other
organizations .................................................

1

□

h. Webinars or teleconferences sponsored by
other organizations........................................

1

i.

Key websites (other than MedQIC) ...............

1

j.

Personal contacts with other QIOs................

1

□
□
□

k. Personal contacts with other health care
organizations .................................................

1

1

l.

Other key information source (Please specify
below) ............................................................
___________________________________

Prepared by Mathematica Policy Research

C.52

□
□

Medium
Value
2

2

□
□

2

□
□
□
□

2

□

2

2

2

2

□
□
□

□

2

□

2

2

2

Low
Value
3

3

□
□

3

□
□
□
□

3

□

3

3

3

3

□
□
□

□

3

□

3

3

3

Did Not
Use
0

0

□
□

0

□
□
□
□

0

□

0

0

0

0

□
□
□

□

0

□

□

0

□

0

0

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

13.

□
□

Yes
No

Go to Question 14

Please identify the tools or other resource(s) and describe how they were adapted.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

Go to Question 16

Please describe the tool(s) or other resource(s) created.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.53

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.54

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ..........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.55

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.56

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

Prepared by Mathematica Policy Research

1

□

2

□

C.57

n

□

1

□

2

□

3

□

SECTION V: STAFFING
20.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

e.
1
2
3
4
5

21.

Some college

|

|

| YEARS

| YEARS

Professional level:

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify) __________________________________________

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.58

22.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Chronic Kidney Disease. For each, please indicate his/her
highest level of education attainment, field of study, and years of relevant QI experience,
and years of experience working with the types of providers or organizations relevant to
this theme.

PERSON #1
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

|

| YEARS

| YEARS

PERSON #2
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

|

| YEARS

| YEARS

Prepared by Mathematica Policy Research

C.59

23.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have
the right substantive expertise and
experience.....................................................

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.60

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
24.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

25.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.
2.

26.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 25.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.61

27.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 25. If only one organization was listed at
Question 25, go to Question 28.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
28.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

29.

□
□

Yes
No

GO TO Q.324

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1.
2.

Prepared by Mathematica Policy Research

C.62

31.

To what extent does the headquarters of the first organization listed in Question 29 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

31.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 29
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 29, then go to Question 32.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
32.

Please list up to three other external organizations whose efforts are proving important
to achieving improvements on this theme. (Please spell out the full name of the
organization.)
1.
2.
3.

Prepared by Mathematica Policy Research

C.63

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
33.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
agree

Agree

Disagree

Strongly
disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.64

If you responded “agree” or “strongly agree” to Question 33, item g, go to Question 34.
Otherwise, go to Question 35.
34.

What types of providers lack motivation to improve on this theme?
1
2
3
4

35.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
36.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.65

If you responded “agree” or “strongly agree” to Question 36, item c, go to Question 37.
Otherwise, go to Question 38.
37.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
38.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.66

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
39.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY
1

□

Standardized announcement to eligible providers from the QIO

2

□

QIO initiated personalized contact with providers

3

□

Provider associations or other organizations outside the
QIO delivered announcement

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)

Prepared by Mathematica Policy Research

C.67

40.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 41.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.68

NA

NA

41.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those who
asked questions but ultimately decided not to participate.
1
2
3

42.

□
□
□

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough provider

GO TO Q.44

43.

Please tell us what types of providers were the most difficult to persuade to participate?

44.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

45.

□
□

Yes
No

GO TO Q.46

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.69

46.

What is the total number of participating providers at present?
________ NUMBER

47.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

48.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL
49.

100%

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.50

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Prepared by Mathematica Policy Research

C.70

Thank you for
completing this survey

50.

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.71

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Drug Safety
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or
email at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Drug Safety

Prepared by Mathematica Policy Research

C.74

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.75

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff? Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree,
disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.76

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.77

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Interactions with Individual Providers
9.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Important of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ...

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated .................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers .

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations .....................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ...........

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI ........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them ......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations .......

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.78

Patient Safety
10.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
n

11.

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this theme?
1
2
3
4
d

12.

□
□
□
□
□
□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

About what percent of QIO staff time devoted to the patient safety drug safety sub-theme
has been spent on activities with (or targeting) the following organizations?
Please round to the nearest percent. Percents should total 100.
a. Medicare providers and practitioners ..................................... |

|

|

| %

b. Medicare Advantage plans .................................................... |

|

|

| %

c. Prescription Drug Sponsor plans (PDPs) under Part D ......... |

|

|

| %

d. Other (Please specify).............................................................. |

|

|

| %

TOTAL 100%

Prepared by Mathematica Policy Research

C.79

13.

How interested did you find each of these types of organizations to be in working on
projects to reduce drug-drug interactions?
Very
Somewhat
interested interested

Little or
no interest

Don’t
know

a. Medicare providers and practitioners ..........

1

□

2

□

3

□

d

□

b. Medicare Advantage plans..........................

1

□

2

□

3

□

d

□

c. Prescription Drug Sponsor plans (PDPs)
under Part D ................................................

1

□

2

□

3

□

d

□

d. Other (Specify) .............................................

1

□

2

□

3

□

d

□

__________________________________
14.

How interested did you find each of these types of organizations to be in working on
projects to reduce prescriptions of potentially inappropriate medications?
Very
interested

Somewhat
interested

Little or no
interest

Don’t know

a. Medicare providers and practitioners ..........

1

□

2

□

3

□

d

□

b. Medicare Advantage plans..........................

1

□

2

□

3

□

d

□

c. Prescription Drug Sponsor plans (PDPs)
under Part D ................................................

1

□

2

□

3

□

d

□

d. Other (Specify) .............................................

1

□

2

□

3

□

d

□

__________________________________
15.

What has been the QIO’s most important contribution thus far under the drug safety subtheme?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.80

16.

What lessons have you learned about partnering with other organizations to improve drug
safety?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

17.

Do you have any suggestions for how CMS could improve the drug safety sub-theme?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.81

Mathematica Reference No.: 06514.180

Ninth Scope of Work
QIO Program
Evaluation: QIO Theme
Leader Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Methicillin Resistant Staphylococcus Aureus
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Methicillin Resistant Staphylococcus Aureus

Prepared by Mathematica Policy Research

C.84

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. contract language at the time of award .........

1

□

2

□

3

□

4

□

c. contract modification(s) since award .............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.85

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff. Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree, disagree,
or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear..............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals.................................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable .....................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care .................

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable .....................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear..........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ..............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ......

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.86

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not
Enough
Contact
to Tell

a. Government Task Leader..................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ..............

1

□

2

□

3

□

4

□

5

□

c. Contract Officer .................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer...........................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.87

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is addressing ................

1

□

0

□

b. Enable design of intervention with high likelihood of success.......

1

□

0

□

c. Identify disparities related to this theme ........................................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere........................

1

□

0

□

n

□

e. Adequately justify the intervention to providers and others ...........

1

□

0

□

n

□

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

a. Reports from the QIOSC that contain data
analysis ...........................................................

1

b. Conference calls convened by the QIOSC .....

1

c. Tools provided by the QIOSC (regardless of
source) ............................................................

1

d. Data from IFMC that the QIO analyzes itself ..

1

e. QualityNet (including MedQIC) .......................

1

f.

QualityNet conferences ...................................

1

□
□
□
□

g. Conferences sponsored by other
organizations ...................................................

1

□

h. Webinars or teleconferences sponsored by
other organizations..........................................

1

i.

Key websites (other than MedQIC) .................

1

j.

Personal contacts with other QIOs..................

1

□
□
□

k. Personal contacts with other health care
organizations ...................................................

1

1

l.

Other key information source (Please specify
below) ..............................................................

Prepared by Mathematica Policy Research

C.88

□
□

Medium
Value
2

2

□
□

2

□
□
□
□

2

□

2

2

2

2

□
□
□

□

2

□

2

2

2

Low
Value
3

3

□
□

3

□
□
□
□

3

□

3

3

3

3

□
□
□

□

3

□

3

3

3

Did Not
Use
0

0

□
□

0

□
□
□
□

0

□

0

0

0

0

□
□
□

□

0

□

□

0

□

0

0

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

□
□

Yes
No

GO TO Q.14

13.

Please identify the tools or other resource(s) and describe how they were adapted.

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.

Prepared by Mathematica Policy Research

C.89

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.90

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.91

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.92

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Patient Safety
20.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
5

21.

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this theme?
1
2
3
4
d

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

Prepared by Mathematica Policy Research

C.93

SECTION V: STAFFING
22.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience:
|

d.

e.

|

| YEARS

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

| YEARS

Professional level:
1
2
3
4
5

23.

□
□
□
□
□
□
□

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify)

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.94

24.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Methicillin Resistant Staphylococcus Aureus. For each,
please indicate his/her highest level of education attainment, field of study, and years of
relevant QI experience, and years of experience working with the types of providers or
organizations relevant to this theme.
PERSON #1
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

|

| YEARS

PERSON #2
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

Prepared by Mathematica Policy Research

|

| YEARS

C.95

25.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.96

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
26.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

27.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.
2.

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 27.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.97

29.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 27. If only one organization was listed at
Question 27, go to Question 30.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
30.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

□
□

Yes
No

GO TO Q.34

Prepared by Mathematica Policy Research

C.98

31.

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1.
2.

32.

To what extent does the headquarters of the first organization listed in Question 31 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

33.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 31
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 31, then go to Question 34.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
34.

Please list up to three other external organizations whose efforts are proving important to
achieving improvements on this theme. (Please spell out the full name of the
organization.)
1.
2.
3.

Prepared by Mathematica Policy Research

C.99

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
35.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
Strongly
agree
Agree
Disagree disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.100

If you responded “agree” or “strongly agree” to Question 35, item g, go to Question 36.
Otherwise, go to Question 37.
36.

What types of providers lack motivation to improve on this theme?
1
2
3
4

37.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
38.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.101

If you responded “agree” or “strongly agree” to Question 38, item c, go to Question 39.
Otherwise, go to Question 40.
39.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
40.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.102

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
41.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY
1

□

Standardized announcement to eligible providers from the QIO

2

□

QIO initiated personalized contact with providers

3

□

Provider associations or other organizations outside the
QIO delivered announcement

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)

Prepared by Mathematica Policy Research

C.103

42.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 43.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.104

NA

NA

43.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those who
asked questions but ultimately decided not to participate.
1
2
3

44.

□
□
□

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough provider

GO TO Q.46

45.

Please tell us what types of providers were the most difficult to persuade to participate?

46.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

47.

□
□

Yes
No

GO TO Q.48

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.105

48.

What is the total number of participating providers at present?
________ NUMBER

49.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

50.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.106

100%

51.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.52

52.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.107

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 7, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Nursing Homes in Need
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.

If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Nursing Homes in Need

Prepared by Mathematica Policy Research

C.110

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.111

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff? Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree,
disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.112

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.113

SECTION II: PATIENT SAFETY – NURSING HOMES IN NEED
8.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
n

9.

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this sub-theme?
1
2
3
4
d

10.

□
□
□
□
□
□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

Think about the nursing home you first worked with under the nursing homes in need subtheme. About what percent of QIO staff time was spent on the following activities related
to the sub-theme?
Please round to the nearest percent. Percents should total 100.
Percent of Time

a. Root cause analysis ...............................................................

|

|

|

| %

b. Developing an action plan ......................................................

|

|

|

| %

c. Training or other interventions ...............................................

|

|

|

| %

d. Finding or preparing educational materials for the NHIN .......

|

|

|

| %

e. Follow-up assessment ...........................................................

|

|

|

| %

f.

|

|

|

| %

Other (Please specify)............................................................

TOTAL

Prepared by Mathematica Policy Research

C.114

100%

11.

Thinking again about the nursing home you first worked with under the nursing homes in
need sub-theme, to what extent do you agree with the following statements?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The root cause analysis led to identification
of problems that QIO staff were wellqualified to assist with ...................................

1

□

2

□

3

□

4

□

b. With the QIO’s assistance, the nursing
home was able to make substantial
progress on the major problems affecting its
quality ............................................................

1

□

2

□

3

□

4

□

c. Nursing home executives and staff seemed
to appreciate QIO assistance ........................

1

□

2

□

3

□

4

□

d. The nursing home was able to carry out its
action plan successfully ................................

1

□

2

□

3

□

4

□

12.

What has been the QIO’s most important contribution thus far under the nursing homes in
need sub-theme?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

13.

What lessons have you learned from your experience to date on the nursing homes in
need sub-theme?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.115

14.

Do you have any suggestions for how CMS could improve the nursing homes in need
sub-theme?
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.116

Mathematica Reference No.: 06514.180

Ninth Scope of Work
QIO Program
Evaluation: QIO Theme
Leader Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Physical Restraints
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Physical Restraints

Prepared by Mathematica Policy Research

C.118

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. contract language at the time of award .........

1

□

2

□

3

□

4

□

c. contract modification(s) since award .............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.119

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff. Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree, disagree,
or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear..............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals.................................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable .....................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care .................

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable .....................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear..........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ..............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ......

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.120

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not
Enough
Contact
to Tell

a. Government Task Leader..................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ..............

1

□

2

□

3

□

4

□

5

□

c. Contract Officer .................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer...........................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.121

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is addressing ................

1

□

0

□

b. Enable design of intervention with high likelihood of success.......

1

□

0

□

c. Identify disparities related to this theme ........................................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere........................

1

□

0

□

n

□

e. Adequately justify the intervention to providers and others ...........

1

□

0

□

n

□

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

a. Reports from the QIOSC that contain data
analysis ...........................................................

1

b. Conference calls convened by the QIOSC .....

1

c. Tools provided by the QIOSC (regardless of
source) ............................................................

1

d. Data from IFMC that the QIO analyzes itself ..

1

e. QualityNet (including MedQIC) .......................

1

f.

QualityNet conferences ...................................

1

□
□
□
□

g. Conferences sponsored by other
organizations ...................................................

1

□

h. Webinars or teleconferences sponsored by
other organizations..........................................

1

i.

Key websites (other than MedQIC) .................

1

j.

Personal contacts with other QIOs..................

1

□
□
□

k. Personal contacts with other health care
organizations ...................................................

1

1

l.

Other key information source (Please specify
below) ..............................................................

Prepared by Mathematica Policy Research

C.122

□
□

Medium
Value
2

2

□
□

2

□
□
□
□

2

□

2

2

2

2

□
□
□

□

2

□

2

2

2

Low
Value
3

3

□
□

3

□
□
□
□

3

□

3

3

3

3

□
□
□

□

3

□

3

3

3

Did Not
Use
0

0

□
□

0

□
□
□
□

0

□

0

0

0

0

□
□
□

□

0

□

□

0

□

0

0

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

□
□

Yes
No

GO TO Q.14

13.

Please identify the tools or other resource(s) and describe how they were adapted.

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.

Prepared by Mathematica Policy Research

C.123

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.124

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.125

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.126

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Patient Safety
20.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
5

21.

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this theme?
1
2
3
4
d

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

Prepared by Mathematica Policy Research

C.127

SECTION V: STAFFING
22.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience:
|

d.

e.

|

| YEARS

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

| YEARS

Professional level:
1
2
3
4
5

23.

□
□
□
□
□
□
□

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify)

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.128

24.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Physical Restraints. For each, please indicate his/her
highest level of education attainment, field of study, and years of relevant QI experience,
and years of experience working with the types of providers or organizations relevant to
this theme.
PERSON #1
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

|

| YEARS

PERSON #2
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

Prepared by Mathematica Policy Research

|

| YEARS

C.129

25.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.130

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
26.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

27.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.
2.

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 27.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.131

29.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 27. If only one organization was listed at
Question 27, go to Question 30.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
30.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

□
□

Yes
No

GO TO Q.34

Prepared by Mathematica Policy Research

C.132

31.

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1.
2.

32.

To what extent does the headquarters of the first organization listed in Question 31 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

33.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 31
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 31, then go to Question 34.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
34.

Please list up to three other external organizations whose efforts are proving important to
achieving improvements on this theme. (Please spell out the full name of the
organization.)
1.
2.
3.

Prepared by Mathematica Policy Research

C.133

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
35.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
Strongly
agree
Agree
Disagree disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.134

If you responded “agree” or “strongly agree” to Question 35, item g, go to Question 36.
Otherwise, go to Question 37.
36.

What types of providers lack motivation to improve on this theme?
1
2
3
4

37.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
38.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.135

If you responded “agree” or “strongly agree” to Question 38, item c, go to Question 39.
Otherwise, go to Question 40.
39.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
40.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.136

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
41.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY
1

□

Standardized announcement to eligible providers from the QIO

2

□

QIO initiated personalized contact with providers

3

□

Provider associations or other organizations outside the
QIO delivered announcement

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)

Prepared by Mathematica Policy Research

C.137

42.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 43.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.138

NA

NA

43.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those who
asked questions but ultimately decided not to participate.
1
2
3

44.

□
□
□

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough provider

GO TO Q.46

45.

Please tell us what types of providers were the most difficult to persuade to participate?

46.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

47.

□
□

Yes
No

GO TO Q.48

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.139

48.

What is the total number of participating providers at present?
________ NUMBER

49.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

50.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.140

100%

51.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.52

52.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.141

Mathematica Reference No.: 06514.180

Ninth Scope of Work
QIO Program
Evaluation: QIO Theme
Leader Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Pressure Ulcers
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Pressure Ulcers

Prepared by Mathematica Policy Research

C.144

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. contract language at the time of award .........

1

□

2

□

3

□

4

□

c. contract modification(s) since award .............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.145

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff. Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree, disagree,
or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear..............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals.................................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable .....................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care .................

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable .....................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear..........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ..............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ......

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.146

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not
Enough
Contact
to Tell

a. Government Task Leader..................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ..............

1

□

2

□

3

□

4

□

5

□

c. Contract Officer .................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer...........................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.147

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is addressing ................

1

□

0

□

b. Enable design of intervention with high likelihood of success.......

1

□

0

□

c. Identify disparities related to this theme ........................................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere........................

1

□

0

□

n

□

e. Adequately justify the intervention to providers and others ...........

1

□

0

□

n

□

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

a. Reports from the QIOSC that contain data
analysis ...........................................................

1

b. Conference calls convened by the QIOSC .....

1

c. Tools provided by the QIOSC (regardless of
source) ............................................................

1

d. Data from IFMC that the QIO analyzes itself ..

1

e. QualityNet (including MedQIC) .......................

1

f.

QualityNet conferences ...................................

1

□
□
□
□

g. Conferences sponsored by other
organizations ...................................................

1

□

h. Webinars or teleconferences sponsored by
other organizations..........................................

1

i.

Key websites (other than MedQIC) .................

1

j.

Personal contacts with other QIOs..................

1

□
□
□

k. Personal contacts with other health care
organizations ...................................................

1

1

l.

Other key information source (Please specify
below) ..............................................................

Prepared by Mathematica Policy Research

C.148

□
□

Medium
Value
2

2

□
□

2

□
□
□
□

2

□

2

2

2

2

□
□
□

□

2

□

2

2

2

Low
Value
3

3

□
□

3

□
□
□
□

3

□

3

3

3

3

□
□
□

□

3

□

3

3

3

Did Not
Use
0

0

□
□

0

□
□
□
□

0

□

0

0

0

0

□
□
□

□

0

□

□

0

□

0

0

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

□
□

Yes
No

GO TO Q.14

13.

Please identify the tools or other resource(s) and describe how they were adapted.

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.

Prepared by Mathematica Policy Research

C.149

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.150

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.151

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.152

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Patient Safety
20.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
5

21.

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this theme?
1
2
3
4
d

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

Prepared by Mathematica Policy Research

C.153

SECTION V: STAFFING
22.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience:
|

d.

e.

|

| YEARS

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

| YEARS

Professional level:
1
2
3
4
5

23.

□
□
□
□
□
□
□

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify)

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.154

24.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Pressure Ulcers. For each, please indicate his/her highest
level of education attainment, field of study, and years of relevant QI experience, and
years of experience working with the types of providers or organizations relevant to this
theme.
PERSON #1
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

|

| YEARS

PERSON #2
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

Prepared by Mathematica Policy Research

|

| YEARS

C.155

25.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.156

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
26.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

27.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.
2.

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 27.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.157

29.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 27. If only one organization was listed at
Question 27, go to Question 30.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
30.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

□
□

Yes
No

GO TO Q.34

Prepared by Mathematica Policy Research

C.158

31.

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1.
2.

32.

To what extent does the headquarters of the first organization listed in Question 31 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

33.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 31
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 31, then go to Question 34.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
34.

Please list up to three other external organizations whose efforts are proving important to
achieving improvements on this theme. (Please spell out the full name of the
organization.)
1.
2.
3.

Prepared by Mathematica Policy Research

C.159

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
35.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
Strongly
agree
Agree
Disagree disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.160

If you responded “agree” or “strongly agree” to Question 35, item g, go to Question 36.
Otherwise, go to Question 37.
36.

What types of providers lack motivation to improve on this theme?
1
2
3
4

37.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
38.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.161

If you responded “agree” or “strongly agree” to Question 38, item c, go to Question 39.
Otherwise, go to Question 40.
39.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
40.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.162

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
41.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY
1

□

Standardized announcement to eligible providers from the QIO

2

□

QIO initiated personalized contact with providers

3

□

Provider associations or other organizations outside the
QIO delivered announcement

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)

Prepared by Mathematica Policy Research

C.163

42.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 43.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.164

NA

NA

43.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those who
asked questions but ultimately decided not to participate.
1
2
3

44.

□
□
□

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough provider

GO TO Q.46

45.

Please tell us what types of providers were the most difficult to persuade to participate?

46.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

47.

□
□

Yes
No

GO TO Q.48

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.165

48.

What is the total number of participating providers at present?
________ NUMBER

49.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

50.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.166

100%

51.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.52

52.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.167

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 7, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Prevention - Disparities
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.

If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Prevention – Disparities

Prepared by Mathematica Policy Research

C.170

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.171

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff? Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree,
disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.172

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.173

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is
addressing...............................................................

1

□

0

□

b. Enable design of intervention with high likelihood
of success ...............................................................

1

□

0

□

c. Identify disparities related to this theme ..................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere..

1

□

0

□

n

□

e. Adequately justify the intervention to providers and
others ......................................................................

1

□

0

□

n

□

Prepared by Mathematica Policy Research

C.174

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

Medium
Value

Low
Value

Did Not
Use

a. Reports from the QIOSC that contain data
analysis .........................................................

1

□

2

□

3

□

0

□

b. Conference calls convened by the QIOSC ...

1

□

2

□

3

□

0

□

c. Tools provided by the QIOSC (regardless of
source) ..........................................................

1

□

2

□

3

□

0

□

d. Data from IFMC that the QIO analyzes itself

1

□

2

□

3

□

0

□

e. QualityNet (including MedQIC) .....................

1

□

2

□

3

□

0

□

f.

QualityNet conferences .................................

1

□

2

□

3

□

0

□

g. Conferences sponsored by other
organizations .................................................

1

□

2

□

3

□

0

□

h. Webinars or teleconferences sponsored by
other organizations........................................

1

□

2

□

3

□

0

□

i.

Key websites (other than MedQIC) ...............

1

□

2

□

3

□

0

□

j.

Personal contacts with other QIOs................

1

□

2

□

3

□

0

□

k. Personal contacts with other health care
organizations .................................................

1

□

2

□

3

□

0

□

1

□

2

□

3

□

0

□

l.

Other key information source (Please specify
below) ............................................................
___________________________________

Prepared by Mathematica Policy Research

C.175

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

13.

□
□

Yes
No

GO TO Q.14

Please identify the tools or other resource(s) and describe how they were adapted.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.176

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as
a health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.177

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If
the activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.178

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.179

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Disparities
20.

In Column A, for each of the following Prevention - Disparities activities, indicate if it is a
major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality with regard to this theme. If an activity is “N/A” in Column A, then skip
Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Obtaining clinical EHR-

based data from
practices ...........................

1

□

2

□

n

□

1

□

2

□

3

□

community health
workers .............................

1

□

2

□

n

□

1

□

2

□

3

□

Implementing DSME for
beneficiaries with
diabetes ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Recruiting and training

c.

Prepared by Mathematica Policy Research

C.180

21.

To meet the goals of providing Diabetes Self-Management Education, what mechanisms
were used by the QIO to recruit beneficiaries?
MARK ALL THAT APPLY
1
2
3

□
□
□

Recruitment through local providers
Recruitment directly from the community
Other mechanism(s) (Please specify)
_________________________________________

22.

Which of the following best describes the geographic area targeted under this theme?
1
2
3
4
5

□
□
□
□
□

All urban or suburban
Mostly urban or suburban
About evenly urban/suburban and rural
Mostly rural
All rural

Prepared by Mathematica Policy Research

C.181

SECTION V: STAFFING
23.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

| YEARS

| YEARS

Professional level:
1
2
3
4
5

24.

Associate’s degree

b.

|
e.

Some college

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify) ___________________________________________

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.182

25.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Prevention Disparities. For each, please indicate his/her
highest level of education attainment, field of study, and years of relevant QI experience,
and years of experience working with the types of providers or organizations relevant to
this theme.

PERSON #1
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

|

| YEARS

| YEARS

PERSON #2
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

|

| YEARS

| YEARS

Prepared by Mathematica Policy Research

C.183

26.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.184

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
27.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

28.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.________________________________________________________________
2.________________________________________________________________

29.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 28.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g. quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

The association tends to work with a different set of providers than
the QIO..............................................................................................

1

□

0

□

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................

1

□

0

□

h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.185

30.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 26. If only one organization was listed at
Question 28, go to Question 31.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

The association tends to work with a different set of providers than
the QIO..............................................................................................

1

□

0

□

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................

1

□

0

□

h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
31.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

32.

□
□

Yes
No

GO TO Q.35

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this
theme and have the greatest potential to affect its success.
1. __________________________________________________________________
2. __________________________________________________________________

Prepared by Mathematica Policy Research

C.186

33.

To what extent does the headquarters of the first organization listed in Question 32 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

34.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 32
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 32, then go to Question 35.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
35.

Please list up to three other external organizations whose efforts are proving important to
achieving improvements on this theme. (Please spell out the full name of the
organization.)
1. __________________________________________________________________
2. __________________________________________________________________
3. __________________________________________________________________

Prepared by Mathematica Policy Research

C.187

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
36.

The following are statements which related to motivation and quality improvement.
For each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
agree

Agree

Disagree

Strongly
disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.188

If you responded “agree” or “strongly agree” to Question 36, item g, go to Question 37.
Otherwise, go to Question 38.
37.

What types of providers lack motivation to improve on this theme?
1
2
3
4

38.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
39.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree,
disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
Know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.189

If you responded “agree” or “strongly agree” to Question 39, item c, go to Question 40.
Otherwise, go to Question 41.
40.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
41.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ............................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline. ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines. ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines. ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given .....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.190

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
42.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY

Standardized announcement to eligible providers from the QIO

3

□
□
□

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)
____________________________________________________________

1
2

QIO initiated personalized contact with providers
Provider associations or other organizations outside the QIO
delivered announcement

Prepared by Mathematica Policy Research

C.191

43.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 44.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.192

NA

NA

44.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those
who asked questions but ultimately decided not to participate.
1
2
3

45.

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

46.

□
□
□

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough providers

GO TO Q.47

Please tell us what types of providers were the most difficult to persuade to participate.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

47.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

48.

□
□

Yes
No

GO TO Q.49

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.193

49.

What is the total number of participating providers at present?
|

50.

|

|

| NUMBER

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

51.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.194

100%

52.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.53

53.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.195

Mathematica Reference No.: 06514.180

Ninth Scope of Work QIO
Program Evaluation:
QIO Theme Leader
Survey
January 7, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Prevention
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.

If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Prevention

Prepared by Mathematica Policy Research

C.198

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. Contract language at the time of award ........

1

□

2

□

3

□

4

□

c. Contract modification(s) since award ............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.199

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff? Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree,
disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals...............................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable ...................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care ...............

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable ...................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ....

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.200

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not Enough
Contact
to Tell

a. Government Task Leader ...................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ................

1

□

2

□

3

□

4

□

5

□

c. Contract Officer ...................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer ............................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

______________________________
7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.201

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is
addressing...............................................................

1

□

0

□

b. Enable design of intervention with high likelihood
of success ...............................................................

1

□

0

□

c. Identify disparities related to this theme ..................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere..

1

□

0

□

n

□

e. Adequately justify the intervention to providers and
others ......................................................................

1

□

0

□

n

□

Prepared by Mathematica Policy Research

C.202

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

Medium
Value

Low
Value

Did Not
Use

a. Reports from the QIOSC that contain data
analysis .........................................................

1

□

2

□

3

□

0

□

b. Conference calls convened by the QIOSC ...

1

□

2

□

3

□

0

□

c. Tools provided by the QIOSC (regardless of
source) ..........................................................

1

□

2

□

3

□

0

□

d. Data from IFMC that the QIO analyzes itself

1

□

2

□

3

□

0

□

e. QualityNet (including MedQIC) .....................

1

□

2

□

3

□

0

□

f.

QualityNet conferences .................................

1

□

2

□

3

□

0

□

g. Conferences sponsored by other
organizations .................................................

1

□

2

□

3

□

0

□

h. Webinars or teleconferences sponsored by
other organizations........................................

1

□

2

□

3

□

0

□

i.

Key websites (other than MedQIC) ...............

1

□

2

□

3

□

0

□

j.

Personal contacts with other QIOs................

1

□

2

□

3

□

0

□

k. Personal contacts with other health care
organizations .................................................

1

□

2

□

3

□

0

□

1

□

2

□

3

□

0

□

l.

Other key information source (Please specify
below) ............................................................
___________________________________

Prepared by Mathematica Policy Research

C.203

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

13.

□
□

Yes
No

GO TO Q.14

Please identify the tools or other resource(s) and describe how they were adapted.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

Prepared by Mathematica Policy Research

C.204

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.205

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.206

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.207

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Prevention
20.

Do you think participating providers found clinical quality reporting to be beneficial to their
ongoing quality improvement efforts?
1
0
d

□
□
□

Yes
No
Don’t know

Prepared by Mathematica Policy Research

C.208

SECTION V: STAFFING
21.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) ___________________________________________

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme
|

|

| YEARS

| YEARS

Professional level:
1
2
3
4
5

22.

Associate’s degree

b.

|
e.

Some college

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify) ___________________________________________

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.209

23.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Prevention. For each, please indicate his/her highest level
of education attainment, field of study, and years of relevant QI experience, and years of
experience working with the types of providers or organizations relevant to this theme.

PERSON #1
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

|

| YEARS

| YEARS

PERSON #2
a.
1
2
3
4
5
6
7

Highest level of educational attainment:

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify) __________________________________________

b.

Field of study, if applicable: _______________________________________

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

|

| YEARS

| YEARS

Prepared by Mathematica Policy Research

C.210

24.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.211

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
25.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

26.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.________________________________________________________________
2.________________________________________________________________

27.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 26.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g. quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

The association tends to work with a different set of providers than
the QIO..............................................................................................

1

□

0

□

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................

1

□

0

□

h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.212

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 26. If only one organization was listed at
Question 26, go to Question 29.
Yes

No

a. The association employs at least one staff member with major
responsibility and time devoted to quality improvement....................

1

□

0

□

b. The association and QIO talk periodically (e.g. quarterly) to avoid
overlap ..............................................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored meetings at
least once per year............................................................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as a
Quality Council or a Quality Institute .................................................

1

□

0

□

e. The association and QIO work jointly on one or more QI efforts
substantial in scope (such as co-sponsoring in-person meetings
focused on QI)...................................................................................

1

□

0

□

f.

The association tends to work with a different set of providers than
the QIO..............................................................................................

g. The association tends to work on QI projects that are entirely
different from the QIO .......................................................................
h. The association primarily focuses on quality reporting issues rather
than quality improvement ..................................................................
Role of Large Provider Organizations
The following questions apply to the state level.
29.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

30.

□
□

Yes
No

GO TO Q.33

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this
theme and have the greatest potential to affect its success.
1. __________________________________________________________________
2. __________________________________________________________________

Prepared by Mathematica Policy Research

C.213

31.

To what extent does the headquarters of the first organization listed in Question 30 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

32.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 30
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 30, then go to Question 33.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
33.

Please list up to three other external organizations whose efforts are proving important
to achieving improvements on this theme. (Please spell out the full name of the
organization.)
1. __________________________________________________________________
2. __________________________________________________________________
3. __________________________________________________________________

Prepared by Mathematica Policy Research

C.214

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
34.

The following are statements which related to motivation and quality improvement.
For each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
agree

Agree

Disagree

Strongly
disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.215

If you responded “agree” or “strongly agree” to Question 34, item g, go to Question 35.
Otherwise, go to Question 36.
35.

What types of providers lack motivation to improve on this theme?
1
2
3
4

36.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
37.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree,
disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.216

If you responded “agree” or “strongly agree” to Question 37, item c, go to Question 38.
Otherwise, go to Question 39.
38.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
39.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have
been working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ............................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline. ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines. ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines. ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given .....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.217

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
40.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY

Standardized announcement to eligible providers from the QIO

3

□
□
□

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)
____________________________________________________________

1
2

QIO initiated personalized contact with providers
Provider associations or other organizations outside the QIO
delivered announcement

Prepared by Mathematica Policy Research

C.218

41.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 42.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.219

NA

NA

42.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those
who asked questions but ultimately decided not to participate.
1
2
3

43.

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

44.

□
□
□

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough providers

GO TO Q.45

Please tell us what types of providers were the most difficult to persuade to participate.
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

45.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

46.

Yes
No

GO TO Q.47

How many providers lobbied the QIO for participation?
1
2
3

47.

□
□
□
□
□

1 to 2
3 to 4
5 or more

What is the total number of participating providers at present?
|

|

|

| NUMBER

Prepared by Mathematica Policy Research

C.220

48.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

49.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.221

100%

50.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.51

51.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.222

Mathematica Reference No.: 06514.180

Ninth Scope of Work
QIO Program
Evaluation: QIO Theme
Leader Survey
January 6, 2010

Ninth Scope of Work QIO Program Evaluation:
QIO Theme Leader Survey
Patient Safety – Surgical Care Improvement Project
This survey is the key mechanism for QIO input into the program evaluation of the 9th SOW.
The information collected about QIO activities, experience, environment, and suggestions for
program improvement will allow quantitative and qualitative analysis to support the Mathematica
Policy Research evaluation team in developing program- and theme-level recommendations to
CMS. Each survey should take 10-20 minutes to complete, and time is a covered expense
under your contract per Section C.4.B.13 of the 9th SOW contract, which requires each QIO to
provide data for evaluation.
Please be assured that your responses to the survey will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
If you have any questions about this survey, please call Sarah Croake at 202-554-7555 or email
at: [email protected].

Please answer all questions on this survey in relation to the following
9th SOW theme:
Patient Safety Theme – Surgical Care Improvement Project

Prepared by Mathematica Policy Research

C.224

SECTION I: 9TH SOW CONTRACT AND COMMUNICATION WITH CMS
Clarity of the Contract for the 9th SOW
1.

Please indicate the level of clarity for each of the following aspects of the 9th SOW
contract for this theme. For each, was it very clear, clear, unclear, or very unclear?
Very
clear

Clarity of…

Clear

Unclear

Very
unclear

a. RFP aims and requirements .........................

1

□

2

□

3

□

4

□

b. contract language at the time of award .........

1

□

2

□

3

□

4

□

c. contract modification(s) since award .............

1

□

2

□

3

□

4

□

d. SDPS or TOPS memos since award ............

1

□

2

□

3

□

4

□

Contract Documentation and Reporting Requirements for the 9th SOW
2.

Please indicate if you agree with each of the following statements about the
documentation and reporting requirements of the 9th SOW contract for this theme.
For each, do you strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS documentation and reporting
requirements are clear ..................................

1

□

2

□

3

□

4

□

b. Required reports to CMS capture
meaningful information about the progress
of the intervention..........................................

1

□

2

□

3

□

4

□

c. The amount of CMS required
documentation and reporting is reasonable ..

1

□

2

□

3

□

4

□

d. The PATRIOT system worked well in the
first six months of the contract ......................

1

□

2

□

3

□

4

□

e. The PATRIOT system worked well after the
first six months of the contract ......................

1

□

2

□

3

□

4

□

3.

In an average month how many total hours are spent fulfilling CMS reporting requirements
for this theme?
If reporting does not occur on a monthly basis, please determine total hours spent over a
single reporting period and divide by the total months in that reporting period to determine
the average hours per month.
|

|

|

| HOURS

Prepared by Mathematica Policy Research

C.225

4.

Of the hours spent in an average month reported above, what percent is spent by senior
staff, mid-level staff, and junior staff. Percents should total 100.
For themes where QIO staff work with providers, mid-level staff would include those who
work directly with providers on theme-relevant improvements, while junior staff would take
supporting roles and senior staff would lead and manage the various efforts.
Percent

a. Senior level staff.....................................................................

|

|

|

|

b. Mid-level staff .........................................................................

|

|

|

|

c. Junior staff..............................................................................

|

|

|

|

Contract Goals and Objectives for the 9th SOW
5.

Please indicate if you agree with each of the following statements about the goals,
objectives and improvement targets of the 9th SOW contract related to this theme
(as modified at the time of this survey). For each, do you strongly agree, agree, disagree,
or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. CMS goals and objectives are clear..............

1

□

2

□

3

□

4

□

b. Resources for this theme are sufficient to
support goals.................................................

1

□

2

□

3

□

4

□

c. Improvement targets set by CMS for this
theme are attainable .....................................

1

□

2

□

3

□

4

□

d. Improvement targets set by CMS represent
meaningful improvements in care .................

1

□

2

□

3

□

4

□

e. The timeframe for meeting improvement
targets is reasonable .....................................

1

□

2

□

3

□

4

□

f.

Method for evaluating the QIO is clear..........

1

□

2

□

3

□

4

□

g. The QIO contract focuses effort on
important areas of quality ..............................

1

□

2

□

3

□

4

□

h. The QIO contract focuses effort on
providers whose improvements will have
substantial impact on quality in the state ......

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.226

Contract Support and Communication for the 9th SOW
6.

For each of the following, please indicate if their knowledge base relative to their
responsibilities was excellent, good, fair, or poor.

Excellent

Good

Fair

Poor

Not
Enough
Contact
to Tell

a. Government Task Leader..................

1

□

2

□

3

□

4

□

5

□

b. Government Theme Leader ..............

1

□

2

□

3

□

4

□

5

□

c. Contract Officer .................................

1

□

2

□

3

□

4

□

5

□

d. CMS Project Officer...........................

1

□

2

□

3

□

4

□

5

□

e. Other CMS personnel (Specify role
below) ..................................................

1

□

2

□

3

□

4

□

5

□

7.

Please indicate if you agree with each of the following statements about the support and
communication for the 9th SOW contract as it relates to this theme. For each, do you
strongly agree, agree, disagree, or strongly disagree?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The CMS Project Officer is supportive and
helpful............................................................

1

□

2

□

3

□

4

□

b. The CMS Project Officer understands the
QIO’s interventions........................................

1

□

2

□

3

□

4

□

c. Oral communication by CMS personnel is
clear ..............................................................

1

□

2

□

3

□

4

□

d. Contract modification(s) required little effort
to implement..................................................

1

□

2

□

3

□

4

□

e. Contract modifications improved the
contract .........................................................

1

□

2

□

3

□

4

□

8.

How often was the communication you had with CMS consistent across different CMS
personnel with whom you interfaced?
1
2
3
4

□
□
□
□

Always consistent
Usually consistent
Rarely consistent
Never consistent

Prepared by Mathematica Policy Research

C.227

SECTION II: SUFFICIENCY OF INFORMATION AND RESOURCES
9.

Thinking about designing effective interventions that are known to work, have you had
sufficient data and information to do the following during the 9th SOW?
Yes

No

N/A

a. Understand the problem the intervention is addressing ................

1

□

0

□

b. Enable design of intervention with high likelihood of success.......

1

□

0

□

c. Identify disparities related to this theme ........................................

1

□

0

□

n

□

d. Identify what interventions are working elsewhere........................

1

□

0

□

n

□

e. Adequately justify the intervention to providers and others ...........

1

□

0

□

n

□

10.

Think about the information you get that helps you shape and refine your intervention over
time so that it targets the right techniques to the right providers during the 9th SOW.
Please rate the value of the information you currently obtain from:
High
Value

a. Reports from the QIOSC that contain data
analysis ...........................................................

1

b. Conference calls convened by the QIOSC .....

1

c. Tools provided by the QIOSC (regardless of
source) ............................................................

1

d. Data from IFMC that the QIO analyzes itself ..

1

e. QualityNet (including MedQIC) .......................

1

f.

QualityNet conferences ...................................

1

□
□
□
□

g. Conferences sponsored by other
organizations ...................................................

1

□

h. Webinars or teleconferences sponsored by
other organizations..........................................

1

i.

Key websites (other than MedQIC) .................

1

j.

Personal contacts with other QIOs..................

1

□
□
□

k. Personal contacts with other health care
organizations ...................................................

1

1

l.

Other key information source (Please specify
below) ..............................................................

Prepared by Mathematica Policy Research

C.228

□
□

Medium
Value
2

2

□
□

2

□
□
□
□

2

□

2

2

2

2

□
□
□

□

2

□

2

2

2

Low
Value
3

3

□
□

3

□
□
□
□

3

□

3

3

3

3

□
□
□

□

3

□

3

3

3

Did Not
Use
0

0

□
□

0

□
□
□
□

0

□

0

0

0

0

□
□
□

□

0

□

□

0

□

0

0

SECTION III: TOOLS AND OTHER RESOURCES
11.

The following are statements about tools and other resources used to support
interventions related to this theme. To what extent do you agree with each of the
following?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. The tools and other resources that are
available to support interventions related to
this theme are of high quality ........................

1

□

2

□

3

□

4

□

b. The tools and other resources that are
available to support interventions related to
this theme were available when we needed
them ..............................................................

1

□

2

□

3

□

4

□

c. The tools and specifications that are
available to support measurement related to
this theme work well ......................................

1

□

2

□

3

□

4

□

12.

Has the QIO made substantial adaptations to available tools or other resources to support
interventions related to this theme?
1
0

□
□

Yes
No

GO TO Q.14

13.

Please identify the tools or other resource(s) and describe how they were adapted.

14.

Has the QIO had to create any tools or other resources to support interventions related to
this theme?
1
0

15.

□
□

Yes
No

GO TO Q.16

Please describe the tool(s) or other resource(s) created.

Prepared by Mathematica Policy Research

C.229

SECTION IV: ACTIVITIES
Collaborative Activities
16.

In Column A, for each of the following collaborative activities, indicate if it is a major or
minor component for this theme. If the activity does not occur, check the corresponding
“N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Forming new collaborations
among providers .................

1

□

2

□

n

□

1

□

2

□

3

□

b. Forming new collaborations
that include health
organizations other than
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

Contributing to existing
collaborations......................

1

□

2

□

n

□

1

□

2

□

3

□

d. Supporting a large
organization (such as a
health delivery
organization, or health
plan) in their efforts to
improve ...............................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.230

Interactions with Individual Providers
17.

In Column A, for each of the following activities that involved interactions with
individual providers, indicate if it is a major or minor component for this theme. If the
activity does not occur, check the corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Problem-solving or
strategizing with individual
providers at their request ....

1

□

2

□

n

□

1

□

2

□

3

□

b. Problem-solving or
strategizing with individual
providers during meetings
the QIO initiated ..................

1

□

2

□

n

□

1

□

2

□

3

□

Making presentations onsite at individual providers ..

1

□

2

□

n

□

1

□

2

□

3

□

d. Interacting with top
leadership of provider
organizations ......................

1

□

2

□

n

□

1

□

2

□

3

□

e. Helping integrate clinical
guidelines into health
information systems ............

1

□

2

□

n

□

1

□

2

□

3

□

Helping providers better
use their health information
systems to support QI .........

1

□

2

□

n

□

1

□

2

□

3

□

g. Discussing providers’ own
performance with them .......

1

□

2

□

n

□

1

□

2

□

3

□

h. Training staff within
provider organizations ........

1

□

2

□

n

□

1

□

2

□

3

□

c.

f.

Prepared by Mathematica Policy Research

C.231

One-to-Many Activities
18.

In Column A, for each of the following group education/meeting activities indicate if it
is a major or minor component for this theme. If the activity does not occur, check the
corresponding “N/A” box.
In Column B, for each activity that occurs, indicate the extent to which it is important to
improving quality or patient safety with regard to this theme. If an activity is “N/A” in
Column A, then skip Column B for that activity.
Column A

Column B
Importance of Activity

Major
Component

Minor
Component

N/A

Very
important

Somewhat
important

Not very
important

a. Providing one-to-many
educational or shared
learning sessions via
telephone ............................

1

□

2

□

n

□

1

□

2

□

3

□

b. Large regional or statewide
in-person meetings .............

1

□

2

□

n

□

1

□

2

□

3

□

Routinely providing
provider-specific data to
providers with benchmarks .

1

□

2

□

n

□

1

□

2

□

3

□

d. Notifying providers of
quality improvementrelated opportunities
sponsored by others ...........

1

□

2

□

n

□

1

□

2

□

3

□

e. Summarizing quality
improvement tips or
information in a QIO or
provider association
newsletter, in paper or
electronic format .................

1

□

2

□

n

□

1

□

2

□

3

□

c.

Prepared by Mathematica Policy Research

C.232

Business Case Focus
19.

In Column A, for the following activity, indicate if it is a major or minor component for this
theme. If the activity does not occur, check the corresponding “N/A” box.
In Column B, if this activity occurs, indicate the extent to which it is important to improving
quality with regard to this theme. If this activity is “N/A” in Column A, then skip Column B
for that activity.
Column A

Column B
Importance of Activity

Major
component

Minor
component

N/A

Very
important

Somewhat
important

Not very
important

a. Developing or

incorporating information
into materials, talks,
consultations, etc.
regarding the business
case for quality
improvement relevant to
this theme............................

1

□

2

□

n

□

1

□

2

□

3

□

Patient Safety
20.

How valuable were the annual in-person meetings sponsored by CMS specific to patient
safety?
1
2
3
4
5

21.

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Did not attend any annual in-person meetings sponsored by CMS

How valuable was the “change package” that CMS developed for this theme?
1
2
3
4
d

□
□
□
□
□

Very valuable
Valuable
Marginally valuable
Not valuable at all
Don’t know

Prepared by Mathematica Policy Research

C.233

SECTION V: STAFFING
22.

Think about the person most responsible for design of the intervention materials and
processes for this theme. For this person, please indicate his/her highest level of
education attainment, field of study, years of relevant QI experience, years of experience
working with the types of providers or organizations relevant to this theme, and
professional level (executive, senior, etc.).
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience:
|

d.

e.

|

| YEARS

Years of experience working with the types of providers or organizations relevant to
this theme:
|

|

| YEARS

Professional level:
1
2
3
4
5

23.

□
□
□
□
□
□
□

□
□
□
□
□

Executive
Senior
Mid-Level
Junior
Other (Please specify)

How many QIO staff have interacted directly, on a frequent basis, with providers and
collaborating organizations for this theme?
|

|

| ENTER NUMBER

Prepared by Mathematica Policy Research

C.234

24.

Think about the one or two people who have interacted most frequently with providers and
collaborating organizations for Surgical Care Improvement Project. For each, please
indicate his/her highest level of education attainment, field of study, and years of relevant
QI experience, and years of experience working with the types of providers or
organizations relevant to this theme.
PERSON #1
a.

Highest level of educational attainment:
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

|

| YEARS

PERSON #2
1
2
3
4
5
6
7

□
□
□
□
□
□
□

Some college
Associate’s degree
Bachelor’s degree
Master’s degree
Professional degree [MD, DPM, DO, PharmD, etc.]
Doctoral degree [EdD, PhD]
Other (Please specify)

b.

Field of study, if applicable:

c.

Years of relevant QI experience: |

d.

Years of experience working with the types of providers or organizations relevant to
this theme: | | | YEARS

Prepared by Mathematica Policy Research

|

| YEARS

C.235

25.

To what extent do you agree with the following statements about staffing for this theme?
Strongly
agree

Agree

Disagree

Strongly
disagree

a. QIO staff assigned to this theme have the
right substantive expertise and experience ...

1

□

2

□

3

□

4

□

b. An adequate number of QIO staff have
been available to perform work on this
theme ............................................................

1

□

2

□

3

□

4

□

c. The QIO has been able to retain key staff
working on this theme (that is, turnover has
not been a problem) ......................................

1

□

2

□

3

□

4

□

Prepared by Mathematica Policy Research

C.236

SECTION VI: IMPACT OF EXTERNAL FACTORS ON THIS THEME
Role of Provider, Professional Associations and/or State Agencies
26.

Which role(s) does the state agency most relevant to this theme play in assuring and
improving quality?

2

□
□

3

□

1

27.

Regulatory oversight
Actively engaged with others (such as the QIO and/or other
provider organizations) to foster quality improvements
Both

Please list up to two provider or professional associations that are stakeholders in this
theme and could potentially affect its success that have been most relevant to this theme.
1.
2.

28.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the first organization
listed at Question 27.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Prepared by Mathematica Policy Research

C.237

29.

The following are items that characterize the types of activities and/or relationships that
may exist between QIO’s and provider and/or professional associations.
Please answer each of the following questions with regard to the second
organization listed at Question 27. If only one organization was listed at
Question 27, go to Question 30.
Yes

No

a. The association employs at least one staff member with
major responsibility and time devoted to quality
improvement ........................................................................

1

□

0

□

b. The association and QIO talk periodically (e.g., quarterly)
to avoid overlap ....................................................................

1

□

0

□

c. QIO staff attend and speak at association-sponsored
meetings at least once per year ...........................................

1

□

0

□

d. The association sponsors a quality-focused entity, such as
a Quality Council or a Quality Institute .................................

1

□

0

□

e. The association and QIO work jointly on one or more
QI efforts substantial in scope (such as co-sponsoring
in-person meetings focused on QI) ......................................

1

□

0

□

The association tends to work with a different set of
providers than the QIO .........................................................

1

□

0

□

g. The association tends to work on QI projects that are
entirely different from the QIO ..............................................

1

□

0

□

h. The association primarily focuses on quality reporting
issues rather than quality improvement ...............................

1

□

0

□

f.

Role of Large Provider Organizations
The following questions apply to the state level.
30.

Do any large provider organizations that are relevant to this theme, such as health
systems, hospitals, or nursing home chains, exist in your state?
1
0

□
□

Yes
No

GO TO Q.34

Prepared by Mathematica Policy Research

C.238

31.

Please list up to two large health care delivery organizations in your state (such as an
integrated delivery system or dominant medical group) that are stakeholders in this theme
and have the greatest potential to affect its success.
1.
2.

32.

To what extent does the headquarters of the first organization listed in Question 31 drive
quality in owned or affiliated organizations in this state?
1
2
3
4
d

33.

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

To what extent does the headquarters of the second organization listed in Question 31
drive quality in owned or affiliated organizations in this state?
If only one provider was listed at Question 31, then go to Question 34.
1
2
3
4
d

□
□
□
□
□

A lot
A moderate amount
A little
Not at all
Don’t know

Other Important External Players
34.

Please list up to three other external organizations whose efforts are proving important to
achieving improvements on this theme. (Please spell out the full name of the
organization.)
1.
2.
3.

Prepared by Mathematica Policy Research

C.239

SECTION VI: QIO VIEWS ON QUALITY IMPROVEMENT
Motivation and Quality Improvement
35.

The following are statements which related to motivation and quality improvement. For
each statement, please indicate if you strongly agree, agree, disagree, or strongly
disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective.
Strongly
Strongly
agree
Agree
Disagree disagree

a. Senior leaders at providers care about their
quality performance related to this theme .....

1

□

2

□

3

□

4

□

b. Providers regularly review data (generated
internally or received from another
organization) on their performance related
to this theme..................................................

1

□

2

□

3

□

4

□

c. The “business case” for quality, when it is
clear, is a key motivator for improvement for
most providers...............................................

1

□

2

□

3

□

4

□

d. Providers perceive a strong business case
for quality improvement on the measures
important to this theme..................................

1

□

2

□

3

□

4

□

e. Ongoing pay-for-performance efforts are a
key motivation for quality improvements in
this state ........................................................

1

□

2

□

3

□

4

□

One or more large health plans “tiers”
providers in their network in ways that
consider their quality performance ................

1

□

2

□

3

□

4

□

g. Many providers lack motivation to improve ...

1

□

2

□

3

□

4

□

h. Motivational speakers (such as key IHI
personnel, a prominent physician, or
leading QI thinkers who are physicians) are
effective motivators for improvement ............

1

□

2

□

3

□

4

□

The number of physician champions is
adequate to help facilitate improvement on
key measures for this theme .........................

1

□

2

□

3

□

4

□

Public reporting is a key motivator for
improvement for most providers....................

1

□

2

□

3

□

4

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.240

If you responded “agree” or “strongly agree” to Question 35, item g, go to Question 36.
Otherwise, go to Question 37.
36.

What types of providers lack motivation to improve on this theme?
1
2
3
4

37.

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Does state-level public reporting relevant to this theme exist in this state?
1
0

□
□

Yes
No

Knowledge and Information and Quality Improvement
38.

The following are statements related to knowledge and information and quality
improvement. For each statement, please indicate if you strongly agree, agree, disagree,
or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Enough information exists for
providers suggesting how they should
improve, once they are motivated and
link (or are helped to link) to this
information .........................................

1

□

2

□

3

□

4

□

d

□

b. Once motivated, providers tend to
seek out and find good information on
how to improve ...................................

1

□

2

□

3

□

4

□

d

□

c. Providers have staff who are
educated or otherwise qualified to
support improvement efforts ..............

1

□

2

□

3

□

4

□

d

□

d. The limitations of provider information
systems remain a large barrier to
improvement ......................................

1

□

2

□

3

□

4

□

d

□

e. Workforce turnover is a large barrier
to improvement ..................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.241

If you responded “agree” or “strongly agree” to Question 38, item c, go to Question 39.
Otherwise, go to Question 40.
39.

What types of providers are not so well educated or qualified to support improvement
efforts?
1
2
3
4

□
□
□
□

Hospitals
Physician practices
Nursing homes
Home health agencies

Poor Performers and Quality Improvement
40.

The following are statements about poor performance and quality improvement. For each
statement, please indicate if you strongly agree, agree, disagree, or strongly disagree.
For each item, please attempt to answer thinking about your theme from a statewide
perspective (that is, thinking about all providers in the state, not just those you have been
working with).
Strongly
agree

Agree

Strongly
Disagree disagree

Don’t
know

a. Poorly performing providers often
have financial and management
problems ...........................................

1

□

2

□

3

□

4

□

d

□

b. Poor performance on a particular
measure is often associated with
physicians who disagree with the
relevant guideline ..............................

1

□

2

□

3

□

4

□

d

□

c. Poor performance on a particular
measure is often associated with
corporate chain managers who
disagree with or do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

d. Poor performance on a particular
measure is often associated with
physicians who do not believe in
establishing care routines based on
guidelines ..........................................

1

□

2

□

3

□

4

□

d

□

e. QIO-initiated help to the poorest
performers is/would be ineffective
because of their limited ability to use
the help given ....................................

1

□

2

□

3

□

4

□

d

□

Prepared by Mathematica Policy Research

C.242

SECTION VII: PARTICIPATING PROVIDERS’ SELECTION PROCESS
41.

Which of the following strategies did you use when you first began recruiting providers for
work under this theme?
MARK ALL THAT APPLY
1

□

Standardized announcement to eligible providers from the QIO

2

□

QIO initiated personalized contact with providers

3

□

Provider associations or other organizations outside the
QIO delivered announcement

4

□

Joint announcement made by QIO and provider associations
or other organizations outside the QIO

5

□

Other (Please specify)

Prepared by Mathematica Policy Research

C.243

42.

The following table contains characteristics of providers that you may have targeted when
you first began recruitments. For each one, first indicate if the QIO targeted it in
recruitment (Column A) and, if so, how successful the recruitment effort was (Column B).

□

If no specific characteristics were considered when targeting providers, check this box
and go to Question 43.
Column A

Provider Characteristics among
those Eligible to Participate

Yes,
targeted
during
recruitment

Column B
How successful were you in
recruiting providers with this
characteristic?

Very
Somewhat
Not
successful successful successful

a. No special characteristics sought –
just any provider who was eligible..

1

□

NA

b. Providers who had past
experience working with the QIO ...

1

□

1

□

2

□

0

□

c. Providers who did not have past
experience working with the QIO ...

1

□

1

□

2

□

0

□

d. Providers known to have
the organizational capacity to
improve ..........................................

1

□

1

□

2

□

0

□

e. Providers above average on the
targeted measures among eligible
providers ........................................

1

□

1

□

2

□

0

□

Providers below average among
eligible providers on the targeted
measures .......................................

1

□

1

□

2

□

0

□

g. Providers viewed as “early
adopters” ........................................

1

□

1

□

2

□

0

□

h. Providers viewed as leader
organizations by their peers ...........

1

□

1

□

2

□

0

□

Providers whose leadership had
higher-than-average commitment
to quality .........................................

1

□

1

□

2

□

0

□

Providers with other special
characteristics (Please specify) ........

1

□

1

□

2

□

0

□

f.

i.

j.

Prepared by Mathematica Policy Research

C.244

NA

NA

43.

Which of these statements best describes the response you received from your initial
recruitment efforts? Please consider both those who decided to participate and those who
asked questions but ultimately decided not to participate.
1
2
3

44.

□
□
□

More providers than needed expressed interest
About the right number of providers expressed interest
Too few providers expressed interest

Which of these statements best describes the level of effort needed to persuade enough
of the right types of providers to commit to participate?
1
2
3

□
□
□

It took a lot of effort to secure enough providers
It took a moderate amount of effort to secure enough providers
It took only a little bit of effort to secure enough provider

GO TO Q.46

45.

Please tell us what types of providers were the most difficult to persuade to participate?

46.

Were any providers so interested in participating that they lobbied the QIO to be sure they
would be included?
1
0

47.

□
□

Yes
No

GO TO Q.48

How many providers lobbied the QIO for participation?
1
2
3

□
□
□

1 to 2
3 to 4
5 or more

Prepared by Mathematica Policy Research

C.245

48.

What is the total number of participating providers at present?
________ NUMBER

49.

At the start of the effort, please estimate how many of the participating providers had:
Number

a. Worked with the QIO on 3 or more projects over the prior 5 years? ........

________

b. Worked with the QIO on at least 1 but not more than 3 projects in the
prior 5 years? ...........................................................................................

________

c. Not worked with the QIO before? .............................................................

________

50.

Of those who began participating with the QIO, please indicate the percent that
participated in the manner indicated below.

Manner of Participation

Percent of Participants

a. Participated actively throughout ..............................

|

|

|

|

b. Participated less over time ......................................

|

|

|

|

c. Intermittent participation ..........................................

|

|

|

|

d. Never participated very actively ..............................

|

|

|

|

TOTAL

Prepared by Mathematica Policy Research

C.246

100%

51.

Did the QIO exclude providers who expressed interest in participating, due to limited
resources?
1

□

Yes, many providers who expressed interest had to be excluded
GO TO Q.52

52.

2

□

Yes, a few providers who expressed interest had to be excluded

0

□

No, no providers who expressed interest had to be excluded

Thank you for
completing this
survey

Approximately how many interested providers were excluded due to limited resources
available through your 9th SOW contract?
____ NUMBER OF PROVIDERS EXCLUDED

Thank you for completing the QIO Theme Leader Survey
Please return your completed survey to the following:

Via mail:

Via fax:

Mathematica Policy Research
c/o Sarah Croake
Suite 550, 600 Maryland Ave., SW
Washington, DC 20024

Attn: Sarah Croake
202-863-1763
Via email attachment:
[email protected]

Prepared by Mathematica Policy Research

C.247