OCSQ Replies to HLQAT OMB Questions
April 15, 2009
OMB comment #1
Page 3 of Supporting Statement part A (SSA) says that the HRSA report showed that the most important drivers of quality improvement include 1) established goals, 2) sufficient resources, and 3) monitoring progress. Other factors included a multidisciplinary approach, non-hierarchical approach to care, evidence-based care, and management of performance outcomes. Will the QIOs be requiring hospitals to demonstrate that these factors are in place?
CMS response
While there is no requirement that hospitals establish goals, allocate sufficient resources, or monitor progress, the intention of utilizing the HLQAT tool is to assist QIOs in facilitating these critical conversations with leadership to garner results in the 9th SOW Patient Safety Metrics. Strong systems lead to strong outcomes. The HLQAT is simply one tool that can assist the QIO in making an assessment on what and where to apply their interventions. The premise, which is supported by the HRSA report is that by making hospital leadership aware of where they are on patient safety that it will be easier for QIOs to assist in the removal of barriers and increase overall acceptance of the clinical practices that QIOs seek to hardwire in the hospitals with which they are working.
OMB comment #2
Page 4 of the SSA says that QIOs will be required to obtain agreement from executive leadership of specific hospitals to participate in the Patient Safety improvement efforts, including the administration of the HLQAT. However, SSA also says that participation of the hospital in HLQAT is voluntary and that QIO’s only requirement is to report whether or not the hospital takes the survey. Please clarify what actually happens if the hospital does not take the survey. What happens to the hospital and what happens to the QIO?
CMS response
The hospital can choose not to take the survey without any concern of sanction from the QIO or CMS. A part of the QIO contract is to encourage hospitals to participate in the survey. As noted in the answer above, if a hospital agrees to work with the QIO, this agreement establishes a sense of commitment to mutual goals on the part of the hospital and the QIO of overall quality improvement and patient safety. If a large number of hospitals in the jurisdiction of a QIO did not take the survey, CMS would question the ability of the QIO to establish such trusting relationships.
OMB comment #3
Page 5 of the SSA does not specify what QIOs do with the survey results, if anything. It seems that the QIOs will encourage hospitals to take HLQAT, but it is unclear whether hospitals will be required or encouraged to do anything with the results. Will the hospitals be required to do anything with the results, other than be encouraged to share them with the state QIOs? And if not, is there evidence that the HLQAT alone has led to quality improvement? Please provide citations and a summary of results.
CMS response
Page 5 of the SSA states, “The QIOs will encourage hospitals to release their scores back to the state QIO in order to achieve a deeper analysis of the relationship between leadership and quality and to receive technical assistance for improvement. The QIO will neither forward the data nor aggregate it with other hospitals.”
The HLQAT alone will not likely produce results however, the hospital self assessment, regular conversations with the QIO and effective practices being implemented by the QIO will produce measurable results. Survey results will not be collected by CMS, however, the results are intended to be utilized by the QIO as part of their assessment for how and with whom to share practices related to patient safety.
The HLQAT development group conducted a pilot test of the survey to determine the instrument's psychometric properties: item response variability, the factor structure of the survey’s a priori leadership dimensions, and the internal consistency reliability of the dimensions. The other purpose of the pilot test was to examine the association between hospital leadership attributes and hospital performance by comparing the high-performing hospitals with low performers.
The sampling plan document dated April 7, 2008 (prepared by Westat) identified the need to collect pilot data from 36 hospitals and 504 respondents to have enough data to examine the psychometric properties of the HLQAT at the individual level, as well as at the hospital level.
The development group randomly selected 200 hospitals and 58 hospitals participated and completed the on-line survey. The response rate was 29% and 939 individual respondents filled out the survey. The minimum sample size requirement was met for this study.
A set of rigorous psychometric analyses was conducted on all items and dimensions. The tool was revised based on the psychometric analysis results with high reliability and satisfied factor loadings.
The association between HLQAT score and hospital quality score was also examined by comparing the HLQAT score between high and low performers.
Five dimensions showed significant difference in HLQAT score between high and low performers.
This is pilot data and has not been released in a public manner.
OMB follow up: This sounds like “information collection” under the PRA. What is the OMB control number that CMS used to survey the 58 hospitals/939 individual respondents?
OMB comment #4
SSA page 5: the respondents to the survey are hospital leaders. How credible are the results if the leaders are being asked to rate themselves? Are lower-level providers (e.g. hospital interns/residents, nurses, other ancillary staff) also being asked to take the survey? If not, why not?
CMS response
In the 9th SOW, for hospitals, two surveys are to be used as part of the assessment process. The HLQAT is specifically focused on hospital leadership. The AHRQ Hospital Patient Safety Survey is geared towards those individuals who work at the patient care level.
OMB comment #5
SSA page 6: the information provided on A10 should be provided to the respondents for this survey so that they understand how their information will be used and what level of privacy they are being provided. Please provide this information on the front page of the HLQAT instrument.
CMS response
This will be done.
OMB follow up: Please make sure to do this for each IC going forward.
OMB comment #6
SSA page 7: what are the burden estimates based on?
CMS response
The burden estimates are based upon preliminary pilot tests done while the instrument was being refined.
OMB comment #7
SSA page 7: the costs do not seem to match up. Page 7 says that the total cost for taking the survey is $2,684,672. Later in the page, the SS says the cost is $1,538,086. Please clarify.
CMS response
The total cost for taking the survey one time is shown in Table 1, as $1,538,086. Other figures are typographical errors and we thank OMB for this correction.
OMB comment #8
SSA page 8: the SS says the total cost to the government is $117,495. However, ROCIS says the total cost will be $95,745. Please clarify.
CMS response
This item was changed at the OS level in ICRAS. We provided an itemization of the cost in the supporting statement by round. Round 1 of the information collection will cost $95,745. Round 2 of the information collection will cost $21,750. The total cost to the government is $117,495 ($95,745 + $21,750).
OMB comment #9
SSA page 8: the response to A16 does not answer the question as to whether, where, and how the data from this survey will be published. Please elaborate.
CMS response
Page 8 states, “CMS expect yes/no results from the survey to be tabulated through July of 2011, or the end of the 9th SOW or when directed by CMS.” The tabulated results will be reported to CMS by the survey contractor as a part of their regular reporting mechanisms. The results will be a part of the survey contractor final report to CMS. CMS does not anticipate the survey contractor reports will be published.
OMB follow up: Please make sure to make this explicit in future supporting statements.
OMB comment #10
SSA page 9: the response to A17 does not answer the question as to when data collection will be completed and whether the expiration date of the OMB control number will be published on the survey instrument. The expiration date does not appear on the draft of the survey that was submitted. Please clarify. Statute requires the expiration date to be printed on the survey materials. Therefore, if the expiration date will not be printed on the survey, a rationale must be provided as to why this is not feasible.
CMS response
Page 9 says, “This is a 9th SOW survey. The 9th SOW ends July 31, 2011.” All work on the 9th SOW, including work on this survey must be complete by the end of the 9th SOW, July 31, 2011.
OMB follow up: Will QIOs be collecting data until July 31, 2011? When will data collection end and data analysis begin?
| File Type | application/msword |
| File Title | HLQAT OMB Responses |
| Author | CMS |
| Last Modified By | Matsuoka_k |
| File Modified | 2009-05-05 |
| File Created | 2009-05-05 |