Supporting Statement A -- Development of a Health Information Rating System 9-18-2012

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions.

According to its authorizing legislation, AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Over the past several years, low health literacy has been identified as an important health care quality issue. Healthy People 2010 defined health literacy as ‘the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions’.^¹ In 2003, the Institute of Medicine identified health literacy as a cross-cutting area for health care quality improvement.^² According to the 2003 National Assessment of Adult Literacy, only 12 percent of adults have proficient health literacy.^³

Persons with limited health literacy face numerous health care challenges. They often have a poor understanding of basic medical vocabulary and health care concepts. A study of patients in a large public hospital showed that 26 percent did not understand when their next appointment was scheduled and 42 percent did not understand instructions to “take medication on an empty stomach”.^⁴ In addition, limited health literacy leads to more medication errors, more and longer hospital stays, and a generally higher level of illness, resulting in an estimated excess cost for the US health care system of $50 billion to $73 billion per year.^⁵

Health care providers can improve their patients’ health outcomes by delivering the right information at the right time in the right way to help patients prevent or manage chronic conditions such as diabetes, cardiovascular disease, hypertension, and asthma. Electronic health records (EHRs) can help providers offer patients the right information at the right time during office visits, by directly connecting patients to helpful resources on treatment and self-management. EHRs can also facilitate clinicians’ use of patient health education materials in the clinical encounter. However, health education materials delivered by EHRs, when available, are rarely written in a way that is understandable and actionable for patients with basic or below basic health literacy ― an estimated 77 million people in the United States.^⁶

In order to fulfill the promise of EHRs for all patients, especially for persons with limited health literacy, clinicians should have a method to determine how easy a health education material is for patients to understand and act on, have access to a library of easy-to-understand and actionable materials, understand the relevant capabilities and features of EHRs to provide effective patient education, and be made aware of these resources and information. Therefore, AHRQ developed a project that includes the following four major tasks: 1) develop a valid and reliable Health Information Rating System (HIRS), 2) create a library of patient health education materials, 3) review EHR’s patient education capabilities and features, and 4) educate EHR vendors and users. This project relates to the first task only; a description of the other tasks is provided in Attachment A.

As a first step, AHRQ has developed a draft HIRS (see Attachment B) using the following rigorous multi-stage approach that draws upon existing rating systems, the evidence base in the literature, and the real-world expertise and experience of a Technical Expert Panel (TEP):

Gather and synthesize evidence on existing rating systems and literature on consumers’ understanding of health information. Seek TEP review of the summary of existing health information rating systems. Develop item pool for each domain - understandability and actionability, defined as follows:

Health education materials are understandable when consumers of diverse backgrounds and varying degrees of health literacy can process and explain key messages.
Health education materials are actionable when consumers of diverse backgrounds and varying levels of health literacy can identify what they can do based on the information presented.

2) Assess the face and content validity of the domains (i.e., understandability and actionability) with the TEP.

3) Assess the inter-rater reliability of the HIRS on English-language health education materials. Seek TEP review of results and provide guidance on how to address discrepancies.

The draft HIRS was used by AHRQ researchers to rate 2 sets of patient health education materials: a set of 6 education materials related to asthma and a set of 6 education materials related to colonoscopy. Each of these 12 health education materials received a score for their understandability and actionability. Some of the materials received good scores on the draft HIRS, meaning that the researchers considered them to be understandable or actionable, and some materials received poor scores on the draft HIRS, indicating that the materials had low understandability or low actionability. The final stage of developing a reliable and valid rating system to assess the understandability and actionability of health education materials is testing with consumers.

This project has the following goals:

1) To assess the construct validity of AHRQ’s draft HIRS. The 12 rated health education materials will be tested with a total of 48 English-speaking consumers. Consumers will review materials and be asked questions to test whether they understand the materials and whether they know what actions to take. The outcome of this testing will be an HIRS that will offer professionals (e.g., clinicians, health librarians, etc.) a systematic method to evaluate and compare the understandability and actionability of health education materials. Since actionability is a new domain, we are testing it distinct from understandability though there is a theoretical relationship between the domains as we have defined them; that is, a material cannot be actionable if it is not first understandable. So actionability may in fact be a sub-domain of understandability. Besides assessing the construct validity, consumer testing will help us determine how to revise and improve the HIRS.

2) Finalize the HIRS and instructions for users, and make them publicly available on AHRQ’s website.

To achieve the goals of the project the following data collections and activities will be implemented:

Demographic Questionnaire – The demographic questionnaire will collect basic demographic information about each participant (see Attachment C). This data will allow the analysis to detect differences in health literacy by population subgroups.

Short Test of Functional Health Literacy in Adults (S-TOFHLA) Questionnaire – The S-TOFHLA will be administered once to all participants to assess their level of health literacy (see Attachment D).
Health Education Materials & Questionnaire – Asthma/Inhaler – This includes a set of educational materials related to asthma and proper use of inhalers. Each consumer will be randomly assigned one of the six following materials (hypertext includes links to the web-based educational materials):

an audiovisual material (understandable and actionable), titled How to use an inhaler by the Utah Department Health Asthma Program
an audiovisual material (understandable and poorly actionable), titled Asthma Triggers by Children’s Healthcare of Atlanta
an audiovisual material (poorly understandable), titled Asthma Inhaler Medication Technique-How to Take an Asthma Inhaler by America’s Allergist
a printable material (understandable and actionable), titled Asthma: How to Use A Metered Dose Inhaler, by FamilyDoctor.org
a printable material (understandable and poorly actionable), titled How to use an inhaler - no spacer, by MedlinePlus
a printable material (poorly understandable), titled Inhaled Asthma Medications: Tips to Remember, by the American Academy of Allergy Asthma & Immunology

After seeing the randomly assigned audiovisual or printable material the participants will be administered a brief questionnaire to assess their understanding of how to use an inhaler and what actions to take based on the material (see Attachment E).

Health Education Materials & Questionnaire - Colonoscopy - This includes a set of educational materials related to colonoscopy. Each consumer will be randomly assigned one of the six following materials (hypertext includes links to the web-based educational materials):

an audiovisual material (understandable and actionable), titled Colonoscopy Patient Education Video by Krames
an audiovisual material (understandable and poorly actionable), titled Colorectal Cancer Awareness by St. Vincent’s Healthcare
an audiovisual material (poorly understandable), titled Prepare for a Colonoscopy by The University of Texas MD Anderson Cancer Center
a printable material (understandable and actionable), titled Getting Ready for Your Colonoscopy by West Chester Endoscopy Suite
a printable material (understandable and poorly actionable), titled Colonoscopy in the National Digestive Diseases Information Clearinghouse (NDDIC)
a printable material (poorly understandable), titled Colonoscopy by the American College of Surgeons Division of Education

After viewing the randomly assigned audiovisual or printable material the participants will be administered a brief questionnaire to assess their understanding of a colonoscopy and what actions to take based on the material (see Attachment F).

This study is being conducted by AHRQ through its contractor, Abt Associates, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

A.2. Purpose and Use of the Information Collected

The data collected from this project will be used to assess the construct validity of and inform revisions to the HIRS. The HIRS will be the first instrument that can assess the understandability and actionability of patient health education materials that can be incorporated into an EHR, including printable and audiovisual materials. Note that the materials to be assessed need not currently be incorporated into EHRs; for now, we are focusing on materials that have the potential to be incorporated into EHRs.

No claim is made that the results from this study will be generalizable in the statistical sense. Rather, the consumer testing will be informative and critical to ensuring we have developed a valid rating system by conducting consumer testing.

A.3. Use of Improved Information Technology and Burden Reduction

The purpose of data collection for this project is to assess consumer understanding of print and audiovisual health education materials provided via EHRs. To the extent that audiovisual materials are administered via a computer and use multimedia, we will administer those materials to consumers using a computer with the required technologies. Consumer testing will not utilize information technology for data collection; rather data collection will involve in-person interviewer-administered questions to assess consumers’ understanding of the content and what actions to take based on the material. For qualitative analysis, we will use NVivo, a qualitative analysis software program.

A.4. Efforts to Identify Duplication and Use of Similar Information

An environmental scan of existing health information rating systems and a panel of experts in EHRs, health literacy and patient education confirmed that no existing systems assess both the understandability and actionability of patient health education materials, and no known system has endeavored to assess materials linked to EHRs. There are some tools for assessing the “readability” and suitability of print materials. ^⁷^,^⁸These assessment tools have been criticized as being subjective, pertaining to print material only, or lacking of evidence that materials rated as easier to read are in fact easier to understand and act upon. Therefore, data collected as part of this study will be unique because it will evaluate the validity of a newly developed health information rating system that assesses both understandability and actionability of print or printable materials and audiovisual materials that can be provided via EHRs.

A.5. Involvement of Small Businesses or other Small Entities

This project does not involve or impact any small entities.

A.6. Consequences if Information Collected Less Frequently

This project is a one-time data collection effort.

A.7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

A.8. Federal Register Notice and Outside Consultations

A.8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on April 2^nd, 2012 for 60 days, and again on October 5^th, 2012 for 30 days (see Attachment G).

A.8.b. Outside Consultations

None.

A.9. Payments/Gifts to Respondents

Consumer participants will be offered a $75 honorarium to compensate them for their interview time (approximately 1.2 hours per consumer).

A.10. Assurance of Confidentiality

Consumer participants will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.

Participants will be informed in the introduction to interviews that their answers will be kept confidential to the extent permitted by law, and prior to the testing informed consent will be obtained (see Attachment H).

Participation will be entirely voluntary, and the study will conform to the requirements of the Privacy Act by omitting individuals’ names, addresses, telephone numbers and other personal identifiers in the final data file.

The firm that will coordinate the data collection, Abt Associates, has conducted numerous projects and surveys involving sensitive information; consequently, facilities and procedures have been developed to maintain respondent confidentiality. All Abt Associates staff who are in contact with human subjects data are required to complete ethical training, which includes training about maintaining the confidentiality of information. Any databases created by Abt Associates will be password-protected, with only the data administrators having write-authority over files. If electronic data transfer is necessary, the data will be transferred in an encrypted and password-protected format via a secure FTP server or by diskette or CD-ROM shipped via a bonded courier.

A.11. Questions of a Sensitive Nature

The consumer testing data collection protocols do not contain any questions concerning sexual behavior and attitudes, religious beliefs, income or proprietary business information. However, data collection will include collecting basic demographic information on consumer participants including: gender, age (in a range), ethnicity, race, marital status, highest grade completed, and questions from the Short Test of Functional Health Literacy in Adults or S-TOFHLA (see Attachment D). In addition we will assess patient understanding of a variety of health education materials suited for different health literacy levels; participants may be asked questions they are unable to correctly answer. Participants will be explicitly informed that their participation is voluntary, information they provide is confidential to the extent provided by law, and they may choose to withdraw from the study or not respond to specific questions without penalty.

A.12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 presents estimates of the annualized burden hours for the participants’ time to participate in this research. The Demographic and S-TOFHLA questionnaires will be completed by all 48 participants and takes 5 and 7 minutes, respectively, to complete. Each of the 48 participants will review 2 different health education materials and then answer the associated questionnaires for each material topic. Participants will review English-language materials related to inhaler use and colonoscopy. To review each material and answer the associated questionnaire requires 30 minutes (15 minutes to review the materials and 15 minutes to complete the questionnaire). The total annualized burden hours are estimated to be 58 hours.

Exhibit 2 presents the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated at $1,237.

Exhibit 1. Estimated annualized burden hours

Data Collection	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Demographic Questionnaire	48	1	5/60	4
S-TOFHLA Questionnaire	48	1	7/60	6
Health Education Materials & Questionnaire – Inhaler	48	1	30/60	24
Health Education Materials & Questionnaire – Colonoscopy	48	1	30/60	24
Total	192	na	na	58

Exhibit 2. Estimated annualized cost burden

Data Collection	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Demographic Questionnaire	48	4	$21.35	$85
S-TOFHLA Questionnaire	48	6	$21.35	$128
Health Education Materials & Questionnaire – Inhaler	48	24	$21.35	$512
Health Education Materials & Questionnaire – Colonoscopy	48	24	$21.35	$512
Total	192	58	na	$1,237

* Based upon the mean wage for all occupations, National Compensation Survey: Occupational wages in the United States May 2010, “U.S. Department of Labor, Bureau of Labor Statistics.”

A.13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

A.14. Estimates of Annualized Cost to the Government

The total cost of this contract to the government is $524,945, and the project extends over 3 years (July 19, 2010 to July 18, 2013). The data collection for which we are seeking OMB clearance will take place from February 1, 2013 to March 31, 2013. Exhibit 3 shows a breakdown of the total cost as well as the annualized cost for the data collection, processing and analysis activity for this entire contract.

Exhibit 3. Estimated Cost

Cost Component	Total Cost	Annual Cost
Project Development	$66,447	$22,149
Data Collection Activities	$129,547	$43,182
Data Processing and Analysis	$129,548	$43,183
Publication of Results	$131,571	$43,857
Project Management	$67,832	$22,611
Total	$524,945	$174,982

A.15. Changes in Hour Burden

This is a new information collection effort.

A.16. Time Schedule, Publication and Analysis Plan

Time Schedule

The project timeline is shown in Exhibit 4 below.

Exhibit 4. Project Timeline

Activity	Time Schedule
Develop Health Information Rating System (HIRS) Stage 1: Gather and synthesize evidence on existing rating systems Stage 2: Assess face and content validity of HIRS with the TEP Stage 3: Assess the inter-rater reliability Stage 4: Test HIRS with diverse consumers Stage 5: Finalize system and instructions	Jan. 2011 – May 2013 Jan. 2011 – Apr. 2011 Apr. 2011 – June 2011 Dec. 2011 – Aug. 2012 Feb. 2013 – Mar. 2013 Mar. 2013 – May 2013
Health Information Rating System (HIRS) Report	May. 2013
Project Final Report	June 2013

Publication Plan

The Health Information Rating System (HIRS) will be posted on AHRQ’s website and disseminated to EHR vendors and users via government and industry websites, professional associations, and associated listservs. Results will also be disseminated through presentations at professional associations and other meetings (e.g., AHRQ Annual Meeting). Results will also be published in at least one peer-reviewed journal. The AHRQ Office of Communications and Knowledge Transfer (OCKT) and the National Resource Center Domain 3 contractors will also be engaged to assist with dissemination of results.

Analysis Plan

The purpose of this data collection is to assess the construct validity of the Health Information Rating System (HIRS) by conducting consumer testing with 48 consumers. Construct validity refers to whether an instrument measures the constructs (i.e., understandability and actionability) it purports to measure or the extent to which what was intended to be measured was actually measured. Therefore, for this project we will assess the construct validity of the HIRS by determining whether consumers have a better understanding of materials rated understandable on the HIRS compared with materials rated poorly understandable on the HIRS. Similarly to determine whether consumers better know what actions to take from materials rated actionable on the HIRS compared with materials rated poorly actionable on the HIRS. To assess the construct validity of the HIRS via consumer testing, we have developed a questionnaire, consisting of several open-ended questions and a few structured questions with scale response options.

For the open-ended questions we will qualitatively analyze the data. We will develop a coding scheme for each question using the content presented in the materials as potential response options, and then qualitatively assess the extent to which the participant was able to respond in his/her own words, for example. The coding will be completed by two trained qualitative researchers using NVivo, a qualitative analysis software program. We will then qualitatively compare the results (i.e., analyzed responses to open-ended questions) for each domain (i.e., understandability and actionability) by material, material topic (i.e., inhaler, colonoscopy), material modality (i.e., printable and audiovisual materials) and for the quality of materials for each domain (i.e., understandable and poorly understandable; actionable and poorly actionable). Exhibit 5 provides a table of hypothetical examples of the qualitative results we could have from analyzing the open-ended questions.

Exhibit 5. Hypothetical Examples of Consumer Testing Qualitative Results

Understandability

After viewing or reading understandable materials, consumers were better able to describe in their own words what happens during a colonoscopy, as compared to consumers who viewed or read poorly understandable materials

Consumers identified more information as difficult to understand for poorly understandable materials compared to understandable materials

Actionability

After viewing/reading actionable materials, consumers were readily able to explain how to prepare for a colonoscopy/use an inhaler as compared with consumers who viewed/read poorly understandable materials

For poorly actionable materials, the majority of consumers especially for printable materials, identified at least one thing that was unclear about how to use an inhaler properly.

For the structured questions with scale response options, we will calculate the descriptive statistics (i.e., frequency distribution; mean, median and mode) for each domain (i.e., understandability and actionability) by material, material topic, material modality (i.e., printable and audiovisual materials) and a total for all observations for the quality of materials for each domain (i.e., understandable and poorly understandable; actionable and poorly actionable). We will examine the differences on the descriptive statistics (e.g., mean) between the levels of quality to determine whether consumers have a better understanding of materials rated understandable compared with materials rated poorly understandable, for example. Exhibit 6 provides a table of hypothetical examples of the quantitative results (summary tables) we could have from analyzing the structured questions with scale response options. Since the consumers are assigned the health education materials at random, we may also use a statistical test to examine whether some of the smaller differences are statistically significant.

Exhibit 6. Hypothetical Examples of Consumer Testing Quantitative Results

	Mean Scores
Understandability	Understandable	Poorly Understandable
“How easy was this material to understand?” (1=very difficult; 10=very easy)
All materials	8.4	3.8
All Audiovisual materials	7.8	3.0
All Printable materials	9.0	4.6
Inhaler/Asthma materials	8.2	3.6
Colonoscopy materials	8.6	4.0
Actionability	Actionable	Poorly Actionable
“How well did this material identify the actions or steps you need to take to use an inhaler properly?” (1=very poor; 10=very good)
All materials	8.8	3.2
All Audiovisual materials	8.4	3.4
All Printable materials	9.2	3.0
Inhaler/Asthma materials	8.7	2.9
Colonoscopy materials	8.9	3.5

A.17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A – Description of All Project Tasks

Attachment B – Health Information Rating System (HIRS)
Attachment C – Demographic Questionnaire

Attachment D – Short Test of Functional Health Literacy in Adults (S-TOFHLA)

Attachment E – Health Education Materials & Questionnaire – Asthma/Inhaler

Attachment F – Health Education Materials & Questionnaire – Colonoscopy

Attachment G – Federal Register Notice

Attachment H – Consent Form

1 U.S. Department of Health and Human Services. Healthy People 2010. Retrieved from http://www.healthypeople.gov/2010/default.htm

2 Institute of Medicine. (2004) Health Literacy: A Prescription to End Confusion. Washington, DC: National Academies Press.

3 National Center for Education Statistics. (2003). National assessment of adult literacy. Retrieved from http://nces.ed.gov/naal/.

4 Williams, M. et al., “Inadequate Functional Health Literacy among Patients at Two Public Hospitals,” Journal of the American Medical Association. 274, no. 21 (1995): 1677–1682.

5 Friedland, R.B. 1988. Understanding Health Literacy: New Estimates of the Costs of Inadequate Health Literacy. Washington, DC: National Academy on an Aging Society.

6 National Assessment of Adult Literacy (NAAL). (2003). Retrieved from: http://health.gov/communication/literacy/issuebrief/#survey

7 McGee J. 2010. Toolkit for Making Written Material Clear and Effective. Part 7: Using readability formulas. Accessed at: https://www.cms.gov/WrittenMaterialsToolkit/Downloads/ToolkitPart07.pdf

8 Doak L, Doak C, Root J. 1993. Suitability Assessment of Materials for evaluation of health-related information for adults. Accessed at: http://aspiruslibrary.org/literacy/SAM.pdf

File Type	application/msword
Author	SubramanianR
Last Modified By	DHHS
File Modified	2012-10-05
File Created	2012-09-18