Supporting Statement A --Use of Deliberative Methods to Enhance Public Engagement 6-14-2012

SUPPORTING STATEMENT

Part A

Use of Deliberative Methods to Enhance Public Engagement in the Agency for Healthcare Research and Quality’s (AHRQ’s) Effective Healthcare (EHC) Program and Comparative Effectiveness Research (CER) Enterprise

June 15, 2012

Agency for Healthcare Research and Quality (AHRQ)

Table of Contents

A. Justification 1

1. Circumstances Making the Collection of Information Necessary 1

2. Purpose and Use of the Information Collection 6

3. Use of Information Technology and Burden Reduction 7

4. Efforts to Identify Duplication and Use of Similar Information 7

5. Impact on Small Businesses or Other Small Entities 8

6. Consequences of Collecting the Information Less Frequently 8

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 8

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 8

9. Explanation of Any Payment of Gift to Respondents 9

10. Assurance of Confidentiality Provided to Respondents 10

11. Justification for Sensitive Questions 10

12. Estimates of Hour Burden Including Annualized Hourly Costs 11

13. Estimates of Annualized Respondent Capital and Maintenance Costs 13

14. Annualized Cost to the Federal Government 13

15. Explanation for Program Changes or Adjustments 13

16. Time Schedule, Publication and Analysis Plan 13

17. Reason(s) Display of OMB Expiration Date is Inappropriate 16

List of Attachments 16

References 18

A. Justification

1. Circumstances Making the Collection of Information Necessary

Public input is increasingly important to assure the future relevance and success of AHRQ’s Effective Health Care enterprise -- for developing the research programs that address priority health care concerns and disseminating evidence in ways acceptable and useful to the public. Although stakeholder engagement has been central to the Effective Healthcare (EHC) program to date, public input has not traditionally been used to inform and guide broad strategies related to the use of evidence to inform decisions.

With this project, AHRQ will generate evidence on the effectiveness and efficiency of four distinct methods of public deliberation and obtain public input on questions related to the conduct and use of comparative effectiveness research (CER). In addition to providing information on a topic central to the Agency’s mission, this project closely ties to AHRQ’s efforts to improve the rigor of research methods, as it will generate methodological evidence through a randomized controlled experiment to find the most effective and efficient deliberative approaches.

This project contributes to the fulfillment of AHRQ’s mission as set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf). AHRQ’s mission is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions.

AHRQ’s statutory authority states that AHRQ shall:

1. promote health care quality improvement by conducting and supporting both research that develops and presents scientific evidence regarding all aspects of health care and the synthesis and dissemination of available scientific evidence for use by policymakers, among others, and

2. conduct and support research, provide technical assistance, and disseminate information on healthcare and on systems for the delivery of such care.

See 42 U.S.C. 299(b)(1)(A), (D), (F), and (G); 42 U.S.C. 299(b)(2); 42 U.S.C. 299a(a)(1) – (4).

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Public deliberation methods are of interest to the Agency because of their focus on obtaining informed perspectives on complex topics similar to those that arise frequently with respect to healthcare and health research decision. Public deliberation convenes members of the public to learn about and discuss a complicated, values-laden issue that cannot be resolved with technical information alone. Its goal is to obtain informed input and meaningful insights regarding the way people think about a topic. In deliberative sessions, participants are provided with information that is intentionally neutral and respectful of the full range of underlying values and experiences. They have the opportunity to debate, learn about others’ views, and refine their own views. Thus, information obtained through public deliberation differs from that collected through surveys or focus groups, which are often designed to obtain more “top of mind” responses and reactions.

Public deliberation includes three core elements:

1. Convening a group of people (either in person or via online technologies to connect people in remote locations),

2. Educating the participants on the relevant issue(s) through educational materials and/or the use of content experts, and

3. Having the participants engage in a reason-based discussion, or deliberation, on all sides of the issue(s).

Several distinct deliberative methods have been developed and used previously. They share the three core elements of public deliberation listed above but differ on key features of implementation such as burden, whether they take place in-person or online, and the use of content experts. Although there is considerable theoretical and case study literature endorsing the value of public deliberation, there has been little empirical research into its effectiveness (Community Forum Deliberative Methods Literature Review, 2010).

AHRQ has two primary objectives with this study:

1. Obtain evidence to guide the Agency’s use of deliberative methods to obtain informed public input.  In this study, AHRQ will first evaluate whether public deliberation is an effective way to obtain informed public input to inform U.S. health care research. Second, AHRQ seeks to identify a feasible set of choices among deliberative methods, which will allow us to use the most efficient methods for reaching a given audience.

 

2. To inform AHRQ research programs and strategies, gather public views regarding how evidence of the effectiveness of medical interventions should be used.  AHRQ has a mission as well as a Congressional mandate to conduct comparative effectiveness research, generating evidence for clinicians, patients, and others to inform medical decisions.  This project will obtain public input on complex, values-based questions underlying beliefs about the appropriate and acceptable ways to use evidence, which will improve the impact of AHRQ’s comparative effectiveness research programs.

To achieve the objectives of this study the following activities and data collection will be implemented:

1) Participant recruitment – A short telephone screening questionnaire, including a brief overview of the study, will be used to recruit persons for the study (see Attachment A - Recruitment and Consent Materials).

2) Educational Materials – Educational materials are designed to inform participants about the topics that are being deliberated and will be provided to all 1,680 participants recruited before the implementation of any of the methods (see Attachment B - Educational Materials), but after the administration of the Knowledge and Attitudes Pre-test Survey (see Attachment C - Knowledge and Attitudes Pre-test Survey). Additional content provided during the deliberative method sessions includes a brief overview of the study and the background materials needed by participants to competently deliberate the issues. Educational materials to be used during the sessions will be sent to participants before the sessions (but after administration of the pre-test); however, participants are not required to read them to participate.

3) Deliberative Discussion Groups and Control Group – The purpose of the discussion groups is to obtain informed and deliberative input from lay people on an important set of issues underlying health care research. Participants will be randomly assigned to one of the four deliberative methods or a control condition. The four methods were selected because they have been previously implemented and vary on key features that may affect the scalability and effectiveness of the methods, including: burden (from two hours to 24 hours), mode of implementation (online versus in-person), role of content experts, and time between sessions allowing participants to reflect, seek additional information on the issues, and communicate informally with other participants.

The subject of the deliberations is the use of research evidence in healthcare decision-making. This deliberative topic encompasses several themes or “variations” that will be elaborated in the deliberations:

1. Use of evidence to encourage better healthcare: Is evidence useful (or, what kind of evidence is useful) to a physician and a patient who are considering a test or treatment that has been found to be ineffective, less effective than another, riskier than another, or for which effectiveness has not been demonstrated?

2. Use of evidence to encourage better value: Is evidence useful (or, what kind of evidence is useful)to a physician and a patient who are considering a test or treatment that is effective even though an equally effective but less expensive alternative is available?

3. Decision-making when evidence shows more complex trade-offs: Is evidence useful (or, what kind of evidence is useful) in treatment decisions that involve the balancing of effectiveness, risk, and value?

The issues involved in each variation will be discussed in the context of specific comparative effectiveness research (CER) examples. These “case studies” illustrate the issues and will be used to promote deliberation about the issues and the values employed by participants in the deliberations. The case studies and other educational materials are described in Attachment B: Educational Materials.

Exhibit 1 presents a summary of the characteristics of the four deliberative methods included in the experiment, followed by a detailed description of each method and the control group.

Exhibit 1. Characteristics of Four Deliberative Methods

Deliberative Method	Mode	Number of Sessions	Length	Comments on relative burden
Brief Citizens’ Deliberation (BCD)	In person	1 session	2 hours	Low burden – shortest total time, single trip to participate
Online Deliberative Polling® (ODP)	Online (synchronous communication)	4 sessions	5 hours – (1.25 hours for each online session)	Low burden --participation online, usually from home.
Community Deliberation (CD)	In person sessions, Online between sessions (asynchronous communication)	2 sessions	6 hours – (includes 2.5 hours for each in-person session + 1 hour online)	Higher burden – two separate trips to participate; additional time online over the intervening week
Citizens’ Panel (CP)	In person	3 sessions	24 hours -- (includes 8 hours for each of the 3 in-person sessions)	Highest burden – three separate trips to participate; days are consecutive

Brief Citizens’ Deliberation (BCD): Taking place in one 2-hour session, this lowest burden method combines the brevity of a focus group with purposeful and structured deliberation. The group is convened in-person (i.e., participants engage with each other face-to-face). Participants have the opportunity to learn about issues and perspectives relevant to the deliberative topic at the beginning of the session and participate in structured discussions throughout, sometimes in small discussion groups. The particular appeal of this method is the relatively small time commitment required of participants (see Attachment D1).

Online Deliberative Polling^® (ODP): A low burden method, ODP requires four 1.25-hour synchronous Internet sessions, in which participants can participate from home, conducted over a four-week period (for a total of 5 hours of participation). Participants engage in moderated discussions via Internet-based audio conferencing. In two of the sessions, questions generated by the participants are posed to several experts in a virtual plenary session (see Attachment D2).

Community Deliberation (CD, formerly referred to as Interrupted Deliberation): A higher burden method, CD requires participants to attend two in-person 2.5-hour sessions, separated by one week. At home between the two in-person sessions, participants are asked to review materials on a website provided by the research team, engage in Internet-based asynchronous discussions with other participants, and pose questions to the experts. Technological features needed to support CD are discussed in more detail below (see Attachment D3).

Citizens’ Panel (CP): The CP method is the highest burden method in the study, capturing the “gold standard” features of deliberation. Based on the citizens’ council and citizens’ jury models, this method differs from the other three deliberative methods in its depth and breadth. It provides for greater exploration of participants’ views and beliefs, expressed through in-depth face-to-face deliberation over the course of three consecutive days (24 hours). The CP is intended to ensure that information, time, scrutiny, deliberation, and independence (Coote & Lenaghan, 1997) are part of the method to a substantial degree, lending itself to a more robust investigation of the topic. The CP includes educational materials, expert testimony, small-group exercises and deliberation, questioning of expert witnesses, and further deliberation (see Attachment D4).

Control. Participants assigned to the control condition will receive educational materials (see Attachment B - Educational Materials) on the deliberative issues via an email link and will complete the Knowledge and Attitudes Pre/Post-test surveys but will not convene in groups to deliberate.

4) Knowledge and Attitudes Pre-test Survey – This survey will measure knowledge of and attitudes about the health issues discussed in the deliberations (see Attachment C - Knowledge and Attitudes Pre-test Survey). It will be administered online to deliberation participants and controls before educational materials are sent or the methods are implemented.

As described, study participants will be provided with educational materials related to the deliberative topic. In order to assess whether or not participants are sufficiently informed on the topics addressed in the materials during the intervention, the Knowledge and Attitudes Survey contains items assessing knowledge of medical research and medical evidence, of comparative effectiveness research, and of healthcare costs. The attitude questions refer to the use of medical evidence in healthcare decision making. They include attitudes about health care decision-making when research findings can provide no support for, or conflict with patient and doctor preferences for particular treatments.

The survey will gather demographic and other information necessary to characterize the study sample, test the success of the randomization, and define population subgroups for which variation in outcomes will be examined. The demographic variables also will be used to control for participant and group characteristics that may influence the outcomes. Although the design involves randomization that should balance these characteristics across groups, including them in the statistical models guards against inadequate results from randomization.

The variables to be measured in the Knowledge and Attitudes Pre-test Survey include:

• Sociodemographic characteristics: gender, age, marital status, education, employment status, household income, race/ethnicity, priority population, languages spoken (in addition to English)

• General health status

• Recent experience with the healthcare system (e.g., seeing a healthcare provider more than three times for the same condition in the last 12 months)

• Health insurance coverage

5) Knowledge and Attitudes Post-test Survey – This survey will measure knowledge of and attitudes about the issues discussed in the deliberations after the deliberations take place (see Attachment E - Knowledge and Attitudes Post-test Survey). It will be administered to deliberation participants and controls within one week following the conclusion of the deliberative methods and will include the same knowledge and attitude questions as the pre-test survey.

6) Deliberative Experience Survey – As described above, the four deliberative methods being tested vary in terms of burden, mode, use of educational materials, and time between deliberative sessions. A one-time survey will be administered to participants in the deliberative methods after implementation of the experimental conditions to capture their experience, including measures of their perceptions about participation and elements of the deliberative process (see Attachment F - Deliberative Experience Survey). In particular, levels of discourse quality and implementation quality achieved will be assessed. Using multi-item scales, the survey will measure the following:

Discourse quality

• Equal participation in the discussions

• Respect for others’ opinions and tolerance of differing perspectives

• Appreciation of perspectives other than their own

• Reasoned justification of ideas: sharing the reasoning or rationale for positions, opinions, beliefs, or preferences

Implementation quality

• Quality of group facilitation

• Quality of the educational materials provided

• Quality of the experts

• Transparency of the process and use of the results

• Participants’ perceived value of method

• Participants’ view of the influence the results will have on research

This study is being conducted by AHRQ through its contractor, the American Institutes of Research (AIR), pursuant to AHRQ’s statutory authority referenced on page 1 of this document. See 42 U.S.C. 299(b)(1)(A), (D), (F), and (G); 42 U.S.C. 299(b)(2); 42 U.S.C. 299a(a)(1) – (4).

2. Purpose and Use of the Information Collection

Information collection will entail transcript review and quantitative surveys. The information from these sources will be used to describe and summarize the input obtained from the participants in the deliberative sessions and present it in reports for AHRQ.

In the proposed study, in which participants will be randomly assigned to alternative deliberative methods and a control group, the information from the surveys will also be used to expand the evidence base for public deliberation by measuring effectiveness of these methods as compared to a typical education approach – distributing educational materials without follow-up deliberation. Effectiveness will be determined in terms of changes in knowledge and attitudes and the participants’ self-rated quality of experience. (The latter includes measures relevant to comparisons between deliberative methods, such as the extent to participants felt all had an opportunity to speak and believed their perspective was heard.)

The analyses described above provide AHRQ with the effectiveness of each method in terms of knowledge, attitudes, and experience. Further, the research team will track the hours and dollars required to deliver each method.  Comparing high burden, high cost methods to low burden, low cost methods will enable us to assess differences in terms of effectiveness relative to participant burden (e.g., total hours, required in-person attendance) and cost.

3. Use of Information Technology and Burden Reduction

The experiment involves testing one method that takes place entirely online: ODP. ODP was developed specifically to reduce participant burden by allowing participants to join remotely and avoid travel. If shown to be effective in the experiment, it will demonstrate a less burdensome method for future implementations of public deliberation.

Another method, CD, uses an online format to allow participants to view the educational materials and communicate asynchronously, at their leisure, with other participants and the content experts between the in-person deliberative sessions.

The pre-test and post-test Knowledge and Attitude surveys will be administered online. Use of an online system, instead of a paper-based or phone-based survey makes completing and submitting the surveys less time-consuming and more convenient for respondents. It can also reduce coding errors, since codes will be automatically linked to the responses.

The use of audio and video recordings and transcript review for qualitative data collection minimizes burden to participants because no additional data collection from subjects is required to report the public input on the issues discussed.

4. Efforts to Identify Duplication and Use of Similar Information

The information to be collected is unique to this project, although public deliberation is not. A recent literature review conducted to inform the study design did not reveal any instance, within the U.S. of the use of deliberation on the issues that are the focus of this project, nor did it identify an evaluation of deliberative methods of such scope (Community Forum Deliberative Methods Literature Review). Likewise, interviews with experts in the fields related to public deliberation and CER and a convened technical expert panel did not reveal any collection of information similar to what is planned for this study.

5. Impact on Small Businesses or Other Small Entities

There will be no collection of information from small businesses or other small entities.

6. Consequences of Collecting the Information Less Frequently

This is a one-time study based on a single time period for data collection.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This request is consistent with the general information collection guidelines of 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on December 1^st, 2011, FRN Vol. 76, No. 231, page 74785-74788, for 60 days (see Attachment H). One comment was received; see Attachment I for the comment and Attachment J for AHRQ’s response.

AHRQ has consulted with staff of American Institutes for Research (AIR), the Center for Healthcare Decisions, the Center for Deliberative Democracy at Stanford University, the Symbolic Systems Group of Stanford University, and Marthe Gold, Shoshanna Sofaer, Ela Pathak-Sen, and Stirling Bryan. AIR staff are experts in study design and quantitative and qualitative methodology. Stanford University has developed the software platform (Deme) to be used in the methods with online components.

Individuals Consulted on Study Design and Implementation

Name (Affiliation)	Telephone Number	Email
Johannes Bos (AIR)	650-843-8110	[email protected]
Kristin Carman (AIR)	202-403-5090	[email protected]
Kirsten Firminger (AIR)	919-918-4507	[email protected]
Steven Garfinkel (AIR)	919-918-2306	[email protected]
Dierdre Gilmore (AIR)	650-843-8139	[email protected]
Jessica Waddell Heeringa (AIR)	202-403-5947	[email protected]
Susan Heil (AIR)	301-592-2227	[email protected]
Maureen Maurer (AIR)	919-918-2308	[email protected]
Marilyn Moon (AIR)	301-592-2101	[email protected]
HarmoniJoie Noel	202-403-5779	[email protected]
Grace Wang (AIR)	650-843-8191	[email protected]
Amy Windham (AIR)	301-592-2165	[email protected]
Manshu Yang (AIR)	919-918-2312	[email protected]
Stirling Bryan (consultant)	604-875-4776 (Canada)	[email protected]
Todd Davies (Symbolic Systems Program, Stanford)	650-723-4091	[email protected]
James Fishkin (Center for Deliberative Democracy, Stanford)	650-723-4611	[email protected]
Marge Ginsburg (Center for Healthcare Decisions)	916-851-2828	[email protected]
Marthe Gold (consultant)	212-650-7794	[email protected]
Ela Pathak-Sen (consultant)	0-145-222-6206 (UK)	[email protected]
Alice Siu (Center for Deliberative Democracy, Stanford)	650-724-1301	[email protected]
Shoshanna Sofaer (consultant)	646-660-6815	[email protected]

9. Explanation of Any Payment or Gift to Respondents

Payments are necessary to compensate participants for transportation costs and lost opportunity costs (i.e., costs associated with other uses of time that must be foregone to allow participation in the study) (Krueger, 2008, p. 77). AIR’s Institutional Review Board (IRB) reviews all payments to ensure that the amount is not so large as to be coercive.

Payments will be prorated based on three steps. First, $25 for the completion of both the Knowledge and Attitudes Pre/post-test Survey. Second, an additional payment of $75 to $325 will be made for participation in the deliberations and completion of the Deliberative Experience Survey that will be administered at the conclusion of the deliberative sessions. The amount of payment will vary depending on the length of time required to participate in the method as the method lengths vary from two hours to three days. The balance of the amount is for the post-survey. The table below lists the payments for each deliberative method.

Payments will be broken into three steps.

Methods	Total Payments
Brief Citizens’ Deliberation (BCD)	$100
Online Deliberative Polling® (ODP)	$125
Community Deliberation (CD)	$160
Citizens’ Panel (CP)	$325
Control	$75

10. Assurance of Confidentiality Provided to Respondents

Individuals will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.

Data protections

The contractor implementing this project, AIR, has a comprehensive and programmatic information security program that is established and documented in the AIR Information Security Policy. This policy is designed to provide a framework for how AIR can meet Federal, State, and industry information security requirements that apply as a result of project work or internal business requirements. Below is a list of the key laws, standards and regulations that affect how AIR must secure data and information technology systems:

• Federal Information Security Management Act (FISMA)

• Health Insurance Portability and Accountability Act (HIPAA)

• Family Educational Rights and Privacy Act (FERPA)

• DoD Information Assurance Certification and Accreditation Act (DIACAP)

• Payment Card Industry (PCI) Data Security Standards

AIR also has a long history in supporting the data security requirements involving Personally Identifiable Information (PII) on project participants for multiple government as well as non-government organizations. AIR’s Institutional Review Board (IRB) has conducted expedited and full-board reviews of research involving human subjects for more than 17 years. AIR is registered with Office of Human Research Protection (OHRP) as a research institution and conducts research under its own Federalwide Assurance (FWA).

This research has been approved by the AIR IRB. A waiver of written consent was granted because the recruitment and consent process will be conducted online. Documentation of IRB approval is in Attachment G - Documentation of IRB Approval.pdf.

11. Justification for Sensitive Questions

Participants will be asked to report household income for the purpose of assessing whether the effects of deliberative methods are similar for participants at different levels of income and to be able to characterize the study population in order to report on the generalizability of study results. No other questions of a sensitive nature or others considered private will be asked of participants. During the discussions, it is possible that personal information related to topics being deliberated (e.g., personal health information) might be shared by participants if they choose to do so, but they will not be asked to provide such information. Comments made by specific participants in the groups will not be linked to any personally identifiable data. Personally identifiable information (PII) will be obtained for the purpose of recruiting and coordinating the logistics of implementing the methods. The PII will be kept in a separate secure database from the study data.

Informed consent will be obtained from all study participants. Because the initial recruitment, consent process, and the Knowledge and Attitudes Pre-test Survey will be conducted online, we requested and were granted a waiver of written documentation of consent.

Survey data will be housed at the American Institutes for Research internal servers and will not be transferred via e-mail or stored on portable devices. Hardcopies of surveys will be maintained in a locked filling cabinet at AIR’s corporate office at 1000 Thomas Jefferson Street, NW, Washington, DC, 20007 and will be destroyed after the project is terminated.

12. Estimates of Hour Burden Including Annualized Hourly Costs

Exhibit 2 shows the estimated annualized burden associated with the respondents’ time to participate in this research. The total annualized burden hours are estimated to be 9,788 hours. The burden estimate comprises the following activities:

Participant Recruitment – The screening questions and materials will be sent to 1,680 participants (to achieve target sample of 1,296 as described in Supporting Statement B, Sample Allocation section). We estimate that it will take 15 minutes to complete the screening.

Educational materials -- Educational materials will be provided to all 1,680 participants recruited before the implementation of any of the methods. We estimate that it will take up to 1 hour to review the materials.

Brief Citizens’ Deliberation (BCD): This method will be tested with 288 participants (24 groups).  Participants will attend a single, 2 hour in-person meeting.

Online Deliberative Polling^® (ODP): This method will be tested with 288 participants (24 groups) and will consist of 4 online 1.25-hour sessions over the course of 4 weeks; in total, this method will take about 5 hours per person.

Community Deliberation (CD): This method will be tested with 288 participants (24 groups).  Participants will attend 2 in-person meetings, lasting 2.5 hours each, a week apart.  Between meetings, participants will be asked, but not required, to access an online platform; estimated time online is an hour.  In total, this method will take about 6 hours per person.

Citizens’ Panel (CP): This method will be tested with 96 participants (4 groups); participants will attend a 3-day (24 hours), in-person meeting.

Knowledge and Attitudes Pre-test Survey: This survey will be administered to 1,680 participants and will take an estimated 30 minutes to complete.

Knowledge and Attitudes Post-test Survey: This survey will be administered to 1,680 participants and will take an estimated 20 minutes to complete.

Deliberative Experience Survey: This survey will be administered to 960 deliberative methods participants at the conclusion of the deliberative method. It will take an estimated 15 minutes to complete.

Exhibit 2. Estimated annualized burden hours

Form Name/Deliberative Method	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Recruitment and Consent Materials	1,680	1	15/60	420
Brief Citizens’ Deliberation (BCD)	288	1	2	576
Online Deliberative Polling® (ODP)	288	1	5	1,440
Community Deliberation (CD)	288	1	6	1,728
Citizens’ Panel	96	1	24	2,304
Educational Materials	1,680	1	1	1,680
Knowledge and Attitudes Pretest Survey	1,680	1	30/60	840
Knowledge and Attitudes Post-test Survey	1,680	1	20/60	560
Deliberative Experience Survey	960	1	15/60	240
Total	8,640	N/A	N/A	9,788

Exhibit 3 shows the estimated burden hours for the respondents’ time to participate in this project. The total annual cost burden is estimated to be $212,792.

Exhibit 3. Estimated annualized cost burden

Form Name/Deliberative Method	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Recruitment and Consent Materials	1,680	420	$21.74	$9,131
Brief Citizens’ Deliberation (BCD)	288	576	$21.74	$12,522
Online Deliberative Polling® (ODP)	288	1,440	$21.74	$31,306
Community Deliberation (CD)	288	1,728	$21.74	$37,567
Citizens’ Panel	96	2,304	$21.74	$50,089
Educational Materials	1,680	1,680	$21.74	$36,523
Knowledge and Attitudes Pretest Survey	1,680	840	$21.74	$18,262
Knowledge and Attitudes Post-test Survey	1,680	560	$21.74	$12,174
Deliberative Experience Survey	960	240	$21.74	$5,218
Total	8,640	9,788	N/A	$212,792

^*Based upon the mean of the wages for 00-000 All Occupations ($21.74), May 2011 National Occupational Employment and Wage Estimates. United States , “U.S. Department of Labor, Bureau of Labor Statistics.” http://www.bls.gov/oes/current/oes_nat.htm#00-0000

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Annualized Cost to the Federal Government

Exhibit 4 below breaks down the costs related to this study. These are the costs associated with the portion of the contract awarded to AIR to conduct the experiment. Since the implementation and evaluation periods will span 24 months, the costs have been annualized by taking the total cost and dividing by 2.

Exhibit 4. Estimated Annualized Cost to the Federal Government

Cost Component	Total Cost	Annualized Cost
Project Management	$60,106	$30,053
Technical Expert Panel	$117,793	$58,896
Technology Tools	$177,580	$88,790
Develop Educational Materials	$368,624	$184,312
Evaluation Plan	$214,566	$107,283
Implement Methods	$1,636,473	$818,237
Conceptual Framework	$50,195	$25,098
Data Processing and Analysis	$566,846	$283,423
Reporting	$135,693	$67,847
Overhead	$1,281,340	$640,670
Total	$4,616,512	$2,308,257

15. Explanation for Program Changes or Adjustments

This is a new collection of information.

16. Time Schedule, Publication and Analysis Plan

Schedule. AIR’s contract for this project lasts from July 28, 2010 to July 27, 2013. Data collection is scheduled to begin in July 2012. Data collection is scheduled to be completed by January 2013.

Publication. Exhibit 5 summarizes the planned reporting and publication activities. AIR’s contract calls for one manuscript suitable for publication in a peer-reviewed health services research journal to be completed 11-13 months following OMB clearance for data collection, depending on when OMB clearance is obtained. Other manuscripts might be produced by contractor or AHRQ staff.

Exhibit 5. Schedule of Reporting Activities

Activity / Deliverable	Time Schedule / Date
Participant recruitment begins	0-1  Month(s) after OMB approval
Data Collection Begins	1 Month after OMB Approval
Data Collection Concludes	7 Months after OMB Approval
Final Project Report / Analyses	8 -12 Months after OMB Approval
Publication submitted	11 -13 Months after OMB Approval

Analysis Plan. Through our analysis plan, we will address the following research questions:

1. Is public deliberation more or less effective than education at changing knowledge and attitudes about the topic?

2. How does burden change effectiveness?

3. Are specific deliberative methods more effective than other ones?

4. Describe the trade-off, if any, in deliberative output (effectiveness) relative to cost and burden. ( I.e., describe the relative efficiency of methods).

Five sets of analyses will be conducted: (1) Analysis of deliberative session transcripts to summarize public input provided by session participants on the deliberative topics, (2) Analysis of transcript data to assess the content of deliberation and how it varies by deliberative method (3) Analysis of pre- and post-deliberation survey data to identify the effects of deliberation on participants’ knowledge of and attitudes about the deliberative topics, (4) Analysis of post-deliberation survey data to compare the quality of the deliberative discourse and implementation, and (5) Description of costs required to implement methods.

Review and Analysis of Transcripts of Audio and Video-Recordings. The transcript data will be analyzed using qualitative analytic techniques. The analysis to meet both study objectives 1 and 2—provide public input on the topics and evaluate the effects of deliberation, respectively—will be combined, because both will be achieved with the same coding and analysis process. Thus, we discuss them together in this section.

Transcripts will be managed, coded, and analyzed using NVIVO 9.0 qualitative data analysis software. We will employ the following process to analyze both the content and implementation aspects of the deliberative method sessions:

1. Define initial analytic framework and a code list to capture key themes related to our aims and research questions.

2. Code a small number of deliberative sessions to “test” the code list and revise it based on this test.

3. Apply codes to all transcripts collected using the final code list.

4. Develop “code summaries” or memoranda for each of the codes or groups of codes used in the analysis.

5. Examine relationships among codes and identify patterns and themes within and among codes to draw conclusions about the data. To ensure reliable and valid conclusions, we will attempt to disprove hypotheses and assumptions, including those derived from our guiding logic model and others that arise from the data itself.

6. Continue to examine the relationships across deliberative methods and identify themes both within and among methods.

We will use transcripts to identify variation in process and outcomes among the deliberative methods by searching for similarities and differences by deliberative method, deliberative modes (in-person or on-line), and other potentially distinguishing factors. We will employ a variety of evidence-based qualitative analysis techniques to draw conclusions from the data (e.g., noting patterns and themes, plausibility, relationships between variables, and finding intervening variables).

Analysis of Survey Data. We will use an experimental design to identify the effects of deliberation and the differences in effects and participant experience by deliberative method and participant characteristics. The surveys will measure the following:

Effectiveness of deliberation is defined in terms of the method’s ability to increase participants’ knowledge of and change participants’ attitudes regarding the issues that are the subject of the deliberations. Although attitude change is an important measure of effectiveness, we have no expectation that deliberation will produce group consensus and no hypotheses for the direction of attitudinal change. However, we will test to see if deliberation moves groups toward consensus and in which direction attitudes change.

Quality of the deliberative discourse refers to how the participants provide reasons for their positions, and interact with each other in the course of the deliberation. It is defined in terms of the level of participation by all group members, the level of respect for other group members’ views, and the degree to which participants constructively deliberate the issues.

Quality of implementation refers to how well the deliberative methods were constructed and implemented and whether participants thought the process was worthwhile.

All quantitative data, including the Knowledge and Attitude Surveys and the Deliberative Experience Surveys, will be managed and analyzed using SAS statistical analysis software.

The statistical analysis will use one of several variations on the following hierarchical linear model (Raudenbush & Bryk, 2002) to test all the above research questions, with the individual participant as the unit of analysis¹. The generic model can be expressed as:

Y = β_0j + β₁X₁ + β₂X₂ + ε                        (1)

Where:

βs are regression coefficients and ε is the unexplained variance, in particular, β_0j is the regression intercept for the jth deliberative group.

Y = the outcome of interest, derived from survey data, constructed as the pre-post change in the value reported on the Knowledge and Attitude Survey to test the effectiveness of deliberation, or the response on the Deliberative Experience Survey to test the discourse quality or implementation quality²;

X₁ = An indicator of the two conditions involved in the comparison being tested (up to 2 levels: given the contrast being tested, the 2 levels can be chosen from two of the experimental conditions, or combinations of the four deliberative methods, e.g., CD + CP. X₁ = 0 for all participants eligible for the contrast and assigned to condition 1; X₁ = 1 for all participants eligible for the contrast and assigned to condition 2;

X₂ = Personal characteristics (e.g. sociodemographic characteristics, general health status, recent experience with the healthcare system, ethnicity, percentage of the participant’s group with congruent ethnicity, insurance coverage).

All the research hypotheses can be tested using the generic model, with a finding that:

β₁ ≠ 0    indicates that the public deliberation is effective, or one deliberative method (or a combination of several deliberative methods) is more effective than another. (Research Questions 1, 2, and 3)

β₂ ≠ 0    indicates that the effectiveness and deliberative experience differ by personal characteristics of the participants.

Description of burden and cost differences among effective methods. For methods determined to be effective, we will describe the trade-off, if any, in effectiveness relative to cost and burden (i.e., we will describe the relative efficiency of methods). In future decisions regarding the use of deliberative methods to provide public input to AHRQ, the cost and burden associated with a method, relative to the type of information that can be obtained, will be a consideration of clear importance.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

AHRQ does not seek this exemption.

List of Attachments

Attachment A: Recruitment and Consent Materials

Attachment B: Educational Materials

Attachment C: Knowledge and Attitudes Pre-test Survey

Attachment D1: Brief Citizens’ Deliberation Facilitator’s Guide

Attachment D2: Online Deliberative Polling Facilitator’s Guide

Attachment D3: Community Deliberation Facilitator’s Guide

Attachment D4: Citizens’ Panel Facilitator’s Guide

Attachment E: Knowledge and Attitudes Post-test Survey

Attachment F: Deliberative Experience Survey

Attachment G: Documentation of IRB Approval

Attachment H: Federal Register Notice

Attachment I: Public Comment

Attachment J: Response to Public Comment

References

1. Agency for Healthcare Research and Quality: Reauthorization Fact Sheet. AHRQ Publication No. 00-P002, December 1999. Agency for Healthcare Research and Quality, Rockville, MD. Retrieved from http://www.ahrq.gov/about/ahrqfact.htm

2. Community Forum Deliberative Methods Literature Review, submitted to AHRQ on June 29, 2011.

3. Coote, A. & Lenaghan, J. Citizens Juries: Theory into Practice Institute for Public Policy Research. London, 1997.

4. Krueger, R.A. & Casey, M.A. (2008). Focus Groups: A Practical Guide for Applied Research (4th ed.). Thousand Oaks, CA: Sage.

5. Raudenbush, S. & Bryk, A. (2002). Hierarchical Linear Models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

6. United States Department of Labor: Bureau of Labor Statistics. (2011). May 2010 National Occupational Employment and Wage Estimates United States [Data sets]. Retrieved from http://www.bls.gov/oes/current/oes_nat.htm#00-0000.

1 The hierarchical linear models are extensions of basic multiple regression models where the regression coefficients are allowed to vary across deliberative groups to account for the intra-class correlation.

2 For continuous outcome measures, we treat the original outcome values as dependent variables, whereas for binary outcome measures, we use logistic regression models by replacing the variable “Y” in Models 1 and 2 with the logit values “Logit(Y)”. The logit function “Logit(Y)” equals to .