Supporting Statement Part A -- Eval of ARRA CER Dissemination Efforts 3-15-2012

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

Research that develops and presents scientific evidence regarding all aspects of health care; and

The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

Initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Today, both patients and their health care providers have many options when deciding on a treatment plan. Information available to patients and their health care providers offers great opportunities for informed decision making. However, the volume of information that needs to be reviewed and synthesized can be daunting. To complicate matters, studies may offer conflicting information or have a conflict of interest (e.g., research sponsored by pharmaceutical companies that make drugs). Sorting through conflicting information requires a background in research that most patients do not have, and physicians have limited time to conduct these reviews. Having a neutral third party review research, draw conclusions, and disseminate findings is necessary to ensure effective health care delivery and consumption of quality care.

AHRQ recognizes the need to fill this gap and has taken a lead role in developing mechanisms for reviewing and disseminating Comparative Effectiveness Research (CER) and findings to clinicians, health care decision makers, purchasers/business decision makers, and consumers through its Effective Healthcare Program (EHCP). CER directly compares the benefits, potential risks, and costs of two or more health care interventions. These direct comparisons allow assessments of how well a health care treatment or intervention works under real-world conditions. AHRQ has paid careful attention not only to how studies are conducted but also to how results are communicated to its audiences.

To augment AHRQ’s existing CER dissemination efforts performed by the Eisenberg Center and other initiatives, AHRQ is conducting four one-time projects to test other ways to disseminate CER results. These four related projects will test new approaches to CER dissemination and promote awareness of the EHCP. Collectively, dissemination efforts will reach AHRQ’s priority audiences of: clinical decision makers, health care system decision makers, purchasers/business decision makers, public policy decision makers, and consumers/patients^¹. The four projects are:

Publicity Center (RFQ No. AHRQ-10-10008). The publicity center will promote attributes of comparative effectiveness research (CER) and disseminate EHCP nationwide through an integrated and strategic approach. It will also encourage the use and adoption of CER in general and of specific CER products. The publicity center will use 1) media and marketing techniques, 2) strategic partnerships, and 3) virtual centers (Web sites with specific CER information in a single interface) to reach consumers, clinical decision makers, health care system decision makers, and purchasers.

Regional Offices (RFQ No. AHRQ-10-10013). The regional offices are charged with enhancing awareness, understanding, and use of CER and EHCP products in health care decision-making at the regional, State, and local levels. Their central strategy is the creation of partnerships that can be used for co-branding, educational opportunities with the partners’ membership, and referrals to AHRQ-sponsored trainings. Partners will reach clinical decision makers, consumers, health care system decision makers, and policymakers.

Academic Detailing (RFQ No. AHRQ-10-10011). The academic detailing contractor will disseminate key messages from EHCP through face-to-face visits with primary care providers and health system directors nationwide.

Continuing Education (RFQ No. AHRQ-10-10009). The continuing education provider will develop and provide online courses for continuing education credit and promote specific CER results to pharmacists, nurse practitioners, physician assistants, nurses, physicians, and other health care professionals.

Through these four projects AHRQ aims to: (1) educate professional and consumer audiences about CER; (2) inform professional and consumer audiences about AHRQ’s EHCP; (3) and inform a wide range of audiences about new EHCP research findings.

This project will evaluate the effectiveness of these four new dissemination efforts. The evaluation has four main goals:

Assess the effectiveness of the four dissemination strategies in creating awareness of CER, specific CER topics, and the EHCP.

Assess the effectiveness of the four dissemination strategies in fostering knowledge and understanding of CER finding, specific CER topics, and the EHCP.

Assess the effectiveness of the four dissemination strategies in promoting utilization, including use of the EHCP materials by consumers and by clinicians in patient care and if usage by clinicians is increasing across time.

Assess the effectiveness of the four dissemination strategies in supporting the benefits of using CER, and specific CER topics, for both patients and health care providers.

To achieve project goals the following data collections will be implemented, each of which apply to all of the above-stated goals:

Clinician Survey -- Conduct three cross-sectional mail surveys with clinicians to measure awareness, understanding, use of the EHCP materials, and benefits of CER. Collecting survey data at multiple time points is critical to assess trends in the outcomes of interest among clinicians and the impact of ongoing and increased dissemination contractor activities. Three data points for the survey will allow us to test if the proportion of clinicians aware of CER and the Effective Healthcare Program is changing over time and if the rate of change is changing. The Survey will be administered at the end of years 1, 3 and 4; the burden for the year 4 data collection is not included in the estimates in Exhibits 1 and 2 since it will be included in a second OMB clearance package to be submitted after year 3. See Attachment A for the questionnaire and Attachment B for related respondent materials, such as the informed consent form, advance/reminder/thank-you letters, etc.

Consumer/Patient Survey -- Conduct two cross-sectional telephone surveys with consumers/patients to measure awareness, understanding, use of the EHCP materials, and benefits of CER. Collecting survey data at multiple time points is critical to assess trends in the outcomes of interest among consumers/patients and the impact of ongoing and increased dissemination contractor activities. Two data points for the survey will allow us to test if the proportion of consumers/patients aware of CER and the Effective Healthcare Program is changing over time. The Survey will be administered at the end of years 1 and 3. A short screener questionnaire will be used to identify eligible respondents. See Attachment C for the screener questionnaire, Attachment D for the consumer/patient questionnaire, and Attachment E for the related respondent materials.

Health Care System Decision Maker Survey -- Conduct one cross-sectional telephone survey with health care system decision makers to measure awareness, understanding, use of the EHCP materials, and benefits of CER. The questionnaire and respondent materials for this data collection are not included in this submission since it occurs in year 4 of the project and have not yet been developed. These materials will be submitted in another OMB clearance package in year 3 of this project. This data collection is mentioned here in order to provide an overview of the entire 5 years of the project; it is not included in the burden estimates in Exhibits 1 and 2.

Clinician Focus Groups -- Conduct six follow-up focus groups with clinicians after the first and third cross-sectional surveys of this audience, The focus groups will be conducted with three clinician segments: (1) those who report awareness of CER and have self-reported use of CER in their clinical practice; (2) those who report awareness of CER and have self-reported non-use of CER in their clinical practice; and (3) those who report no awareness of CER. See Attachment F for the clinician focus group moderator guide. One moderator guide will be used for each focus group. By asking the same questions to each clinician segment, who will have been targeted by all four dissemination contractors, differences among answers are more likely to be attributed to the segmentation criteria and eliminate bias through different questions. Two focus groups will be conducted for each of the three segments. The clinician focus groups will be conducted by telephone. The focus groups will be administered at the end of year 2 and during year 5; the burden for the year 5 data collection is not included in the estimates in Exhibits 1 and 2 since it will be included in a second OMB clearance package to be submitted after year 3. See Attachment G for the related respondent materials.

Consumer/Patients Focus Groups -- Conduct twelve follow-up focus groups with consumers/patients after the first cross-sectional survey of this audience, at the end of year 2 of the project. The focus groups will be conducted with three consumer/patient segments: (1) those who report awareness of CER and have self-reported use of CER in medical decision making (see Attachment H); (2) those who report awareness of CER and have self-reported non-use of CER in medical decision making (see Attachment I ); and (3) those who report no awareness of CER (see Attachment J ). Four focus groups will be conducted for each of the three segments. A single screening questionnaire will be used to recruit participants (see Attachment K). The consumer/patient focus groups will be conducted by telephone. See Attachment L for the related respondent materials.

Mini Focus Groups -- Conduct six mini telephone focus groups (i.e., with 6-8 individuals), in year 3 of the project. The team proposes do conduct two focus groups each with the following audiences: health care system decision makers (moderator guide included in Attachment M), purchasers (moderator guide included in Attachment N), and policymakers (moderator guide included in Attachment O). The team will use the focus groups to determine how people receive and interpret CER-related materials and verbal information and how they adopt new behaviors based on information they receive. See Attachment P for the related respondent recruitment materials.

Focus Groups -- Conduct 18 focus groups with health care system decision makers, purchasers, and policy makers.. The focus groups will be conducted with three segments for each audience: (1) those who reported awareness of CER and have self-reported use of CER ; (2) those who reported awareness of CER and have self-reported non-use of CER; and (3) those who report no awareness of CER. Two focus groups will be conducted for each of the three segments with each audience (health care system decision makers, purchasers, and policy makers). The focus groups will be conducted by telephone. The screener, moderator guides, and respondent recruitment materials for this data collection are not included in this submission since it occurs in year 5 of the project and have not yet been developed. These materials will be submitted in another OMB clearance package in year 3 of this project. This data collection is mentioned here in order to provide an overview of the entire 5 years of the project; it is not included in the burden estimates in Exhibits 1 and 2.

This study is being conducted by AHRQ through its contractor, IMPAQ International, LLC and its subcontractor, Battelle Memorial Institute, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to clinical practice, including primary care and practice-oriented research. 42 U.S.C. 299a(a)(1) and (4).

2. Purpose and Use of Information

AHRQ will use the survey and focus group data to assess trends and the effectiveness of the four complementary and different dissemination methods to inform current and future dissemination of the EHCP. Specific attention will be given to changes in audience awareness, understanding, behavior change/use, and benefits of CER. Collecting data at multiple times will enable AHRQ to determine whether increased dissemination contractors’ activities over time is associated with any change in CER awareness, knowledge, use, or benefit. Finally, collecting data from five audiences (i.e., clinicians, consumers/patients, health system decision makers, purchasers, and policy makers) will enable AHRQ to assess the effectiveness of its CER-related dissemination efforts among its target populations.

3. Use of Improved Information Technology

This project will use several forms of information technology to minimize respondent burden, enhance data quality, and maximize cost-effectiveness:

Computer Assisted Telephone Interviewing (CATI) will be used to conduct telephone surveys with consumers/patients and health care system decision makers. Telephone interviews are more cost-effective and impose less burden on respondents than do in-person interviews.

Telephone focus groups will be used to collect qualitative follow-up information from clinicians, consumers/patients, and health care system decision makers who respond to the surveys. Additionally, focus groups will also be conducted with healthcare system decision makers, purchasers, and policymakers in years 3 and 5 to learn about how people receive and interpret CER-related materials For all audiences, telephone focus groups are the best method to ensure that selection is not limited to a small number of market areas. Moreover, conducting the focus groups by telephone eliminates travel burden and reduces costs.

4. Efforts to Identify Duplication

The surveys and focus groups will collect key information from AHRQ’s target populations for CER-related dissemination efforts. No other data collection effort has been conducted or has been planned to collect similar information. In addition, no administrative data source exists that can provide the required information on the level of awareness, understanding, use, and perceived benefits of CER.

5. Involvement of Small Entities

The surveys and focus groups will only involve individuals; therefore, they will not pose a burden to small businesses or other small entities.

6. Consequences if Information is Collected Less Frequently

The purpose of this one-time project is to evaluate the effectiveness of four dissemination and implementation strategies that AHRQ uses to foster awareness, knowledge, use, and benefits of CER as part of the EHCP over the five-year project.

Currently, there is no other data source that collects this information for AHRQ. Collecting data at multiple time points is necessary to achieve the objectives of the evaluation and to measure the outcomes of interest. Moreover, the ability of respondents to recall CER-related information that they hear or see would be impaired if the data were collected less frequently.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on July 27^th, 2011 for 60 days, and again on November 14^th, 2011 for 30 days (see Attachment Q). No comments were received.

8b. Outside Consultations

AHRQ consulted with many individuals in developing the evaluation design, data collection plan, and study instruments. This includes AHRQ program staff, staff from each of the four dissemination contractors, and staff from the evaluation contractor (IMPAQ International and its subcontractor, Battelle). AHRQ also spoke with researchers at Mathematica Policy Research who are conducting an ASPE evaluation project that involves surveys with some of the same target audiences that are part of this evaluation. No unresolved problems were identified by any of these individuals. The individuals that were consulted are listed in the table below.

Name	Affiliation	Telephone Number

Berkowitz, Alicia	Evaluation Contractor – IMPAQ	443-718-4343
Siegel, Sari	Evaluation Contractor – IMPAQ	443-539-9764
Bollino, Camellia	Evaluation Contractor – IMPAQ	443-718-4356

Young, Julie	Evaluation Contractor – IMPAQ	443-539-9766

Berkowitz, Judy	Evaluation Contractor – Battelle	404-460-1449
Derzon, Jim	Evaluation Contractor – Battelle	703-248-1640
Rose, John	Evaluation Contractor – Battelle	919-544-3717

Rhoda, Dale	Evaluation Contractor – Battelle	614-424-4944
Carey, Nicole	Duty First Consulting	240-535-2945
Esposito, Dominick	Mathematica Policy Research	609-275-2358

9. Payments/Gifts to Respondents

There is a large methodological literature that suggests offering monetary incentives can significantly improve survey response rates^²^,^³^,^⁴^,^⁵. For this evaluation, AHRQ is proposing an incentive structure that is tailored to the audiences from whom data will be collected in order to maximize response rates. AHRQ believes that the proposed incentives are sufficient to achieve the desired response rates. The proposed incentive structure is as follows:

Clinician Surveys. Clinicians will be offered a $50 incentive for completing the mail survey. Several studies of the effects of monetary incentives on survey response rates among physicians indicate that incentive levels between $25-$100 yield the best results.^⁶^,^⁷^,^⁸^,^⁹ Recent physician surveys conducted by the contractor (Battelle) using the proposed methods, including the $50 incentive, have yielded response rates in the range of 70-80%.

Consumer/Patient Surveys and Health Care System Decision Maker Surveys. Consumers/patients and health care system decision makers will not be offered an incentive for participating in the telephone survey.

Clinician Focus Groups. Clinicians will be offered a $150 incentive for participating in the telephone focus groups. Focus group with highly-paid professional participants (engineers, physicians, attorneys, upper-level managers, etc.) often require higher incentive amounts in the $100 to $500+ range.^¹⁰ The clinician audience consists of highly paid health care providers, and the planned focus groups require a substantial amount of time for participation (1.5-2 hours), reasonable participation rates with the lowest possible cost will be achieved with a $150 incentive level.

Consumer/Patient, Health Care System Decision Makers, Purchasers, and Policy Maker Focus Groups. Consumers/patients and health care system decision makers will be offered a $75 incentive for participating in the telephone focus groups. This is a standard value used. Incentives offered to focus group participants of similar audiences are $75.^¹¹

10. Assurance of Confidentiality

All individuals who participate in the study will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. All respondent materials used for the project will include the following language: “Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c).”

A limited set of respondent identifying information will be collected for the study. All data items that identify respondents will be kept only by the contractor, IMPAQ International, and its subcontractor, Battelle, for use in conducting mail and telephone surveys and focus groups. Any data received by AHRQ will not contain personal identifiers, thus precluding individual identification. Completed mail surveys will be stored in locked file cabinets. All electronic files will be password protected and accessible only to authorized project staff. Measures to safeguard data will be emphasized in written and verbal training procedures for project personnel, and all project personnel will sign an Assurance of Confidentiality statement. In administering the surveys, a link between respondents and their respective ID numbers will be maintained. This link will be used for tracking survey mailings and responses, making follow-up contacts, and recruiting focus group participants. The links between respondent contact information and ID numbers will be stored securely and separately. The links between survey ID numbers and identifying information, including the respondent’s contact information, will be destroyed upon completion of data collection.

11. Questions of a Sensitive Nature

The data collection instruments being submitted for clearance do not include questions of a sensitive or personal nature. The project will not collect Social Security Numbers, Medicare Numbers, or Medicaid Numbers.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondent’s time to participate in this evaluation. The total burden hours are estimated to be 2,451.

Clinician Surveys: The design for the clinician survey consists of three cross sectional waves (only 2 of which are included in the estimates here, as explained in section 1), each wave having 1,926 respondents for a total of 3,852 across the two waves included in this information collection request. The survey will take no longer than 20 minutes to complete.

Consumer/Patient Surveys: The design for the consumer/patient survey consists of two cross-sectional waves, each wave having 1,000 respondents for a total of 2,000 across both waves. The screener will take no longer than 5 minutes to complete. The survey will take no longer than 20 minutes to complete.

Clinician Focus Groups: Six follow-up focus groups with clinicians will be conducted by telephone twice; once after the first and again after the third cross-sectional surveys of this audience (only one of which is included in the estimates here, as explained in section 1). Focus group participants will have completed the survey and will have expressed interest in participating in a telephone focus group. For each of the two rounds of focus groups, twelve clinicians will be recruited for each of six focus groups. Focus groups will last one hour.

Consumer/Patient Focus Groups: Twelve follow-up focus groups with consumer/patients will be conducted by telephone after the first cross-sectional survey of this audience. Focus group participants will have completed the survey and will have expressed interest in participating in a telephone focus group. Eight people will be in each focus group. The screener will take no longer than 5 minutes to complete. Focus group will last approximately 90 minutes.

Mini Focus Groups with Other Key Audiences: Mini telephone focus groups will be conducted with three key audiences: (1) health care system decision makers, (2) purchasers, and (3) policy makers. Each focus group will consist of six participants. The screener will take no longer than 5 minutes to complete. Focus group will last approximately 60 minutes.

The estimated annualized cost burden associated with the respondent’s time to participate in this evaluation is shown in Exhibit 2. The total cost burden is estimated to be $143,369.

Exhibit 1: Estimated Annualized Burden Hours

Data Collection Activity	Number of Respondents	Number of Responses per Respondent	Hours per Response	Total Burden Hours
Clinician Survey	3,852	1	20/60	1,284
Consumer/Patient Survey Screener Survey	2560 2000	1 1	5/60 20/60	214 667
Clinician Focus Groups	72	1	60/60	72
Consumer/Patient Focus Groups Screener Focus Group	120 96	1 1	5/60 90/60	10 144
Mini Focus Group with Health Care System Decision Makers Screener Focus Group	16 12	1 1	5/60 60/60	1.3 12
Mini Focus Group with Purchasers Screener Focus Group	16 12	1 1	5/60 60/60	1.3 12
Mini Focus Group with Policymakers Screener Focus Group	16 12	1 1	5/60 60/60	1.3 12
Total	8,784	n/a	n/a	2,431

Data Collection Activity

Number of Respondents

Number of Responses per Respondent

Hours per Response

Total Burden Hours

Clinician Survey

3,852

20/60

1,284

Consumer/Patient Survey

Screener

Survey

2560

2000

5/60

20/60

214

667

Clinician Focus Groups

60/60

Consumer/Patient Focus Groups

Screener

Focus Group

120

5/60

90/60

144

Mini Focus Group with Health Care System Decision Makers

Screener

Focus Group

5/60

60/60

1.3

Mini Focus Group with Purchasers

Screener

Focus Group

5/60

60/60

1.3

Mini Focus Group with Policymakers

Screener

Focus Group

5/60

60/60

1.3

Total

8,784

n/a

2,431

Exhibit 2: Estimated Annualized Cost Burden

Data Collection Activity	Number of Respondents	Total Burden Hours	Average Hourly Wage Rate*	Total Cost Burden
Clinician Survey	3,852	1,284	$88.46	$113,583
Consumer/Patient Survey Screener Survey	2560 2000	214 667	$20.90 $20.90	$4,473 $13,940
Clinician Focus Groups	72	72	$88.46	$6,369
Consumer/Patient Focus Groups Screener Focus Groups	120 96	10 144	$20.90 $20.90	$209 $3,010
Mini Focus Groups with Health Care System Decision Makers Screener Focus Group	16 12	1.3 12	$43.74	$57 $525
Mini Focus Groups with Purchasers Screener Focus Group	16 12	1.3 12	$46.59	$61 $560
Mini Focus Groups with Policymakers Screener Focus Group	16 12	1.3 12	$43.74	$57 $525
Total		2,451	n/a	$143,369

Data Collection Activity

Number of Respondents

Total Burden Hours

Average Hourly Wage Rate*

Total Cost Burden

Clinician Survey

3,852

1,284

$88.46

$113,583

Consumer/Patient Survey

Screener

Survey

2560

2000

214

667

$20.90

$4,473

$13,940

Clinician Focus Groups

$88.46

$6,369

Consumer/Patient Focus Groups

Screener

Focus Groups

120

144

$20.90

$209

$3,010

Mini Focus Groups with Health Care System Decision Makers

Screener

Focus Group

1.3

$43.74

$57

$525

Mini Focus Groups with Purchasers

Screener

Focus Group

1.3

$46.59

$61

$560

Mini Focus Groups with Policymakers

Screener

Focus Group

1.3

$43.74

$57

$525

Total

2,451

n/a

$143,369

*Based upon the mean of the average wages, National Compensation Survey: Occupational wages in the United States May 2009, “U.S. Department of Labor, Bureau of Labor Statistics.” Hourly wage rates for clinicians were estimated using the mean wage for internists (occupation code 29-1063). Hourly wage rates for consumers/patients were estimated using the mean wage for all occupations (occupation code 00-0000) since participants in the consumer groups may have a wide range of jobs and occupations. Hourly wage rates for health system decision makers and policymakers were estimated using the mean wage for medical and health services managers (occupation code 11-9111). Hourly wage rates for purchasers were estimated using the mean wage for purchasing managers (occupation code 11-3061). These rates were obtained in January 2011 at the following website: http://www.bls.gov/oes/current/oes_nat.htm#b29-0000.

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

The total cost to the Government for this information collection is $1,631,562 over the first three years of this five year project; the costs associated with years four and five will be included in the renewal submission. Exhibit 3 provides a breakdown of these costs.

Exhibit 3: Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$252,033	$84,011
Data Collection Activities	$871,374	$290,458
Data Processing and Analysis, and Reports to AHRQ	$84,981	$28,327
Project Management	$175,023	$58,341
Overhead^¹²	$248,151	$82,717
Total^¹³	$1,631,562	$543,854

15. Changes in Hour Burden

This is a new collection of information.

16. Time Schedule, Publication, and Analysis Plans

16a.Time Schedule

The tentative time schedule for data collection activities is provided in Exhibit 4. These dates may change depending on when OMB grants clearance.

Exhibit 4: Time Schedule

Date(s)	Activity
11/2010 – 8/2011	Develop evaluation instruments
8/2011	Submit clearance request to OMB
1/2012 – 3/2012	Conduct Wave 1 clinician mail survey
1/2012 – 3/2012	Conduct Wave 1 consumer/patient telephone survey
7/2012 – 9/2012	Conduct clinician focus groups
7/2012 – 9/2012	Conduct consumer/patient focus groups
1/2013 – 3/2013	Conduct Wave 2 clinician mail survey
1/2013 – 3/2013	Conduct Wave 2 consumer/patient telephone survey
4/2013 – 5/2013	Conduct mini focus groups with health care system decision makers, purchasers, and policymakers
1/1/2013	Interim report 1
7/1/2013	Interim report 2
9/30/2013	Present final data analysis
10/15/2013	Final report – Phase 1
7/14 – 1/15	Conduct Wave 3 clinician mail survey
7/14 – 1/15	Conduct health care system decision maker survey
7/14 – 1/15	Conduct clinician focus groups
7/14 – 1/15	Conduct focus groups with health care system decision makers, purchasers, and policy makers.
4/10/15	Interim report 1
7/15/15	Interim report 2
9/16/15	Final report – Phase 2

16b. Publication Plans

There are currently no specific plans to publish results based on these data.

16c. Analysis Plans

Clinician Survey. The clinician survey will provide information that can be used to estimate general trends in the key study metrics among the U.S. population of clinicians through cross-sectional surveys at three points in time. The surveys will be a clear source of insight into the “secular trend” of CER awareness and use.

The clinician surveys will be used to estimate changes in the proportion of clinicians who are aware of CER broadly, aware of EHCP specifically, consider themselves to seek out and understand CER products and research (generally and AHRQ-specific), consider CER to be beneficial, consider themselves to incorporate CER into their practice, and consider themselves to discuss CER with their patients. The surveys will also seek to elucidate some of the barriers to CER consumption and adoption.

The statistical goal of the survey analysis will be to estimate proportions and changes in those proportions across the three waves of the survey. Multiple sources of CER information are available to the curious clinician, and trends in CER awareness and use are probably on the rise. By estimating important proportions at three points in time, estimations not only of the rate of change of CER awareness and use are possible, but also whether that rate is accelerating or decelerating between the three survey time points. The analysis of proportions will account for complex features of the sample, including stratification by clinical specialty. The responses will be weighted using the appropriate proportion of the clinician population that each response represents (calculated using weights from the sampling frame), and results will be reported with confidence intervals computed with the survey estimation commands of Stata.

The clinician survey will also include measures of exposure to the dissemination contractors’ activities, which will (1) estimate the reach of those activities, (2) control for activity exposure when estimating the key metrics in the general clinician population, and (3) identify “non-exposed” clinicians who may serve as an ad hoc comparison group.

Proportions will be reported on the sample as a whole and on subsets of the sample stratified on exposure to the activities of the dissemination contractors. It is expected that the dissemination efforts that will have the most impact (continuing education courses and academic detailing) will have the smallest reach (i.e., the fewest people “touched” by the activity). Thus, it is expected to find only a few people in the nationally representative survey of clinicians who indicate that they were exposed to these activities directly. Among the persons surveyed, the greatest source of EHCP awareness may be the marketing activities, though these will have the smallest effect on clinical behaviors. The survey sample size was selected to have statistical power to reliably detect a four percent change in CER awareness at the national level. In any analysis of subsets of the sample, the magnitude change that is reliably detectable will be larger than that.

Consumer/Patient Survey. The analysis of consumer/patient survey data will focus on trends and changes in the areas of awareness, knowledge, behavior change/use, and benefits of CER, specific CER topics, and the EHCP. This approach will enable AHRQ to determine whether the passage of time, and increased dissemination contractor activity, is associated with any change in CER awareness, knowledge, use, or benefit.

The statistical goal of the survey analysis will be to estimate proportions and changes in those proportions across the two survey waves. The analytic approach will utilize statistical packages such as SAS, STATA, and SPSS to develop descriptive as well as inferential statistics as appropriate. After the data collection is complete, the survey data will be cleaned prior to beginning the analysis. The data cleaning approach will include handling confidential personal identifiers, identifying outliers, imputing missing values, recoding selected variable values to standardize meaning, checking for duplicate records, cross-checking for internal consistency, and documenting computer programs for archiving.

To identify trends and changes in awareness, knowledge, behavior change/use, and benefits of CER, the analytic approach will be to run frequencies and cross tabulations of key variables to identify what percentage of respondents were aware of CER or using CER. Crosstabs or subgroup analyses also will be run, as appropriate, to identify awareness levels among priority target populations of interest to AHRQ. This kind of analysis might reveal that a particular subgroup has unusually low awareness of CER—a finding that would be useful in planning future dissemination efforts for that particular subgroup.

Health Care System Decision Maker Survey. The analysis of the health care system decision maker survey data will focus on awareness, knowledge, behavior, use, and benefits of CER, specific CER topics, and the EHCP during year 4 of the project. At this point, the four dissemination projects will have been disseminating information for 3.5 years.

The analytic approach will utilize statistical packages such as SAS, STATA, and SPSS to develop descriptive as well as inferential statistics as appropriate. After the data collection is complete, the survey data will be cleaned prior to beginning the analysis. The data cleaning approach will include handling confidential personal identifiers, identifying outliers, imputing missing values, recoding selected variable values to standardize meaning, checking for duplicate records, cross-checking for internal consistency, and documenting computer programs for archiving.

Clinician Focus Groups. The clinician focus group data (in the form of transcripts based on the audio recordings) will be analyzed using qualitative content analysis and constant-comparison techniques. For the first step of analysis, a debriefing will be conducted after each focus group (within one day) to discuss and summarize key findings related to the research questions. The debriefings can include the remote observers either through conference call or online collaboration technology (e.g., LiveMeeting, WebEx). The notes from the debriefings will facilitate rapid reporting of the initial findings (as needed) and will facilitate the in-depth analyses of the focus group data. A topline report will be prepared within one week of finishing the focus groups.

All transcripts will be reviewed to ensure accuracy and completeness, and as initial preparation for in-depth analyses. Analysis and data management will be performed using a qualitative data analysis software package (such as NVivo) that accommodates multiple analysts and the integration of external structured/quantitative data. Because of the small number of analysts and focus groups (six), formal processes for evaluating inter-rater reliability for the qualitative coding will not be utilized. Regular communication and small teams will be used to develop and continuously refine a coding scheme for analyzing the data. This close collaboration will ensure a common understanding of how codes are defined and applied, and reduce coding inconsistencies.

Consumer/Patient, Health Care System Decision Makers, Purchasers, and Policy Maker Focus Groups.. The focus group data (in the form of transcripts based on the audio recordings) will be analyzed using qualitative content analysis and constant-comparison techniques. For the first step of analysis, a debriefing will be conducted after each focus group (within one day) to discuss and summarize key findings related to the research questions. The debriefings can include the remote observers either through conference call or online collaboration technology (e.g., LiveMeeting, WebEx). The notes from the debriefings will facilitate rapid reporting of the initial findings (as needed) and will facilitate the in-depth analyses of the focus group data. A topline report will be prepared within one week of finishing the focus groups.

The focus group responses will be analyzed using a multi-method approach. Content analysis techniques and NVivo qualitative analysis software will be used to strengthen the analysis. Themes, trends, and outliers across and within the populations regarding CER awareness, understanding, and use/non-use of the CER “brand and tools” will be identified.

NVivo software allows the answer to each focus group question to be coded and generate reports based on the findings. NVivo allows analysis and coding of notes and transcripts as well as audio recordings of the focus groups (if necessary). Initially, the software will be used to produce a word-frequency report, through which frequently occurring concepts in the focus group data will appear. Then, focus group responses will be coded: the responses first will be attributed to their respective respondents and then will be coded to reflect the main point of each response. Once coding of the main points is complete, reports on the frequency of and relationships between the codes will be generated. The coding frequency reports will be independently reviewed to ensure a consensus on the overall main points made in each focus group. The codes then will be organized into a hierarchical structure: each point will be coded as a consensus theme (agreed on by the majority of participants within the group), a frequent theme (commonly mentioned by different participants), or a rare theme (mentioned by only one or a small number of participants). Illustrative quotes for each point will be included with the key issue.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A – Clinician Questionnaire

Attachment B – Clinician Survey Respondent Materials

Attachment C—Consumer or Patient Screener Questionnaire

Attachment D—Consumer or Patient Questionnaire

Attachment E – Consumer or Patient Respondent Materials

Attachment F -- Clinician Focus Group Guide

Attachment G -- Clinician Focus Group Respondent Materials

Attachment H – Consumer or Patient Focus Group Guide -- Aware with use

Attachment I – Consumer or Patient Focus Group Guide -- Aware without use

Attachment J – Consumer or Patient Focus Group Guide -- Unaware of CER

Attachment K – Consumer or Patient Focus Group Screener Questionnaire

Attachment L – Consumer or Patient Focus Group Respondent Materials

Attachment M – Focus Group Moderator Guide for Health Care System Decision Makers

Attachment N – Focus Group Moderator Guide for Purchasers

Attachment O – Focus Group Moderator Guide for Policymakers

Attachment P – Focus Group Moderator Respondent materials for Health Care System Decision Makers, Purchasers, and Policy Makers

Attachment Q – Federal Register Notice

1 Clinical decision makers include doctors, nurses, pharmacists, and other allied health professional organizations. Health care system decision makers include people working for health plans, integrated health systems, insurance companies, hospitals, group practices, and long-term care institutions, pharmaceutical and other health product firms that are developing new drugs, medical devices, tests, or ways to deliver health care. Purchasers/business decision makers include employers, corporations, business coalitions, and unions, vendors of health information technology focused on clinical decision support. Public policy decision makers include Federal and State policymakers such as Medicare and Medicaid, policy advisory groups such as the Institute of Medicine, professional societies, health care associations, and Quality Improvement Organizations. Consumers/patients include patients, their caregivers, and patient advocacy groups.

2 S. A. Everett, J. H. Price, A. W. Bedell, and S. K. Telljohann. The Effect of a Monetary Incentive in Increasing the Return Rate of a Survey to Family Physicians. Eval Health Prof, June 1, 1997; 20(2): 207 ‐ 214.

3 R. L. Collins, P. L. Ellickson, R. D. Hays, and D. F. Mccaffrey. Effects of Incentive Size and Timing on Response Rates to a Follow‐Up Wave of a Longitudinal Mailed Survey. Eval Rev, August 1, 2000; 24(4): 347 ‐ 363.

4 E. Ryu, M. P. Couper, and R. W. Marans. Survey Incentives: Cash vs. In‐Kind; Face‐to‐Face vs. Mail; Response Rate vs. Nonresponse Error. Int. J. Public Opin. Res., March 1, 2006; 18(1): 89 ‐ 106.

5 M. F. Teisl, B. Roe, and M. E. Vayda. Incentive Effects on Response Rates, Data Quality, and Survey Administration Costs. Int. J. Public Opin. Res., September 1, 2006; 18(3): 364 ‐ 373.

6 S. A. Everett et al, 1997, ibid.

7 Kasprzyk, D., Montano, D. E., St Lawrence, J. S., & Phillips, W. R. (2001). The effects of variations in mode of delivery and monetary incentive on physicians' responses to a mailed survey assessing STD practice patterns. Eval Health Prof, 24(1), 3-17.

8 Dykema, J., Stevenson, J., Day, B., Sellers, S. L., & Bonham, V. L. (2011). Effects of Incentives and Prenotification on Response Rates and Costs in a National Web Survey of Physicians. Evaluation & the Health Professions, 34(4), 434-447

9 Montaño DE, Kasprzyk D, Hall IJ, Richardson LC, Greek A, and Ross L. Effect of incentive amount and telephone follow-up on response to a physician survey: findings from a prostate cancer screening survey of primary care physicians. Evaluation and the Health Professions. (under review)

10 Krueger, R.A., Casey, M.A. (2009) Focus Groups: A Practical Guide for Applied Research, 4^th Edition. Sage Publications, Inc.

11 Lovejoy, Kristin, Handy, Susan . (2008). A case for measuring individuals’ access to private-vehicle travel as a matter of degree: lessons from focus groups with Mexican immigrants in California. Transportation, 34, 601-612.

12 Overhead only includes overhead; it does not include fringe, G&A, subcontractor handling fee, nor fee.

13 Total reflects unloaded labor and overhead; other direct costs are not included.

Page 2

File Type	application/msword
File Title	OMB Clearance Application
Author	hamlin-ben
Last Modified By	CTAC
File Modified	2012-03-19
File Created	2012-03-19