Summmary State EGAPP. 06BI

Determining Stakeholder Awareness and the Use and Impact of Products Developed by the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Model Project

OMB Package

Agency:

Centers for Disease Control and Prevention

National Office of Public Health Genomics

CDC Principal Investigator / Project Officer:

Linda A. Bradley, PhD, FACMG

National Office of Public Health Genomics

Centers for Disease Control and Prevention

4770 Buford Highway, Mailstop K-89

Atlanta, GA 30341

Phone: 770-488-8399

Fax: 770-488-8336

e-mail: [email protected]

Table of Contents

Section Page

A. Justification 4

A.1. Circumstances Making the Collection of Information Necessary 4

A.1.1 Background on the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Model Project 4

A.1.2 Background on Key EGAPP Stakeholders 7

A.1.3 Surveying Stakeholders: The Evaluation Component of the EGAPP Pilot Project 8

A.2. Purpose and Use of Information Collection 9

A.3. Use of Improved Information Technology and Burden Reduction 10

A.4. Efforts to Identify Duplication and Use of Similar Information 12

A.5. Impact on Small Businesses or Other Small Entities 14

A.6. Consequences of Collecting the Information Less Frequently 15

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 15

A.9. Explanation of Any Payment or Gift to Respondents 18

A.10. Assurance of Confidentiality Provided to Respondents 18

A.11. Justification for Sensitive Questions 19

A.12. Estimates of Annualized Burden Hours and Costs 20

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers 20

A.14. Annualized Cost to the Federal Government 21

A.15. Explanation for Program Changes or Adjustments 23

A.16. Plans for Tabulation and Publication and Project Time Schedule 23

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate 25

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions 26

B. Collections of Information Employing Statistical Methods 26

B.2. Procedures for the Collection of Information 34

B.3 Methods to Maximize Response Rates and Deal with Non-Response 38

B.4. Tests of Procedures or Methods to be Undertaken 39

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 39

List of Attachments 40

ATTACHMENT A: STAKEHOLDER CATEGORIES 41

ATTACHMENT B: PUBLIC HEALTH SERVICE ACT Error! Bookmark not defined.

ATTACHMENT C: INTRODUCTORY NOTICE - Draft Memo to Stakeholders Error! Bookmark not defined.

ATTACHMENT D: EGAPP SURVEYS Error! Bookmark not defined.

EGAPP Internal Designations Error! Bookmark not defined.

(Respondents will be directed to the appropriate survey, Error! Bookmark not defined.

all designated on the web as “EGAPP Survey”) Error! Bookmark not defined.

Attachment D1 - Healthcare Provider Survey Error! Bookmark not defined.

Attachment D2 - Policy/Payer Survey Error! Bookmark not defined.

Attachment D3 - Healthcare Purchaser Survey Error! Bookmark not defined.

Attachment D4 - Policy Survey Error! Bookmark not defined.

Attachment D5 - General Survey Error! Bookmark not defined.

ATTACHMENT E: FEDERAL REGISTER NOTICES Error! Bookmark not defined.

ATTACHMENT F: IRB APPROVAL Error! Bookmark not defined.

A. Justification

A.1. Circumstances Making the Collection of Information Necessary

A.1.1 Background on the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Model Project

The success of the Human Genome Project has led to increasingly rapid translation of genomic information into clinical testing applications. Genetic tests for about 1,200 diseases have been developed, with more than 1,000 currently available in clinical practice. Most are used for diagnosis of rare genetic diseases, but a growing number have population-based applications, including carrier identification, predictive testing for inherited risk for common diseases, and pharmacogenetic testing for variation in drug response. These tests have the potential for broad public health impact.

Most genetic testing currently offered in the United States does not involve the use of US Food and Drug Administration (FDA) approved test kits. Most are laboratory developed or “home brew” assays, and are marketed by laboratories as clinical laboratory services with limited regulatory oversight. A number of issues have been raised about this current status of genetic testing implementation, including the need to develop evidence to establish the validity and utility of genetic tests before the tests are commercialized. There is also an increasingly urgent need for timely and reliable information that allows healthcare professionals to clearly identify those genetic tests that have demonstrated validity and utility, and to use them appropriately. Recommendations on the development and clinical implementation of safe and effective genetic tests have been produced by advisory panels, professional organizations, and clinical experts since 1995.^1,2,3 However, a coordinated process for effectively integrating genetic tests into clinical practice and health policy is still needed.

The EGAPP project was initiated in late 2004 in direct response to the need to develop a coordinated interagency process for the systematic evaluation of the validity and utility of genetic tests, as articulated by the HHS Secretary’s Advisory Committee on Genetic Testing⁴ and, later, by the Secretary’s Advisory Committee on Genetics, Health and Society (SACGHS). The February, 2006 SACGHS report on Coverage and Reimbursement of Genetic Tests and Services made nine recommendations to the Secretary that could “help improve appropriate access to and utilization of health-related genetic tests and services in both public and private health insurance programs.”⁵ The first recommendation, on Evidence-Based Coverage Decisionmaking, stated that the Secretary should task an appropriate group to address the evidence on validity and utility of genetic tests and identify evidentiary gaps; they specifically cited the CDC EGAPP Working Group as an example of such a group. SACGHS continues to request periodic updates on the status of the EGAPP Project.⁶ In September, 2006, a representative of the Secretary’s Personalized Healthcare Initiative sought information on EGAPP and attended an EGAPP Working Group Meeting; information on EGAPP has also been requested and provided to Senator Barack Obama’s legislative staff.

The stated goal of the EGAPP pilot project is to establish a systematic, evidence-based process for assessing genetic tests or other applications of genomic technology (e.g., gene expression profiles, proteomics, microarrays) in transition from research to practice. The EGAPP model project integrates methods and approaches used by other systematic evaluation and appraisal processes, such as the Agency for Healthcare Research and Quality supported U.S. Preventive Services Task Force, and CDC’s Community Guide. The interagency EGAPP Steering Committee advised CDC that feedback should be solicited from key stakeholders on the products of this pilot project before making decisions about the most effective way to move forward with plans for a sustainable “EGAPP-like” program.

The first step in the development of the EGAPP pilot project was to recruit an independent, non-federal Working Group; established in April, 2005, the EGAPP Working Group is composed of 13 multidisciplinary experts. From May to October, 2005, the focus of the Working Group was on identifying, prioritizing and selecting specific genetic tests for evidence review, and on determining the methodology and standards to be used in the systematic reviews. Once tests are selected by the Working Group, CDC’s National Office of Public Health Genomics commissions the reviews, and the Working Group oversees the review process and provides technical assistance to the reviewers as part of Technical Expert Panels. Three reviews were commissioned in the fall of 2005 and one in February, 2006; two reviews are being commissioned in late 2006, and one in early 2007. Once evidence reports are delivered to the Working Group, panel members begin the process of deliberation on the quantity, quality and generalizability of the evidence, and of developing recommendations about the appropriate use of the tests based on the evidence. Note that the timeframe for completion of systematic evidence reviews and development of clinical recommendations can range from 6 to 14 months, so products are not anticipated until the end of 2006; products of the pilot project will come out in a staggered fashion over a period from November, 2006 to Spring, 2008 (see Figure A.6 in section A.6).

Anticipated primary products of the EGAPP process include:

• evidence reports on specific tests;

• published summaries of the evidence reports; and

• published recommendations of the EGAPP Working Group based on the evidence reports.

Secondary products will consist of shorter and less technical multi-media informational messages developed for specific target audiences from the evidence reports and Working Group recommendations.

A.1.2 Background on Key EGAPP Stakeholders

The process of developing evidence and recommendations related to implementation of genetic tests and other applications of genomic technology has many important stakeholders, including health care providers, medical professional organizations, health care payers and purchasers, health policy makers (non-regulatory), consumers, public health, test developers from industry/biotechnology, academic researchers, clinical laboratories, and regulatory bodies. While reports, recommendations, and general information developed by the EGAPP project will be available to, and have some relevance to, all of these stakeholders, limitations of time and resources necessitated an early decision that the evaluation component of the EGAPP pilot project would focus on four key target audiences that have the most immediate need for the information: healthcare providers, healthcare payers and purchasers, healthcare policy makers⁷, and, ultimately, healthcare consumers.

As noted above, products and information will be released by the EGAPP Working Group as completed, at different times during the pilot project. Each specific product will almost certainly reach different stakeholder groups at different times, necessitating a staggered timeline for survey activities (see Figure A.1 and Figure A.6). For example, health care providers and health care payers are now actively seeking evidence-based information on emerging genetic tests and have indicated that they will immediately consider evidence reports and recommendations as released.⁸ However, lengthy and technical evidence reports and articles published in the scientific literature are less likely to reach, or be of interest to, certain other stakeholders, such as some healthcare policy makers or consumers. As noted above, translation into, and dissemination of, secondary products (e.g., more engaging informational messages) are likely to be needed to reach some key stakeholder groups. So, information is expected to “trickle down” over time from those involved in health care to policy makers and consumers (Figure A1).

A.1.3 Surveying Stakeholders: The Evaluation Component of the EGAPP Pilot Project

The plan for surveying key stakeholders described in this OMB package represents the largest and most important component of the overall project evaluation plan being conducted for the CDC National Office of Public Health Genomics (NOPHG) as part of assessing the success and impact of the EGAPP pilot project. The survey study will be conducted in collaboration with a consultant, Judith L. Johnson, PhD, under a CDC task order with the McKing Consulting Corporation. McKing Consulting Corporation will work with CDC to design the study, collect data for the study, conduct data analyses, and develop written reports of results.

There are no legal or administrative requirements that necessitate collection of this data. However, feedback from key stakeholders on the value and impact of the EGAPP process and products is crucial to: 1) inform CDC and other collaborating HHS agencies about which methods and approaches would be most effective for the future development of a sustainable process for assessment of the safety and effectiveness of emerging genetic tests; 2) to investigate the potential impact of such a program on the integration of genomics into public health and health care; and 3) to ensure that development of such a process is a good use of vital health care resources.

Authorization for this data collection is drawn from Section 301 of the Public Health Service Act (42 U.S.C. 241) Research and Investigations (Attachment B). Specifically, 241 (a)(1) on the collection and dissemination of information on the practical application of research, and 241 (a)(4) on securing the advice and assistance of experts. In this case, stakeholders such as healthcare providers, healthcare payers and purchasers, policy makers, and consumers are the experts, based on their unique ability to provide feedback on the usefulness of the information developed through the EGAPP process in making medical and personal decisions about appropriate use of new genetic tests.

Evaluation of the EGAPP pilot project also addresses broader CDC research goals related to cross cutting research on human genomics in public health, specifically building the genomic evidence base in public health practice and the “genomics bridge” between public health research and preventive medicine. See Section A.2 below for specific questions.

A.2. Purpose and Use of Information Collection

This information collection among key stakeholder groups will provide stakeholder feedback on the value and impact of the EGAPP products developed and disseminated (e.g., evidence reviews, published summaries, Working Group recommendations, informational messages). Principal questions include⁹:

• How successful was EGAPP in engaging stakeholders and in building awareness of this initiative and the need for evidence-based information on genetic tests?

• How successful was EGAPP in disseminating products and in partnering with other organizations in dissemination?

• Did EGAPP products succeed in addressing the informational needs of key stakeholder groups with regard to new genetic tests?

• Did EGAPP products bring about, or contribute to, changes in practice (e.g., medical decision-making, development of practice guidelines, adoption of EGAPP Working Group recommendations) or in coverage and reimbursement of reviewed genetic tests?

A.3. Use of Improved Information Technology and Burden Reduction

The stakeholder surveys will be as brief as possible and will be conducted electronically using SurveyMonkey, a flexible and well-described survey development tool for creating surveys and exporting collected data.¹⁰ Features include the ability to create conditional logic (i.e., skip patterns), to require answers to certain questions, and to randomize answer choices. The ability to branch questions based on response provides a streamlined process to direct respondents to the appropriate surveys and to move most effectively through the surveys. SurveyMonkey meets Safe Harbor and EU Data Protection Requirements.¹¹ Survey subjects will receive a brief statement about the project with a request to participate (See Attachment C) via an e-mail notice or interaction with the planned EGAPPreviews.org website (waiver approved by HHS in November, 2006 for implementation in early 2007).¹² A link will be provided within the message to an online survey which they will complete anonymously.

The study is being limited to key stakeholder groups for which the evidence reports and recommendations on genetic tests have the most relevance, and that can contribute the most pertinent feedback on the EGAPP pilot project. To minimize the number of questions on each survey and avoid including questions not applicable to some respondents, six focused sets of questions have been developed. By answering a short series of basic questions (first question set), respondents will determine eligibility for the study and self-identify by profession, organization and role into the five categories of stakeholders: health care providers, healthcare payers, healthcare purchasers, policy makers, and consumers/web visitors (each category has a set of questions). The responses will link them to the appropriate survey instrument: Healthcare Provider, Policy/Payer, Policy, Purchaser, or General survey (Appendix D). Each survey instrument is composed of two or three modules, depending on the roles of the individuals/ organizations to be surveyed. For example, the Policy/Payer Survey contains the basic questions plus Policy and Payer questions, and is targeted to organizations such as health plans and insurers that both set policy about the use of tests and make decisions about coverage. Responses to the on-line survey are completely anonymous.

Most survey responses will be multiple-choice or other categorical options, allowing respondents to easily indicate their choice through a single mouse click. Numbers of questions will range from 12 in the Healthcare Purchaser survey to 17 in the Policy/Payer survey. Optional write-in areas will be provided where appropriate to allow elaboration, but a written response will not be required.

Surveys include sets of specific questions about awareness and usefulness of specific products (e.g., evidence reports and recommendations on specific genetic tests) on three topics published/disseminated at the times of the surveys (see Table A.6 for the expected delivery dates on reviews currently in progress). Using SurveyMonkey, respondents who have not seen any products on a specific topic will be automatically redirected to the introductory question for the next topic, thereby answering only one question for each unfamiliar topic.

The study will be conducted in four parts, as illustrated by Figure A.6. Surveys of healthcare providers and payers will be conducted twice; the first survey of these groups will be about six months after the release of the first EGAPP products, and a second survey will occur one year later. Policy makers, healthcare purchasers, targeted consumers and general website visitors will also be surveyed twice; for these groups, the first survey will be about one year after the release of the first EGAPP products, and the second survey will be one year after that. The sampling methods for selecting the survey respondents from the five groups of stakeholders are explained in Section B of this package (Collections of Information Employing Statistical Methods). Since the pools of some respondent groups are relatively small (e.g., healthcare payers, purchasers, and policy makers), some of the same respondents may be asked to complete both the first and second surveys. Note that the organizations themselves will designate the appropriate individuals to complete Policy, Policy/Payer and Purchaser surveys as representatives of the organization. For the Healthcare Provider survey, the pool of respondents is very large for all groups except genetic counselors. Therefore, it is possible, but unlikely, that the same individual respondents will be selected for both surveys. Surveying stakeholder groups twice supports investigation of increasing awareness and use of products, allows the potential use of early results to improve development and/or dissemination of later products, ensures that later products will be assessed, and at the same time, minimizes the possibility of poor recall due to a lengthy time lag between release of early products and surveys.

A.4. Efforts to Identify Duplication and Use of Similar Information

The evidence-based reviews on genetic tests being conducted through the EGAPP project are among the first conducted in the field of genomics, and are the first undertaken by a group focused only on genomic applications. As such, they are unique and of national relevance and significance to public health and many other stakeholder groups. In addition, very little evaluation has been done to date to assess the value and impact of evidence reports on clinical practice.

The CDC-funded ACCE Project conducted five evidence reviews on genetic tests between 2000 and 2004, but did not survey stakeholders to assess value and impact.^13,14 The U.S. Preventive

Table A.1. Estimated Product Release and Survey Dates

Tests Under Review	Start Date	Estimated Products Release Date	Estimated first inclusion in Surveys
Tests for Ovarian Cancer Detection and Management	Sept 2005	Oct 2006 – Jan 2007	Apr 2007 – Mar 2008
CYP450 testing – depression, SSRIs	Oct 2005	Nov 2006 – Jan 2007	June 2007
HNPCC testing	Oct 2005	Dec 2006 – Feb 2007	June 2007
UGT1A1 testing	Feb 2006	Jan 2007	June 2007
Gene expression profiling for breast cancer management	Nov 2006	Jun – Aug 2007	June 2008
CYP450 testing – pain management, codeine	Jan 2006	Oct – Nov 2007	June 2008
Cardiogenomic profiling	Feb 2007	Jun – Aug 2007	June 2008

Services Task Force (USPSTF) has reviewed two genetic tests.^15,16 The USPSTF has a legislated and established role in providing recommendations to family physicians, but has not conducted formal surveys of health care providers or other stakeholders on the impact of their recommendations on genetic tests.¹⁷ A search of the published literature identified one recent report that describes a study on the success of dissemination of evidence reports on non-genetic topics prepared through the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Centers (EPC) Program.¹⁸ They based their review on 22 semi-structured telephone interviews conducted in 2004 with EPC directors, representatives of AHRQ, and participants from partner organizations. A further search of published (e.g., PubMed) and grey literature did not reveal any other examples of studies being conducted on the value and impact of evidence-based reviews and associated clinical recommendations, though the potential value of such information is recognized by CDC and other agencies.

Figure A.2. Estimated data collection timeline

A.5. Impact on Small Businesses or Other Small Entities

For this survey, stakeholders qualifying as small businesses are expected to be limited to physicians in solo and group practices and to a few small businesses in which individuals that function as purchasers of health care may be surveyed. Of note, physicians and healthcare payers and purchasers are among the stakeholders who are actively seeking the type of information to be provided by the EGAPP Project. As previously stated, the surveys will be as brief as possible and will utilize skip pattern logic to minimize burden. Most survey responses will be multiple-choice or another form of categorical option, allowing respondents to easily indicate their choice through a single mouse click; use of write-in fields will be optional. Medical professional organizations such as the American Medical Association (AMA) and the American Academy of Family Physicians (AAFP) have expressed interest in the EGAPP process and in participating in the evaluation to be conducted.¹⁹

A.6. Consequences of Collecting the Information Less Frequently

From the perspectives of CDC, the EGAPP Steering Committee, and the EGAPP Working Group, there are two major consequences of not seeking feedback from stakeholders about the value and impact of the pilot project: 1) Engaging and seeking feedback from key stakeholders is a essential step for the development of a transparent and credible public health sponsored process for assessment of emerging genetic tests; and 2) the information generated through these surveys will inform the development of the most effective sustainable process in the future.

Less frequent data collection would not provide an opportunity for feedback on the early and late products of EGAPP. There are no legal obstacles to reducing the burden.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

None of the special circumstances are applicable for the current project; the request fully complies with the regulation.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A. Federal register - Notice was published in the Federal Register, Volume 71, pages 36343-36344, from June 26 to August 26, 2006 to solicit comments on this information collection prior to its submission to the Office of Management and Budget (OMB) as required by 5 CFR 1320.8(d) (Attachment E).

Summary of public comments – No comments were received.

B. The following persons were consulted in 2005 and 2006 regarding the conduct of this study:

Evaluation Consultants

Judith L. Johnson, PhD President JohnSilver Associates Phone: (207) 784-0301 Email: [email protected]

Lynn Short, PhD, MPH Executive Director Analytic Systems Associates, Inc. Phone: 770-736-8867 Email: [email protected]

EGAPP Working Group

Alfred O. Berg, MD, MPH Chair, Department of Family Medicine University of Washington Phone: 206-543-3101 Email: [email protected] Katrina Armstrong, MD, MSCE Director of Research Leonard Davis Institute of Health Economics University of Pennsylvania School of Medicine Phone: 215-898-0957 Email: [email protected] Jeffrey Botkin, MD, MPH Associate Vice President for Research Professor of Pediatrics and Medical Ethics University of Utah Phone: 801-581-7170 Email: [email protected] Ned Calonge, MD, MPH Chief Medical Officer and State Epidemiologist Associate Professor Colorado Department of Public Health and Environment and University of Colorado Health Sciences Center Phone: 303-692-2662 Email: [email protected] James Haddow, MD Director, Division of Medical Screening Women & Infants Hospital Phone: 207-657-7888 Email: [email protected] Maxine Hayes, MD, MPH Washington State Department of Health Phone: 360-236-4018 Email: [email protected] Celia Kaye, MD, PhD Senior Associate Dean, Education School of Medicine University of Colorado at Denver and Health Sciences Center Phone: 303-315-0567 Email: [email protected] The following persons were consulted in 2005 and 2006 regarding the objectives of this study:

Kathryn A. Phillips, PhD Prof. of Health Economics and Health Services Research School of Pharmacy, Institute for Health Policy Studies, and UCSF Comprehensive Cancer Center University of California, San Francisco Phone: 415-502-8271 Email: [email protected] Margaret Piper, PhD, MPH Associate Director Blue Cross/Blue Shield Association Technology Evaluation Center Phone: 312-297-5895 Email: [email protected] Carolyn Sue Richards, PhD, FACMG Scientific Director, OHSU Molecular Diagnostic Center Director, OHSU DNA Diagnostic Laboratory Oregon Health & Science University Phone: 503-494-4416 Email: [email protected] Joan A. Scott, MS, CGC Deputy Director Genetics and Public Policy Center Johns Hopkins University Phone: 202-663-5975 Email: [email protected] Ora Strickland, PhD Nell Hodgson Woodruff School of Nursing Emory University Phone: 404-727-7941 Email: [email protected] Steven Teutsch, MD, MPH Executive Director of Outcomes Research Merck & Co., Inc. Phone: 215-652-2788 Email: [email protected]

EGAPP Steering Committee

Muin Khoury, MD, PhD Director National Office of Public Health Genomics, CDC Phone: 770-488-8510 Email: [email protected] Robert L. Becker, MD, PhD Director, Division of Immunology and Hematology Devices Office of In Vitro Diagnostic Device Evaluation and Safety U.S. Food and Drug Administration Phone: 240-276-0493 Email: [email protected] Joe Boone, Ph.D. Associate Director for Science Division of Laboratory Services Office of Public Health Partnerships National Center for Health Marketing, CDC Phone: 770-488-8080 Email: [email protected] Linda Bradley, PhD Geneticist / Technical Monitor for EGAPP National Office of Public Health Genomics, CDC Phone: 770-488-8399 Email: [email protected] Peter Briss, PhD Chief, Community Guide & Evidence Branch Epidemiology Program Office, CDC Phone: 770-488-8338 Email: [email protected] Ralph Coates, PhD Associate Director for Science Division of Cancer Prevention and Control, CDC Phone: 770-488-3003/4226 Email: [email protected]

Phyllis Frosst, PhD Science Policy Analyst National Institutes of Health National Human Genome Research Institute Phone: 301-496-0609 Email: [email protected] Scott Grosse, PhD  National Center on Birth Defects & Developmental Disability, CDC Phone: 404-498-3074 Email: [email protected] Dr. Suzanne Feetham, PhD, RN, FAAN HRSA Center for Quality Phone: 301-443-0458 Email: [email protected] Steve I. Gutman, MD Director, In-Vitro Diagnostics Device Evaluation and Safety FDA/DHHS Center for Devices and Radiological Health Phone: 301-594-3084 Email: [email protected] Debra Leonard, MD, PhD Weill Medical College of Cornell University Phone: 212-746-2041 Email: [email protected] Gurvaneet Randhawa, MD, MPH Center for Outcomes and Evidence Agency for Healthcare Research and Quality Phone: 301 427-1619 Email: [email protected] James A. Rollins, MD Centers for Medicaid & Medicare Services Department of Health and Human Services Phone: 410 786-4695 Email: [email protected]

Other individuals with whom we discussed this project include:

Karen Edwards, PhD Public Health Genetics Program Dept. of Epidemiology Director, Center for Genomics and Public Health University of Washington Phone:  (206) 616-1258 Email:  [email protected]   Toby Citrin, PhD Co-Director, Center for Genomics and Public Health University of Michigan School of Public Health Phone: (734) 936-0936 Email:  [email protected]

Litjen Tan, MD Director, Infectious Diseases, Immunology and Molecular Medicine American Medical Association Phone: (312) 464-4147 Email: [email protected]   Rick Carlson, JD Department of Health Services University of Washington School of Public Health and Community Medicine Phone: (206) 545-7294 Email:  [email protected]

C. There were no major problems that could not be resolved during consultation.

D. There were no other public contacts or opportunities for public comment provided or required.

A.9. Explanation of Any Payment or Gift to Respondents

There will be no payments or gifts made to respondents.

A.10. Assurance of Confidentiality Provided to Respondents

The CDC Privacy Act Coordinator has reviewed this project and has determined that the Privacy Act is not applicable. CDC will not receive identifiable response data. In the majority of cases (with the exception of the General Survey), respondents are speaking from their roles as representatives of their organizations.

Respondents will be advised of privacy safeguards. (See attachment C)

Although identifiable information may be used for recruitment and follow-up purposes by the evaluation contractor, McKing Consulting, and contact persons for respondent organizations, the response data collection process for the EGAPP project does not involve collection of personal identifiers or other sensitive information. This is true for all surveys and all respondent groups. Although the Survey Monkey online data collection system provides the option of obtaining respondents’ e-mail addresses, this option will not be selected. The Survey Monkey system collects and uses IP addresses for system administration and record-keeping purposes, but IP addresses will not be provided to CDC or the evaluation contractor. Survey responses cannot be linked or traced to any unique respondent identifiers. Privacy safeguards for the study will be described in the recruitment letter. Additional information about Survey Monkey is available at http://www.surveymonkey.com.

By agreement with CDC, the McKing consultant, Dr. Judith Johnson, will maintain the SurveyMonkey account, enable the privacy setting for SurveyMonkey respondents as noted above, implement the on-line surveys, and oversee McKing personnel who send out recruitment letters. Dr. Johnson is a professional evaluator who abides by program evaluation standards.²⁰ She will ensure that contact lists provided by organizations are safeguarded, and are appropriately destroyed after use. Dr. Johnson will also be the ONLY person who has access to the survey responses; only summary data and analyses will be provided to CDC. After data analysis is complete, original survey responses will be deleted.

CDC’s Human Research Protection Office has determined that the proposed data collection is exempt from the requirements of 45 CFR part 46 pertaining to IRB review and approval (Attachment F).

A.11. Justification for Sensitive Questions

Although website respondents are asked whether the information is relevant to their personal interests or work interests, survey questions do not ask direct questions about personal medical history. As previously noted, responses cannot be linked or traced to respondent identifiers. For healthcare providers, questions relating to their personal knowledge of genetic tests and related current practice decisions might be viewed as sensitive by a subset of respondents who are reluctant to disclose gaps in their understanding. However, the rapidly evolving field of genetics is a topic of interest and much discussion among healthcare providers, who openly admit lack of needed information, so this is unlikely to be a significant sensitivity relative to the importance of understanding the usefulness of the information provided by EGAPP.

No respondents will be asked to specify race or ethnicity. Therefore, an exemption to the HHS policy on Inclusion of Race and Ethnicity in DHHS Data Collections is requested for the following reasons: 1) Respondents to the Policy/Payer, Purchaser and Policy surveys are responding on behalf of the organizations which they represent, and, in this case, an individual respondent’s race and ethnicity is considered immaterial to the data collection and would not be analyzed if collected. 2) Given the limited resources and relatively short time frame of the project, respondents to the Healthcare Provider and General surveys will be identified through medical professional organizations and consumer advocacy/disease-specific support groups, respectively, as described in Section B.1. Based on the sample sizes and the groups surveyed, it is unlikely that analyses of race and ethnicity data would reflect the general population. Only very limited demographic information directly related to understanding relevance to the respondent of the topics under review will be collected and analyzed.

A.12. Estimates of Annualized Burden Hours and Costs

Table A.12.1 presents the type and number of respondents, frequency of response, average burden per response, and total burden to respondents. Table A.12.2 presents the annualized cost to respondents. The burden was estimated as follows:

• The number of respondents for survey data collection was derived from the statistical sampling procedures described in Section B of this package.

• The hours per response for the survey were derived by pilot testing the surveys with no more than eight subjects each.

• Annualized costs were based on hourly rates as determined by the National Industry-Specific Occupational Employment and Wage Estimates.²¹

The cost to respondents (Table A.12.2) who participate in the study will be in terms of their time only. The survey will take about 10 minutes based on a timed pretest with two persons per survey. Respondents will participate once per year; therefore, the total annual burden will be 448.52 hours. The total burden for two surveys, one year apart, would then be 897.04 hours.

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

There is no direct cost to respondents or to record keepers.

A.14. Annualized Cost to the Federal Government

Costs for this project include personnel for planning and designing the study, working with identified organizations to recruit the sample, collecting and analyzing the data, and reporting. The government costs include personnel costs for federal staff involved in the oversight, study design, OMB and IRB review, initiation of contact with stakeholder groups, data interpretation, report writing and presentation development, estimated at approximately 5 percent of a GS-14 scientist

Table A 12.1. Estimated Annualized Respondent Burden Hours

Type of Respondent	Survey Name	Number of Respondents	Number of Responses per Respondent	Average Time per Response (hours)	Response Burden (in hours)
Healthcare Providers Primary Care Providers22 Specialists Genetic Counselors Mid-level Practitioners Nurses	Healthcare Provider Survey	385 385 200 385 385	1 1 1 1 1	10/60 10/60 10/60 10/60 10/60	64.17 64.17 33.33 64.17 64.17
Healthcare Payers and Purchasers Healthcare Payers Healthcare Purchasers	Policy/ Payer Survey Purchaser Survey	100 3119	1 1	10/60 10/60	16.67 5.17
Healthcare Policy Makers	Policy Survey	50	1	10/60	8.33
Consumers Group members Website visitors	General Survey	385 385	1 1	10/60 10/60	64.17 64.17
Total Burden					448.52

Table A 12.2. Annualized Cost to Respondents

Type of Respondents	Number of Respondents	Response Burden per Respondent (in hours)	Hourly Wage Rate23	Respondent Cost
Healthcare Providers Primary Care Providers Specialists Genetic Counselors Mid-Level Practitioners Nurses	385 385 200 385 385	10/60 10/60 10/60 10/60 10/60	$75.18 $75.18 $28.03 $29.85 $26.77	$4824.3006 $4824.3006 $934.23 $1,915.47 $1,717.83
Healthcare Payers and Purchasers Healthcare Payers (General Managers) Healthcare Purchasers (General Managers)	100 31	10/60 10/60	$44.99 $44.99	$749.98 $232.59
Healthcare Policy Makers (General Managers)	50	10/60	$44.99	$374.7667
Consumers Group members Website visitors	385 385	10/60 10/60	$18.00 $18.00	$1,155.06 $1,155.06
Total				$17,883.58

for one year. Costs for contract labor hours include planning and design, development of OMB, IRB and study protocols, communication and working with stakeholder organizations to achieve sampling of members and identification of appropriate representatives, development of forms on Survey Monkey, data collection, data preparation, data cleaning, data analysis, and report development and dissemination. The overall cost of this research to the Federal Government is presented in the following table.

Table A 14.1 Estimated Annualized Cost to the Government

Labor:	Cost:
CDC personnel for oversight, communications, OMB protocol development, report writing, presentations, publications.	$3,750.00
Contract labor for planning and design, OMB, IRB and other protocol development, communications, development of forms on Survey Monkey, purchasing membership lists, sampling, data collection, preparation, entry, cleaning and analysis, report writing, presentations, publications.	$60,000.00
Other direct costs:
Copies, binding, presentation materials	$500.00
Communications - email, mailing	$500.00
Total estimated annual contract costs	$64,750.00

A.15. Explanation for Program Changes or Adjustments

This is a new data collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Table A 16.1 Project Time Schedule Task	Time Schedule
Develop surveys	May 2006 (complete)
Develop distribution protocols April and October 2007 Surveys April and October 2008 Surveys	January 2007
OMB submission	June 1, 2006
60-day Federal Register Notice 30-day Federal Register Notice OMB approval	June 26 - August 26, 2006 (complete) When submitted to HHS (written and pending) 10 months after OMB submission
First Survey Distribution (SD1)24 - Healthcare Providers and Payers
Initiate communication with identified respondent organizations	Immediately after OMB approval
Identify appropriate respondents and achieve sampling plan with respondent organizations.	Within 1 month after OMB approval
Notify respondents – Send request by post and/or e-mail with link to survey; two follow-up memos at 2-week intervals, as needed	2 months after OMB approval
Final data cleaning and analysis – SD1	6 months after OMB approval
Data interpretation, written summary to Working Group and Steering Committee – SD1	7-8 months after OMB approval
Possible changes to review or distribution methodology based on Working Group and staff deliberation on summary	8-9 months after OMB approval
Second Survey Distribution (SD2) – Policy Makers, Purchasers and Consumers
Initiate communication with identified respondent organizations	Within 4-6 months after OMB approval
Identify appropriate respondents and achieve sampling plan with respondent organizations.	Within 6-8 months after OMB approval
Notify respondents – Send request by post and/or e-mail with link to survey; two follow-up memos at 2-week intervals, as needed	8 months after OMB approval
Final data cleaning and analysis - SD2	12 months after OMB approval
Data interpretation, written summary to Working Group and Steering Committee – SD2	13-14 months after OMB approval
Possible changes to review or distribution methodology based on Working Group and staff deliberation on summary	14-15 months after OMB approval
Third Survey Distribution (SD3) - Repeat survey of Healthcare Providers and Payers
Identify appropriate respondents and achieve sampling plan with respondent organizations	Within 10 months after OMB approval
Notify respondents – Send request by post and/or e-mail with link to survey; two follow-up memos at 2-week intervals, as needed	14 months after OMB approval
Final data cleaning and analysis - SD3	Within 20 months of OMB approval
Data interpretation, written summary to Working Group and Steering Committee – SD3	21-22 months after OMB approval
Possible changes to review or distribution methodology based on Working Group and staff deliberation on summary	22-23 months after OMB approval
Fourth Survey Distribution (SD4) - Repeat survey of Policy Makers, Purchasers and Consumers
Identify appropriate respondents and achieve sampling plan with respondent organizations.	Within 16 months of OMB approval
Notify respondents – Send request by e-mail with link to survey (mail may be needed in a small proportion of surveys); two follow-up email memos at 2-week intervals, if needed.	20 months after OMB approval
Final data cleaning and analysis - SD4	Within 26 months of OMB approval
Data interpretation, written summary to Working Group and Steering Committee – SD4	27-28 months after OMB approval
Final Report and Dissemination of Results
Presentation of findings to Steering Committee and EGAPP Working Group	As required and appropriate
Development of publications and presentations Development of recommendations for a sustainable genetic test assessment process based in part on stakeholder response	As appropriate As appropriate

After all the data have been collected and analyzed, a final report and summary of the findings will be prepared. The report will include general descriptive analyses of aggregated data and a summary of quantitative findings.

Publications resulting from this research activity will be developed in collaboration with the EGAPP project team. Publications will be submitted to journals and external sources only after the findings have been presented to project staff, the EGAPP Working Group, and the EGAPP Steering Committee. The information will be primarily descriptive in nature, although comparisons within selected sub-groups will be conducted using appropriate statistical techniques (e.g., Analysis of Variance, Chi-Square, T-test). The data will be analyzed using SPSS software.

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

No exemptions are being requested.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are being requested.

B. Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Process

The EGAPP survey instruments to be used in the first survey of each group are included, along with specific objectives, in Appendix D; each survey specifically solicits responses on three of the first tests reviewed and released (CYP450 testing in patients with depression treated with SSRI drugs, HNPCC testing and colorectal cancer, and UGT1A1 testing in colorectal cancer patients treated with irinotecan). As shown in Table A.1, additional topics will be reviewed and released during the interval following the first two sets of surveys. It is planned that the second surveys will include one of the same topics for a longitudinal view (CYP450 testing and depression), along with two new topics to be decided. Note that the wording of the questions will remain exactly the same, with only the test name changed in the set of questions.

The evaluator is responsible for closely monitoring data collection and data review on a weekly basis to allow interim assessment. This monitoring of data collection will occur throughout the entire collection period, but especially during the initial stages when problems might be detected and corrected. During the administration of the surveys, the evaluator will report monthly to project staff on evaluation progress, the number of respondents to each survey to date, and response trends as they develop. Analysis will be ongoing throughout the data collection period. Email reminders will be sent to the sample to increase response rate. Once the requisite sample size has been reached, the evaluator will provide a summary and analysis of the data collected.

Descriptive statistics will be utilized to summarize the information collected, including categorization of the users of EGAPP materials, how different stakeholders learned about EGAPP, the potential relevance and usefulness of EGAPP information to individual users or organizations, respondents’ awareness and exposure to EGAPP-sponsored evidence reports and EGAPP Working Group recommendations on specific genetic tests, their ratings of the value of the evidence reports and recommendations on specific tests named in the survey (e.g., CYP450, HNPCC), and any reported examples of impact of EGAPP products on practice (e.g., healthcare provider decision-making, use of information to develop practice guidelines or determine use and coverage of reviewed tests). Quantitative analysis of responses to some questions may identify differences between some groups of stakeholders in awareness of and interest in EGAPP, uptake of products, and integration of information developed into decision-making. Qualitative analysis will be conducted on open-ended responses that specifically elicit alternative responses. The findings of analyses will be reported in interim summary reports to the EGAPP Working Group and CDC project staff.

At the end of the data collections, final summarization and analysis will again be performed and a final report will be issued. The final report will include the evaluation methodology and findings. The final report will be delivered within 4-5 months after the close of data collection. The evaluator will be available for presentations, meetings with project staff, and/or consultation about the entire project as the project staff deem necessary.

Respondents

As described in Section A1, the surveys will focus on four categories of stakeholders deemed by project advisors to have the most immediate need and interest in EGAPP reports and recommendations: healthcare providers, targeted consumers, healthcare payers and purchasers, and policy makers. Study subjects will include representative subsets of these broad categories.

Surveys of these five groups will be conducted twice as indicated in Figure A.6. Independent random samples of healthcare providers and targeted consumers will be identified from the membership of professional organizations and health advocacy groups for both the first and second surveys. With the exception of one subset of health care providers (i.e., genetic counselors), the pool of respondents is very large, so it is unlikely, that the same respondents will be randomly selected for both surveys.

For healthcare payer, healthcare purchaser, and policy maker groups, respondents will be selected by, and respond on behalf of, their organizations. The pools of these respondent groups are relatively smaller, and some are important enough to be included in both surveys. For example, the American Academy of Family Physicians is an important policy-making group that represents more than 90,000 physicians who provide about half of routine care in the United States; Blue Cross Blue Shield cumulatively accounts for more than 88 million covered lives. Therefore, such organizations will be asked to complete both the first and second surveys. Therefore, it has been assumed that individuals selected by such organizations to provide their responses would be the same for both surveys.

Healthcare Providers

Relevant subcategories of healthcare providers to be surveyed are physicians, genetic counselors, mid-level practitioners and nurses (refer to Attachment A, Figure 1).

Physicians are further stratified into primary care providers (e.g., family physicians, general practitioners, internists, obstetrician/gynecologists, doctors of osteopathy) and specialists. As noted, primary care providers and physicians from selected relevant specialty groups (e.g., American College of Medical Genetics) will be included in both Healthcare Provider surveys; sub-specialty groups will be included based on the genetic test reviews released. For example, because the first set of evidence reviews completed will include topics related to depression and breast and colorectal cancer, specialists included for the initial survey would include psychiatrists, obstetrician/gynecologists, oncologists and gastroenterologists. As additional evidence reviews are conducted, appropriate specialty groups will be identified for the second surveys.

Access to healthcare providers is planned primarily through medical professional organizations. If working through professional organizations leads to lower response rates than anticipated, the alternative plan is access through insurers (e.g., Blue Cross Blue Shield Association) and HMOs (e.g., Intermountain Healthcare, Kaiser Permanente). The evaluator and her project team will contact the organizations to determine the best method for approaching members. Large professional organizations that have expressed a need for this kind of information on genetic testing and their interest in being involved in evaluating the products (American Medical Association, American Academy of Family Physicians) will be contacted first. The cost for contacting the organizations is included in the contract labor (Table A14.1), and the burden to the organizations is expected to be minimal. These organizations routinely deal with requests to survey members, and have indicated that they can easily direct our evaluation staff to the right person to facilitate participation.

Each organization that agrees to take part in the survey will either provide a list of contact information for a random sample of their membership or contact the random sample of members directly; in either case, the subjects will receive an e-mail letter that includes a brief introductory statement (Attachment C) and a link to the survey. In some cases, it may be necessary to purchase a onetime distribution list (estimated costs included in the contract labor; Table A14.1). In the infrequent cases where members can only be approached through mailed announcements (e.g., some organizations restrict access to email addresses and do not offer the service of distributing to members), member responses will be sought by providing the website address in a written mailed letter (cost included in table A14.1).

It is not possible at this point to provide a complete list of all organizations and health plans, since decisions have not been made about future topics. However, Tables B.1.1 and B1.2 provide examples of healthcare provider professional organizations and their membership figures. This list represents professional organizations within each category (e.g., physicians, healthcare providers, healthcare organizations) from which selections will be made for inclusion in the study; those professional organizations to be contacted first are bolded. Should additional subjects be needed, Table B.1.3 provides examples of health plans and their size, provided as covered lives and number of associated physicians (if available).

Table B 1.1. Relevant Examples of Physician Professional Organizations

Organization	Membership
American Medical Association (AMA)	250,00025
American College of Physicians (ACP)	119,00026
American Academy of Family Physicians (AAFP)	94,00027
American College of Obstetricians and Gynecologists (ACOG)	49,00028
American Osteopathic Association (AOA)	56,00029
American College of Medical Genetics (ACMG)	1,38030
American Society of Clinical Oncology (ASCO)	23,51931
American College of Gastroenterology	8,50032
American Gastroenterology Association (AGA)	15,00033
American Psychiatric Association (APA)	35,00034
American College of Clinical Pharmacy	7,72335

Genetic counselors have a strong interest in the EGAPP process and can be accessed through the National Society of Genetic Counselors listserve. Mid-level practitioners will include nurse practitioners, physician assistants, and certified nurse midwives (latter group included because a planned topic involves breast cancer). Nurses represent the final category of health care providers. All will be accessed through professional organizations. Table B 1.3 lists the relevant professional organizations and their membership. Healthcare providers will be asked to complete the Healthcare Provider survey (Attachment D 1).

Table B 1.2. Professional Organizations for Non-Physician Healthcare Providers

Organization	Membership
National Society of Genetic Counselors (NSGC)	2,10036
American Academy of Nurse Practitioners (AANP)	20,000 & 119 groups37
American Academy of Physician Assistants (AAPA)	39,52338
American College of Nurse Mid-Wives (ACNM)	6,60039
International Society of Nurses in Genetics (ISONG)	32040
Nursing Organizations Allliance	69 organizations41

Table B 1.3. Relevant Examples of Healthcare Organizations

Organization	Membership (covered lives/number of physicians if available)
Health Plans / HMOs
Intermountain Health Care (IHC)	500,000 / 40042
Kaiser Permanente43 - Colorado	440,000 / >700
Kaiser Permanente - Northern California	3.2 million / 4,400
Kaiser Permanente - Southern California	3.1 million / 3,600
Group Health Cooperative Health Care	590,000 / access to 6,00044
HarvardPilgrim Healthcare	924,000 / access to 22,00045
Insurers46
Aetna	13.6 million
CIGNA	9.7 million
Blue Cross Blue Shield	>88 million
UnitedHealth	>18 million / access to 433,000
WellPoint (BCBS licensee in 14 states)	28 million
Umbrella Groups
America’s Health Insurance Plans (AHIP)47	1,300 member companies
Blue Cross Blue Shield Association (BCBSA)48	38 member BCBS plans

Healthcare Payers & Purchasers

Healthcare payers include the same group of health plans, HMOs, and insurers described in section B 1.2 (examples in Table B 1.2). In section B 1.2, these groups were described as a secondary approach to provide access to healthcare providers. For this purpose, representatives of the organizations who are responsible for decision making on coverage/reimbursement for genetic tests would be targeted. Such organizations could be approached individually, but the primary plan will be to access them more effectively through sponsorship of the survey by umbrella organizations. For example, access to a large number of health plans/payers is possible through America’s Health Insurance Plans (1,300 members) and the Blue Cross Blue Shield Association (38 BCBS plans). Because most of these organizations also set policy, the representatives will be asked to complete the Policy/Payer Survey (Attachment D 2).

Healthcare purchasers are a heterogeneous group that includes small businesses, large corporations, the government, and group purchasing organizations. The Health Industry Group Purchasing Association (HIGPA) is a broad-based trade association that represents 31 purchasing organizations (e.g., for-profit and not-for-profit corporations, purchasing groups, associations, multi-hospital systems, health care provider alliances). HIGPA has expressed interest in the products of the EGAPP Project and claims a very high response rate for their sponsored surveys.⁴⁹ Member representatives of purchasing organizations will be asked to complete the Purchaser Survey (Attachment D 3).

Policy Makers

As noted in the page 6 footnote, policy makers targeted for this study are non-federal, non-regulatory decision makers from professional organizations that set policy for members and develop clinical/”best practice” guidelines (examples in Table B 1.1), public health programs, and organizations that advocate policy (e.g., Secretary’s Advisory Committee on Genetics, Health and Society, Genetics and Public Policy Center). Public health workers who influence and implement policy will be accessed through a program that has interest in two early EGAPP reviews of cancer-related genetic tests. The National Comprehensive Cancer Control Program (NCCCP) is comprised of representatives from each state and two tribes. CDC’s Comprehensive Control Cancer Branch will facilitate the dissemination of the Policy Survey to a subset of NCCCP Program Directors by informing them of the survey and aiding in the distribution. Two federal policy groups, FDA and CMS, also set policy, but their perspective on the quality and usefulness of EGAPP products will be sought through their participation in the EGAPP HHS interagency Steering Committee, not through these surveys (rulemaking based on EGAPP information is unlikely until the process has been validated).

As described in the previous section, one representative from each of the policy organizations who has responsibility for decision making with regard to policy for genetic testing will be sought and asked to complete the Policy Survey (Attachment D4). As noted in the section above on Healthcare Payers, a representative of organizations that both set policy and determine coverage/ reimbursement will be asked to complete the Policy/Payer Survey (Attachment D2).

Consumers

As previously noted, secondary translation and dissemination of information from the evidence reviews and EGAPP Working Group recommendations will almost certainly be needed for consumers. Based on limited resources and the relatively short time frame of the project, an early decision was made not to attempt to sample the “general public”, but to limit the evaluation to targeted groups of consumers who are considered likely to seek out this type of information on emerging genetic tests. The consumers to be targeted are those who belong to consumer advocacy groups (both genetics and general medicine focused) or disease-specific support groups related to the topics under review (e.g, Colon Cancer Alliance related to the review on testing for heritable forms of colon cancer) (refer to Attachment A, Figure 4). As an example, the Genetic Alliance Access Action Team is currently organizing an online and ongoing discussion on the topic of genetic testing. We expect to be able to work with these organizations to utilize such existing web-based communication systems to reach consumers and direct them to the secure Survey Monkey link. Table B 1.4 lists relevant examples of these different types of consumer groups; primary contacts are bolded. Representatives of consumer groups will be asked to complete the General Survey (Attachment D 5).

In addition, there will be two periods when visitors reviewing information on the EGAPP website will be invited to respond to a survey. The first questions will identify the background of each visitor; those website visitors that belong to other identifiable groups will be skipped to the appropriate specific survey. For example, a health care provider would be skipped from the General to the Healthcare Provider survey questions. Consumers will complete the General Survey. For each of the two survey periods, the request to complete the survey will remain on the website for 3 months or until the planned sample size for consumers is reached. Although these respondents will comprise a convenience sample and responses will be completely voluntary, it is anticipated that respondents will represent a population of interested parties using the website.

Table B 1.4. Relevant Examples of Consumer Organizations

Organization	Membership
Advocacy Organizations - Genetics focused
Genetic Alliance	> 600 organizations50
Council for Responsible Genetics	N/A51
Personalized Medicine Coalition	74 organizations52
Advocacy Organizations - General medicine
Center for the Advancement of Health	N/A53
Partnership for Prevention	46 organizations54
Center for Accelerating Medical Solutions (Faster Cures)	N/A
Topic Specific
Colon Cancer Alliance (CCA)	27,00055
National Mental Health Association (NMHA)	340 affiliate organizations56

B.2. Procedures for the Collection of Information

Timing of Surveys

Four survey distributions will be staggered at intervals of six months (see Table A 16.1, Project Time Schedule and Figure A.6). Feedback from healthcare providers and payers suggests that they are the most interested and ready to receive and use evidence reports and Working Group recommendations.⁵⁷ Therefore, they will be the subjects of Survey Distribution 1, beginning about 6 months after the release of the first products. This group will be resurveyed one year later (Survey Distribution 3).

As previously described, consumers, policy makers, and healthcare purchasers are expected to receive and be impacted by information developed by EGAPP somewhat later. Therefore, these groups will be the subjects of Survey Distribution 2, beginning six months after the initiation of Survey Distribution 1 (one year after the first release of products). This group will be resurveyed one year later (Survey Distribution 4). The two periods of time when the General Survey is offered to visitors on the EGAPP website will begin at the same times as Survey Distributions 2 and 4.

Numbers of Respondents

The goal is to secure a response rate above 30%; however, we expect that the response rate will be higher. Projecting what the response rate will be is difficult, as research on survey methodology suggests that the rate could be as low as 18% or as high as 80%.^58,59 Response rates in excess of 70% have been achieved in surveys on a variety of topics from crime victimization to attitudes about community growth⁵⁹, but are uncommon in healthcare-related surveys. A recent review article on physician response to surveys found that response rates as low as 34% were reported; a study comparing response rates among surgeons for internet versus mailed questionnaires found a 45% response rate among the internet group. ^60,61 One professional organization contacted by McKing staff, the American Academy of Family Physicians, reported survey response rates of 40-50%; however, it has been their experience that two contacts (original plus a reminder) were needed to reach this response rate. It is known that people who are notified that their survey has not yet been received are more likely to respond than when a deadline is set for survey return (response rates 38% and 23%, respectively).⁵⁹

The basic assumption underlying successful research is that “a person is most likely to respond to a questionnaire when the perceived costs of doing so are minimized, the perceived rewards are maximized, and the respondent trusts that the expected perceived rewards will be delivered”.⁵⁹ Dillman also suggests general principles in constructing successful questionnaires and that a follow-up reminder should be sent to all recipients of the survey one and three weeks later. Using these techniques for survey research, it is anticipated that our response rate will be above 30%. Because no incentives will be provided, response rates will to some extent reflect levels of interest among stakeholder groups.

For the healthcare provider and consumer groups in which individuals will be randomly selected from members of professional and advocacy organizations for inclusion in the study, the Margin of Error was calculated to obtain representative samples of each group for each survey period. Using a formula for large populations, it was determined that for the most conservative estimates, 385 respondents would be required in each of the subcategories to obtain a 5% Margin of Error with a confidence level of 95%.⁶²

Using the reported conservative response rate of 40% for this group, over-sampling to 965 respondents per healthcare provider subcategory was determined (see Table A 12.1) and would provide for a total of 1930 respondents surveyed per healthcare provider category in two surveys. Efforts will be made to maximize response rates to ensure validity and generalizability of results. In particular, general practitioners and specialists are considered a critical response group. To ensure an adequate rate of response, a second distribution of surveys is planned for physicians in Survey 1 and Survey 3 if the response rate is less than 30%. The number of respondents included in a second distribution for each survey would be proportional; that is, sufficient respondents will be targeted within each group to increase the response rate to 30% (based on the initial response rate).

Both the American Academy of Nurse Practitioners and the American Academy of Physicians Assistants have more than 20,000 members, but less is known about response rates. However, obtaining the 40% level of response should be possible for mid-level practitioners and for nurses. Numbers of genetic counselors are much smaller; an oversample of 400 per survey period should provide 190 respondents per survey period for a 7% margin of error with a 95% confidence interval (also assuming a 40% response rate).

For the healthcare payer, purchaser, and policy maker categories, only one individual will be surveyed per organization. Due to the interest shown by healthcare payers in the EGAPP pilot project and their expressed desire to participate in the evaluation, we are expecting a response rate of at least 50%. Assuming a 50% response rate, an oversampling of 200 healthcare payer organizations per survey period would provide 100 respondents per survey period for a 9.4% margin of error with a 95% confidence interval. This estimate was obtained using the most conservative estimate of response distribution (.5) and an approximate number of 1,300 healthcare payer organizations in the U.S. Though there are 1,300 healthcare payer organizations, a small number of these groups provide the majority of covered lives and will be the primary contacts for the surveys (see Table B 1.2).

There are fewer organizations currently setting or debating policies related to genetic testing. These are mainly professional organizations (e.g., American College of Medical Genetics, American Society of Clinical Oncologists), policy groups (e.g., Genetics and Public Policy Center, Secretary’s Advisory Committees), and payers (e.g., health plans, insurers). The objective of surveying representatives of 50 such organizations is based primarily on expert consultation, expecting to recruit responses from representatives of 10 professional organizations, 10 policy groups, and 30 payers. There are far fewer purchasing organizations, so responses will be solicited through mainly through HIGPA.

On the website, the General Survey will be posted for three months per survey period or until the representative sample of 385 consumers visiting the website has been received per survey period. Numbers of weekly web hits will be compared to weekly rates of survey uptake to gain some measure of non-response.

Descriptive Statistics

Descriptive statistics will be used to summarize results of surveys for all stakeholder groups. Additional quantitative statistical methods will be employed to compare responses among provider groups (e.g., primary care providers, specialists, mid-level practitioners, nurses). Similarly, healthcare providers, healthcare payers, consumers and policy makers will be compared to investigate differences in awareness of and interest in EGAPP, uptake of products, and integration of information developed into decision-making. Analysis of variance techniques will be used with continuous or Likert-type scaled items, and Cross-tabs / Chi-Square analyses will be used with dichotomous data.

Power Calculations

Factor A in Table B.2.1 below represents power calculations for four healthcare provider groups (primary care providers, specialists, mid-level practitioners, nurses). Using the proposed design, there is sufficient power to detect small differences in responses between these primary user groups.

Table B.2.1: Power Calculations for Four Healthcare Provider Groups

Table B 2.2 shows that any five groups may be analyzed with sufficient power to detect statistically significant differences in response at the p < 0.05 level (moderate) if there are at least 45 respondents per group.

Table B 2.2. Power Calculations for the Five Healthcare Provider Groups

The data will be collected by Dr. Johnson, contracted through McKing Consulting Corporation. She will work closely with stakeholder member organizations to ensure the most appropriate methods for identifying and accessing constituents. Preliminary conversations with professional organization representatives revealed that various methods may be required for organizationally approved purposes, from renting or purchasing member lists to contacting members on behalf of or through the organization (considered in contractor costs). Staggering surveys at six month intervals will match dissemination of information to likely stakeholder access and allow better project management, as the evaluation team will be able to devote more time to ensure that the recruitment of each stakeholder group is properly addressed.

B.3 Methods to Maximize Response Rates and Deal with Non-Response

Online survey methodology employing Survey Monkey was selected as the most suitable means of collecting data for this wide range of stakeholders. This online system can make the survey much easier and shorter for those needing to skip questions for unfamiliar topics.

Some studies have suggested that response rates for web-based surveys are not as high as those for mailed paper surveys; others, however, have found no differences.^63,64,65 One significant study found that a web-based survey application achieved a comparable response rate to a mail hard copy questionnaire when both were preceded by an advance notification.⁶⁶ Informal contacts by EGAPP staff with organization representatives at conferences and meetings have indicated interest in EGAPP and anticipation of the products; it is expected that most organizations that are approached will be willing to participate in or facilitate the study. Some organizations report that members are more willing to participate if their organization endorses the study or sends out announcements, and the project team will work with organizations to garner their support in this endeavor. It is hoped that email letters, received from the organization, CDC, and EGAPP announcing the study and preceding an e-mailed link to the survey, will be well received.

Initial contacts at specific organizations will initially be sent a brief introductory email announcing the project and specifying the evaluation staff member who will be making contact to discuss the EGAPP pilot project and the survey study. It is anticipated that potential respondents (e.g., organization members) will be emailed an initial announcement and invitation to participate containing a link to the survey. The procedure may vary somewhat depending on recommendations from and involvement with each organization. It is intended that members will be sent two email reminder requests to ensure that those willing to participate and provide feedback will have an opportunity.

B.4. Tests of Procedures or Methods to be Undertaken

To explore the feasibility of working with healthcare and professional organizations, a number have been informally contacted at scientific meetings and by telephone by EGAPP staff and consultants to discuss possibilities and options. To ensure appropriate content and format, draft survey instruments were reviewed by multiple members of the project team and project consultants during development. The project consultants included representatives from the applicable stakeholder groups (e.g. healthcare provider, payer organization representative), with their feedback resulting in several changes in terminology and wording of questions. In addition, the instruments were pre-tested with nine individuals of various backgrounds to determine response time. Because the questions relate to information that is not yet available, the individuals pre-testing the instruments were asked to record the amount of time taken to read each question and response, and consider the various response options.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The design and development of the projected statistical analyses and calculations to ensure appropriate sample sizes were completed by McKing consultant, Lynn M. Short, PhD, MPH. Separate online surveys were developed by Dr. Judith Johnson for each stakeholder group and sub-group to ensure appropriate tracking of response sources. Data collection activities and data analysis will be overseen by Dr. Johnson.

List of Attachments

ATTACHMENT A: STAKEHOLDER CATEGORIES - Figures 1-4

ATTACHMENT B: PUBLIC HEALTH SERVICE ACT

ATTACHMENT C: INTRODUCTORY NOTICE

ATTACHMENT D: STAKEHOLDER SURVEYS

ATTACHMENT D: D1 thru D5 SURVEYS

EGAPP Internal Designations

(Respondents will be directed to the appropriate survey,

all designated on the web as “EGAPP Survey”)

D1. Survey

1. Healthcare Provider Survey (HPS)

Healthcare Providers includes:

• Primary Care Providers

• Specialists

• Genetic Counselors

• Mid-level Practitioners

• Nurses

D2. Survey

2. Policy/Payer Survey (PPS) includes

• Healthcare Payers

D3. Survey

3. Purchaser Survey (PS) includes:

• Healthcare Purchasers

D4. Survey

4. Policy Survey (PoS) includes:

• Policy Survey(Healthcare Policy Makers)

D5. Survey

5. General Survey (GS) includes:

• Consumers

Group Members

Website Visitors

EGAPP Paragraph (referenced in surveys)

ATTACHMENT E: FEDERAL REGISTER NOTICE

ATTACHMENT F: IRB EXEMPTION LETTER

ATTACHMENT A: STAKEHOLDER CATEGORIES

Figures 1- 4

1 Holtzman NA, Watson MS. Promoting Safe and Effective Genetic Testing in the United States. Final Report of the National Institute of Health - Department of Energy (DOE) Task Force on Genetic Testing, 1997; http://www.genome.gov/10001733 .

2 Enhancing the Oversight of Genetic Tests: Recommendations of the SACGT, 2000. http://www4.od.nih.gov/oba/sacgt/reports/oversight_report.htm, accessed November 29, 2005.

3 Burke W, Atkins D, Gwinn M, Guttmacher A, Haddow J, Lau J, Palomaki G, Press N, Richards CS, Wideroff L, Wiesner GL. Genetic Test Evaluation: Information Needs of Clinicians, Policy Makers, and the Public. Am J Epidemiol 2002;156:311-18.

4 Enhancing the Oversight of Genetic Tests: Recommendations of the SACGT; January 2001. http://www4.od.nih.gov/oba/sacgt/gtdocuments.html.

5 Coverage and Reimbursement of Genetic Tests and Services; February, 2006. http://www4.od.nih.gov/oba/sacghs/reports/reports.html#coverage.

6 Most recently November 14, 2006; webcast http://www4.od.nih.gov/oba/SACGHS/meetings/Nov2006/SACGHSNov2006meeting.htm.

7 Figures 1-4 in Attachment A illustrate the types of individuals and organizations that fall into these broad categories; further description is included in section B.1. Notes: 1) “Policy makers” targeted for this study are non-federal, non-regulatory decision makers from organizations that set standards and produce clinical guidelines for healthcare providers, public health programs, and organizations that advocate policy (e.g., Secretary’s Advisory Committee on Genetics, Health and Society; Johns Hopkins Genetics and Public Policy Center). Two federal policy groups, FDA and CMS, also fall into this category, but their perspective on the quality and usefulness of EGAPP products will be sought through their participation in the EGAPP HHS interagency Steering Committee, not through these surveys (rulemaking based on EGAPP information is unlikely until the process has been validated). 2) Certain stakeholder organizations appear in more than one category. For example, health plans and health insurers provide an alternative path to reach individual healthcare providers to assess their awareness of EGAPP products and the potential impact of the evidence and recommendations on their medical decision-making. However, individuals and committees within those same health plans and health insurers would also be policy makers and/or payers, responsible for setting organizational policy on utilization and/or coverage of genetic tests and related medical care.

8 Genetics and Public Policy Center Professional Guidelines Meeting, February 1, 2006. www.dnapolicy.org/resources/Professional_Guidelines_Meeting_Summary.pdf .

9 Specific objectives for each survey can be found in Attachment D.

10 Westin, K. Product Review. Digital Web Magazine. August, 22, 2005. http://www.digital-web.com/articles/survey_monkey/.

11 SurveyMonkey website: http://surveymonkey.com/AdvancedFeatures.asp.

12 Recruitment of participants by Dr. Johnson and collaborating partners (e.g., professional and other organizations, health plans) will be done primarily by email and web announcements, but, for a small number of organizations, may involve letters distributed by regular mail that contain information about the surveys and web links to surveys. See section B.6 for a more detailed description of methods.

13 Haddow JE, Palomaki GE: ACCE: A Model Process for Evaluating Data on Emerging Genetic Tests. In: Human Genome Epidemiology: A Scientific Foundation for Using Genetic Information to Improve Health and Prevent Disease. Khoury M, Little J, Burke W (eds.), Oxford University Press, pp. 217-233, 2003.

14 Introduction to ACCE: A CDC Sponsored Project, http://www.cdc.gov/genomics/gtesting/ACCE.htm, accessed February 28, 2006.

15 US Preventive Services Task Force. Genetic risk assessment and BRCA mutation testing for breast and ovarian cancer susceptibility: Recommendation Statement. Ann Intern Med. 2005 Sep 6;143(5):355-61.

16 US Preventive Services Task Force. Screening for Hemochromatosis: Recommendation Statement. Ann Intern Med 2006 Aug 1;145(3):204-08.

17 Personal communication, USPSTF Chair (also a member of the EGAPP Working Group).

18 Matchar DB, Westermann-Clark EV, McCrory DC, Patwardhan M, Samsa G, et al. Dissemination of Evidence-based Practice Center Reports. Ann Intern Med 2005;142:1120-5.

19 Personal communication from Linda Bradley’s discussions with representatives of AMA, AAFP and AAPA.

20 Standards for Evaluations of Educational Programs, Projects and Materials (1981); The Joint Committee Program Evaluation Standards (1994); and American Evaluation Association, Task Force on Guiding Principles for Evaluators, William R. Shadish, Chair, (1994).

21 U.S. Department of Labor Bureau of Labor Statistics. National Industry-Specific Occupational Employment and Wage Estimates. Accessed 2/28/06.

22 Primary Care Providers include family physicians, internists, obstetrician/gynecologists, general practitioners, and doctors of osteopathy.

23 U.S. Department of Labor Bureau of Labor Statistics. Mean Hourly Estimated Income Based on November 2004 National Industry-Specific Occupational Employment and Wage Estimates. Accessed 2/28/06.

24 See Data CollectionTimeline (Figure 6)

25 Personal communication with Erin Woods, February 2006

26 Website: http://www.acponline.org/college/aboutacp/aboutacp.htm?hp, accessed 2/28/06

27 Website: http://www.aafp.org/x7637.xml, accessed 2/28/06

28 Website: http://www.acog.org/from_home/acoginfo.cfm, accessed 3/6/2006

29 Website: http://www.do-online.osteotech.org/index.cfm?PageID=aoa_main, accessed 2/28/06

30 Personal communication with organization representative, February 2006

31 Personal communication with Deanna Hatcher of ASCO, February 2006

32 Website: http://www.acg.gi.org/physicians/about.asp, accessed 3/6/06

33 Website: http://www.gastro.org/wmspage.cfm?parm1=254, accessed 3/6/2006

34 Website: http://www.psych.org/about_apa/, accessed 3/6/2006

35 Personal Communication with Crystal Fields of ACCP, February 2006

36 NSGC Website: http://www.nsgc.org/, accessed 2/28/06/

37 Website: http://www.aanp.org/NR/rdonlyres/ejkt2t2ktqgeu3qvch5hqmq53feeus57x7xjvtczfepxlpu2

spjnaat6e56owslmp76ovjrx6qiiikhussnfwp6mcqh/2006+Fact+Sheet+3-06-06.pdf, accessed 3/6/2006

38 Website: http://www.aapa.org/glance.html, accessed 3/6/2006

39 Personal Communication with ACNM representative, February 2006

40 Personal Communication with ISONG representative, February 2006

41 Website: http://www.nursing-alliance.org/member.cfm and personal communication with Nursing Organizations Alliance representative, 3/7/2006.

42 Website: http://intermountainhealthcare.org/xp/public/aboutihc/, accessed 2/28/06.

43 Kaiser National Stats (and source for regional data): https://newsmedia.kaiserpermanente.org/kpweb/fastfactsmedia/entrypage2.do#3, accessed 2/28/06.

44 Website: http://www.ghc.org/about_gh/co-op_overview/index.jhtml, accessed 2/28/06.

45 HarvardPilgrim Website: http://www.harvardpilgrim.org/portal/page?_pageid=213,65233&_dad=portal&_schema=PORTAL, accessed 2/28/06.

46 Health Care Delivery Covered Lives - A Summary of Findings (for 2005). Health Care Delivery Program, Harvard University. www.ksg.harvard.edu/cbg/hcdp/ numbers/Covered%20Lives%20Summary.pdf.

47 AHIP Website: http://www.ahip.org/content/default.aspx?bc=31/.

48 BCBSA Website: http://www.bcbs.com/whoweare/index.html.

49 Personal communication with Nancy Hughes of HIGPA, February, 2006.

50 The Genetic Alliance “represents more than 600 advocacy, research and healthcare organizations that represent millions of individuals with genetic conditions and their interests.” http://www.geneticalliance.org/ws_display.asp?filter=about_who_we_are, accessed 2/28/06.

51 Non-profit organization that “works to distribute accurate information and represent the public interest on emerging issues in biotechnology.” They have staff and a Board of Directors.

52 Personalized Medical Coalition is made up of industry, government agencies, research agencies, patient advocacy groups, venture capital, industry & trade associations and strategic partners.

53 The Center for the Advancement of Health is “an independent nonprofit corporation that translates to the public the latest evidence-based research on health, health care, prevention and chronic disease management.” They have staff and a Board of Directors.

54 Personal Communication with Partnership for Prevention representative, February 2006

55 Website: http://www.ccalliance.org/what/about/about.html, accessed 3/7/2006.

56 Website: http://www.nmha.org/, accessed 3/7/2006.

57 Genetics and Public Policy Center. Genetics and Public Policy Center. www.dnapolicy.org/resources/Professional_Guidelines_Meeting_Summary.pdf accessed February 1, 2006.

58 Oppenheimer J, Nelson HS. Skin Testing: A Survey of Allergists. Ann Allergy Asthma Immunol. 2006: 96:19-23.

59 Dillman D., Mail and Other Self-Administered Questionnaires. Handbook of Survey Research, Peter H. Rossi, James D. Wright, and Andy B. Anderson Eds. Academic Press, Inc., Harcourt Brace Jovanovich, San Diego, 1983, pp. 359-377.

60 Kellerman SE, Herold J. Physician response to surveys. A review of the literature. American Journal of Preventive Medicine 2001;20(1):61-67.

61 Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, Devereaux PJ, Guyatt GH. Internet Versus Mailed Questionnaires: A Randomized Comparison. Journal of Medical Internet Research 2004;6(3):e30).

62 The following formula for large populations was used: n = (t² x p(1-p)) / m², where n = required sample size; t = confidence level (95% = 1.96); p = estimated prevalence of awareness, usefulness, etc. (0.5 is most conservative); and m = margin of error at 5% (standard value of .05).

63 Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, Devereaux PJ, Guyatt GH. Internet Versus Mailed Questionnaires: A Randomized Comparison. Journal of Medical Internet Research 2004;6(3):e30.

64 Yun GW, Trumbo CW. Comparative Response to a Survey Executed by Post, E-mail, and Web Form. Journal of Computer-Mediated Communication 2000; 6(1).

65 Thompson, L. F., Surface, E. A., Martin, D. L., & Sanders, M. G. (2003). From paper to pixels: Moving personnel surveys to the Web. Personnel Psychology, 56 (1), 197-227.

66 Kaplowitz MD, Hadlock TD, Levine R. A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly 2004; 68(1): 94