Supporting Statement Part B -- Eval of ARRA CER Dissemination Efforts 3-15-2012

Three cross-sectional mail surveys of clinicians will be conducted to assess their awareness, understanding, use, and perceived benefits of CER, specific CER topics, and the EHCP over time. The target population will include doctors who are primary care physicians (i.e., internists, family and general practitioners, pediatricians), hospitalists, or other specialists who spend the majority of their time in direct patient care. The statistical samples will be purchased from Medical Marketing Services (MMS) Inc. MMS maintains a list of physicians and allied health professionals derived from the American Medical Association (AMA) Masterfile. The AMA Physician Master File is the most comprehensive list of physicians in the United States (including both members and non‐members of the AMA). In addition to names and addresses, various demographic and practice related variables will be obtained for the selected samples. MMS updates its list weekly through several techniques, including publication mailings from the AMA; therefore, each provider’s address is current and specialty type known. This removes the need to rescreen individuals for eligibility based on specialty type because the MMS will have already identified providers who meet that as a selection criterion. Exhibit 1 describes the universe and sample for the clinician survey.

Exhibit 1: Clinician Survey Universe and Sample

Clinician	Universe	Number Sampled	Anticipated Response Rate	Number of Anticipated Responses
Patient care physicians in the U.S.	Hundreds of thousands	7,704 in total (2,568 in each of three survey waves)	75%	5,778 in total (1,926 in each of three survey waves)

Consumer/Patient Surveys

Two cross-sectional telephone surveys with consumers/patients will be conducted to assess their awareness, understanding, use, and perceived benefits of CER, specific CER topics, and the EHCP over time. Both surveys will be conducted by telephone, which will enable data to be collected from a large, representative sample of targeted consumers/patients across the United States.

For each of the two waves of the survey, a representative sample of 3,200 households will be purchased from a sample vendor. AHRQ’s evaluation contractor for this project has long-standing relationships with several nationally-known sample providers that could provide the samples from across the country, including Survey Sampling International (SSI) and Genesys.

Sampled households will be screened to identify eligible survey participants. Since the consumer/patient survey aims to collect data from individuals who receive health care services, respondents will be considered eligible if they meet the following criteria: (1) they have visited a doctor or other health care professional in the past 12 months, and (2) the visit resulted in some type of treatment. Focusing on a national sample, but as appropriate, the AHRQ’s evaluation contractor will review the geographic areas, targeted medical condition, or consumer populations that the dissemination contractors have targeted. This information may be used to refine the sample to those areas and populations. Names, addresses, and telephone numbers will be obtained for the selected samples. Using these criteria, a screening response rate of 40 percent is expected (for a total of 1,280 eligible participants). The target response rate among those screened as eligible is 80 percent (1,000 completed surveys). Exhibit 2 describes the universe and sample for the consumer/patient survey.

Exhibit 2: Consumer/Patient Survey Universe and Sample

Consumer/Patient

Universe

Number Sampled

Anticipated Response Rate

Number of Anticipated Responses

Residents of the U.S.

Millions

6,400 for screening (3,200 in each of two survey waves)

2,560 for survey

(1,280 in each of two survey waves)

40 percent for screening

80 percent for survey

2,000 in total (1,000 in each of two survey waves)

Health Care System Decision Maker Surveys

One cross-sectional telephone survey with health care system decision makers will be conducted to assess their awareness, understanding, use, and perceived benefits of CER, specific CER topics, and the EHCP. Health care system decision makers include people working for health plans, integrated health systems, insurance companies, hospitals, group practices, and long-term care institutions, pharmaceutical and other health product firms that are developing new drugs, medical devices, tests, or ways to deliver health care. The survey will be conducted by telephone.

The description of the sampling process and methodology for the health care system decision makers is not included in this OMB submission. It will be submitted as part of the renewal application for Years 4 and 5 of the project.

Clinician Focus Groups

Six telephone follow-up clinician focus groups will be conducted after completion of the first cross-sectional survey, and six telephone follow-up clinician focus groups will be conducted after the third cross-sectional survey. Participants will be identified from among those who participated in the mail survey. The mail survey will include a question asking for permission to contact the participant again for possible inclusion in a one-hour telephone focus group session. Those who answer that they would be interested in participating will be asked to provide their contact information on a separate form included in the survey packet. It is expected that more people will be willing to participate than can be accommodated in the focus groups. From among those that indicate they would be interested in participating in focus groups, the clinicians will be segmented into the following groups: (1) those who report awareness of CER and have self-reported use of CER in their clinical practice; (2) those who report awareness of CER and have self-reported non-use of CER in their clinical practice; and (3) those who report no awareness of CER. Two focus groups will be conducted per segment. Additional criteria, including how they answered other survey CER awareness and use questions, specialty type, gender, and years in practice will be considered in identifying participants. As needed, if counterintuitive results exist in the survey, focus group participants can be selected based on their responses to survey items where focus group responses can provide additional clarification.

For clinicians, telephone focus groups are the best method so that selection is not limited to a small number of market areas. Since many clinicians have demanding schedules, the telephone focus group format removes travel time and allows them to fit the focus group into their otherwise busy schedule. Based on experience, to ensure six focus group participants, 12 clinicians will be recruited for each focus group. It is expected that more people will be willing to participate than can be accommodated in the focus groups. To help increase attendance, an incentive of $150 per person will be offered.

Consumer/Patient Focus Groups

Twelve telephone follow-up consumer focus groups will be conducted after completion of the first cross-sectional survey. Participants will be identified from among those who participated in the telephone survey. The survey will include a question asking for permission to contact the participant again for possible inclusion in a 1.5 hour telephone focus group session. From among those who indicated they would participate in a focus group, additional selection criteria will be applied. Ten consumers will be recruited for each focus group, and it is expected that eight will attend. Participant recruitment will take no longer than five minutes. The focus groups will last approximately 90 minutes.

Responses to the survey will be used to segment the consumers by four regions and three CER awareness categories within each region. The four Regions are:

Region 3: Philadelphia, PA (urban)
Region 4: Atlanta, GA (urban)
Region 7: Kansas City, MO (semi-rural)
Region 9: San Francisco, CA (urban)

The three CER awareness categories are: (1) those who report awareness of CER and have self-reported use of CER in medical decision making; (2) those who report awareness of CER and have self-reported non-use of CER in medical decision making; and (3) those who report no awareness of CER. Four focus groups will be conducted per segment. In selecting participants, additional criteria may also be considered, including how they answered CER awareness questions, and particular demographic or medical characteristics. As needed, if counterintuitive results exist in the survey, focus group participants can be selected based on their responses to survey items where focus group responses can provide additional clarification.

Based on experience, to ensure eight focus group participants, ten will be invited to the group (i.e., 80 percent attendance rate). To help increase attendance, an incentive of $75 per person will be offered. This is a standard value used. Incentives offered for focus group participants of similar audiences are $75.^¹

For this audience, telephone focus groups are highly effective in understanding the perceptions and experiences of the participants. The interactive and dynamic nature of the discussions will help foster insights and provide an opportunity to probe for more information. To ensure that the study accommodates people who work during the day, lunchtime and evening time slots will be offered. An experienced moderator will conduct the focus group sessions, and a note taker will take notes on the discussion. The focus groups also will be audio recorded. Written consent for audio taping will be obtained from participants.

Mini Focus Groups with Health Care System Decision Maker, Purchasers, and Policy Makers

The team will conduct six mini telephone focus groups in year 3 of the project. These will include two focus groups with each of the following audiences: health care system decision makers purchasers, and policymakers. The focus groups will be used to determine how people receive and interpret CER-related materials and verbal information, and adopt new behaviors based on information they receive. Eight participants will be recruited for each focus group, and it is expected that six will attend. Participant recruitment will take no longer than five minutes. The focus groups will last approximately 60 minutes.

Based on experience, to ensure six focus group participants, eight will be invited to the group (i.e., 80 percent attendance rate). To help increase attendance, an incentive of $75 per person will be offered. This is a standard value used. Incentives offered for focus group participants of similar audiences are $75.^²

Health Care System Decision Maker, Purchasers, and Policy Maker Focus Groups

The description of the sampling process and methodology for the health care system decision makers, purchasers, and policy maker focus groups are not included in this OMB submission. It will be submitted as part of the renewal application for Years 4 and 5 of the project.

2. Information Collection Procedures

Clinician Surveys

The clinician surveys will be mail-based paper surveys (as described in Section 1 above) that will allow the estimation of general trends in the key study metrics among the U.S. population of clinicians, through cross-sectional surveys at three points in time. The surveys will be one clear source of insight into the so-called “secular trend” of CER awareness and use. They will be used to estimate changes in the proportion of clinicians who are: aware of CER broadly; specific CER topics; aware of the EHC Program; consider themselves to seek out and understand CER products and research (generally and AHRQ-specific); consider CER to be beneficial; consider themselves to incorporate CER into their practice; and consider themselves to discuss CER with their patients. The surveys also will seek to elucidate some of the barriers to CER consumption and adoption.

The statistical goal of the survey analysis will be to estimate proportions and changes in those proportions across the three waves of the survey. There are multiple sources of CER information available to the curious clinician and trends in CER awareness and use are probably on the rise. Estimating important proportions at three points in time will allow for the estimation of not only the rate of change of CER awareness and use, but also whether that rate is accelerating or decelerating between the three survey time points. The analysis of proportions will account for complex features of the sample including possible stratification by clinical specialty. The responses will be weighted using the appropriate proportion of the clinician population that each response represents (calculated using weights from the sampling frame) and results will be reported with confidence intervals computed with the survey estimation commands of Stata (Stata Version 11 Survey Data Reference Manual, Stata Press, 2009, ISBN 1-59718-062-9).

A power calculation was conducted assuming a 1-sided test to detect a 4% change in a proportion with 80% power at alpha = .05. The power calculations require an estimate of the “pre-intervention” proportion in the population (0-100%) for the outcome of interest. However, due to the lack of existing estimates on the outcomes, the most conservative pre-intervention estimate (50%) was chosen. For each cross-sectional survey, this yielded a sample size of 1,926. Assuming a response rate of 75%, an initial sample of 2,568 clinicians is required to achieve the desired final sample size.

Item non-response (when persons fail to answer individual questions) will be addressed using multiple imputations in Stata if the proportion of missing values is judged to be substantial. Unit non-response (when persons fail to respond at all) will be addressed by adjusting sample survey weights to be sure that each respondent represents the appropriate proportion of the population to which inferences are generalized.

Data Entry and Integrity. The procedures that will be used for entering data from hard copy instruments are as follows:

File layouts are submitted to a data preparation manager in written form with clear specifications as to columns, data types, missing values codes, and editing requests (range and logic checks, automatic filling of skip patterns).
All keying is 100% verified by a data entry operator other than the original keyer. Discrepancies between the two keying efforts are flagged and resolved by a supervisor. Keying error rates are tracked.
Any problems encountered during the keying process are referred to the data preparation manager. A decision log is maintained to keep a thorough and complete record of all decisions made during this process.
Decisions made by data keyers or difficulties encountered during keying are clearly documented by attaching suitable notations to the source documents and posting the change in the decision log.

At each survey administration an advance letter, invitation letters, surveys, informed consent statement, and reminder postcards will be mailed to 2,568 clinicians. With an expected response rate of 75 percent, this will yield a total of 1,926 completed surveys. A $50 incentive will be included with the mailing of the survey; the survey will be sent via Federal Express letter package addressed to the clinician directly.

Consumer/Patient Surveys

The consumer/patient surveys will be conducted by telephone. The surveys will be conducted using computer-assisted telephone interviewing (CATI) technology. This technology allows for consistent data collection, with all interviewers having access to identical scripts, survey instruments, answers to frequently asked questions, and program-related materials. The telephone survey system that will be used for this project handles both inbound and outbound calls through a call scheduler and an automatic call distribution system that ensures timely delivery of each call. Additionally, the CATI system prioritizes cases based on the outcome of the most recent call.

The statistical goal of the survey analysis will be to estimate proportions and changes in those proportions across the two cross-sectional surveys. There are multiple sources of CER information available to consumers and trends in CER awareness and use are probably on the rise. Estimating important proportions at two points in time will allow for the estimation not only of the proportions of consumers/patients aware and using CER, but also rate the rate of change of CER awareness and use. The analysis will also focus on trends and changes in the areas of awareness, knowledge, behavior change/use, and benefits of CER, specific CER topics, and the EHCP. This approach will enable AHRQ to determine whether the passage of time, and increased dissemination contractor activity, is associated with any change in CER awareness, knowledge, use, or benefit. The first cross-sectional survey analysis will concentrate more heavily on awareness and understanding, whereas the second survey analysis will concentrate more on behavior change and benefits.

The analysis will be conducted using statistical packages such as SAS, STATA, and SPSS to develop descriptive as well as inferential statistics (including regression analysis) as appropriate. After the data collection is complete, the survey data will be cleaned prior to beginning the analysis. Data cleaning procedures will include identifying outliers, imputing missing values, recoding selected variable values to standardize meaning, checking for duplicate records, cross-checking for internal consistency, and documenting computer programs for archiving.

To identify trends and themes in the areas of awareness, understanding, behavior change/use, and benefits of CER, the analysis will produce frequencies and cross tabulations of key variables to identify what percentage of respondents were aware of CER or using CER. Crosstabs or subgroup analyses also will be conducted, as appropriate, to identify awareness levels among priority target populations of interest to AHRQ. This kind of analysis might reveal that a particular subgroup has unusually low awareness of CER—a finding that would be useful in planning future dissemination efforts for that particular subgroup.

Item non-response (when persons fail to answer individual questions) will be addressed using multiple imputations if the proportion of missing values is judged to be substantial. Unit non-response (when persons fail to respond at all) will be addressed by adjusting sample survey weights to be sure that each respondent represents the appropriate proportion of the population to which inferences are generalized.

At each survey administration, an advance letter will be mailed to a sample of 3,200 consumers/patients. Sample members will be contacted by telephone and asked to answer some brief screening questions to determine eligibility for the survey. With an expected screener response rate of 40 percent (1,280 eligible participants) and a survey response rate of 80 percent, this will yield a total of 1,000 completed surveys each wave. If needed, reminder notifications will be mailed to individuals who cannot be reached by phone. Consumers/patients who complete the survey will be mailed a thank you letter.

3. Methods to Maximize Response Rates

Clinician Surveys

Clinicians who spend most of their time in direct patient care are a particularly difficult group to survey. For this reason, the study will use a version of Dillman’s Tailored Design Method (1999)^³ modified by Battelle^⁴^,^⁵ to maximize response rates. The following methods will be used to enhance the response rate: 1) sending an advanced letter along with a return postcard to confirm the clinician’s eligibility and mailing address; 2) mailing the survey via express mail to increase the likelihood that the clinician receives the survey; and 3) following up with four reminders as necessary (i.e., a postcard reminder, a telephone prompting call, a second survey mailing, and a second telephone prompting call). For the telephone prompting, messages will be left on an answering machine or with the physician’s assistant to encourage participation. The survey package will include a cover letter emphasizing the importance of the survey, a questionnaire, a postage‐paid return envelope, and a $50 cash incentive (1^st mailing only).

A tracking system will be developed to note survey milestones (e.g., date survey materials mailed, date surveys received, survey status) and to generate reports. Using similar survey methods, AHRQ’s contractor, as shown in Exhibit 3, has achieved response rates ranging from 70 to 91 percent with clinician populations.

Exhibit 3: Battelle Studies with Clinician Surveys

Study	Universe	Sample	Survey Mode	Incentive Payment	Response Rate
National Survey of Endoscopic Capacity	National list of facilities from manufacturers	National sample of 1800 facilities, stratified by region, urban/rural	Telephone screening, mail survey, telephone reminder	$40	74 percent
State Survey of Endoscopic Capacity	National list of facilities from manufacturers	Universe of facilities in 15 states	Telephone screening, mail survey, telephone reminder	$40	78 – 91 percent
Sexually Transmitted Disease (STD) Treatment, Reporting, Infection Control Practices	AMA Masterfile	National sample of 7,300 physicians from 5 specialties in urban, suburban and rural areas with high and low STD prevalence.	Mail survey & reminders and 2^nd survey mailing	$15	70 percent
Primary Care Providers’ Role in HIV/STD Prevention	Masterfiles: AMA AANP ACNM	Sample of 800 primary care providers from 5 specialty groups	Mail survey & reminders, 2^nd survey mailing	$50	80 percent
HPV Provider Survey: Knowledge, Attitudes, and Practices About Genital HPV Infection and Related Conditions	Masterfiles: AMA AANP AAPA ACNM	National sample of 7,000 clinicians from 9 specialties	Mail survey & reminders and 2^nd survey mailing	$50	81 percent
Assessment and Monitoring of Breastfeeding-related Maternity Care Practices in the U.S.	AHA Annual Survey of Hospitals AABC	Universe of birth centers and hospitals in the United States and its territories (approximately 3,300 facilities) that provide postpartum care	Telephone screening, mail survey, telephone reminder	None	82 percent

AMA = American Medical Association; AANP = American Association of Nurse Practitioners; ACNM = American College of Nurse Midwives; AAPA = American Academy of Physician Assistants; AHA = American Hospital Association; AABC = American Association of Birth Centers

Consumer/Patient Surveys

To maximize response rates for the consumer/patient survey, the following methods will be used: 1) sending an advance letter to sampled consumers that explains the purpose and importance of the study and informs them that they will be contacted by telephone to participate in a brief survey; 2) making up to six attempts to contact the sample members during the field period; 3) sending mail follow-up notifications to respondents who cannot be reached by phone; 4) providing comprehensive interviewer training and certification; 5) establishing a toll-free, 24-hour access phone number for survey completion; 6) executing carefully managed sample management; and 7) employing best survey practices for securing cooperation, averting refusals, and maximizing response rates.

Health Care System Decision Maker Surveys

The methodology for maximizing the health care system decision maker survey response rate is not included in this OMB submission. It will be submitted as part of the renewal application for Years 4 and 5 of the project.

4. Tests of Procedures

Each survey instrument will be pre-tested with up to nine members of each of the audiences with whom the survey will be administered (clinicians and consumers/patients). The purpose of the pre-test (often called “cognitive testing”) is to obtain comments and advice about the format, appropriateness, and relevance of the survey questions and response categories (e.g., to ensure that the wording of questions is understandable and the answer categories are clear and comprehensive); and to verify that any skip patterns are easy to follow.

Pretest respondents will be asked to complete the survey, record the time it takes to complete, write comments about the relevance of the survey questions, and note any difficulties they have understanding questions or directions. Following the pretest, a researcher will conduct a brief interview with the respondent to discuss any problems that were encountered in completing the survey. The researcher will review the comments and ask for recommendations on changing the questionnaire to address these comments. For issues with question wording, the participant will be asked for suggestions on how to revise the question to make it easier to understand. For critical outcome questions, questions will be asked to determine how the participant interpreted the question.

Following the pre-test, any resulting changes to the survey instrument will be submitted to OMB.

5. Statistical Consultants

Exhibit 4 lists the individuals who contributed to or reviewed the research design and sampling plan for the study:

Exhibit 4: Statistical Consultants

Name	Affiliation	Telephone Number

Carroll, William	AHRQ	301-427-1647
Berkowitz, Alicia	Evaluation Contractor – IMPAQ	443-539-9764
Siegel, Sari	Evaluation Contractor – IMPAQ	443-539-9764
Young, Julie	Evaluation Contractor – IMPAQ	443-539-9766
Berkowitz, Judy	Evaluation Contractor – Battelle	404-460-1449
Derzon, Jim	Evaluation Contractor – Battelle	703-248-1640
Rhoda, Dale	Evaluation Contractor – Battelle	614-424-4944
Rose, John	Evaluation Contractor – Battelle	919-544-3717

1 Lovejoy, Kristin, Handy, Susan . (2008). A case for measuring individuals’ access to private-vehicle travel as a matter of degree: lessons from focus groups with Mexican immigrants in California. Transportation, 34, 601-612.

2 Lovejoy, Kristin, Handy, Susan . (2008). A case for measuring individuals’ access to private-vehicle travel as a matter of degree: lessons from focus groups with Mexican immigrants in California. Transportation, 34, 601-612.

3 D.A. Dillman. Mail and Internet Surveys: The Tailored Design Method. Canada: John Wiley & Sons, 1999.

4 D. Kasprzyk, D.E. Montano, J.S. Lawrence, W.R. Phillips. The Effects of Variations in Mode of Delivery and Monetary Incentive on Physicians’ Responses to a Mailed Survey Assessing STD Practice Patterns. Evaluation and Health Professionals, 2001; 24(1): 3 -17.

5 Montaño DE, Kasprzyk D, Hall IJ, Richardson LC, Greek A, and Ross L. Effect of incentive amount and telephone follow-up on response to a physician survey: findings from a prostate cancer screening survey of primary care physicians. Evaluation and the Health Professions. (under review)

Page 2

File Type	application/msword
File Title	OMB Clearance Application
Author	hamlin-ben
Last Modified By	CTAC
File Modified	2012-03-19
File Created	2012-03-19