EC OMB Clearance - Supporting Statement Part B EB_REV 08122016

SUPPORTING STATEMENT

Part B

Eisenberg Center Voluntary Customer Survey Generic Clearance

for the Agency for Healthcare Research and Quality

February 24, 2016

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 5

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 7

5. Statistical Consultants 7

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

Formative research activities will consist of gathering data to guide the development and implementation of products and activities. Semi-structured interviews, and in some cases focus groups, will be used for formative assessments with clinicians, consumers, and policymakers to inform the development of the summary products. Formative interviews will also be conducted for determining optimal dissemination channels for the products, and in relation to a unique dissemination research project to be conducted annually. These interviews are not intended to yield results generalizable to the population at large, but are targeted assessments with selected participants meant to ensure that products and/or activities are targeted, tailored, feasible, and accepted by potential users. Formative assessments will employ purposive sampling to deliberately choose participants based on their characteristics (e.g., professional role, knowledge, experience, medical condition). Different types of purposive sampling methods will be employed based on the purpose of the assessment; for example, maximum variation sampling may be used in formative interviews with consumers to capture a wide range of perspectives, while expert sampling may be used to select participants for formative interviews with clinicians and policymakers.

Once draft summary products have been developed, they will undergo pretesting using cognitive interviews with 10-15 clinicians, policymakers, and/or consumers to assess clarity of presentation and content, understandability, and usability. Also referred to as “think-aloud” interviews, these assessments will be used to test how target audiences perceive and interpret the information presented in the summaries, to identify any potential problems in the materials, and in some cases to estimate learning that may occur with the product. Occasionally two rounds of interviews may be needed during testing. Potential interviewees will be selected through typical case purposive sampling approaches with the goal of identifying a sample that is reflective of the target audience (i.e., consumer, clinicians, or policymakers). Participants will be recruited using the procedures described above.

Interview participants will be recruited by either the project PI or clinics and health systems affiliated with Baylor College of Medicine, the American Institutes of Research, or other contract partners. In some cases interview candidates will be invited through the respective clinic, and only their first names and available time slots forwarded to the Eisenberg Center or a qualified contractor for conducting the assessments anonymously.

Evaluation surveys will be administered to representatives of the targeted audiences (i.e., consumer, clinician, policymaker) who have accessed the health information products or the decision aids. Audience members will self-select through invitations offered on the AHRQ web site for participation in the initial survey, or through an e-mail notification system for participation in the follow-up survey. To increase the likelihood of gathering valid data for evaluating the products, a brief latency period (e.g., ~2 weeks) will be selected which will allow time for the respondent to have completed the process of gathering health information and possibly begun using the information, while still being able to recall the source from which the information was drawn. The initial survey will request an email address, which will allow for matching the initial and follow-up data by respondent to compare intended versus realized uses and perceived value of the information. The surveys are not expected to yield representative responses from the target populations because participation is optional with no incentive available, and thus nonresponse bias is a potential threat to validity. However, the data are expected to provide useful information given the assessment constraints imposed, such as the potential for selection bias to be operative, anticipated high levels of motivation associated with self-directed searching for health information, audience characteristics associated with availability of Internet/computer access, potential confirmation bias, and other limitations. Evaluation surveys will be used in the annual dissemination research project to assess success or failures in effective dissemination of the products through the selected channels. Follow-up interviews will be conducted for the summary products, decision aids, and the annual dissemination research projects. These interviews will generally be used to complement the evaluation surveys by allowing for in-depth exploration of findings suggested from the survey data.

Clinicians who have completed CME accrediting requirements and wish to obtain CME credit will be asked to complete a follow-up outcomes assessment survey 2-3 months following participation in the online activity. Because the requirements associated with seeking and obtaining CME credit are more likely to impress upon the learner a more defined memory of the source of the information and learning gained, a 2-3 month period was selected to allow sufficient time for implementing and self-evaluating resultant changes in clinical behaviors and to identify any barriers to implementation. The survey will be conducted anonymously to reduce the likelihood of response biases such as the tendency to provide socially desirable responses.

Advocacy organizations and AHRQ partners will be asked to provide feedback in the form of surveys and interviews. These data will not be collected anonymously and thus may suffer from social desirability biases. Biases will be minimized by explaining to the participant the purposed uses of the data and the general approach of presenting the data anonymously and in aggregate whenever possible.

Given the purposeful nature of the activities, it is unlikely that statistical measures will generally be employed, with the possible exception of one or more of the dissemination research projects to be designed and implemented on an annual basis. In instances when there will be an existing list of "customers" readily available for sampling (e.g., mailing lists for publications or recipients of particular materials or services within known customer groups), appropriate probability sampling techniques will be used to select samples. Separate clearance will be requested for any project that necessitates determining formal sampling design, power estimation, and statistical analysis.

2. Information Collection Procedures

All information collections will be conducted in a manner that is consistent with the following guidelines:

• Participation will be completely voluntary, and non-participation will have no effect on eligibility for or receipt of future AHRQ-sponsored health services research.

• Appropriate sample sizes will be determined for each activity to ensure that burden is minimized while reliable estimates are produced to the extent possible under purposive sampling designs.

• Information collections will be limited to those needed to assess customer reaction to planned or currently available products, services, and collaborations.

• Given the voluntary nature of the information collections, efforts will be made to obtain the highest possible response rates. Efforts will also be made to assess non-response bias as feasible.

3. Methods to Maximize Response Rates

The design of each information collection will include approaches to maximize response rates when feasible, while retaining the voluntary nature of the effort, consistent with appropriate survey methodology. Remuneration will be provided to consumers, clinicians, policymakers, and health system/hospital/clinic personnel for assessment activities in which significant time investments are required. Clinician non-respondents to outcomes assessment surveys will be sent a single follow-up reminder, which the Eisenberg Center has found in prior internal research (unpublished work) improved response rate for this audience by 10-11%, without engendering negative perceptions of the contacting organization.

4. Tests of Procedures

It is anticipated that most information collections will begin with efforts by Eisenberg Center staff or in some cases by interview or focus groups to identify the views/concerns of potential customers. Formal pretesting will be carried out using fewer than 10 participants and in a manner consistent with the specific assessment mechanism (e.g., survey design, interview protocol).

5. Statistical Consultants

Input from statisticians regarding the development, design, conduct, and analysis of information collections will be sought if needed. This statistical expertise will be available from AHRQ statisticians/contractors. Technical assistance in survey design and statistics may, in some cases, be sought through The National Center for Health Statistics.