Supporting Statement Part B -- Generic Pretesting Clearance 2014

SUPPORTING STATEMENT

Part B

Questionnaire and Data Collection Testing, Evaluation, and Research for the Agency for Healthcare Research and Quality

OMB Number 0935-0124

Version: September 5^th, 2014

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 3

4. Tests of Procedures 3

5. Statistical Consultants 5

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The purpose of collections under this generic clearance is not to obtain data per se, but rather to obtain information to develop new questions, questionnaires and tools and to identify problems in instruments currently in use.

In general, these activities are not intended to yield results that are statistically sound. It is expected that these studies will rely heavily on qualitative techniques to meet their objective. This clearance request is limited to research on data collection, toolkit development, and estimation procedures and reports and does not extend to the collection of data for public release or policy formation. See Attachment B for examples of surveys conducted under this clearance.

These research activities will generally employ non probability samples. While on occasion, samples from the Medical Expenditure Panel Survey may be used, these would be the exception. Most studies would be conducted with convenience sampling. “Quota” sampling may be used to insure that the convenience sample includes enough persons with particular characteristics to insure representation of the target audience and/or a reasonable degree of diversity in key demographic characteristics.

Depending on the particular project, samples may be recruited through advertising (for instance when a project requires respondents with particular rare characteristics) or through places of employment or medical care when those places serve the target population. Telephone samples may be selected with random digit-dialing (RDD) techniques, or with stratified sampling of telephone exchanges.

Response rates are generally not applicable to convenience sampling because this type of sampling results in a non-probability sample which is non-representative of the population. Over-recruiting will be used to compensate for non-response.

2. Information Collection Procedures

The following types of research activities may be employed for these evaluation-type methods and techniques under this general clearance: Mail surveys conducted by mail or via email which may include telephone non-response follow-up; telephone surveys; web-based surveys; focus group; automated data collection (e.g. testing of database software, CAPI software or other automated technologies); and cognitive testing.

Professionally recognized procedures will be followed in each information collection activity to ensure high quality data. Examples of these procedures may include

• Monitoring by supervisory staff a certain percent of telephone interviews;

• Data-entry from mail or paper-and-pencil surveys will be computerized through scannable forms or checked through double-key entry;

• Observers will monitor focus groups, and focus group proceedings will be recorded; and

• Data submitted through on-line surveys will be subjected to statistical validation techniques (such as disallowing out-of-range values).

• Cognitive interviewing techniques including think-aloud techniques and debriefings.

Each request under this umbrella clearance will specify the specific procedures to be used.

Participation will be fully voluntary and non-participation will have no affect on eligibility for, or receipt of, future AHRQ health services research. Specific testing and evaluation procedures, when used, will be described when we notify OMB as to actual evaluation methods and techniques. The consent procedures will be customized for each collection but will include assurances of confidentiality and the legislative authority for the activity. If the encounter will be recorded, the respondent’s permission will be obtained before beginning the interview.

Recruitment – Respondents will be recruited by means of advertisements in public venues or through techniques that will be replicated in the main survey – for instances a survey on physician communication designed to be administered following an office visit might be pretested using the same procedure. Each submission will specify the specific recruitment procedure to be used

Screening - When screening is required (as for quota sampling), the screening will be as brief as possible and the screening questionnaire will be provided as part of the submission.

Collection methods - The particular collection methods used will vary but may include the following

• Individual in-depth interviews – In-depth interviews will commonly be used to insure that the meaning of a questionnaire or strategy is understood by the respondent to the questionnaire or user of the tool. When in-depth interviewing is used, the interview guide will be provided.

• Focus groups – Focus groups will be used to obtain insights into beliefs and understandings of the target audience early in the development of a questionnaire or tool. When focus groups are used, the discussion guide will be provided.

• Gatekeeper review of tools – in some instances tools designed for patients may be reviewed in depth by medical providers or other gatekeepers to provide feedback on the acceptability and usability of a particular tool. This would usually be in addition to pretesting of the tool by the actual patient or other user.

• Record abstractions - on occasion the development of a tool or other information collection requires interaction with records rather than individuals.

• “Dress rehersal” of a specific protocol. In some instances the proposed pretesting will actually be a walkthrough of the intended data collection procedure. In these instances, the request will mirror what is expected to be the larger scale data collection.

3. Methods to Maximize Response Rates

The design of testing and evaluation procedures will include approaches to maximize response rates and the quality of the collected information, while retaining the voluntary nature of the effort.

AHRQ has a demonstrated track record of focusing very hard on shortening its data collection and estimation procedures to keep respondent burden down and to reduce costs because private sector sponsors administer and pay for survey fielding for a number of AHRQ tools such as the CAHPS®.

AHRQ will, on a case-by-case basis, consider modest remuneration for survey respondents and focus group participant’s time and travel. In such cases, the remuneration will typically not exceed $50 per individual. AHRQ may ask for a higher amount for hard to survey population, including physicians. Remuneration for focus group participation is a recognized standard industry practice, without which, it would be difficult to achieve appropriate and adequate participation.

4. Tests of Procedures

Before each information collection is implemented, a contractor will pilot test the instrument(s) and method of data collection. Lessons from the pilot test will be identified and changes as necessary will be incorporated into the method and instrument. All pilot tests will involve no more than nine individuals unless OMB clearance is sought for more than nine.

5. Statistical Consultants

Each program will obtain input from statisticians as to the development, design, conduct, and analysis of the testing and evaluation methods planned for the data collections and estimation procedures. This statistical expertise will be available from AHRQ statisticians/contractors. Technical assistance in data collection and estimation procedures may be sought, in some cases, outside of AHRQ from experts in other Federal Agencies or outside the government.

The following individuals at AHRQ will review submissions under this generic submission:

Doris Lefkowitz

Erwin Brown

5