Supporting Statement Part B -- Pilot Test of the High Value and Efficiency Culture Surveys 1-10-2013

Table of Contents

B. STATISTICAL METHODS 2

1. Potential Respondent Universe and Sample Selection Method 2

2. Information Collection Procedures 6

3. Methods to Maximize Response Rate 6

4. Tests of Procedures 7

5. Statistical Consultation and Independent Review 7

B. STATISTICAL METHODS

1. Potential Respondent Universe and Sample Selection Method

Value and Efficiency (VE) Surveys.

Cognitive Interviews

Cognitive interviews will be done on the draft Hospital and Medical Office Value and Efficiency Surveys (VE Surveys) before larger-scale pretesting is done. Nine cognitive interview participants will be selected from both hospitals and medical offices that will vary by type, size and geographic location. We will use participant recruitment methods such as sending research participation flyers to hospitals and medical offices. Sites will be identified through Health Research & Educational Trust (HRET) and through recommendations by technical expert panel members or their affiliated organizations. For hospitals, we aim to conduct cognitive testing with 3 managers, 3 nurses, and 3 technicians. Similarly, we will aim to conduct cognitive testing with 4 physicians and 5 medical assistants in the medical offices.

Pretest Study Sample The purposes of the larger scale pretest sample design are two-fold: 1) to obtain enough pilot data at the hospital and medical office site level and individual respondent level to allow for sufficient N to examine the multilevel psychometric properties of the surveys (item nonresponse, item response variability, factor structure, reliability and construct validity), and 2) to include a variety of hospitals and medical offices that vary by type and size, as well as geographic region.

Since the goals for the data are to examine the psychometric properties of the survey, not to produce national estimates, purposive sampling will be used. Purposive sampling will ensure adequate variability on important hospital and medical office characteristics given the small number of hospitals and medical offices included in the pilot study.

Anticipating that the surveys will have about 50 items, for psychometric and factor analysis purposes we will need at least 10 respondents for every item included in the survey, or at least 500 total respondents for each survey. In addition, to conduct analyses at the site level, we need at least 40 sites per survey. The number of sites and number of individuals sampled were determined by these criteria.

Hospital VE Survey. HRET will work with State Hospital Associations (SHAs) to identify and recruit 42 hospital sites which is sufficient for multilevel analysis purposes. Hospitals will be selected to vary by bed size and teaching status, and will be selected from various geographic areas of the U.S. Hospitals will also be selected based on information about whether they have implemented efficiency initiatives such as Lean or Six Sigma, with some hospitals having experience with these initiatives and others not. This will provide variability on the value and efficiency measures. As shown in Table 1, we will sample 160 staff per hospital to ensure there are enough respondent-level data from each hospital. A total of approximately 6,720 hospital staff will be surveyed for the pilot test (160 staff x 42 hospitals) (see Table 2). With a response rate of 60%, the number of completed surveys will be 4,032, or about 96 responses per hospital. Participating hospitals will be asked to select the 160 staff through systematic sampling; stratifying hospital staff by work area and staff position within work area, hospitals will select 160 individuals from a random start on the list.

Table 1. Hospitals by bed size and teaching status: 42 pilot hospitals

Hospital Bed Size (# of beds)	Type of Hospital		Total Pilot Hospitals
	Teaching	Non-teaching
Small (49)	7	7	14
Medium (50 - 100)	9	9	18
Large (200 +)	5	5	10
Total	21	21	42

Table 2. Individuals surveyed within 42 hospitals: 6,720 individuals*

Hospital Bed Size (# of beds)	Type of Hospital		Total Individuals Surveyed
	Teaching	Non-teaching
Small (49) assumes 160 clinical staff sampled	7 x 160=1,120	7 x 160=1,120	2,240
Medium (50 - 100) assumes 160 clinical staff sampled	9 x 160=1,440	9 x 160=1,440	2,880
Large (200 +) assumes 160 clinical staff sampled	5 x 160=800	5 x 160=800	1,600
Total	3,360	3,360	6,720

* A 60% response rate = 4,032 completed hospital surveys for analysis purposes

Medical Office VE Survey. We will need to recruit 96 medical offices to obtain enough data at the individual respondent level for analysis purposes (approximately 504 responses). Given the small number of staff within each medical office, a census of all clinical and non-clinical staff will be conducted within each medical office. In addition, a Medical Office Information Form on background characteristics will be completed by a medical office manager at each site.

The Medical Group Management Association (MGMA) and American Medical Group Association (AMGA) will help recruit a range of medical office sizes and specialties (see Table 3). It is estimated that a total of approximately 840 individuals will be surveyed for the pilot test (see Table 4). With a response rate of 60%, the number of completed surveys will be 504.

Table 3. Medical offices by size and type: 96 pilot medical offices

Medical Office Size (# of clinical staff)	Type of Medical Office			Total Pilot Medical Offices
	Primary Care Specialty	Other Single Specialty	Multi-specialty
Small (3-5)	8	8	8	24
Medium (6-10)	16	16	16	48
Large (11+)	8	8	8	24
Total	32	32	32	96

Table 4. Individuals surveyed within 96 medical offices: 840 individuals*

Medical Office Size (# of clinical staff)	Type of Medical Office			Total Individuals Surveyed
	Primary Care Specialty	Other Single Specialty	Multi-specialty
Small (3-5) assumes 4 clinical staff	4 x 8 = 32	4 x 8 = 32	4 x 8 = 32	96
Medium (6-10) assumes 8 clinical staff	8 x 16 = 128	8 x 16 = 128	8 x 16 = 128	384
Large (11+) assumes 15 clinical staff	15 x 8 = 120	15 x 8 = 120	15 x 8 = 120	360
Total	280	280	280	840

* A 60% response rate = 504 completed medical office surveys for analysis purposes

Communicating About Value Checklist (CV Checklist)

Three data collection activities will be undertaken for this task: survey data collection, cognitive testing of a draft checklist, and a pretest of the checklist. Data will be collected from both hospitals and medical offices, but these sites will be different than the sites participating in activities associated with the VE Surveys. The sites will be different to avoid confusion and overburdening participating sites.

The purposes of the data collection are two-fold: 1) to obtain data to determine the best content and focus for the checklist, and 2) to obtain data and feedback about the checklist from providers using it. Therefore, since the goals of the data collection are not to produce national estimates on checklist use, but obtain information to make the checklist a better tool, purposive sampling will be used. Purposive sampling will ensure adequate variability of providers from both hospitals and medical offices to enable the checklist to be used by providers in both settings.

Communicating About Value Survey to Identify Checklist Items

The purpose of the Communicating About Value (CV) Survey is to glean information that shall be useful in the creation of the Communicating About Value Checklist to assist clinicians in addressing value issues when talking with their patients. The survey will be sent to 280 providers of various types (40 Physicians, 40 Registered Nurses, 20 Social Workers, 20 Health Educators) and with 40 patients from both hospitals and medical offices. A 50 percent response rate is expected (N = 160). The survey will ask providers and patients about what the best content and focus for the checklist should be.

To recruit respondents for the survey, HRET will contact several State Hospital Associations (SHA) to determine their interest in working with HRET to identify hospitals interested in participating in this study. Hospitals willing to participate will further identify providers willing to complete the survey.

Medical offices will be identified with assistance from the Medical Group Management Association (MGMA) and the American Medical Group Association (AMGA). A point of contact at each medical office will be identified and responsible for further identifying providers willing to complete the survey.

Craigslist will serve as the recruitment medium for the 40 patients needed. Ads will target two Craigslist populations in various states across the country: patients seen in hospitals and patients seen in medical offices. The advertisements will be checked and reposted every eight days. Respondents will be screened to determine eligibility. Eligible participants include:

• Any patient who has been ill or injured and in need of treatment by a physician, physician assistant, advanced practice registered nurse, or other health care provider in the last four months.

• Any patient 18 years of age or older.

• Any patient without severe cognitive impairments (e.g., Alzheimer’s patients).

• Any patient without severe sensory-motor impairments.

• Only patients able to read the survey.

Table 5. Sample for the CV Survey

	Total Number of Individuals to be Surveyed
	Hospitals	Medical Offices
Physicians	40	40
Nurses	40	40
Social Workers	20	20
Health Educators	20	20
Patients	40	40
Total	160	160

Cognitive Interviews of the Draft CV Checklist

Once CV checklist content and items have been identified from the survey, a draft checklist will be produced and 18 cognitive interviews will be conducted with 4 physicians, 4 nurses, 2 social workers, 2 health educators, and 6 patients across both hospital and medical office settings to understand the cognitive processes the respondent engages in when using the CV checklist. To recruit cognitive interview participants, we will use the same process that was used to select participants for the CV survey.

Pretest for the CV Checklist

The CV checklist along with a checklist evaluation will be administered to a purposive sample of 250 physicians in hospitals and 250 physicians in medical offices. A 20 percent response rate is expected to yield 100 completed checklist evaluations. HRET will again use the same purposive recruitment and selection process to identify potential respondents.

The review and field-test findings will be systematically considered and addressed and a final checklist will be prepared. Responses will be compiled in a database, a summary of our analyses prepared, including response rates, and information submitted along with the database, which will include a final version of the CV checklist to AHRQ and the Technical Expert Panel (TEP).

Table 6. Sample for the CV Checklist

	Total Number to be Surveyed
	Hospitals	Medical Offices
Physician	250	250
Total	250	250

2. Information Collection Procedures

VE surveys. The pilot survey data collection for both the Hospital and Medical Office VE surveys will include these steps:

• Mailing the VE surveys (which include a cover letter) to the hospital/medical office point of contact

• Hospital/medical office point of contact distributes survey packets to staff

• A reminder notice distributed to nonrespondents

• Distribution of a second questionnaire to nonrespondents

CV checklist. The CV survey data collection will include these steps:

• Mailing the Communicating about Value surveys (which include a cover letter) to the purposive sample

• A reminder notice distributed to nonrespondents

• Distribution of a second questionnaire to nonrespondents

The CV Checklist data collection will include these steps:

• Mailing the CV checklist and evaluation (which include a cover letter and instructions) to the purposive sample

• A reminder notice distributed to nonrespondents

• Distribution of a second questionnaire to nonrespondents

3. Methods to Maximize Response Rate

VE surveys. Cognitive interview participants will receive cash remuneration ($175 for hospital administrators/executives and medical office physicians and $100 cash for other clinical staff). In addition, we are using site-level remuneration of $100 for medical offices, and we are following up with a reminder notice and second questionnaire to nonrespondents in both hospitals and medical offices.

CV survey and checklist. Patients will be offered a $5 incentive for completing the CV survey. Cognitive interview participants will receive cash remuneration for interviews about the CV checklist ($175 for physicians and $100 for other participants). Every effort will be taken to minimize the burden on survey respondents. In addition, we are following up with reminder notices and second questionnaires to nonrespondents.

4. Tests of Procedures

The procedures for this specific project have not been subjected to testing. However, both HRET and Westat have conducted many similar projects and are using well-established research methods with this project.

5. Statistical Consultation and Independent Review

Input from statistical analysts was used to develop the study design and plans for data analysis, including:

Joann Sorra, PhD, Senior Study Director, Westat, 301-294-3933

Naomi Dyer, PhD, Senior Study Director, Westat, 301-610-8842

10