Supporting Justification for OMB Clearance Package
Evaluation of an Intervention to Increase Colorectal Cancer Screening in Primary Care Clinics
Submitted by:
Judith W. Lee, PhD
Centers for Disease Control and Prevention
National Center for Chronic Disease Prevention and Health Promotion
Division of Cancer Prevention and Control
Epidemiology and Applied Research Branch
4770 Buford Highway, NE, Mailstop K-55
Atlanta, Georgia 30341
Voice: 770-488-4864
Fax: 770-488-4639
Email: [email protected]
May 2007
Table of Contents
Page
1. Circumstances Making the Collection of Information Necessary 1
2. Purpose and Use of the Information Collection 3
3. Use of Improved Information Technology and Burden Reduction 3
4. Efforts to Identify Duplication and Use of Similar Information 5
5. Impact on Small Businesses or Other Small Entities 5
6. Consequences of Collecting the Information Less Frequently 5
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 6
8. Comments in Response to the Federal Register Notice and Efforts
to Consult Outside the Agency 6
9. Explanation of Any Payments or Gifts to Respondents 8
10. Assurance of Confidentiality Provided to Respondents 9
11. Justification for Sensitive Questions 10
12. Estimates of Annualized Burden Hours and Costs 11
13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers 13
14. Annualized Cost to the Federal Government 13
15. Explanation for Program Changes or Adjustments 14
16. Plans for Tabulation and Publication and Project Time Schedule 14
17. Reason(s) Display of OMB Expiration Date is Inappropriate 18
18. Exceptions to Certification for Paperwork Reduction Act Submissions 18
1. Respondent Universe and Sampling Methods 19
2. Procedures for the Collection of Information 23
3. Methods to Maximize Response Rates and Deal with Nonresponse 30
4. Tests of Procedures or Methods to be Undertaken 34
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 36
Bibliography 37
List of Tables
Page
Table A.12B Estimated Cost to Respondents 13
Table A.14A Cost to the Government 13
Table A.16A Time Schedule 14
Table A.16B1 Example Shell Table of Study Results 16
Table A.16B2 Expected Mean Squares 17
Table B.1A Estimated Size of Respondent Universe and Proposed Study Sample 20
Table B.1C Estimated Number of Respondents 23
Table B.2B-3 Respondent Sample Size Required Per Clinic at Various
Levels of Statistical Power for the Three Main Outcome Analyses 28
List of Attachments
Attachment 1: Authorizing Legislation
Attachment 2: Copy of Federal Register Notice
Attachment 3: CDC IRB Approval
Attachment 4: Survey Instruments
Attachment 5: Survey Cover Letters
Attachment 6: Study Reminder Postcards
Attachment 7: Clinic-Focused Intervention Training Curriculum Summary and Objectives and Presentations
Attachment 8: Patient-Focused Intervention Packet
Evaluation of an Intervention to Increase Colorectal Cancer Screening in Primary Care Clinics
1. Circumstances Making the Collection of Information Necessary
Centers for Disease Control and Prevention (CDC) requests approval by the Office of Management and Budget (OMB) for collection of information to evaluate the effect of an intervention to increase colorectal cancer screening in primary health care settings. This project is one in a series of research projects undertaken by Division of Cancer Prevention and Control (DCPC) at CDC in the area of colorectal cancer control that includes epidemiologic, surveillance, and behavioral research. The study is authorized by Section 301 of the PHS Act (42 U.S.C. 241). (see Attachment 1). Approval is requested for pre- and post-intervention surveys of patients and post-intervention surveys of clinicians and clinic support staff in two Managed Care Organizations (MCO) that have been selected to participate in the intervention.
Colorectal cancer (CRC) is the third most frequent form of cancer and the third leading cause of cancer-related deaths among both men and women in the United States.1 The natural history of CRC, with progression from adenomatous polyps to cancer over a period of approximately 10-15 years, provides an excellent window of opportunity for screening and intervention. When CRC is detected at a localized stage, five-year survival is 90 percent, but survival drops to 68 percent if CRC is detected at a regional stage and to ten percent if detected at a distant stage.1 Screening is beneficial for: 1) detection and removal of precancerous polyps, resulting in patients recovering without progression to a diagnosis of cancer, and 2) early detection of CRC for more effective treatment and improved survival.2 Survival of patients with localized disease is extremely good. Most patients (63%), however, have regional or distant metastases at the time of CRC diagnosis.3 Currently, then, only 39 percent of CRC is discovered at the localized stage. Regular CRC screening among people aged 50 years and older has been recommended by the U.S. Preventive Services Task Force, the National Cancer Institute, and the American Cancer Society.4-6 The National Committee for Quality Assurance has added a CRC screening measure to the 2004 Health Plan Employer Data and Information Set (HEDIS®).
Regular CRC screening among individuals age 50 years and older, fecal occult blood test (FOBT) in the past 12 months, lower endoscopy in the past 10 years, or a combination of both tests, has great potential for both primary and secondary cancer prevention. For instance, case-control studies indicate that flexible sigmoidoscopy is effective and cost-effective in the early detection and prevention of CRC.5,7 However, national surveys indicate that colorectal cancer screening rates in the U.S. are disappointingly low, with between 24 - 53 percent of age-appropriate respondents reporting having had a screening test within the appropriate interval 8.
Research shows many behavioral and structural factors impact cancer screening rates among physicians and patients. Studies conducted among clinicians have shown that system barriers, practice and demographic characteristics, and various behavioral factors, including attitudes and perceptions of colleagues’ support, are the most important factors in determining whether clinicians recommend CRC screening.9-18 Researchers have also shown that factors such as continuity of care, patient attitudes and other psychosocial factors, awareness and perceived risk for cancer, patient discomfort, patient demographic variables, and family history are associated with cancer screening rates.19-24 Low CRC screening rates may be in part due to: 1) low public awareness of the benefits of screening, 2) failure by many clinicians to recommend screening to their patients, and 3) the absence of efficient surveillance and reminder systems in many primary care settings.
2. Purpose and Use of Information Collection
There is clearly a need for new interventions to increase the rates of screening for CRC that will ultimately increase the number of such cancers that are prevented, or detected early and successfully treated. CDC and its partners the Center for Medicare and Medicaid Services and the National Cancer Institute, designed and implemented “Screen for Life,” a national multimedia CRC screening promotion campaign designed to increase public awareness of the seriousness of CRC and the importance of routine screening. An important next step planned by DCPC is an intervention that builds upon public awareness campaigns such as “Screen for Life” by encouraging and supplying the necessary tools for delivery of CRC screening messages within the primary care setting. This intervention is designed with two complementary components to respectively impact the beliefs, attitudes and behavior of both patients and clinic staff (clinicians and clinic support staff). The clinic staff component also is designed to encourage office system changes to assist the clinician in identification of patients due for screening and to remind the clinician to offer CRC screening.
This study employs a two-component intervention targeted to 1) patients and 2) clinicians and clinic support staff. For the patient intervention, eligible patients (patients aged 50-80 years and due for regular CRC screening) in the intervention arms of the study will receive a letter describing the study from their primary care provider and materials based upon the “Screen for Life campaign” including CRC educational materials and revised, easier to read FOBT instructions. This packet of information will be sent to patients who are scheduled to see their primary care provider for a non-acute ambulatory care visit approximately two weeks before their scheduled medical appointment. Patients will complete a survey assessing their attitudes and behaviors regarding CRC screening pre-and post intervention. Patients not eligible for this study include those with a prior diagnosis of CRC, CRC polyps, ulcerative colitis, Crohn’s disease, a family history of CRC, or younger than 50 years of age or older than 80 years of age.
For the clinician and clinic support staff intervention, clinic personnel in the intervention arms of the study will receive three hours of training via two facilitated educational sessions. Across both sites, the clinic staff eligible to participate in the intervention include clinicians, nurses, medical assistants, and those persons responsible for scheduling medical appointments. In session 1, participants will review the latest statistics, evidence, and CRC screening guidelines, participate in exercises to enhance skills to begin conversations about CRC screening with their patients, and develop skills in motivational interviewing and reflective listening. In session 2, participants will identify and develop office surveillance and reminder systems that will increase or facilitate identification of patients eligible for CRC screening and patient tracking and follow-up after CRC screening. Clinicians and clinic support staff will complete a survey post-intervention to assess their attitudes, behaviors, and practices regarding CRC screening.
This intervention has been designed and prepared to be implemented and evaluated in two MCOs: Henry Ford Health System (HFHS) in Detroit, Michigan, and Lovelace Health Systems (LHS) in Albuquerque, New Mexico. The evaluation study has been designed to test the relative effects of a clinic‑focused intervention (the intervention arm involving only clinicians and clinic support staff), a combined clinic‑ and patient‑focused intervention (the intervention arm involving clinicians and clinic support staff and the patients), and a “usual care” control group on colorectal cancer screening rates in primary care clinics.
HFHS has 21 primary care clinics and LHS has 10 primary care clinics that are available to be selected and assigned to the study conditions. A conservative estimate (see section B.2 for a further discussion of this issue) indicated that a total of 19.3 clinics would be sufficient power. Thus, 20 will be sufficient to evaluate a single arm intervention effect against the control condition. We plan to include 21 clinics to assign 7 clinics to each study arm and to ensure sufficient power in the unlikely event of loss of a clinic to the study (though we conservatively have already discounted the numbers of patient surveys per clinic).
The effect of the intervention on the primary outcome of CRC screening can be assessed by using MCO electronic claims or administrative data alone. Each MCO will query their electronic record data associated with patient visits for non-acute ambulatory care visits. This data will be used to compute CRC screening rates to evaluate whether the intervention components result in increased screening rates compared to control clinics. In addition, a very important goal of the evaluation is to assess the impact of the intervention on secondary outcomes (appointment making) and on intermediate outcomes (attitudes, beliefs, opinions, perceived social support) measured in patient surveys. This data will also allow us to carry out structural analyses to determine whether and how changes in key intermediate outcomes among patients lead to increased CRC screening. Additionally, we will test the post-intervention impact of the intervention for clinicians and clinic support staff as measured by the surveys administered to the clinicians and clinic support staff. The proposed data collection activities that form the basis for the present request to OMB are essential to this evaluation. The proposed surveys will provide the baseline and follow-up data necessary to document colorectal cancer screening attitudes, beliefs, and opinions prior to the intervention, and to assess changes in these attitudes, beliefs, and opinions that can be attributed to the intervention. The surveys were designed to measure the intermediate outcomes identified in formative work as potentially very important in determining CRC screening behavior. Understanding prior attitudes, beliefs, opinions, and perceptions of social support, and measuring resultant change after the intervention, will assist in the development of enhanced interventions that will target the intermediate outcomes that are found in this study to have the greatest impact on increased CRC screening.
CDC, working with Battelle, HFHS, and LHS, designed a two-component intervention to increase CRC screening in primary care clinics. A three-arm randomized control trial has been designed to evaluate the effect of the intervention on both primary outcomes (receipt of CRC screening) and intermediate outcomes (attitudes, beliefs, opinions, perceived social support). These findings will help provide needed information about the effectiveness of intervention strategies to increase CRC screening. The proposed survey data collection from clinicians, patients, and clinic support staff at HFHS and LHS primary care clinics will be used to:
• Describe and assess CRC knowledge, attitudes, beliefs, opinions, and screening behavior among patients, at baseline (pre-intervention).
• Describe and assess CRC screening outcomes (CRC screening and intermediate outcomes) among clinicians, clinic support staff, and patients at follow-up (post-intervention).
• Compare post-intervention CRC screening outcomes among clinicians, patients and clinic support staff in the different intervention arms, including in control clinics, to determine the relative effects of the intervention components.
Without this research, CDC will have little information to understand the effectiveness of these interventions in changing attitudes, beliefs, opinions and rates of colorectal cancer screening. The evaluation design will allow CDC to assess the relative effectiveness of the patient- and clinic-focused intervention on intermediate outcomes and CRC screening when implemented within the primary care setting. Findings concerning change in intermediate outcomes and associated change in CRC screening will help focus efforts to enhance the intervention in the future. If efficacious, the intervention may be adapted for other primary health care settings and promotion of other cancer screening. Findings from the proposed data collection and analyses consequently are essential to guide future CDC DCPC activities in its mission to promote CRC screening, and to allocate program resources.
3. Use of Improved Information Technology and Burden Reduction
Use of improved technology will be used for mail surveys of clinicians, patients, and clinic support staff. Additionally, electronic record systems will be used to identify appropriate patients to receive surveys and intervention packets, and to capture data to compute CRC screening rates. These systems are described below.
Mail Survey Data Collection
Surveys of patients will be carried out prior to the intervention and after the intervention. For patients, the initial survey mailing will be followed by a reminder postcard after two weeks, a second survey mailing to non-respondents at four weeks after the initial survey mailing, and a final mailing to non-respondents one month after the second mailing. A pre- and post-intervention cohort design will be used to compare study arm change in the clinician primary behavioral outcome of CRC screening rate, which will be obtained pre- and post-intervention from electronic encounter and claims data. Additionally, clinicians and clinic support staff will complete post-intervention surveys. For clinicians and clinic support staff, post-intervention survey mailings will be followed by a reminder postcard after two weeks, a second survey mailing to non-respondents at four weeks after the initial survey mailing, and a second reminder postcard to non-respondents at six weeks after the initial survey mailing. A study-specific computerized tracking and reporting system has been designed to monitor all phases of the study. A database will hold all respondent information and track the study’s progress through all phases. The data management system will track the mailing dates for the surveys and postcards and flags will be set to initiate follow-up mailings and reminder postcards. The receipt of a completed survey, or a refusal, will be logged into this computerized control system. From this system, electronic progress reports will be generated on a weekly basis. This system will reduce respondent burden by ensuring that clinicians, patients and clinic support staff are contacted at appropriate time points and that those who have responded are not contacted with reminders. Access to all study databases will be limited to authorized study personnel.
Electronic or web-based completion of the surveys was considered but was dismissed for the following reasons. Many patients at LHS and HFHS are lower income and are unlikely to have internet access at home. Thus it is much easier for patients to complete a mailed survey than to have to find access to the internet. Clinicians may have greater access than clinic support staff to computers. However, such access is often on a shared computer, which would not be conducive to privacy. Most clinicians prefer a mailed survey, which they can put down when they are busy and pick up again to resume at convenient times (e.g., between patients). 10
Clinicians in a busy practice cannot easily complete the survey in this manner with a web-based survey, particularly if they need to access a shared computer. Thus, a mailed survey was selected as the best data collection modality for patients, clinicians and clinic support staff in the managed care organizations selected for this study.
We have also taken particular care to design the survey instruments to collect only the minimum information necessary to achieve the goals of the project and to adhere to appropriate grade level specifications. This was accomplished by completing an extensive literature review and by conducting qualitative interviews with clinicians, clinic staff and patients to identify the most important factors to measure with respect to the project goals and to design questions to measure those factors. Practicing clinicians, clinic support staff and patients also provided advice about the questions that are most relevant to them and questions that could be deleted from earlier drafts of the survey. (See section B.4)
Use of Electronic Records
Electronic records from the MCOs will be used to: 1) identify appropriate patients to receive the patient surveys and appropriate patients to receive the patient-focused intervention, and 2) obtain data to compute CRC screening outcome measures. The data capture capacity of each MCO is listed below.
Lovelace Health Systems: LHS is able to electronically capture all FOBT tests carried out at the Lovelace Lab. In addition, all colonoscopy, flexible sigmoidoscopy, and barium enema data can be captured system-wide through the encounter and claims data. This information can be linked to membership data to obtain contact information for sending patient surveys and CRC intervention materials.
Henry Ford Health System: HFHS maintains extensive centralized repositories of computerized historical databases that include: 1) encounter and claims records; 2) the Clinical Management Information system (CMIS); 3) a Master Patient Index (MPI), and 4) an HMO membership file (HMOM). Administrative patient encounter records are maintained in the Corporate Data Store (CDS). The CDS includes comprehensive patient encounter data for outpatient, emergency department, and inpatient care delivered by Henry Ford Medical Group physicians practicing at Henry Ford Hospital and any of its affiliated 30 urban/suburban ambulatory care clinics. For each outpatient encounter, information on date of visit, diagnoses (ICD-9-CM diagnostic codes), physician delivering care, and procedures delivered (CPT-4 and ICD-9-CM codes) are compiled.
For the pre-intervention survey, the MCOs will use electronic records to identify the sampling frame of all patients aged 50 to 80 who visited each study clinic within the past year, after also excluding patients with diagnosis codes for CRC, ulcerative colitis, Crohn’s disease or a family history of CRC. At HFHS the research office is part of the health system and thus has Health Insurance Portability and Accountability Act (HIPAA) authorization to use the electronic records for patient selection. At LHS, the research organization (Lovelace Clinic Foundation) is a separate entity and therefore a HIPAA authorization form will be sent to patients with the patient survey mailing.
During the 12-month intervention period, the MCOs will use electronic data to identify all patients aged 50 to 80 who make an appointment for a non-acute ambulatory care visit, and they will use lab test and procedure code data to identify the patients who are due for CRC screening based on National Committee for Quality Assurance (NCQA) guidelines. In the clinics assigned to the patient-focused intervention, these patients will be sent the patient intervention packet prior to the scheduled medical exam. In all clinics these patients will form the sampling frame for the post-intervention patient survey.
Finally, the MCO electronic data will be used to compute the primary outcome of CRC screening rate for each clinic and each study arm. All patients seen for a non-acute ambulatory care visit during the one-year intervention period, and who are due for CRC screening based on NCQA guidelines, will be identified. The electronic lab test and procedure code data for these patients will be used to compute screening rates. These data queries and computations will not include patient identification and therefore will be anonymous. In addition, the CRC screening lab test and procedure code data for patients who participate in the post-intervention patient survey will be obtained and will be linked to survey data to validate patient self-report of CRC screening.
4. Efforts to Identify Duplication and Use of Similar Information
CDC has undertaken a series of research projects in the area of colorectal cancer control, including epidemiologic, surveillance, and behavioral research. DCPC researchers have identified key gaps in clinician practice with respect to CRC screening and follow-up to positive CRC screening results25 and DCPC has led efforts in CRC education and screening initiatives through its work on “Screen for Life”. However, this work has led DCPC to conclude that public education messages alone are insufficient to increase colorectal cancer screening to acceptable levels. There is an urgent public health need to conduct an intervention that tests patient education and provider training and this has become a priority for DCPC. To evaluate the effects of this intervention on screening outcomes, it is important to implement the data collection activities proposed here. The intervention evaluation planned has not previously been done and is important to carry out in order to determine whether theory-driven provider training alone is effective in increasing physician motivation and behavior with respect to providing CRC screening, as well as to determine whether patient education further increases provider motivation and actual provision of CRC screening. Thus, the data collection to evaluate this intervention has not been previously carried out and is essential to determine whether the intervention is effective.
5. Impact on Small Businesses or Other Small Entities
The information requested is the minimum required to meet the study objectives. It is restricted to clinicians, patients, and clinic support staff of two large established health maintenance organizations.
No small businesses will be involved in this study.
6. Consequences of Collecting the Information Less Frequently
This request is for a one time study. If the data collection effort is not conducted, CDC will not have information to understand the effectiveness of the proposed intervention to increase colorectal cancer screening. This information is essential to guide future CDC colorectal cancer screening promotion efforts and the allocation of resources for future colorectal cancer research.
In order to assess any change that occurs as a result of the intervention, one fourth of baseline patient respondents, will be asked to complete the survey twice; once prior to the intervention and a second time post-intervention. All clinicians and clinic staff will be asked to complete the survey post-intervention.
There are no legal obstacles to reduce the burden.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
All guidelines are met and this data collection fully complies with the regulation.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A. A 60-day Federal Register Notice was published in the Federal Register, on September 5, 2006, vol.71, No. 171, pp 52334-52335 (see Attachment 2). There were no public comments.
B. The study protocol, including the survey instruments, sampling plan and data collection procedures, was designed in collaboration with: 1) researchers at Battelle, Centers for Public Health Research and Evaluation through Contract No. GS-23F-8167H / Order No. 200-2002-F-00838 874-1, and 2) researchers and clinicians at HFHS and LHS through subcontracts. The Division of Cancer Prevention and Control at CDC has an internal work group that provided oversight and guidance on issues concerning the design of the survey instrument, the sampling design, and data collection procedures.
The CDC staff who participated in the development of the research protocol are:
Judith W. Lee, Ph.D.
Behavioral Scientist
CDC/NCCD/DCPC/EARB
4770 Buford Highway NE
Mailstop K-55
Atlanta, GA 30341
(770)488-4864
Mary C. White, Sc.D., MPH
Branch Chief
CDC/NCCD/DCPC/EARB
4770 Buford Highway NE
Mailstop K-55
Atlanta, GA 30341
(770) 488-3032
Steven Leadbetter, MS
Mathematical Statistician
CDC/NCCD/DCPC/EARB
4770 Buford Highway NE
Mailstop K-55
Atlanta, GA 30341
(770) 488-4143
The Battelle staff who participated in the development of the research protocol are:
Daniel Montaño, Ph.D.
Health Research Leader
Battelle Seattle Research Center
Centers for Public Health Research and Evaluation
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109-3598
(206) 528-3105
Danuta Kasprzyk, Ph.D
Health Research Leader
Battelle Seattle Research Center
Centers for Public Health Research and Evaluation
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109-3598
(206) 528-3106
Alvah Bittner, Ph.D.
Statistical Consultant
Battelle Seattle Research Center
Centers for Public Health Research and Evaluation
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109-3598
(206) 528-3263
Linda Carlin, Ph.D.
Health Research Scientist
Battelle Seattle Research Center
Centers for Public Health Research and Evaluation
1100 Dexter Avenue North, Suite 400
Seattle, WA 98109-3598
(206) 528-3040
LHS staff who participated in the development of the research protocol are:
Margaret J. Gunter, PhD
Research Director
Lovelace Clinic Foundation
2309 Renard Pl. SE, Suite 103
Albuquerque, NM 87106
(505) 262-7857
Cleveland Sharp, MD
Family Physician
Lovelace Clinic Foundation
2309 Renard Pl. SE, Suite 103
Albuquerque, NM 87106
505-888-8500
Richard Ming, MD
Gastroenterologist
Lovelace Clinic Foundation
2309 Renard Pl. SE, Suite 103
Albuquerque, NM 87106
505-244-7000
Shelley Carter, BSN, MPH, MCRP
Project Coordinator
Lovelace Clinic Foundation
2309 Renard Pl. SE, Suite 103
Albuquerque, NM 87106
(505) 262-7199
HFHS staff who participated in the development of the research protocol are:
Jennifer Elston Lafata, PhD
Research Director
Center for Health Services Research
Henry Ford Health System
1 Ford Place, 3A
Detroit, MI 48202
(313) 874-5454
Keoki Williams, MD
Clinical Epidemiologist
Center for Health Services Research
Henry Ford Health System
1 Ford Place, 3A
Detroit, MI 48202
(313) 874-5454
Tamir Ben-Menachem, MD
Gastroenterologist
UMDNJ-RWJ Medical School
125 Paterson Street-MEB 478
(732) 235-6513
Christina Moon, MA
Project Coordinator
Center for Health Services Research
Henry Ford Health System
1 Ford Place, 3A
Detroit, MI 48202
(313) 874-7432
Nine practicing physicians, nine patients and four clinic support staff participated in a review of the survey instruments designed for physicians (clinicians), patients, and clinic support staff, respectively, in the Fall of 2001. They each completed the appropriate questionnaire in order to measure respondent burden and provide comments and advice about the format, appropriateness, and relevance of survey questions. After significantly revising the instruments, a second pretest was conducted with clinicians, clinic support staff, and patients at HFHS and LHS. In-person and telephone interviews were conducted with six clinicians and seven clinic support staff at HFHS and LHS between April and July, 2004. Telephone interviews were conducted with a total of seven patients at HFHS and LHS between July and August, 2004. Recommendations from these pretest participants were incorporated into the final questionnaires (See section B.4). The review and pretest were anonymous so participant names are not included here.
9. Explanation of Any Payments or Gifts to Respondents
There is clear and consistent evidence that monetary incentives significantly increase response rates in most surveys, and experts on survey methods such as Kasprzyk, Montaño, St. Lawrence & Phillips, and Dillman recommend their use.26-28
Obtaining high survey response rates is particularly difficult from busy professionals like physicians. Several studies specifically designed to test the effects of incentives on physician survey response rates have confirmed the importance of monetary incentives. One study by Everett, Price, Bedell, and Telljohann found that response rates were 18% higher among physicians receiving incentives (63% vs. 45%).29 Another study by Tambor et al. found significantly more physicians responded when a $25 incentive was provided compared with a no incentive control group (62.0% vs. 18.3%).30 A third study by Berk, Edwards, and Gay divided physicians into three groups: Group 1 received a monetary incentive on the initial mailing, Group 2 received a monetary incentive on a second mailing to non-responders, and Group 3 received no incentive. 31 Response rates for the 3 groups were 63%, 50%, and 40%, respectively. Gunn and Rhodes, and Weber, et al tested incentives of $0, $25 and $50, and found increased physician response rates for higher incentives.32,33 Similarly, Kasprzyk et al. tested incentives of $0, $15 and $25 and found increased response with higher incentives (27%, 75% and 81% respectively).26 In 2000, primary care clinicians affiliated with two large managed care organizations were provided a $25 gift check incentive to complete a mail survey on STD care practices that required about 20 minutes to complete, resulting in a response rate of 82%.34 A recent national survey of clinicians on HPV infection management provided a $50 incentive and obtained 81% response.35
Berry and Kanouse also tested the timing of incentive payments and found that a $20 payment with the survey mailing resulted in significantly higher response (78%) than a promise of payment after return of the completed survey (66%).36 Clearly, it is best to provide incentives at the time the survey is sent rather than upon return of the completed survey. Therefore, the monetary incentives for this study will be included in the initial pre- and post-intervention survey packet sent to clinicians.
An incentive amount of $50 will be provided to clinicians. This amount was selected for two reasons. First, studies have found that as the amount of incentive increases, response rates also increase. Too small an amount is easy to ignore and may be insignificant, particularly to clinicians whose salaries tend to be higher than many other professionals. A $50 incentive is not likely to be ignored by a clinician and is near the optimal amount used by studies that have found positive associations between incentive amount and response rate. Second, there is evidence that response rates drop when the incentive approaches the cost of replacing the clinician’s revenue in their practice (i.e., what their time is worth to the practice). The selected incentive amount of $50 will avoid being viewed as reimbursement for the clinician’s time since it is far enough below the typical amount that the clinician’s time is worth to the practice ($150 to $250 per hour).37
In addition to clinicians who provide medical appointments, clinic support staff will be asked to complete a survey regarding clinic practices, beliefs, and opinions. The survey is similar to but shorter than the one designed for clinicians. Incentives have also been shown to be effective in the survey of medical support staff.38 Therefore, clinic support staff will also receive a monetary incentive, but in the amount of $25. As with clinicians, this amount is high enough to not be ignored by clinic support staff yet it does not surpass the amount that their time is worth to the practice (i.e., the value of clinic support staff to the total clinic revenue).
Studies assessing the effect of incentives to maximize response rates to surveys among the general population have found that, like physicians, the average survey participant is more likely to respond to survey participation requests if a monetary incentive is included.27,28
Several studies have shown that modest incentives are an effective way to maximize response rates among survey samples. In a study to assess the effect of the use of pre-payment versus post-payment of incentives on return rates to a longitudinal mail survey, Collins et al. found that a pre-payment of $20 or a post-payment of $25 had a positive effect on return rates compared to a post-payment control of $20, increasing the response rates by 3 to 7 percentage points.39 In another study, James and Bolstein found that modest monetary incentives of $1, $5 or $10 increased response rates to a mail survey between 2 and 12 percentage points.40 Similarly, Lesser et al. found that the inclusion of a pre-paid incentive of two dollars was just as effective at increasing response rates by 19 to 31 percentage points.41 Another study assessing the effectiveness of using a $2 or $5 incentive to maximize response rates to a mail survey of a community sample of health plan enrollees found that an incentive of $5 significantly increased response rates compared to a $2 incentive (74.3% vs. 67.4%).42 Finally, a mailed survey of patients obtaining care in primary care clinics in Washington State provided a $10 incentive and obtained 71% response.43 It is therefore believed that an incentive amount of $10, to be included with the pre- and post-intervention patient surveys, will be sufficient to maximize return rates among patients.
10. Assurance of Confidentiality Provided to Respondents
The CDC Privacy Act Coordinator has reviewed this study and has determined that the Privacy Act is or is not applicable. The applicable Privacy Act system of records is 09-20-0136, “Epidemiologic Studies and Surveillance of Disease Problems.”
Every effort will be made to maintain the privacy of participant data. The MCOs will carry out all survey mailings. Thus, only a few staff in the research offices of the MCOs will have access to the names of clinicians, clinic support staff and patients selected for the surveys. Collected survey data will not contain participant names, rather surveys will be assigned a unique identification number. The identification numbers will be placed on the surveys for the purposes of tracking and follow-up. Only this unique identification number, not the names of survey participants, will be entered into the survey data file. The tracking databases maintained by each MCO research office will be the only places where there will be a link between the unique survey identification number and the participant’s name. No survey data will be in the tracking databases. Access to the tracking databases will be limited to MCO research personnel who are responsible for activities associated with participant tracking (i.e., sending out reminder postcards). Raw survey data will not be shared with health care professionals at participating MCOs. Reports and publications of collected data will be presented in aggregate form only. In addition to these privacy procedures, all study personnel have been trained regarding the principles of human subjects protection.
Surveys will be stored by ID number only. Hard copies of surveys will be kept in locked file cabinets when not being edited or keyed. Prior to filing and to being sent to data keying, each survey will be carefully checked for any identifying information. If any identifying information is found it will be removed or blacked out from the survey. Data files will be backed up and only accessed by Battelle employees. Once data quality assurance measures (e.g., checking that all mailings have been sent and accounted for, checking for coding errors) are completed, the tracking file information that would allow linking of individuals to their survey data will be destroyed. Statements describing procedures to maintain respondent privacy are included on the survey instrument introduction pages and the survey cover letters (see Attachments 4 and 5).
The study protocol was submitted for CDC IRB review and was approved. CDC IRB approval is provided in Attachment 3. CDC will not receive any personal identifiers on respondents. The study protocol was also approved by the Battelle IRB and the two MCO IRBs. The MCO IRB reviews strictly enforce HIPAA requirements to protect patient privacy. At HFHS the research office is part of the health care system and therefore can access HFHS electronic records to select patients for the pre- and post-intervention surveys. At LHS, Lovelace Clinic Foundation (LCF) is a separate entity from the health system. Thus, LCF has obtained a HIPAA waiver to access the electronic records to select patients for the surveys, and will send a HIPAA authorization form to patients along with the survey. Selection of patients who are scheduled for a non-acute ambulatory care visit and due for CRC screening, and sending these patients the patient intervention materials, is considered to be part of provision of care at HFHS and LHS so HIPAA authorization is not required.
11. Justification for Sensitive Questions
The survey instruments with introductory information on the cover pages are found in Attachment 4. The survey cover letters are found in Attachment 5. There are no questions on this survey that are generally considered to be personally sensitive, such as personal practices related to sexual behavior, religious beliefs, or alcohol or drug use. While not sensitive information, some clinicians may feel anxious about being evaluated on their knowledge, attitudes, and practices, particularly if they are inconsistent with the most recently released medical information, or national clinical practice guidelines. However, most clinicians view national clinical practice guidelines as recommendations and do not view them as mandated practice standards.44 This was also confirmed by clinicians in the qualitative and pretest phases used to design the survey. To reduce potential anxiety about acknowledging practice inconsistent with national guidelines, clinician respondents are reminded on the survey cover page that CDC is seeking information on a variety of approaches to colorectal cancer screening. Similarly, patients may feel some discomfort about acknowledging that they have not obtained CRC screening. To reduce potential patient anxiety the patient survey cover letter indicates that CDC is interested in finding out why people do or do not have CRC tests and that their names will not be associated with their responses. These issues are addressed in the survey cover letters (Attachment 5) and the survey instrument Introductions (Attachment 4).
12. Estimates of Annualized Burden Hours and Costs
A. Clinicians and Clinic Support Staff
Surveys will be conducted with clinicians and clinic support staff in 21 primary care clinics at LHS and HFHS. All clinicians and clinic support staff will be asked to complete surveys post-intervention. Approximately 155 clinicians provide patient care in the 21 clinics, averaging 7.4 clinicians in each clinic. All 155 clinicians from the 21 clinics will be sent the post-intervention clinician survey, and 140 (90%) are expected to complete it. Although a maximum of 10% staff attrition is expected over the 1-year intervention period, clinicians and clinic support staff who leave LHS or HFHS are expected to be replaced. Thus, all 155 clinicians practicing in the 21 clinics will be sent the post-intervention survey, and 140 (90%) are expected to complete it.
Similarly, assuming approximately 7.4 clinic support staff in each clinic, it is expected that about 155 clinic support staff from all 21 clinics will be sent the post-intervention clinic support staff survey, and 140 (90%) will complete it. Although an estimated rate of attrition for clinicians and clinic support staff has been presented, every effort will be made to retain all clinicians and clinic support staff who participated in the pre-intervention facilitated educational sessions.
B. Patients
The pre-intervention patient survey will be sent to 4,725 patients (225 per clinic) and approximately 157 patients (70%) per clinic will respond, totaling 3,307 patient respondents across the 21 clinics. Of these pre-intervention survey respondents, it is estimated that there will be 70% continuous enrollment (2,314 total patients or 110 per clinic) and that 60% (1,388 total patients or 66 per clinic) will visit their clinics for a non-acute ambulatory care visit and will be due for CRC screening during the intervention period. These 1,388 patients will be sent a post-intervention patient survey, and it is estimated that approximately 46 patients (70%) per clinic, totaling 972 patients, will complete the post-intervention survey. Thus, it is estimated that 2,335 (3,307-972) patients will complete only the pre-intervention survey, while 972 will complete both the pre-intervention and post-intervention surveys.
Finally, an additional sample of 3,336 patients (158 per clinic) will be sent the post-intervention survey and approximately 2,335 patients (70%) will respond (approximately 111 per clinic). Thus, 2,335 patients will complete only the post-intervention survey. Section B.1 describes the survey sampling plan and rationale.
Table A.12C1 presents the estimated numbers of clinician, clinic support staff, and patient respondents to pre- and/or post-intervention surveys and the response burden. The numbers listed for clinicians and clinic support staff assume a best case of no attrition with all required respondents participating in the requested surveys.1 There are five distinct data collections: (1) clinicians will complete the clinician post-intervention survey; (2) clinic support staff will complete the clinic support staff post-intervention survey; (3) one group of 2335 patients will complete only the pre-intervention survey; (4) a different group of patients (N=2335) will complete only the post-intervention survey; and (5) a different group of patients (N=972) will complete both the pre-intervention survey and the post intervention survey. The time required to complete the surveys is estimated from pilot interviews conducted with nine practicing clinicians, five clinic support staff, and nine patients who responded to their appropriate surveys. After a review by 9 or fewer individuals, the response burden for clinicians is estimated at 70 hours, the response burden for clinic support staff is estimated at 58 hours, and the response burden for patients is estimated at 2,204 hours. The total estimated response burden for the data collection effort is 2,332 hours.
Table A.12C-1. Annualized Burden and Costs
Respondents |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden (in hours) |
Clinicians |
140 |
1 |
30/60 |
70 |
Clinic Support Staff |
140 |
1 |
25/60 |
58 |
Patients Surveyed Pre-Intervention Only |
2,335 |
1 |
20/60 |
778 |
Patients Surveyed Pre- and Post-Intervention |
972 |
2 |
20/60 |
648 |
Patients Surveyed Post-Intervention Only |
2,335 |
1 |
20/60 |
778 |
Total |
|
|
|
2,332 |
Table A.12C2 shows the estimated cost to respondents. Based on an average pay rate of $80/hour, the estimated cost burden for clinicians is $5,600. Based on an average pay rate of $25/hour, the estimated cost burden for clinic support staff is $1,458 (Dictionary of Occupational Titles, US Department of Labor, 2001).45 Based on an average pay rate of $12/hour, the estimated cost burden for patients is $26,456. Thus, the total cost burden for the data collection effort is estimated to be $33,514.
Table A.12C-2. Annualized Cost to Respondents
Type of Respondent |
Number of respondents |
Response burden (in hrs.) |
Hourly rate |
Respondent cost |
Clinicians |
140 |
70 |
$80 |
$5,600 |
Clinic Support Staff |
140 |
58 |
$25 |
$1,450 |
Patients |
6614 |
2,204 |
$12 |
$26,448 |
Total |
6894 |
2,322 |
$17.00 1 |
$33,498 |
1 Weighted mean hourly rate
13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
There is no direct cost to respondents or record keepers.
14. Annualized Cost to the Government
This project has been fully funded by CDC. The project costs are shown in Table A.141. These total costs include (1) contract costs for Battelle and the participating MCOs to make final preparations for intervention implementation, conduct the pre-intervention surveys, implement the intervention, conduct the post-intervention surveys, compile electronic record data from MCOs, clean and analyze the data, and write reports on study findings, and (2) the cost of CDC staff to provide oversight to the study. The total contract cost for carrying out the project is $2,034,939 over a period of about 32 months after receiving OMB clearance. Table A.141 shows these contract costs broken down by personnel and other intervention and data collection costs (including MCO subcontracts printing, mailing, incentives, and data entry). The annualized contract cost is $763,102. The CDC oversight costs include personnel costs of Federal employees involved in oversight, analysis, report preparation, and administrative support, estimated at $75,466 (25% of a FTE at GS-13 plus 5% of a FTE at GS-15 plus 5% of a FTE at GS-14) over the entire 32-month project period. Thus, the annualized CDC oversight cost is $28,300. There are no travel, equipment, or printing costs for the government. Thus, the total cost to the government over 32 months is $2,110,405, and the average annualized cost to the government is $791,402.
Table A.14-1. Annualized Cost to the Federal Government
|
Total 32-Month Cost |
Annualized Cost |
Contract Costs: |
|
|
Personnel |
$810,138 |
$303,802 |
Other intervention and data collection costs |
$1,224,801 |
$459,300 |
Total Contract Cost |
$2,034,939 |
$763,102 |
CDC Oversight |
$75,466 |
$28,300 |
Total Cost |
$2,110,405 |
$791,402 |
15. Explanation for Program Changes or Adjustments
This is a new data collection.
16. Plans for Tabulation and Publication and Project Time Schedule
The
study will proceed in four main phases after receiving OMB clearance:
1) Pre-Intervention Data Collection, 2) Intervention, 3)
Post-Intervention Data Collection, and 4) Data Analysis and Report
Writing. The pre-intervention patient survey sample will be
drawn and preparation for survey data collection will occur during
the first month after OMB clearance.
Pre-intervention surveys of patients
subsequently will
be carried out within four months after OMB clearance. Upon
completion of the pre-intervention surveys, the intervention will
begin and will be carried out for 12 months. Subsequently, the
post-intervention patient survey will be conducted quarterly among
patients who have a non-acute ambulatory care visit, beginning four
months after initiation of the intervention and concluding four
months after the end of the 12-month intervention period. On a
different schedule, the post-intervention clinician and clinic
support staff surveys will be carried out 12 months after initiation
of the intervention. Electronic record data capture to compute CRC
screening outcome rates will take place nine months after the end of
the intervention period, allowing sufficient time for patients seen
at the end of the intervention period to receive endoscopic screening
and for these data to enter the record systems. Data analysis and
report writing will be carried out beginning six months after
completion of the post-intervention surveys through three months
after final capture of electronic record data. Report writing will
be ongoing during the data analysis, and a Final Report will be
complete within four months after completion of analyses. Thus, the
intervention evaluation survey data collection will be complete
within 19 months of OMB clearance, and analysis and reports will be
complete within 32 months of OMB clearance. Table A.16-1
provides a summary of the study activities and the months following
OMB clearance during which they will take place.
Activity |
Time Period (Months after OMB Clearance) |
Sample preparation |
Month 1 |
Pre-Intervention Surveys |
Months 2 to 4 |
Intervention |
Months 5 to 16 |
Post-Intervention Patient Surveys |
Months 8 to 20 |
Post-Intervention Clinician and Staff Surveys |
Months 17 to 18 |
Electronic Record Capture |
Months 9 to 25 |
Data Analysis |
Months 20 to 28 |
Report Writing |
Months 24 to 32 |
Battelle will submit a draft report on the survey findings to CDC and will then prepare a Final Report that incorporates CDC comments and recommendations on the draft report. The Final Report will present the results of analyses of the clinician, patient, and clinic support staff surveys to evaluate the effect of the intervention on opinions, attitudes, and screening behavior. The report will also present results of analyses to assess the effect of the intervention using CRC screening rates obtained from electronic record data. The Final Report will be used as a basis for publishing the findings from this study in peer-reviewed journals.
Data analysis will be conducted in three stages after completion of the post-intervention data collection: 1) descriptive analyses, 2) outcome analyses, and 3) structural analyses. The data will be analyzed to determine the impact of the intervention on two main classes of outcome measures: CRC screening behavioral outcomes (primary [e.g., completion of CRC screening] and secondary [e.g. CRC screening discussion, CRC screening appointment]) and intermediate outcomes (e.g., attitudes, beliefs, opinions, social influence). Process measures of exposure to the intervention will also be included at the third stage of analysis. Similar analytic procedures and stages will be used for data from the clinician, clinical staff, and patient surveys, though there will be some variation in these analyses due to the outcomes assessed, the evaluation design, and sampling methods as described in Section B.1.
As described in Section B.1, a patient cohort design will be used to collect patient survey data to assess change in patient intermediate outcomes. However, because of the larger sample size required to assess the effect of the intervention on patient dichotomous behavioral outcomes, the patient cohort design is inefficient to assess these outcomes (see Section B.1). Therefore, the effect of the intervention on patient reported behavioral outcomes will be assessed using post-intervention patient survey measures only. The analysis of patient behavioral outcomes consequently utilizes a cross-sectional design, comparing post-intervention data across the study arms.
Below we describe the three main analytic stages.
1. Descriptive Analyses
In the first stage, descriptive analyses will be conducted on all clinician, clinic support staff, and patient survey items. Variances and distributions of all items will be examined after each survey, both as part of the data assurance process and to set the stage for later analyses. During the pre-intervention and post-intervention survey descriptive analyses, formal item analyses will also be conducted on sets of items thought to be measuring related constructs. Inter-item correlations will be conducted on all items thought to be measuring related constructs to ensure that convergent validity exists, yet items account for separate variances. Cronbach alphas will be calculated for all scaled items to determine which items to retain or reject in the subsequent analyses. Descriptive analyses will ensure the basis for subsequent analyses.
2. Outcome Analyses
Table A.16C2-1. Example Shell Table of Study Results
Outcome Measures
|
Study Condition Means |
Two-Way ANOVA MS, df, F, and p-values |
||
Clinic Only |
Clinic & Patient |
Control |
||
Screened by any CRC screening method |
|
|
|
|
-Screened with FOBT |
|
|
|
|
-Screened with flexible sigmoidoscopy |
|
|
|
|
-Screened with colonoscopy |
|
|
|
|
Belief 1 about FOBT (mean) |
|
|
|
|
Belief (x) about FOBT (mean) |
|
|
|
|
Belief 1 about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief (x) about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief 1 about colonoscopy (mean) |
|
|
|
|
Belief (x) about colonoscopy (mean) |
|
|
|
|
Additional intermediate outcome measures…. |
|
|
|
|
Rate of ordering any CRC Screening |
|
|
|
|
-Rate of ordering FOBT |
|
|
|
|
-Rate of ordering flexible sigmoidoscopy |
|
|
|
|
-Rate of ordering colonoscopy |
|
|
|
|
Belief 1about FOBT (mean) |
|
|
|
|
Belief (x) about FOBT (mean) |
|
|
|
|
Belief 1 about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief (x) about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief 1 about colonoscopy (mean) |
|
|
|
|
Belief (x) about colonoscopy (mean) |
|
|
|
|
Additional intermediate outcome measures…. |
|
|
|
|
Distribution of FOBT (mean) |
|
|
|
|
Belief 1about FOBT (mean) |
|
|
|
|
Belief (x) about FOBT (mean) |
|
|
|
|
Belief 1 about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief (x) about flexible sigmoidoscopy (mean) |
|
|
|
|
Belief 1 about colonoscopy (mean) |
|
|
|
|
Belief (x) about colonoscopy (mean) |
|
|
|
|
Additional intermediate outcome measures… |
|
|
|
|
The primary behavioral outcome analysis, conducted at the clinic level, will employ analysis-of-variance (ANOVAs) on patient screening measures obtained on the post-intervention patient survey and validated with clinic medical records data (potentially including the baseline rates as covariates). Though FOBT, flexible sigmoidoscopy and colonoscopy rates are of individual interest, the variable of greatest interest will be the overall screening rate (i.e., screening with any screening method). Table A.16C21 presents a table shell to illustrate how these findings will be presented. As shown in this table, two-way (Clinic X Patient Intervention) ANOVAs will be used to test whether there are main and/or (CxP) interaction effects (analogous Repeated-measures ANOVAs “RANOVAs” will be applied when baseline data are available). Owing to potential interest, power calculation was conducted with regard to potential post-hoc paired comparisons among any individual pairs of study arms (See Section B.2). It is expected that the clinic intervention alone arm will result in higher screening than the control arm, and that the combined patient and clinician intervention arm will result in higher screening than the clinic intervention alone.
Next, analyses will be conducted to assess the effect of the intervention on intermediate patient outcomes (e.g., knowledge, attitudes, beliefs, and opinions) measured on the patient survey. Table A.16C1 illustrates how these findings will be presented. Here, it is anticipated that RANOVAs will be conducted on each of these measures assessed pre- and post-intervention. Of note, repeated measures analysis is appropriate in this study because of the cohort design, and has substantial statistical power advantages over a post-intervention only analysis. RANOVA is expected to be very appropriate for this purpose given its well-known robustness in pre-post settings, particularly after selected transformations (e.g., Scheffe’, 1959, Ch. 10).46 These analyses will provide an initial assessment of both the intermediate and behavioral outcome efficacy of the intervention. Table A.16C2 delineates the general structure initially assumed for the analysis (e.g., Winer et al, 1991, Table 5.19)47 where, for purposes of simplification, pre-post data are assumed to have been converted into simple or the potentially more powerful regressed ‘difference scores’ (Cronbach and Furby, 1970).48
Table A.16C2-2. Expected Mean Squares
[Clinics (factor B) is nested under Interventions (factor A)]
Source of Variation df Expected Mean Square Test-Ratio
__________________________________________________________________________________________________________________________________
1. A p – 1 2 + nr 2 + nqr 2 (1)/(2)
2. B w. A p(q – 1) 2 + nr 2 (2)/(3)
3. Within Cell p q(n – 1) 2
3. Structural Analyses
The third stage of analysis, using regression models applied at the individual participant level within clinics, will examine the direct relationships between the intermediate and behavioral outcomes. Broadly, the model would support the structure of successive outcomes shown in the figure below.
Structure of Outcomes based upon the Integrated Behavioral Model (IBM)
Intervention |
|
Exposure |
|
Intermediate Outcomes |
|
Behavioral Outcomes |
Consistent with this structure, our approach would be to first evaluate the post-intervention Intermediate outcomes (attitudes, beliefs and opinions) as a function of their pre-intervention values, intervention study arm, and intervention exposure obtained from process measures and the surveys. The patient data analysis will include assessments of any added effects of age and gender, as well as clinic characteristics. In turn, the post-intervention behavioral outcomes would be evaluated as a function of both their pre-intervention values and intermediate outcomes, with assessments of any added effects of age and gender, as well as clinic characteristics. Finally, our approach would separately evaluate both the overall CRC screening status (0,1) and status of the individual screening tests (0,1) of each patient using logistic regressions. Variables in these analyses would include intervention arm, gender and intermediate outcomes as predictive variables, with assessments of any added effects of age and clinic characteristics. These analyses altogether will provide information regarding the impact of the intervention on exposure and intermediate outcomes leading to behavioral screening outcomes.
17. Reasons Display of OMB Expiration Date is Inappropriate
There is no request for an exemption from displaying the expiration date for OMB approval.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exemptions being requested for this clearance
1 With attrition the total burden would be the same except that some individuals will participate in either the pre- or post-intervention survey alone, while fewer will participate in both surveys. Patients will complete pre-and post-intervention surveys; clinicians and clinic staff will complete post-intervention surveys only.
| File Type | application/msword |
| File Title | Supporting Justification for OMB Clearance Package |
| Author | Dvv1 |
| Last Modified By | shari steinberg |
| File Modified | 2009-10-14 |
| File Created | 2009-10-14 |