Supporting Statement Part B 20160605

SUPPORTING STATEMENT

Part B

Collection of Information for: Agency for Healthcare Research and Quality’s (AHRQ) Guide to Improving Patient Safety in Primary Care Settings by Engaging Patients and Families – Evaluation

Version June, 2016

Agency for Healthcare Research and Quality (AHRQ)

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 5

3. Methods to Maximize Response Rates 5

4. Tests of Procedures 6

5. Statistical Consultants 6

B. Collections of Information Employing Statistical Methods

The data collection planned involves focus groups and interviews, all of which will collect only qualitative and contextual data. Quantitative data collection will not provide the type of feedback on the interventions or the Guide required to address the objectives of this contract.

The information to be collected is primarily qualitative. Interview and focus group data will be thematically analyzed using standard techniques for thematic review. This will include a generation of content codes using an iterative process involving transcript review, generation of preliminary codes, team review, revision, application of codes to transcripts, elaboration, and continued application and elaboration as needed. When the team concurs that the code list captures all themes identified in the transcripts, a trained research assistant will code all transcripts. Another team member will also code a subset of at least 10% of the transcripts to check coding consistency. Results of both reviews will be entered into a case record form within the REDCap™ database. Descriptive statistics will be provided to include simple frequencies and percentages for other evaluations involving categorical and continuous data such as the assessment of representativeness (age, gender, education, race/ethnicity, site of care, provider and patient profiles, etc.), and other measures for process evaluations. Student’s t-test will be used for continuous variables and the chi-square test for categorical variables for any bivariate analyses. Any statistical results will be based on two-sided tests with significance level of 0.05.

1. Respondent universe and sampling methods

The data collection methods proposed under this effort are primarily qualitative in nature and will not require quantitative analysis. Recruitment and sampling methods are described below

Approach to Interviews and Focus Groups

Recruitment of Participants

This collection is for focus groups with practice staff as well as patients and family members and cognitive interviews with primary care providers in two distinct geographic locations to enhance the diversity of the informants and generalizability of the results. Patients and family members, practice staff, and primary care providers within the communities of practices supported by AHRQ contractor MedStar Health (MD, DC, VA) and their subcontractor Clinical Directors Network (CDN) (NY), will be offered the opportunity to participate as key informants for the interventions. These partners were selected due to their broad geographic reach, the diverse patient populations served, experience in delivering educational and behavior change interventions to diverse audiences, familiarity with the process of research and rigorous evaluation, and leadership’s willingness to serve as champions for identifying and recruiting pilot practices to engage in the process. The project teams at MedStar Health and CDN have significant experience and success in recruiting diverse patient and provider populations to participate in research activities.

In addition to these criteria, each partner has member practices that serve high proportions of AHRQ priority populations. This approach has an additional benefit in that members of the evaluation team from MedStar and CDN will be able to conduct focus groups and cognitive interviews in the community at minimal cost across both geographic regions. This will facilitate stakeholder recruitment for the evaluation activities and yield a more generalizable product. The table below summarizes key characteristics of the MedStar Health and CDN practice networks to help characterize the sampling universe for this effort.

Characteristics of Practices Available to Evaluate Intervention Materials

	MedStar Health	CDN
Geographic Location	Maryland, Virginia, District of Columbia	17 states (NY, NJ, ME, MA, NH, VT, RI, CT, PA, MD, VA, WV, FL, OK, UT, CA, TX) and three territories (Puerto Rico, Washington DC, US Virgin Islands)
Practice Locations	Urban, inner city, suburban, critical access, rural	Urban, inner city, suburban, critical access, rural, FQHC
Number of Practices	300 MedStar Health-owned practices 1,040 affiliated practices 85 medium/large PCP 21 predominantly serving AHRQ priority populations	CDN is a not-for-profit clinician membership organization, practice-based research network (PBRN), and clinician training organization, founded to provide peer-initiated activities for clinicians practicing in low income, minority, and other underserved communities. CDN has over 230 member practices.

Numbers of Participants

Patients and Family Members –Up to 6 patients and/or family members will be recruited to participate in focus groups for the Guide field test. These patients and/or family members will be recruited from each of the 12 practices participating in the Guide field test. It is anticipated that patient or family members will participate in the evaluation during both the initial and the follow-up intervention, whereby obtaining information on each of the two interventions that the practice has agreed to adopting. Thus, over the Guide field testing period, up to 72 patients and family members in total will be recruited to participate in focus groups. Each participant will be asked to commit to both the Evaluation 1 and Evaluation 2 focus groups (as described in supporting statement A).

Practice Staff – Up to 6 primary care practice staff members will be recruited to participate in focus groups during Guide field testing. Thus, over the Guide field testing period, up to 72 practice staff members will be recruited to participate in focus groups from the 12 primary care practices that are engaged in the field testing of the Guide. Each participant will be asked to commit to both the Evaluation 1 and Evaluation 2 focus groups.

Primary Care Providers – Up to 2 primary care providers will be recruited to participate in cognitive interviews for each intervention. This will include up to 24 providers within the 12 primary care practices that will interviewed for the Guide field testing activities. Each primary care provider will be asked to commit to both the initial and the follow-up evaluations of an individual intervention.

2. Information Collection Procedures

This collection involves analysis of mainly qualitative and some descriptive quantitative project data to evaluate intervention effectiveness. Qualitative data will be taped during interviews and focus groups and transcribed and analyzed using standard techniques for thematic review. Content codes will be generated using an iterative process involving transcript review, generation of preliminary codes, team review, revision, application of codes to transcripts, elaboration, and continued application and elaboration as needed. When the full team concurs that the code list captures all themes identified in the transcripts, a trained research assistant will code all transcripts. Another team member will also code a subset of at least 10% of the transcripts to check coding consistency. Descriptive statistics, simple frequencies and percentages for other evaluations involving categorical and continuous data such as the assessment of representativeness (age, gender, education, race/ethnicity, site of care, provider and patient profiles, etc.), and other measures for process evaluations will be provided. Student’s t test for continuous variables and the chi-square test for categorical variables for any bivariate analyses will be conducted. Where possible depending on the intervention, multiple weekly observations will be obtained before and after the intervention comes on line to allow for interrupted time series (ITS) analysis or statistical process control (SPC) approaches to strengthen the analysis. All statistical results will be based on two-sided tests with significance level of 0.05.

3. Methods to Maximize Response Rates

In order to maximize response rates and increase the likelihood of participation due to the time commitment required of participants, all potential participants will be offered an incentive as described in supporting statement A. The amount of the cash incentive is based on the industry standard to recruit and retain participants for this type of data collection in the metropolitan areas described in section 1 above. As this is primarily qualitative data collection, the data collected is expected to be contextual in nature and provide insight into the perspectives of representative patients, family members and practice staff in each geographic locale. These perspectives will be representative of those for the practices in which the Guide and its respective interventions will ultimately be disseminated.

4. Tests of Procedures

Because this data collection is primarily qualitative there will be no pretesting. The goals of this evaluation as described in supporting statement A will be reasonably met using the planned methodology (focus groups and interviews). Questions on these instruments were developed in collaboration with subject matter experts and content experts and we expect that they are phrased in such a way as to obtain the necessary information required for this type of evaluation and provide satisfactory results.

5. Statistical Consultants

This research will involve the collection of primarily qualitative data. Any analysis will be completed by our study team. There will be no statistical consultants used

6