TS_Primary_Care_OMB_Supporting Statement Part B_FINAL

SUPPORTING STATEMENT

Part B

“Evaluation of the Implementation of TeamSTEPPS in Primary Care Settings (ITS-PC).”

Version: April 14, 2014

Agency for Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 4

4. Tests of Procedures 5

5. Statistical Consultants 5

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The TeamSTEPPS in Primary Care Post-Training Survey will be distributed to all individuals who have participated in the TeamSTEPPS in Primary Care Training program. It is anticipated that 150 participants will be trained during April – December 2014. We will not employ sampling, as the Agency for Healthcare Research and Quality (AHRQ) seeks to obtain feedback from everyone who participates in the program (i.e., a census survey). A 75 percent response rate or higher is anticipated.

To inform the survey results, semi-structured interviews will be conducted with 9 individuals who participated in the TeamSTEPPS in Primary Care Training program and responded to the Post-Training Survey.

Some of the limitations of this study, which will also be noted in the final report, include the following:

• Retrospective analysis of post-training experiences. This is not an empirical study; therefore, neither causality nor generalizability can be established;

• Survey non-response, which can limit AHRQ’s understanding of the concerns and issues participants encountered when trying to apply TeamSTEPPS tools and concepts in their home organizations or organizations that they support;

• Lack of access to contact information for non-respondents;

• Limited number of interviews that can be conducted per training class due to time and budgetary constraints; and

• Inability to survey participants who have changed jobs since TeamSTEPPS in Primary Care Training. The survey will be administered six months after training.

Despite the limitations in this study, the data collected will provide valuable information that will help refine the TeamSTEPPS in Primary Care curriculum to support primary care practices’ efforts to improve patient safety. By understanding the factors that facilitate or inhibit the use of tools or the spread of knowledge, AHRQ will better understand the needs of primary care practices and will be better prepared to address their future concerns, issues, and needs for improving patient safety.

2. Information Collection Procedures

The TeamSTEPPS in Primary Care Post-Training Survey will be administered to all individuals who have participated in the TeamSTEPPS in Primary Care Training program using the Web-based questionnaire included in Attachment A. The participant questionnaire will be sent to all training participants via email. All correspondence to training participants is presented in Attachments C-F. These correspondences contain a hyperlink to enable easy access to the online questionnaire.

The TeamSTEPPS in Primary Care Post-Training Survey will be accessible to training participants 24 hours a day, for a total of 30 days. On entrance into the questionnaire, respondents will view an introduction page that explains the questionnaire objectives and stresses the importance of participation. Respondents can respond to the questionnaire items by clicking on pre-coded options for closed-ended items and typing in boxes for any open-ended items. The survey requires 20 minutes to complete.

Completed responses to the questionnaires will be backed up daily onto IMPAQ’s dedicated data collection server and will also be printed and stored in a locked cabinet. Data responses will be checked visually by researchers and analysts on a regular basis to assure that data are entered appropriately into the database.

Following data collection, questionnaire responses will be compiled and assessed formally for data quality to produce a finalized database for statistical analyses. Incomplete response data poses a substantial threat to confident interpretation of the study results. The general approach to handling incomplete response data is to salvage as much data as possible using multiple techniques for examining patterns of missing data. Given the scope of the training participant questionnaire, IMPAQ will review questionnaire items with a substantial proportion of omitted responses. The precise cutoff percentage is typically chosen once the distribution of missing data has been established. It will be determined whether responses are missing in a manner that relates to other observable values. If data are determined to be missing in a manner that affects the interpretability of the responses, descriptive statistics and point estimates of relations among variables may be adjusted to account for missing data. This may be accomplished by using multiple imputation and full information maximum likelihood estimation techniques.

The better under the survey results, semi-structured interviews will be conducted with nine survey respondent by phone. An experienced IMPAQ interviewer will conduct all interviews after obtaining written informed consent. Each interview will last about 60 minutes. A trained IMPAQ staff member will take notes on the interview; the interviews will not be transcribed. Notes from the interviews will be stored in secured computer servers at IMPAQ.

3. Methods to Maximize Response Rates

To effectively bolster the response rate, time-staggered notices of the opportunity to participate in the questionnaire will be sent via email. If at any point an invited respondent refuses to participate, he or she will not be contacted again through any means.

During the data collection period, invited respondents who have yet to respond to the training participant questionnaire will be re-contacted via email reminding them of the opportunity to participate and the importance of their feedback regarding the training program. The re-contact notice sent via email will provide the hyperlink to access the questionnaire, the estimated time (in minutes) it will take to respond, the impending deadline for submission of their responses, and additional information regarding privacy of responses and confidentiality of personal information. The re-contact notice is provided in Attachment D.

It is important to note that the most common way to ensure an acceptable response rate is to provide introductory information through advance notice from a known and respected source – in this case, AHRQ. For the purposes of this study, the AHRQ Project Officer will distribute an e-mail to all invited respondents to provide advance notice of the study and the importance of participation (see Attachment C).

4. Tests of Procedures

A prior version of the TeamSTEPPS in Primary Care Post-Training Survey was piloted. The pilot test involved: (1) a series of two to three cognitive laboratory interviews with AHRQ staff members and (2) a field test of the survey with a subsample of no more than nine potential respondents. The results were used to refine the questionnaire prior to field testing. Content changes to align this version of the questionnaire with the TeamSTEPPS in Primary Care curriculum have been reviewed by subject matter experts.

5. Statistical Consultants

IMPAQ International, LLC will serve as the primary consultants for statistical aspects of the design and analysis of the Web questionnaire data. Dr. David P. Baker, Chief Program Officer at IMPAQ, is the primary point of contact for statistical design and analyses. He can be reached at [email protected] or 443-539-9773.