Download:
pdf |
pdfEVALUATION OF THE STATE
HEALTH INFORMATION EXCHANGE
COOPERATIVE AGREEMENT
PROGRAM
National Survey on Health Information
Exchange in Clinical Laboratories:
Cognitive Interviewing Report
PRESENTED TO:
The Office of the National
Coordinator for Health
Information Technology
(ONC)
200 Independence
Avenue, SW
Suite 729-D
Washington, DC 20201
(202) 205-3754
JULY 31, 2012
PRESENTED BY:
Authors: Felicia LeClere,
Rene Bautista, Lindsay Virost
NORC at the
University of Chicago
Prashila Dullabh
Health IT Program Area Director
4350 East-West Highway, 8th Floor
Bethesda, MD 20814
(301) 634-9418
Acknowledgements
Many people assisted in this research effort and the development of this report.
Prashila Dullabh and Susan Schechter played an integral role in developing the framework for the
interviews, reviewing this report, and helping to implement the changes in the future questionnaire. In
addition, the authors would like to thank Mary Kennedy and Ron Ranauro, from the College of American
Pathologists, Lynn Bryan, NorthEast Community Laboratory Alliance (NECLA), and Greg Farnum, HLN
Consulting LLC., for their help recruiting participants.
Table of Contents
Executive Summary ........................................................................................................ 1
Introduction ..................................................................................................................... 3
Methods .......................................................................................................................... 4
Sample definition ......................................................................................................... 4
Recruitment ................................................................................................................. 6
Interviewing methods and protocols ............................................................................ 7
Results ............................................................................................................................ 7
Issues of Comprehension............................................................................................ 8
Recommendation
................................................................................................... 12
Issues of Information Retrieval .................................................................................. 13
Recommendation
................................................................................................... 15
Issues of Communication of Answers ....................................................................... 16
Recommendation
................................................................................................... 18
Conclusion .................................................................................................................... 18
Limitations ................................................................................................................. 18
Summary Findings .................................................................................................... 18
Final Recommendations................................................................................................ 19
Appendices
Appendix A: Cognitive Interviewing Protocol – Version A (Hospitals)……………………..….20
Appendix B: Cognitive Interviewing Protocol – Version B (Non-hospitals: regional &
independent)……………………………………………………………………………………37
Appendix C: Final Tabulation of Coded Responses……………………………………………50
Executive Summary
This document summarizes a series of interviews designed to elicit the cognitive processes respondents
use to complete two versions of the National Survey on Health Information Exchange in Clinical
Laboratories. Cognitive interviewing is an established technique to pre-test survey instruments and
materials, such as cover letters and supplemental information, which will be fielded in a larger sample via
hardcopy mail-out. While not as robust or effective as a field test, cognitive research techniques can
provide useful information to questionnaire designers seeking to improve wording and response options
for complex measures or difficult questions to answer. For this pre-test, we used both personal contacts
and data from the Clinical Laboratory Improvement Amendments (CLIA) database collected by the
Centers for Medicare &Medicaid Services to identify potential respondents from small and large
independent and hospital medical laboratories, which were administratively independent or affiliated with
a centralized entity. In July 2012, we conducted one round of interviews for each instrument; nine
interviews were conducted for a hospital based instrument and three interviews were conducted for an
instrument designed for independent labs. Four experienced, NORC cognitive interviewers were trained
for this project, working from structured interview protocols. We used both quantitative coding and openended notes to identify questions where there are problems in comprehension, information retrieval, or
communication of the response.
These two instruments seek to collect information which will provide crucial state and national data on
the status and volume of laboratory information exchange occurring electronically, reporting capacity,
and other key measures of electronic exchange. These measures of the new and complex concepts that
surround the development of a nationwide health IT infrastructure are essential to inform major changes
in the operations and information systems of clinical laboratories.
The major findings of this cognitive testing effort suggest that both the administrative structure of
laboratories and the health information exchange process are variable and in a state of transition. We
found particular difficulties in defining the terms “affiliated physicians” and “final billable tests” across
all respondents. Respondents were also uncertain how to respond if their laboratory was affiliated with
other laboratories either administratively or technically. Technical terms such as “free text” or “coded
format” presented problems if the respondent was not in the IT/LIS departments of the labs or if they did
not consult with staff on the technical side of the laboratory. Similar comprehension and reporting
experiences occurred for reporting on transaction volumes, particularly where the laboratories did not
collect the information in the normal course of business practice or it was not part of the lab work flow.
Cognitive Interviewing Report | 1
In those instances, respondents estimated the answers using a variety of techniques. Finally, several
questions included mutually exclusive response categories that did not match respondents’ experiences
because they fell into more than one of the categories. Other smaller definitional and informational issues
occurred throughout the questionnaire, all of which have simple solutions.
We conclude this report with a summary of the limitation of cognitive interviewing as a singular pre-test
methodology as well as a full list of recommendations for modification of both the questionnaire and the
cover letter. In summary, while additional testing might prove useful, we believe that due to cost and time
constraints, this exercise has provided sufficient information to revise the questionnaires and proceed with
the survey process upon approval from the Office of Management and Budget.
Cognitive Interviewing Report | 2
Introduction
Cognitive interviewing represents one important tool for pretesting survey instruments before they go into
the field. This qualitative methodology is designed to address the cognitive processes respondents use
when they answer survey questions. There are many cognitive interviewing techniques which can run the
gamut from loosely organized “think aloud” interviews to structured “verbal probing” to elicit particular
conceptual domains.
In this application, we used structured verbal probing for a variety of reasons including our decision to
mail a copy of the survey in advance of the interview and our interest in focusing on specific concerns we
had with words, concepts, and phrases in draft questions and response categories. Figure 1 illustrates the
survey cognition model used to build our testing protocols.
Figure 1. Theoretical Survey Response Model for the National Survey on Health Information Exchange in
Clinical Laboratories
Comprehension
Information retrieval
Source decision
Memory-based
estimates
Record look-up
Judgment
Communication
Adapted from Sherman & Cantor, 1991, Toward a Response Model in
Establishment Surveys, Measurement Error in Surveys, Biemer et. al (ed).
Cognitive Interviewing Report | 3
We were particularly interested in the comprehension of keys terms, the retrieval of factual information,
and the communication of that information into a response to a survey question. Because much of the
terminology in the electronic exchange of health information continues to emerge, we wanted to test
whether respondents to this survey have a shared understanding of the concepts of interest or have
consistent access to the information required. The aim of the cognitive protocols was to elicit the range of
concepts and definitions used by respondents. The second challenge of this survey, which may be unique
to establishment or institutional surveys, was whether data are routinely collected in labs that conform to
the categories we have created for this purpose. The exercise of recalling specific numbers is difficult if
there is not a “map” from the concepts to the types of data normally collected in business practice.
Finally, because there was substantial variability in the size and functioning of labs, we needed to
understand whether we have provided labs with the appropriate answer choices to communicate their
particular responses.
The protocols, which are included in Appendices A and B, provided for both narrative and coded
responses by the cognitive interviewers. We analyzed the data coded by interviewers in the interview
protocols and supplemented the analysis with selective review of audio recordings of the interviews. We
also conducted informal debriefings with the interviewers, which provided qualitative impressions and
opinions about respondent difficulties and possible solutions. These elements allowed us to understand
both the nature and the depth of the cognitive issues that have arisen in the questionnaires and develop
recommended improvements to the instruments. The final tabulation of coded responses appears in
Appendix C.
Methods
In this section, we describe the method by which we selected and recruited candidate laboratories for the
cognitive interview process. We also describe the recruitment process, interviewer training, and the final
completed sample of respondents.
Sample definition
While cognitive interviewing does not require a statistically valid sampling plan, we chose a recruitment
strategy that would maximize our ability to recruit a broad range of laboratories. Our goal was to conduct
nine interviews in hospital labs using the hospital-based instrument and nine interviews with independent
labs using the independent-based instrument. The heterogeneity of medical laboratories and their degree
of technical sophistication prompted us to select as complex a sample as possible. To aid in sample
Cognitive Interviewing Report | 4
selection, we geocoded the CLIA data file and also matched the hospital labs to data from the American
Hospital Survey. We wanted to recruit large and small laboratories as well as those that were part of a
multi-site laboratory group and those that had a single site and operated independently.
For independents, we eliminated all laboratories that we could not geocode and then identified labs in
metropolitan and non-metropolitan counties as a proxy for small and large labs. We also looked at
organization names from the CLIA database to identify whether or not the laboratory belonged to a larger
affiliate. The final step was to match the resulting file to a list received from the College of American
Pathologists that provided a name and address for the laboratory director. The final matched file was
used to generate potential respondents within the respective subgroups described in Table 1 shown below:
Table 1: Cognitive Interview Recruitment Goals for Independent Labs
Small
Large
Total
Multisite
2
2
4
Single site
3
2
5
Total
5
4
9
For hospitals, we followed a similar plan using data from the American Hospital Association 2011 survey
of hospitals. First, we used the geocoded addresses on the CLIA database and, then, used fuzzy matching
software to match to addresses on the AHA survey1. We eliminated hospitals that did not match and
cross classified the resulting list using two variables: “total number of beds” and “whether more than 1
lab was matched to the hospital system.” Again, the final step was to match the resulting file to a list
received from the College of American Pathologists that provided a name and phone number for the
laboratory director. We found email addresses through Google searches. The final matched file was used
to generate potential respondents within the subgroups described in Table 2 shown below:
Table 2: Cognitive Interview Recruitment Goals for Hospital Labs
Small
Large
Total
1
Multisite
2
2
4
Single site
3
2
5
Total
5
4
9
Cognitive Interviewing Report | 5
Fuzzy matching allows for incomplete text strings or abbreviations to be used in the matching process. It is a useful method when
the matching items are text strings or free form texts which addresses often are because of the mode of entry.
Recruitment
Methods
We selected respondents who represent both large and small independent and hospital laboratories with
and without multiple sites. NORC used leads within the field, including contacts within laboratory
associations, to help reach out to potential respondents. In addition, NORC supplemented these leads by
selecting, at random, potential respondents with the desired characteristics from the CLIA and AHA
datasets described above. Additional contact information for potential respondents was obtained via web
searches. In total, we made 10 cold calls, sent 66 initial contact emails, 56 follow-up emails, 33 follow-up
calls, and 37 mail packets consisting of hard copies of the cover letter and applicable survey instrument.
Initial contact emails, sent from the Project Director’s mailbox, informed potential respondents of the
survey and invited them to participate in the survey instrument testing. The email also provided a link to a
Doodle calendar, where respondents could anonymously select the most convenient available time slot.
Follow up emails were sent, at minimum, two days following the initial contact email. Follow-up phone
calls generally took place two days after the follow-up emails.
Final Result
NORC scheduled 14 interviews, and successfully completed 12 interviews. We achieved our goal for
testing the hospital-based instrument (conducted nine interviews) but fell short of achieving our goal to
test the independent-based instrument (conducted three interviews). Two scheduled respondents did not
dial-in to their interviews at the designated time and did not respond to requests to reschedule their
interviews. Table 3 displayed below provides the characteristics of the labs that participated.
Table 3: Final Laboratory Respondent Characteristics
Small Hospital
Large Hospital
Small
Independent
Large
Independent
Total
Multisite
4
2
0
Single site
1
2
0
2
1
8
4
Cognitive Interviewing Report | 6
Interviewing methods and protocols
NORC developed two scripted protocols to help guide the interviews, one for the hospital laboratory
questionnaire (Version A—Hospitals) and another for the independent laboratory questionnaire (Version
B—Independents). Each protocol consisted of an introduction to the survey and the purpose of the
cognitive interview, request for permission to record, and questions and probes to gain feedback on the
cover letter and each survey item.
The NORC interviewing team consisted of four NORC staff members, all of whom had previous
experience conducting cognitive interviews. Prior to the interview period, interviewers went through a
three-hour training session. During the training, each question and associated probes in the protocols was
carefully reviewed to ensure the interviewers understood the protocol and could administer it easily. The
training also provided an opportunity for interviewers to familiarize themselves with the questionnaires
and ask questions of the survey development team to obtain further content knowledge. Interviewers were
also instructed on how to open the conference line and begin/end recording via the Conference Plus
recording feature associated with the conference line number.
Upon receiving respondents’ notice of the intent to participate, NORC emailed a meeting invitation with
conference line information to the respondent and respective interviewer. In addition, if not previously
done, NORC mailed the respondent a packet including hardcopies of the cover letter and survey
instrument. Prior to the interview, NORC sent a meeting reminder with an electronic copy of the
questionnaire attached to ensure that the respondent has access to the material. All calls were recorded,
with the respondents’ permission. Following the interview, the recording files were downloaded into a
secure folder. Each respondent was assigned a random identification number to ensure anonymity. All
protocols were coded by the interviewers and compiled for analysis.
Results
This section provides an overview of the main issues discovered during the cognitive interviews. Results
are divided into three sections based on the three sources of error previously discussed in the conceptual
model: issues of comprehension about concepts and terms, issues of information retrieval, and issues of
the communication of responses. Each of these sections discuss key aspects of both questionnaires
(Version A and B); namely, cover letter, Section 1: General Information, Section 2: Current Systems and
Cognitive Interviewing Report | 7
Technical Architecture, Section 3: Methods Used to Send Lab Results and Volume of Exchange, Section
4: Lab Exchange Standards, and Section 5: Implementation of Guidelines and Regulations.
For each element, we start by summarizing main findings, then, we provide recommendations for
consideration to improve the survey instrument.
Issues of Comprehension
Cover Letter
Most respondents expressed that the wording and language used in the cover letter are clear. The
following statement seemed to work well: “This survey should be completed by the lab director, the lab
manager, or another individual who is most knowledgeable about your laboratory’s information exchange
capacities.” Respondents appreciated that the cover letter mentions assistance from other staff members
is expected throughout the survey process.
Respondents interpreted the term “most knowledgeable” person as the person with the most experience in
the lab, or as the lab employee with technical experience, in particular a staff member from the IT or LIS
departments. In cases where respondents attempted to complete the questionnaire without first reviewing
the survey, they sought help from others midway through. In cases where the respondent examined the
questionnaire first, the survey instrument was passed along to the person thought to be most
knowledgeable, in particular, an IT or LIS manager. There were some cases in which the person taking
ownership of the questionnaire had to contact people from other departments, such as billing, suggesting
that the term “most knowledgeable” can also include staff from the accounting department.
An identified source of confusion in the cover letter was to the abbreviation “NORC.” Since there is no
description of the acronym, it might potentially lead to confusion about the solicitor or the request.
Respondents also mentioned the cover letter should give an estimation of the time it takes to complete the
survey that is as accurate as possible. Some respondents also suggested that the letter should provide
direct examples to illustrate the value of the survey rather than the vague reference to collecting “crucial
state and national data.” Several respondents also suggested addressing respondents with correct titles
whenever possible (i.e., Dr. vs. Mr.).
Cognitive Interviewing Report | 8
Section 1: General Information
Overall, respondents did not have problems understanding questions in Section 1 in both Versions (A—
hospitals and B—regional and independent). There were several terms, nonetheless, that did have
comprehension issues. Some respondents indicated that it would be more useful if the list of definitions
provided as part of the questionnaire was placed at the beginning of the questionnaire. Respondents did
not notice the list until they finished the questionnaire.
For instance, responses to Q1 (Version A—Hospitals) suggest the term “non-academic affiliated
laboratory” may be unclear for some respondents because no definition is provided. Also, responses to Q5
(Version A—Hospitals)/Q6 (Version B—Independents) suggest that the phrase “perform any of the
following types of clinical pathology testing” could be confusing because, in some cases, labs take orders
for tests that they do not conduct in house. Responses collected about Q2 (Version B—Independents)
suggest that some confusion may arise because it is not clear the intent of this question. One respondent
defined reference lab as a lab that cannot perform all tests/lab works so they have to send out to a
different lab while another respondent understood it to mean a lab that provides both clinical and
anatomical testing for hospitals and other labs.
Section 2: Current Systems and Technical Architecture
Section 2 in both instruments did not have a large number of comprehension issues. Respondents found
the introductory statement helpful and successful in serving as a reminder that respondents should consult
with LIS\IT staff members to complete this section.
Responses to questions concerning whether a Lab information System (LIS)/Lab Information
Management Systems (LIMS) system supports multiple or single laboratories (Q7 [Version A—
Hospitals]/Q8 [Version B—Independents]) suggest that confusion arises when multiple laboratories are
located in a single facility. Some respondents suggested that asking for the physical location of the
facility (to convey the idea of multiple sites), or providing a definition of multiple sites, would help make
the meaning of the question more accurate.
The response category “commercial system” in Q9 (Version A—Hospitals)/Q10 (Version B—
Independents) also created confusion. Some respondents interpreted the term as “off-the-shelf” software
purchased from a vendor but managed by the lab, while other respondents understood it as software
developed and managed by a third party vendor.
Cognitive Interviewing Report | 9
Section 3: Methods Used to Send Lab Results and Volume of Exchange
Responses to survey items in Section 3 suggest there are terms not consistently understood across
respondents. In particular, responses to Q10 (Version A—Hospitals) suggest the expression “physicians
affiliated with your hospital” is subject to several interpretations including: “paid physicians,” “outreach
doctors,” “physicians approved,” “physicians on staff,” “physicians with a professional relationship with
the hospital,” or “physicians employed by the hospital.” In the instance of a small hospital lab,
“midwives” were also confused with physicians. Further, some respondents suggested that the term
“ordering provider” might be preferable to “ordering physicians.” In some states other providers, i.e.,
registered nurses, not just physicians, are credentialed to order lab results.
The interpretation of the term “physicians affiliated with your hospital” has a direct link to the numbers
reported in the volume questions. For instance, a respondent chose not to answer Q11 (Version A—
Hospitals) because the “physicians affiliated with the hospital” was interpreted as “doctors directly
employed by the hospital.” For this respondent, no doctors were directly employed by the hospital in
2011 so the respondent reported that there were no billable tests sent to “ordering physicians.”
The phrase “final billable tests” was subject to comprehension issues as well. In the context of Q10
(Version A—Hospitals), the phrase had several interpretations such as “tests that went to quality control,”
“billable tests that are ordered by physicians,” “test lab performed and for which they bill,” and “tests that
had received a result and could be billed.” Some respondents suggested alternative and presumably
clearer concepts could be “reportable tests,” “ordered tests,” “test results, not test,” “test results not bills”
“total final billable ordered tests,” “tests including both in-patient and out-patient testing,” “captive work
versus outreach work,” or even “CPT codes.” Respondents indicated that depending on the concept to be
measured (whether billable test, reportable tests or other kind of tests), different departments are likely to
be involved in reporting numbers.
Furthermore, responses to Q10 (Version A—Hospitals) suggest that labs may have different
interpretations of the two expressions combined in a single question stem (“ordering physicians affiliated
with your hospital” and “total final billable tests”). Some labs interpreted the question as the number of
bills sent to physicians, apparently overlooking or giving less importance to the word “tests.” In
particular, a respondent mentioned that they do not bill physicians employed by the hospital, as they
typically bill insurance companies or patients. It seems that for some respondents the term “affiliated
with your hospital” would not be clear until they reach Q14 (Version A—Hospitals). One of the
Cognitive Interviewing Report | 10
respondents included all billable test in Q10 (Version A—Hospitals) but had to amend the answer once
the respondent tried to complete answer Q14 (Version A—Hospitals).
Responses to Q11 (Version B—Independents) indicate similar problems with the term “final billable
tests” to those found in Q10 (Version A—Hospitals). The comprehension issue again lay in
interpretations of “final billable tests” or what constitutes “final billable tests.” In the case of Q11
(Version B—Independents) unlike hospitals, the expression “ordering physician” did not pose problems.
It is consistently understood as “the doctor who requested the work,” or the “the doctor whose signature
was on the requisition.”
Responses to Q11 (Version A—Hospitals) /Q12 (Version B—Independents) suggest that the terms “non
discrete data (free-text format),” “electronic discrete data (no free-text format),” and “electronic mixed
format (free text and non-free text format)” are consistently understood. Some responses suggest that in a
few instances these terms may present comprehension issues. In particular, “free-text and “no free-text”
appeared to be confusing. One respondent was not able to infer the meaning from the description
provided. It seems the respondent was not able to determine which files are searchable and which ones are
not. This respondent felt it would be useful to include a related reference as to whether the format is
searchable or not, as part of the question stem. This was not an issue for respondents with a technical
background. For specialized respondents (LIS/IT respondents), the “free text vs. non-free text” prompt
was straightforward.
The phrase “coded format such as LOINC or SNOMED CT” was consistently understood overall, but it
may have some comprehension issues for some respondents. In particular, responses to Q12 (Version
A—Hospitals)/Q13 (Version B—Independents) suggests that the word “coded format” may not be clear
to respondents without a technical background. In one instance, a respondent indicated that the term
“coded format” is confusing because they have their own interpretation of the term (using codes in their
Orchard Laboratory Information System), but they have the notion that there is a more nationally
recognized code format for results as well. The respondent further indicated that a better phrasing for Q12
(Version A) would be “Are you sending codified formats such as LOINC or SNOMED on a lab
interphase to an: HIS/ HIE / outside physician practice/EHR.” The interviewer’s motivation for providing
this suggestion is that the interviewer assumed that the intention of the question is to inquire about
preparation for Stage 2 Meaningful Use, where labs are expected to send LOINC codes in addition to
results. For this laboratory, the term “lab interphase” would be important to better understand the
intention of the question.
Cognitive Interviewing Report | 11
Section 4: Lab Exchange Standards
Respondents, generally, understood the questions in Section 4 and did not find them difficult to answer.
Respondents are familiar with whether their lab uses HL7, though some respondents were not familiar
with the different Versions of the standard. Similarly, respondents understood the meaning of LOINC and
were aware of their labs’ LOINC capabilities and experiences.
Section 5: Implementation of Guidelines and Regulations
Many respondents noted that they are unfamiliar with the “LRI guide” and although the definition was
helpful, they suggested it did not provide enough information to ensure their understanding. One
respondent suggested that a web link to additional information would be useful. Some respondents looked
up additional information (via Google) to better understand the question. The respondents’ lack of
understanding was reflected in their inability to confidently answer Q21 (Version A—Hospitals)/ Q18
(Version B—Independents). Even after they understood the definition of LRI guide, many respondents
said that they were unfamiliar with it.
Definitions and End Instructions
Most respondents did not use the definitions, though they noted that they were complete and would be
helpful. One respondent suggested that it would be helpful to refer to the definitions at the beginning of
the survey so that people are aware of them.
Respondents noted that the instructions to return the survey were clear and that they would prefer to mail
their completed surveys. Some respondents suggested that they would ultimately prefer an electronic
survey because it would be easier and increase the likelihood that IT staff would be willing to cooperate.
Recommendation
Add a note in the cover letter to the respondent to review the questionnaire first.
Describe NORC and its role in the cover letter.
Move definitions up front or to a separate brochure or instruction page.
Define multiple laboratories as those who share a common administrative structure.
Cognitive Interviewing Report | 12
For hospitals, change “physicians affiliated with the hospital” to “physicians with a professional
relationship to the hospital.”
Change “ordering physicians” to “ordering practitioners.”
Change final billable tests to “final test results” in all instances including questions that measures
the size of the lab overall. Further, include an accompanying definition to describe the intention
and meaning of the phrase.
Drop question about LRI guide.
Issues of Information Retrieval
Cover Letter
No issues related to information retrieval are noticed in the cover letter.
Section 1: General Information
Most respondents did not have problems recalling the information necessary to answer the questions in
Section 1. For Q3 (Version A—Hospitals)/ Q4 (Version B—Independents), respondents noted that this is
information most lab managers would not need to look up. A few respondents reported they consulted
other documents (e.g., the budget) to report the number of FTE in their labs. They noted that obtaining
this information was not demanding.
Section 2: Current Systems and Technical Architecture
Respondents did not have trouble recalling the information to answer the questions in this section. A few
respondents noted that it might be helpful to have the assistance of the LIS manager for these questions,
as suggested on the survey. One respondent suggested that IT staff could assist with these questions.
Section 3: Methods Used to Send Lab Results and Volume of Exchange
Responses to survey questions in this section indicate some respondents experienced difficulties
retrieving information to answer these questions.
Cognitive Interviewing Report | 13
Some respondents experienced problems retrieving relevant information regarding volume questions
because they have data for a ‘Fiscal Year’ and not a ‘Calendar Year.’ Some respondents would have
provided information nevertheless, without noting the reported reference period of 2011.
Responses to volume questions suggest in some questions respondents would consult records while in
other questions, they would just estimate numbers. Not all information is organized in a way that can be
easily retrieved. In particular, responses to Q10 (Version A—Hospitals)/Q11 (Version B—Independents)
suggest that respondents would attempt first to obtain this information from records, and if not available,
they would estimate. Responses to Q13 (Version A—Hospitals) and Q14 (Version B—Independents)
suggest that respondents will not have this information readily available in their records (i.e., organized
by standards such as LOINC and SNOMED CT). Consequently, they would estimate numbers for Q13
(Version A—Hospitals)/Q14 (Version B—Independents).
Responses to Q12 (Version A—Hospitals)/Q13 (Version B—Independents) also suggest that their ability
to retrieve information is also related to the way the flow of information works. In particular, respondents
seem to have problems providing information regarding option (c) and (d) when results are sent to an HIE
systems at the state level, given that the questions focuses on information sent to ordering physicians.
Options (a) and (b) appear less problematic. In one instance, a respondent from a hospital lab indicated
that they have a web portal through which physicians are able to access the hospital HIS system and it
does contain laboratory results. However, the hospital lab does not send LOINC codes to the core HIS
system. The lab is beginning to implement the process of sending results to the State HIE, and
maintaining a map of lab tests, results and LOINC codes. Further, the respondent mentioned that the
process for sending information to outside physicians is by first sending information to the State HIE,
which then sends the information on to outside physicians’ practices software. The interviewer expressed
that this process may be different across states.
Responses to Q11 (Version B—Independents) suggest that multisite independent labs would have
problems retrieving information (i.e., running a query) to report how many total final billable tests were
sent to “ordering physicians” as criterion. That is, it appears the volume records cannot be easily sorted,
grouped or filtered based on “ordering physician.” Independent labs are more used to retrieving
information based on “traditional” schemes, or references such as “patient billing,” “client billing,”
“hospital billing,” or “insurance billing.” As respondents tried to figure out the best way to retrieve
volume data during the interviews, they indicated that thorough searches would be needed through
various records or databases to extract data involving ordering physicians. Respondents’ answers suggest
that if they have not conducted exhaustive searches, they underestimate their volume transactions by
Cognitive Interviewing Report | 14
exchange category. Since all interviews were conducted over the telephonenot site visits,
interviewers were not able to determine the accuracy of the estimates.
Section 4: Lab Exchange Standards
Many respondents said that much of information for Section 4 is derived from their day-to-day activities
and responsibilities, so it is easy for them to recall. Respondents were able to recall the necessary
information to answer the questions in this section. Some respondents suggested that Q20 (Version A—
Hospitals)/ Q17 (Version B—Independents) —issues concerning adoption of LOINC codes— may
require assistance from other staff, such as the LIS manager. While most respondents knew whether or
not their lab used HL7, as asked in Q18 (Version A—Hospitals)/ Q15 (Version B—Independents), many
had to consult with IT staff to provide what HL7 version they use. Similarly, one respondent suggested
that they would reach out to the systems manager to answer Q19 (Version A—Hospitals).
Section 5: Implementation of Guidelines and Regulations
Respondents did not express trouble retrieving information for this section, though, as previously noted,
many expressed unfamiliarity with “LRI guides” in Q21 (Version A—Hospitals)/ Q18 (Version B—
Independents) which hindered their ability to answer the question. Once they understood the question,
they did not have trouble answering it. Respondents did not have trouble recalling information for Q22
(Version A—Hospitals)/ Q19 (Version B—Independents), or sub-question Q22a (Version A—Hospitals)/
Q19a (Version B—Independents). In one instance, the respondent suggested adding a link relevant for
this definition.
Recommendation
Provide a check box at the beginning of Section 3 that allows them to choose the reporting
reference period to be calendar year or fiscal year (if they differ).
Revise Q12 (Version A—Hospitals)/ Q13 (Version B—Independents) to eliminate options c and
d as they seem to be inconsistent with laboratory work flow.
Cognitive Interviewing Report | 15
Issues of Communication of Answers
Cover Letter
No issues related to communication of answers are noticed in the cover letter.
Section 1: General Information
Respondents noted little difficulty in articulating their information with the provided answer choices in
Section 1. However, respondents suggested that the answer options for Q1 (Version A /B) did not
completely describe their type of lab. For instance, one respondent noted that her lab is affiliated with a
university, but ultimately selected “Hospital or Health System” as the best answer to describe the lab.
Another respondent, using Version B—Independents, whose lab was owned by a national headquarters,
was unsure whether to select “commercial laboratory” or “other” to indicate this business model.
Responses to Q2 (Version A—Hospitals) / Q3 (Version B—Independents) suggest that in the category
“Other,” some respondents would report as job title “Chief Information Officer.”
Regarding Q3 (Version A—Hospitals)/ Q4 (Version B—Independents), some respondents noted that they
have full-time equivalents (i.e., two people working half time, which equates to one fulltime person) and
other employees, such as couriers and non-professional staff, who they excluded from their response. In
addition, respondents noted that current ranges for Q3 (Version A—Hospitals) are low. Some respondents
suggested that their number of FTEs is as high as 100, although one respondent noted that their lab had
over 600 FTE. A respondent from an independent lab noted that it was not clear whether to answer for the
local lab or for the national headquarters.
A few respondents also noted that there are other categories that can be included as answer choices to Q5
(Version A—Hospitals)/ Q6 (Version B—Independents). Suggested categories included:
immunohematology, flow cytometry, molecular, urinalaysis, coagulation, and bacteriology.
Section 2: Current Systems and Technical Architecture
There were few communication problems evident in Section 2. One minor issue that a respondent
reported for Q8 (Version A—Hospitals) was that the respondent’s lab uses a commercial version for some
types of tests and an in-house version for others, thus a response for ‘mixed’ LIS would be most accurate.
Another respondent uses different software packages depending on the type of tests.
Cognitive Interviewing Report | 16
Section 3: Methods Used to Send Lab Results and Volume of Exchange
Responses to Q10a (Version A—Hospitals) /Q11a (Version B—Independents) suggest that labs would be
able to communicate their answers using the current ranges in the questionnaire, overall. A respondent
from a multisite independent lab nonetheless, thought that some labs would not be conducting anywhere
near 500,000 tests. Thus, the respondent suggested having an intermediate range with a 250,000 cut-off.
Similarly, a respondent from a small hospital lab indicated that options in Q10a (Version A—Hospitals)
are very high. This respondent indicated that in the respondent’s state, only one lab is likely to complete
over 500,000 in a year. Further, the respondent suggested using more detailed response options; namely,
(1) 0-250,000, (2) 250,000-500,000, and (3) 500,000 or more.
Responses to Q13a (Version A—Hospitals)/Q14a (Version B—Independents) suggest that the range
provided should be larger to allow respondents to find the category that best fits their lab. No specific
suggestions were provided by respondents for these questions.
Section 4: Lab Exchange Standards
Respondents had few communication issues in Section 4. One respondent suggested not limiting
respondents to one response option, rather these “select-one-box” questions should be “check all that
apply” to accommodate various combinations of standards. In addition, one respondent reported that their
lab does not report LOINC codes, but they do enter LOINC codes. In this instance, the respondent is able
to provide information for Q20 (Version A—Hospitals), despite the instructions to skip to Q21 (Version
A—Hospitals). For Q20 (Version A—Hospitals), one respondent chose answer #2 (Our laboratory does
not have LOINC codes/mapping for some of our tests) as it best reflected the lab’s circumstances,
although the respondent noted that the lab does have LOINC codes for all of the tests, but not all of the
parts of the tests have codes. The respondent suggested changing or adding ‘only part of the tests’ to the
answer option.
In response to Q16 (Version B—Independents), one respondent suggested that we may want to capture
the percentage of lab results done in LOINC/SNOMED because, as the respondent noted, there are not
LOINC codes available for all existing lab tests.
Cognitive Interviewing Report | 17
Section 5: Implementation of Guidelines and Regulations
Respondents expressed few communication issues in Section 5. A few respondents noted that the
response to Q22 (Version A—Hospitals) is dependent on whether it is legal to share laboratory
information with patients, which is not a detail currently captured. One respondent also explained that the
lab results are posted to a patient portal that is supported by the hospital, but the lab itself does not support
or provide access to the portal. Regarding Q21 (Version A—Hospitals), one respondent mentioned they
are piloting the LRI guide, but have not fully implemented it.
Recommendation
Q 13a (Version A–Hospitals)\ Q14a (Version B—Independents), for labs that belong to a larger
entity, add an explanatory line that suggests that they should report for their own lab.
Expand questions that ask for numbers of billable tests (or results) to 4 categories with over
500,000 as the top of the range.
Allow for a check all the apply on Q 20(Version A–Hospitals)\ Q17 (Version B—Independents).
For Q22 (Version A–Hospitals), add a category that says “ it is not legal in my state to send lab
results directly to patients.”
For Q22 (Version A–Hospitals), refine question wording so it is clear that results sharing should
come directly from the lab.
Conclusion
Limitations
Cognitive interviewing, which does not include probability-based samples and does not fully reproduce
field conditions, has limitations as a true test of the functionality of the questionnaire. For this particular
instrument, there are three aspects of our design that limit the value of the data collected from this
exercise in cognitive interviewing. First, as noted, our sampling plan did not yield an adequate number of
small independent laboratories. Given that this is a study of the spread of technological change, this will
have inferential consequences for the comprehension of technical terms. Smaller, remote laboratories
will likely not have the same understanding of terms and their business models are likely to be
substantially different. Similarly, small sample sizes limit the variability we observed both in the
Cognitive Interviewing Report | 18
cognition of terms and the types of technological systems used to transmit lab results. Finally, we chose to
use a formalized interviewing process rather than the more informative “think aloud” interviewing, which
allows respond to provide substantial narrative substance to their responses. More structured cognitive
interviewing carries with it many benefits but shares with the main questionnaire the problem of limiting
our knowledge of the cognitive processes by which respondents arrive at their responses.
Taken together, these difficulties with cognitive interviewing and our particular design suggest that the
questionnaire revisions suggested by these interviews should be carefully reviewed for substantive and
technical fit to the subject matter and the analytic goals.
Summary Findings
Overall, the cognitive interviews suggest that there is still substantial variation in the interpretations of
key terms in this instrument. The most important source of confusion is the appropriate term to be used
for questions that seek to measure volume of test results. The primary difficulty appears to be identifying
the summary term to be used for the lab tests and whether it should be anchored in the patient, the billing
unit, or the single elements of any test panel. The term “affiliated physician” appears to pose smaller but
similar definitional problems for the hospital version of the questionnaire. Variations in interpretation
may reflect true variations in how physicians are attached to hospitals. Similarly, the administrative
structure of laboratories, which may include a group of affiliated but distinct laboratories or fully
centralized and administratively integrated collection of physical laboratory locations, cause both
cognitive issues and uncertainties in reporting. Finally, there are several instances in which the
questionnaire did not offer enough response categories or restricted response options and, thus, limited the
respondents’ ability to find an appropriate way to map their experience to the questionnaire.
Final Recommendations
Cover Letter
Add a note in the cover letter to the respondent to review the questionnaire first.
Describe NORC and its role in the cover letter.
Move definitions up front or to a separate brochure or instruction page.
Cognitive Interviewing Report | 19
Section 1: General Information
None
Section 2: Current Systems and Technical Architecture
For Q7 (Version A—Hospitals)/ Q8 (Version B—Independents) define multiple laboratories as
those who share a common administrative structure.
Section 3: Methods Used to Send Lab Results and Volume of Exchange
Provide a check box at the beginning of Section 3 that allows them to choose the reporting
reference period to be calendar year or fiscal year (if they differ).
For hospitals, change physicians affiliated with the hospital to “physicians with a professional
relationship to the hospital.”
Change “ordering physicians” to “ordering practitioners.”
Change final billable tests to “final test results” in all instances including questions that measures
the size of the lab overall. Further, include an accompanying definition to describe the intention
and meaning of the phrase.
Expand volume of test ranges to 4 categories with over 500,000 as the top of the range.
Revise questions Q12 (Version A—Hospitals)/ Q13 (Version B—Independents) to eliminate
options c and d as they seem to be inconsistent with laboratory work flow.
Section 4: Lab Exchange Standards
Allow for a check all the apply on Q20 (Version A—Hospitals)/ Q17 (Version B—Independents).
Section 5: Implementation of Guidelines and Regulations
For Q22 (Version A—Hospitals), add a category that says “ it is not legal in my state to send lab
results directly to patients.”
For Q22 (Version A—Hospitals), refine question wording so it is clear that results sharing should
come directly from the lab.
Cognitive Interviewing Report | 20
Drop question about LRI guide.
Q 13a (Version A–Hospitals)\ Q14a (Version B—Independents), for labs that belong to a larger
entity, add an explanatory line that suggests that they should report for their own lab.
Cognitive Interviewing Report | 21
Appendix A: Cognitive Interviewing Protocol – Version A
(Hospitals)
COGNITIVE INTERVIEWING PROTOCOL
2012 NATIONAL SURVEY ON HEALTH
INFORMATION EXCHANGE IN CLINICAL
LABORATORIES
Version A (Hospitals)
Lab ID
Appointment Date (MM/DD/YY)
_____/_____/_____/
Appointment Time
_____:_____ AM/PM
Interviewer name
Additional observations
Cognitive Interviewing Report | 22
Calling instructions
Interviewer: Make sure you are in a quiet room. Please call the designed telephone number:
1-866-688-8062
Passcode: 9071660
Moderator Code: 5022660
Make sure the respondent can hear you well on the phone.
Introduction
[Interviewer read]: Hello. My name is [YOUR NAME]. I work at NORC at the University of Chicago. We are very
grateful for your time and willingness to participate. NORC is developing a survey questionnaire that will be used in a
national survey of clinical laboratories funded by the U.S. Department of Health and Human Services (DHHS), Office
of the National Coordinator for Health Information Technology (ONC).
The survey will assess the electronic exchange of key laboratory information across the United States. To help
improve the self-administered questionnaire, NORC is conducting interviews to assess the ease of navigation of the
questionnaire; the clarity of the questionnaire items; the level of difficulty involved with collecting the requested
information; the amount of time it takes to complete the questionnaire; and to find out whether we have missed
anything in the various topics we are interested in.
We recently sent you a mailing with a letter describing the purpose of the survey and a copy of the hardcopy survey
questionnaire. Were you able to complete the survey questionnaire?
Interviewer: If yes, thank the respondent for completing the questionnaire ahead of time. If no, say that it is
ok if he/she did not complete it ahead of time, and let them know that they can fill it out as you go through the
questions. If the respondent does not have copy of the questionnaire, mention that there’s an electronic copy
attached to the meeting invitation they recently received.
[Interviewer read]: This conversation should take about an hour. Your participation involves answering questions
about your lab’s electronic exchange capabilities. While there is no direct benefit to you or your lab for participating in
this survey, your feedback will help make critical improvements to the survey questionnaire. All your feedback is very
important to us, and we want to make sure we don’t miss any comments.
We would like to audiotape our discussion today. I want to assure you, however, that all of your comments are strictly
confidential and will be used for research purposes only. Your participation in this study is entirely voluntary. You can
refuse to answer any question and you can stop the interview at any time. If at any point during the interview you wish
to withdraw as a participant, please just tell me and we will stop.
May I audiotape our conversation?
Interviewer: If R provides consent, please continue. If not, DO NOT audiotape but continue with the interview
stating that the conversation is not being audiotaped. Press *2 to begin recording. Once a recording is
initiated, all participants are informed by an automated message that the conference is being recorded. To
stop the recording, again press *2 or the recording will automatically end when the last participant hangs up.
Do you have any questions before I begin?
Interviewer: Answer any respondent questions or concerns.
Do you agree to participate in this study as I have described it?
[Interviewer: If yes, thank the respondent and proceed. If not, thank the respondent for his/her time and
finish.
If the respondent wants to learn more about the project, feel free to provide Dr. Prashila Dullabh’s contact
information: [email protected], or (301) 634-9418. Also, if the respondent asks for our IRB approval
number, feel free to share that our research was submitted for review and determined not to be human
subjects research by the NORC Institutional Review Board (IRB00000967), under its Federal Assurance
Cognitive Interviewing Report | 23
#FWA00000142. For questions about their rights as a study participant, they may call Kathleen Parks, the
NORC IRB Administrator, at toll-free number 866-309-0542.
Interview
[Interviewer read]: We will divide the interview into two sections today. The first part is about the materials we sent
to you in a mail package a few days ago, including the cover letter. Then, we’ll talk about the questionnaire.
Let’s talk about the cover letter first. Please turn to the cover letter.
Interviewer: Although we will use your notes and recordings to obtain detailed feedback for the write-up,
please use the coding scheme provided throughout this protocol to facilitate the summary of main findings.
Do not read the coding scheme to the respondent.
Probe CL1: This cover letter is very similar to the Version that we will use for the actual survey. Is there anything
unclear or confusing that you’ve noticed in our letter? If yes, which parts?
1
2
R pointed out confusing parts
R did not point out confusing parts
Probe CL2: Please tell me how you figured it out the best person to answer this questionnaire?
1
2
R experienced difficulties figuring out the best person
R did not experience difficulties figuring out the best person
Probe CL3:
What does the phrase “most knowledgeable person about your laboratory’s information exchange capacities” mean
to you?
1
R seemed certain about the meaning of the phrase
2
R seemed uncertain about the meaning of the phrase
Cognitive Interviewing Report | 24
Probe CL4:
Is there anything you would change or re-phrase in this cover letter to improve our chances of cooperation?
1
2
R provided a suggestion
R did not provide a suggestion
[Interviewer read]: Let’s now talk about the questionnaire.
As you may have noticed, this Version is a working document. Therefore, some instructions and sections are still
missing, and some numbers are yet to be added. However, we have developed questions and we want to find out
how well they will perform when asked in the actual survey.
Interviewer: If the respondent completed the questionnaire in advanced, please ask how long it took to
complete it. Otherwise, go to the next question.
[Interviewer read if R completed the questionnaire in advanced]: Before covering these specific
questions, though, let me ask you, how long did it take you to complete the questionnaire?
ProbeT |____|hr |____|mins
Thank you. Now, I’d like ask you some specific questions about our questionnaire. So, please turn to the
questionnaire.
Interviewer: The current protocol was created under the assumption that R completed the questionnaire in
advance. But if R couldn’t complete it ahead of time, be aware that you’ll have to rephrase a bit some of the
questions. For example, instead of asking “Did you receive assistance from other lab members to answer this
question” , you’ll have to ask: “Would you have to ask for assistance to other lab members to answer this
question”
Probe Q1 (Section 1, Q1):
Let’s talk about Question 1. What was your answer to this question? |____|
Please tell me how you figured out the best description for your lab’s affiliation or ownership?
Was it difficult to answer this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
2
|____| No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q2 (Section 1, Q2):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
Cognitive Interviewing Report | 25
[IF R ANSWERED “Other” ASK:] You indicated that your title is [FILL IN RESPONDENT’S ANSWER]. Is this a job
title that you think exist also in other laboratories, or it’s unique to your lab?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
2
|____| No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q3 (Section 1, Q3):
What was your answer to this question? |____|
How did you estimate the number of FTE in your lab?
How hard was it to figure out your answer? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
2
|____| No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q4 (Section 1, Q4):
What was your answer to this question? |____|
How hard was to identify the description of the type of testing your lab performs? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED “1” TO Q4, GO TO PROBE Q6]
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
1
2
2
|____| No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Cognitive Interviewing Report | 26
Probe Q5 (Section 1, Q5):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____| E.|____|
How hard was it to figure out your answer to each item? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes
If yes, What’s the position of the person who helped you to answer these questions?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
2
|____|No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S2 (Section 2, Intro):
I want to ask you about the paragraph we have at the beginning of section 2. In this part, we suggest working with the
LIS or LIMS manager.
How effective do you think this suggestion is? Why?
Did the suggestion lead you to ask for help to other staff members in your lab?
1
R asked for help to others based on the suggestion
2
R did not ask for help to others based on the suggestion
Probe Q6 (Section 2, Q6):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the expression “Laboratory Information System” mean to you?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED 2 or 3 TO Q6, GO TO PROBE Q10]
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
1
2
2
|____|No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q7 (Section 2, Q7):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the expression “multiple laboratories” mean to you?
Cognitive Interviewing Report | 27
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
2
|____|No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q8 (Section 2, Q8):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the tem “commercial system” mean to you?
Did you receive assistance from other lab members to answer this question? 1|____| Yes
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED 2,3 or 4 TO Q8, GO TO PROBE Q10]
1
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
question
Probe Q9 (Section 2, Q9):
What were your answers to these items?
1.|____|
6.|____|
2.|____|
7.|____|
3.|____|
8.|____|
4.|____|
9.|____|
5.|____|
10.|____|
1
2
2
|____|No
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
11.|____|
12.|____|
13.|____|
14.|____|
15.|____|
How hard was it to figure out your answer to each item? Why?
Which item was the most problematic? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Cognitive Interviewing Report | 28
Probe Q10 (Section 3, Q10):
What was your answer to this question? |_______________________________|
How hard was it to understand instructions to this question? Why?
What does the expression “total final billable tests” mean to you?
When your read the phrase “ordering physicians affiliated with your hospital” what came to you mind?
Was the term “calendar year 2011” confusing to you? Why?
[IF R ANSWERED “Don’t know” to Q10, GO TO PROBE Q10a; otherwise, ask the following questions]
Did you provide your answer based on records or did you estimate the number?
1
Records
2
Estimates
Was it hard to retrieve information for calendar year 2011 (from January 2011 to December 2011)? Why?
How did you figure out the total billable tests your lab sent to ordering physicians affiliated with your hospital?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Even though you didn’t select “Don’t Know”, how reasonable you think is the range provided in Q10a? Would you
suggest changing the current range to better capture information? How?
1
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
question
Probe Q10a (Section 3, Q10a):
What was your answer to this question? |____|
How did you figure out the answer to this question?
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
How reasonable is the range provided in the question? Would you suggest changing the current range to better
capture information? How?
Cognitive Interviewing Report | 29
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q11 (Section 3, Q11):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to each item? Why?
What does the expression “final billable tests” mean to you?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
In your opinion, is the expression “electronic non-discrete data” confusing? Why?
Is the expression “electronic discrete data” confusing? Why?
Is the expression “electronic mixed format” confusing? Why?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q12 (Section 3, Q12):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to each item? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
In your opinion, is the expression “coded format such as LOINC or SNOMED CT” confusing? Why?
Is there anything you would change in this question? What? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Cognitive Interviewing Report | 30
Probe Q13 (Section 3, Q13):
What was your answer to this question? |_______________________________|
How hard was it to understand the meaning of this question? Why?
[IF R ANSWERED “Don’t know” to Q13, GO TO PROBE Q13a; otherwise, ask the following questions]
Did you provide your answer based on records or did you estimate the number?
1
Records
2
Estimates
Was it hard to retrieve this information for calendar year 2011 (from January 2011 to December 2011)? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Even though you didn’t select “Don’t Know”, how reasonable you think is the range provided in Q13a? Would you
suggest changing the current range to better capture information? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q13a (Section 3, Q13a):
What was your answer to this question? |____|
How did you figure out the answer to this question?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
How reasonable is the range provided in the question? Would you suggest changing the current range to better
capture information? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Cognitive Interviewing Report | 31
Probe Q14 (Section 3, Q14):
As you surely noticed Question 14 through Question 17 are similar to previous questions. The focus of questions 14
to 17, however, is now on ordering physicians not affiliated with your hospital.
When you hear the phrase “ordering physicians not affiliated with your hospital”, what comes to you mind?
What was your answer to question Q14? |_______________________________|
How did you figure out the answer to this question?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Regarding Q14a, in your view, how reasonable is the range provided in the question? Would you suggest changing
the current range to better capture information? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q15 (Section 3, Q15):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
Earlier, we discussed a set of similar questions (in Q11). Q15 is now focused on physicians not affiliated to your
hospital. Did you experience any problems —other than those mentioned in Q11— with the items in Question 15?
Which problems?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q16 (Section 3, Q16):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
We discussed similar issues in Question 12. Did you experience problems other than those mentioned in Q12 with
the items in Question 16? Which problems?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
Cognitive Interviewing Report | 32
1
2
R seemed uncertain about the answers to
these survey questions
R seemed certain about the answers to these
survey questions
1
2
R expressed difficulties to answer these
survey questions
R did not express difficulties to answer these
survey questions
Probe Q17 (Section 3, Q17):
What was your answer to this question? |_______________________________|
As before, this question is similar to Q13. Did you experience problems other than those mentioned in Q13 with the
items in Question 17? Which problems?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S4 (Section 4, Intro):
At the beginning of section 4, we provide a brief definition of HL7.
Did you find this definition helpful or not for the understanding of subsequent questions? Why?
1
R indicated the definition was helpful
2
R indicated the definition was not helpful
Probe Q18 (Section 3, Q18):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q19 (Section 3, Q19):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Cognitive Interviewing Report | 33
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q20 (Section 3, Q20):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S5 (Section 5, Intro):
At the beginning of section 5, we provide a brief definition of the “LRI guide.”
Did the definition help you or not understand the meaning of subsequent questions? Why?
1
R indicated the definition was helpful
2
R indicated the definition was not helpful
Probe Q21 (Section 5, Q21):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q22 (Section 5, Q22):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Cognitive Interviewing Report | 34
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q22a (Section 5, Q22a):
Which were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to these questions? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
R seemed to experience difficulties to
understand the meaning of the survey items
1
2
R did not seem to experience difficulties to
understand the meaning of the survey items
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
items
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey items
Probe QD (Definitions):
You may have noticed that we provide a list of definitions in the back of the questionnaire (ordered alphabetically).
Did you use this list? 1|____| Yes 2|____|No
Did you find these definitions helpful or not? Why?
If there any other definition that you can think of that can be useful in this list?
1
R indicated the list of definitions was helpful
2
R indicated the list of definitions was not helpful
Probe EQ (Instructions to return the questionnaire by mail):
You may have noticed that we provided instructions to return the questionnaire upon completion. I’d like to have your
feedback about the instructions we provided in the last page. Are these instructions clear enough in your
perspective?
1
R indicated the instructions are clear enough
2
R indicated the instructions are not clear enough
End of interview
Interviewer: Thank the respondent for his/her time and his/her help. Ask if there is anything else R would like
to add. To stop the recording, again press *2 or the recording will automatically end when the last participant
hangs up.
These are all the questions I had for you. We sincerely appreciate your time and help. Your answers are very helpful
to improve our questionnaire. Is there anything else you would like to add?
Cognitive Interviewing Report | 35
Thank you again, and have a good day.
Cognitive Interviewing Report | 36
Appendix B: Cognitive Interviewing Protocol – Version B
(Non-hospitals: regional & independent)
COGNITIVE INTERVIEWING PROTOCOL
2012 NATIONAL SURVEY ON HEALTH
INFORMATION EXCHANGE IN CLINICAL
LABORATORIES
Version B (Non-hospitals:
regional & independent)
Lab ID
Appointment Date (MM/DD/YY)
_____/_____/_____/
Appointment Time
_____:_____ AM/PM
Interviewer name
Additional observations
Cognitive Interviewing Report | 37
Calling instructions
Interviewer: Make sure you are in a quiet room. Please call the designed telephone number:
1-866-688-8062
Passcode: 9071660
Moderator Code: 5022660
Make sure the respondent can hear you well on the phone.
Introduction
[Interviewer read]: Hello. My name is [YOUR NAME]. I work at NORC at the University of Chicago. We are very
grateful for your time and willingness to participate. NORC is developing a survey questionnaire that will be used in a
national survey of clinical laboratories funded by the U.S. Department of Health and Human Services (DHHS), Office
of the National Coordinator for Health Information Technology (ONC).
The survey will assess the electronic exchange of key laboratory information across the United States. To help
improve the self-administered questionnaire, NORC is conducting interviews to assess the ease of navigation of the
questionnaire; the clarity of the questionnaire items; the level of difficulty involved with collecting the requested
information; the amount of time it takes to complete the questionnaire; and to find out whether we have missed
anything in the various topics we are interested in.
We recently sent you a mailing with a letter describing the purpose of the survey and a copy of the hardcopy survey
questionnaire. Were you able to complete the survey questionnaire?
Interviewer: If yes, thank the respondent for completing the questionnaire ahead of time. If no, say that it is
ok if he/she did not complete it ahead of time, and let them know that they can fill it out as you go through the
questions. If the respondent does not have copy of the questionnaire, mention that there’s an electronic copy
attached to the meeting invitation they recently received.
[Interviewer read]: This conversation should take about an hour. Your participation involves answering questions
about your lab’s electronic exchange capabilities. While there is no direct benefit to you or your lab for participating in
this survey, your feedback will help make critical improvements to the survey questionnaire. All your feedback is very
important to us, and we want to make sure we don’t miss any comments.
We would like to audiotape our discussion today. I want to assure you, however, that all of your comments are strictly
confidential and will be used for research purposes only. Your participation in this study is entirely voluntary. You can
refuse to answer any question and you can stop the interview at any time. If at any point during the interview you wish
to withdraw as a participant, please just tell me and we will stop.
May I audiotape our conversation?
Interviewer: If R provides consent, please continue. If not, DO NOT audiotape but continue with the interview
stating that the conversation is not being audiotaped. Press *2 to begin recording. Once a recording is
initiated, all participants are informed by an automated message that the conference is being recorded. To
stop the recording, again press *2 or the recording will automatically end when the last participant hangs up.
Do you have any questions before I begin?
Interviewer: Answer any respondent questions or concerns.
Do you agree to participate in this study as I have described it?
[Interviewer: If yes, thank the respondent and proceed. If not, thank the respondent for his/her time and
finish.
If the respondent wants to learn more about the project, feel free to provide Dr. Prashila Dullabh’s contact
information: [email protected], or (301) 634-9418. Also, if the respondent asks for our IRB approval
number, feel free to share that our research was submitted for review and determined not to be human
subjects research by the NORC Institutional Review Board (IRB00000967), under its Federal Assurance
Cognitive Interviewing Report | 38
#FWA00000142. For questions about their rights as a study participant, they may call Kathleen Parks, the
NORC IRB Administrator, at toll-free number 866-309-0542.
Interview
[Interviewer read]: We will divide the interview into two sections today. The first part is about the materials we sent
to you in a mail package a few days ago, including the cover letter. Then, we’ll talk about the questionnaire.
Let’s talk about the cover letter first. Please turn to the cover letter.
Interviewer: Although we will use your notes and recordings to obtain detailed feedback for the write-up,
please use the coding scheme provided throughout this protocol to facilitate the summary of main findings.
Do not read the coding scheme to the respondent.
Probe CL1: This cover letter is very similar to the Version that we will use for the actual survey. Is there anything
unclear or confusing that you’ve noticed in our letter? If yes, which parts?
1
2
R pointed out confusing parts
R did not point out confusing parts
Probe CL2: Please tell me how you figured it out the best person to answer this questionnaire?
1
2
R experienced difficulties figuring out the best person
R did not experience difficulties figuring out the best person
Probe CL3:
What does the phrase “most knowledgeable person about your laboratory’s information exchange capacities” mean
to you?
1
R seemed certain about the meaning of the phrase
2
R seemed uncertain about the meaning of the phrase
Probe CL4:
Is there anything you would change or re-phrase in this cover letter to improve our chances of cooperation?
1
2
R provided a suggestion
R did not provide a suggestion
Cognitive Interviewing Report | 39
[Interviewer read]: Let’s now talk about the questionnaire.
As you may have noticed, this Version is a working document. Therefore, some instructions and sections are still
missing, and some numbers are yet to be added. However, we have developed questions and we want to find out
how well they will perform when asked in the actual survey.
Interviewer: If the respondent completed the questionnaire in advanced, please ask how long it took to
complete it. Otherwise, go to the next question.
[Interviewer read if R completed the questionnaire in advanced]: Before covering these specific
questions, though, let me ask you, how long did it take you to complete the questionnaire?
ProbeT
|____|hr |____|mins
Thank you. Now, I’d like ask you some specific questions about our questionnaire. So, please turn to the
questionnaire.
Interviewer: The current protocol was created under the assumption that R completed the questionnaire in
advance. But if R couldn’t complete it ahead of time, be aware that you’ll have to rephrase a bit some of the
questions. For example, instead of asking “Did you receive assistance from other lab members to answer this
question” , you’ll have to ask: “Would you have to ask for assistance to other lab members to answer this
question”
Probe Q1 (Section 1, Q1):
Let’s talk about Question 1. What was your answer to this question? |____|
Please tell me how you figured out the best description for your lab’s affiliation or ownership?
Was it difficult to answer this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q2 (Section 1, Q1):
Let’s talk about Question 1. What was your answer to this question? |____|
Please tell me how you figured out the best description for your lab’s affiliation or ownership?
Was it difficult to answer this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
Cognitive Interviewing Report | 40
question
survey question
Probe Q3 (Section 1, Q3):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
[IF R ANSWERED “Other” ASK:] You indicated that your title is [FILL IN RESPONDENT’S ANSWER]. Is this a job
title that you think exist also in other laboratories, or it’s unique to your lab?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q4 (Section 1, Q3):
What was your answer to this question? |____|
How did you estimate the number of FTE in your lab?
How hard was it to figure out your answer? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q5 (Section 1, Q5):
What was your answer to this question? |____|
How hard was to identify the description of the type of testing your lab performs? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED “1” TO Q5, GO TO PROBE Q7]
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
1
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
Cognitive Interviewing Report | 41
question
survey question
Probe Q6 (Section 1, Q6):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____| E.|____|
How hard was it to figure out your answer to each item? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S2 (Section 2, Intro):
I want to ask you about the paragraph we have at the beginning of section 2. In this part, we suggest working with the
LIS or LIMS manager.
How effective do you think this suggestion is? Why?
Did the suggestion lead you to ask for help to other staff members in your lab?
1
R asked for help to others based on the suggestion
2
R did not ask for help to others based on the suggestion
Probe Q7 (Section 2, Q7):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the expression “Laboratory Information System” mean to you?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED 2 or 3 TO Q7, GO TO PROBE Q11]
R seemed to experience difficulties to
understand the meaning of the survey
question
2
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q8 (Section 2, Q8):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the expression “multiple laboratories” mean to you?
Cognitive Interviewing Report | 42
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q9 (Section 2, Q9):
What was your answer to this question? |____|
How hard was it to figure out your answer? Why?
What does the tem “commercial system” mean to you?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
[IF R ANSWERED 2,3 or 4 TO Q8, GO TO PROBE Q11]
1
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
2
understand the meaning of the survey
question
Probe Q10 (Section 2, Q10):
What were your answers to these items?
1.|____|
6.|____|
2.|____|
7.|____|
3.|____|
8.|____|
4.|____|
9.|____|
5.|____|
10.|____|
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
11.|____|
12.|____|
13.|____|
14.|____|
15.|____|
How hard was it to figure out your answer to each item? Why?
Which item was the most problematic? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q11 (Section 3, Q11):
What was your answer to this question? |_______________________________|
How hard was it to understand instructions to this question? Why?
Cognitive Interviewing Report | 43
What does the expression “total final billable tests” mean to you?
When your read the phrase “ordering physicians” what came to you mind?
Was the term “calendar year 2011” confusing to you? Why?
[IF R ANSWERED “Don’t know” to Q11, GO TO PROBE Q11a; otherwise, ask the following questions]
Did you provide your answer based on records or did you estimate the number?
1
Records
2
Estimates
Was it hard to retrieve information for calendar year 2011 (from January 2011 to December 2011)? Why?
How did you figure out the total billable tests your lab sent to ordering physicians?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
Even though you didn’t select “Don’t Know”, how reasonable you think is the range provided in Q11a? Would you
suggest changing the current range to better capture information? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q11a (Section 3, Q11a):
What was your answer to this question? |____|
How did you figure out the answer to this question?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
How reasonable is the range provided in the question? Would you suggest changing the current range to better
capture information? How?
Cognitive Interviewing Report | 44
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q12 (Section 3, Q12):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to each item? Why?
What does the expression “final billable tests” mean to you?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
In your opinion, is the expression “electronic non-discrete data” confusing? Why?
Is the expression “electronic discrete data” confusing? Why?
Is the expression “electronic mixed format” confusing? Why?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q13 (Section 3, Q13):
What were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to each item? Why?
Did you receive assistance from other lab members to answer these questions? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer these questions?
In your opinion, is the expression “coded format such as LOINC or SNOMED CT” confusing? Why?
Is there anything you would change in this question? What? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q14 (Section 3, Q14):
What was your answer to this question? |_______________________________|
Cognitive Interviewing Report | 45
How hard was it to understand the meaning of this question? Why?
[IF R ANSWERED “Don’t know” to Q13, GO TO PROBE Q13a; otherwise, ask the following questions]
Did you provide your answer based on records or did you estimate the number?
1
Records
2
Estimates
Was it hard to retrieve this information for calendar year 2011 (from January 2011 to December 2011)? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No No
If yes, What’s the position of the person who helped you to answer this question?
Even though you didn’t select “Don’t Know”, how reasonable you think is the range provided in Q14a? Would you
suggest changing the current range to better capture information? How?
1
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
2
understand the meaning of the survey
question
Probe Q14a (Section 3, Q14a):
What was your answer to this question? |____|
How did you figure out the answer to this question?
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
How reasonable is the range provided in the question? Would you suggest changing the current range to better
capture information? How?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S4 (Section 4, Intro):
At the beginning of section 4, we provide a brief definition of HL7.
Did you find this definition helpful or not for the understanding of subsequent questions? Why?
1
R indicated the definition was helpful
2
R indicated the definition was not helpful
Probe Q15 (Section 3, Q15):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Cognitive Interviewing Report | 46
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q16 (Section 3, Q16):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q17 (Section 3, Q17):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe S5 (Section 5, Intro):
At the beginning of section 5, we provide a brief definition of the “LRI guide.”
Did the definition help you or not understand the meaning of subsequent questions? Why?
1
R indicated the definition was helpful
2
R indicated the definition was not helpful
Cognitive Interviewing Report | 47
Probe Q18 (Section 5, Q18):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q19 (Section 5, Q19):
What was your answer to this question? |____|
How hard was it to figure out your answer to this question? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
2
R seemed to experience difficulties to
understand the meaning of the survey
question
R did not seem to experience difficulties to
understand the meaning of the survey
question
1
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
question
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey question
Probe Q19a (Section 5, Q19a):
Which were your answers to these items?
A.|____| B.|____| C.|____| D.|____|
How hard was it to figure out your answer to these questions? Why?
Did you receive assistance from other lab members to answer this question? 1|____| Yes 2|____|No
If yes, What’s the position of the person who helped you to answer this question?
1
R seemed to experience difficulties to
understand the meaning of the survey items
1
2
R did not seem to experience difficulties to
understand the meaning of the survey items
2
R seemed to experience difficulties to retrieve
relevant information to answer the survey
items
R did not seem to experience difficulties to
retrieve relevant information to answer the
survey items
Probe QD (Definitions):
You may have noticed that we provide a list of definitions in the back of the questionnaire (ordered alphabetically).
Did you use this list? 1|____| Yes 2|____|No
Did you find these definitions helpful or not? Why?
If there any other definition that you can think of that can be useful in this list?
Cognitive Interviewing Report | 48
1
R indicated the list of definitions was helpful
2
R indicated the list of definitions was not helpful
Probe EQ (Instructions to return the questionnaire by mail):
You may have noticed that we provided instructions to return the questionnaire upon completion. I’d like to have your
feedback about the instructions we provided in the last page. Are these instructions clear enough in your
perspective?
1
R indicated the instructions are clear enough
2
R indicated the instructions are not clear enough
End of interview
Interviewer: Thank the respondent for his/her time and his/her help. Ask if there is anything else R would like
to add. To stop the recording, again press *2 or the recording will automatically end when the last participant
hangs up.
These are all the questions I had for you. We sincerely appreciate your time and help. Your answers are very helpful
to improve our questionnaire. Is there anything else you would like to add?
Thank you again, and have a good day.
Cognitive Interviewing Report | 49
Appendix C: Final Tabulation of Coded Responses
Independent Hospital
survey Q#
survey Q#
CL1
CL1
CL2
CL2
CL3
CL3
CL4
CL4
Q1
Q2
Q3
Q1
n/a
Q2
Q4
Q3
Q5
Q4
Q6
Q5
S2
S2
Question
This cover letter is very similar to the Version that
we will use for the actual survey. Is there anything
unclear or confusing that you’ve noticed in our
letter? If yes, which parts?
Please tell me how you figured it out the best person
to answer this questionnaire?
What does the phrase “most knowledgeable person
about your laboratory’s information exchange
capacities” mean to you?
Is there anything you would change or re-phrase in
this cover letter to improve our chances of
cooperation?
Which option below best describes your
laboratory’s organizational affiliation or ownership?
Is your laboratory a reference laboratory?
What is your job title?
How many full time employees (FTEs) currently
work in your laboratory at this site?
Which option below best describes the type of
testing your laboratory performs?
Does your laboratory perform any of the following
types of clinical pathology testing?
In this part, we suggest working with the LIS or
LIMS manager.
How effective do you think this suggestion is? Why?
Did the suggestion lead you to ask for help to other
staff members in your lab?
Number
reporting
comprehension Total
issues
(N=12)
Number
reporting
retrieval Total
issues
(N=12)
10
12
n/a
n/a
10
11
n/a
n/a
3
12
n/a
n/a
9
12
n/a
n/a
10
2
11
12
2
12
9
2
10
11
2
11
10
12
11
11
11
11
9
10
8
11
10
10
8
10
n/a
n/a
Cognitive Interviewing Report | 50
Independent Hospital
survey Q#
survey Q#
Q7
Q6
Q8
Q7
Q9
Q8
Q10
Q9
Q10
(question
specific to
affiliated
ordering
physicians)
Q10a
(question
specific to
affiliated
ordering
physicians)
Q11
(question
specific to
affiliated
ordering
physicians)
Q11
Q11a
Q12
Number
reporting
comprehension Total
issues
(N=12)
Number
reporting
retrieval Total
issues
(N=12)
10
12
11
11
8
11
9
10
11
11
9
10
7
10
5
9
Approximately, how many total final billable
tests did your lab send to ordering physicians
during calendar year 2011? Please consult your
records to answer this question. If records are not
available, please provide your best estimate.
6
11
5
10
Would the number of total final billable tests
be 499,999 or less, or 500,000 or more?
1
1
0
1
Did your laboratory send final billable tests to
ordering physicians during calendar year 2011
using any of the following methods?
6
10
6
9
Question
Does your laboratory currently use a Laboratory
Information System (LIS/LIMS)?
Does your Laboratory Information System
(LIS/LIMS) support multiple laboratories or only
one laboratory?
What type of Laboratory Information System (LIS/
LIMS) does your laboratory currently use?
Which vendor(s) supplies your Laboratory
Information System(s) (LIS/LIMS)?
Cognitive Interviewing Report | 51
Independent Hospital
survey Q#
survey Q#
Q12
(question
specific to
affiliated
ordering
Q13
physicians)
Q14
Q14a
Question
Number
reporting
comprehension Total
issues
(N=12)
Did your laboratory send final billable tests in
coded format such as LOINC or SNOMED CT
standard to ordering physicians during calendar
year 2011 using any of the following methods?
6
Approximately, how many final billable tests coded
Q13
based on standards such as LOINC or SNOMED CT
(question
did your lab send during calendar year 2011 to
specific to ordering physicians using electronic health records
affiliated
(EHR)? Please consult your
ordering
records to answer this question. If records are not
physicians) available, please provide your best estimate.
5
Q13a
(question
specific to
affiliated
ordering
Would the number of final billable tests be
physicians) 124,999 or less, or 125,000 or more?
2
Approximately, how many total final billable tests
did your lab send to ordering physicians not
affiliated with your hospital during calendar year
2011?
n/a
Q14 &
Q14a
n/a
Q15
Would the number of total final billable tests
be 499,999 or less, or 500,000 or more?
Did your laboratory send final billable tests to
ordering physicians not affiliated with your hospital
during calendar year 2011 using any of the
Number
reporting
retrieval Total
issues
(N=12)
11
7
10
8
4
8
2
1
2
4
6
4
6
5
5
5
5
Cognitive Interviewing Report | 52
Independent Hospital
survey Q#
survey Q#
n/a
Q16
n/a
Q17/Q17a
S4
S4
Q15
Q18
Q16
Q19
Q17
Q20
S5
S5
Question
following methods?
Did your laboratory send final billable tests in coded
format such as LOINC or SNOMED CT standard to
ordering physicians not affiliated with your hospital
during calendar year 2011 using any of the
following methods?
Approximately, how many final billable tests coded
based on standards such as LOINC or SNOMED CT
did your lab send during calendar year 2011 to
ordering physicians not affiliated
with your hospital using electronic health records
(EHR)?/ Would the number of final billable tests be
124,999 or less, or 125,000 or more?
At the beginning of section 4, we provide a brief
definition of HL7. Did you find this definition
helpful or not for the understanding of subsequent
questions? Why?
Which of the following messaging standards are
currently in use by your laboratory for result
reporting?
Which vocabulary standard for lab result reporting
is currently in use by your laboratory?
Which one of the following issues concerning
adoption of LOINC codes for sending results best
describes the current situation in your laboratory?
At the beginning of section 5, we provide a brief
definition of the “LRI guide.” Did the definition
help you or not understand the meaning of
Number
reporting
comprehension Total
issues
(N=12)
Number
reporting
retrieval Total
issues
(N=12)
5
5
5
5
1
1
1
1
2
9
n/a
n/a
10
11
7
11
9
11
9
11
5
7
6
7
4
10
n/a
n/a
Cognitive Interviewing Report | 53
Independent Hospital
survey Q#
survey Q#
Q18
Q21
Q19
Q22
Q19a
Q22a
QD
QD
ED
ED
Question
subsequent questions? Why?
Has your laboratory implemented the LRI guide
for lab result content and format?
Does you laboratory allow patients, or their
representatives direct access to their laboratory
results?
If YES, please indicate which of the following
methods of delivery your laboratory anticipates
using to deliver results directly to patients or
patients’ representatives.
You may have noticed that we provide a list of
definitions in the back of the questionnaire (ordered
alphabetically). Did you use this list? Did you find
these definitions helpful or not? Why?
You may have noticed that we provided instructions
to return the questionnaire upon completion. I’d like
to have your feedback about the instructions we
provided in the last page. Are these instructions
clear enough in your perspective?
Number
reporting
comprehension Total
issues
(N=12)
Number
reporting
retrieval Total
issues
(N=12)
9
10
6
10
8
9
8
9
3
4
3
4
2
9
n/a
n/a
1
9
n/a
n/a
Cognitive Interviewing Report | 54
| File Type | application/pdf |
| File Title | Microsoft Word - Attachment C - Cognitive Interviewing Report |
| Author | colicchia-michael |
| File Modified | 2012-10-03 |
| File Created | 2012-10-03 |