Appointment Wait Time Guidance

CMS-10803 - AWT-SSS-Tech-Guide-QHP-FFE-508.pdf

Essential Community Provider-Network Adequacy (ECP/NA) Data Collection to Support QHP Certification (CMS-10803)

Appointment Wait Time Guidance

OMB: 0938-1415

Document [pdf]

Download: pdf | pdf

Appointment Wait Time Secret Shopper
Survey Technical Guidance for Qualified
Health Plan (QHP) Issuers in the Federallyfacilitated Exchanges (FFEs)
September 2024

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Document Change Log
Background
•

September 18, 2024

Telehealth definition has been added for more clarity.

Section A. Identifying the Population to Be Surveyed
• Information added to denote data elements included in the provider
population file.
• Information added to note that more than one phone number per provider
may be used for no more than three contact attempts.
• Information added to the definition of a nonresponsive provider. Third parties
are to spend no more than a maximum of 20 minutes of interactive voice
response (i.e., phone tree navigation).
Section B. Sample Selection
• Information added to clarify that providers should be identified by a unique
National Provider Identifier in the provider population file.
• Information added to clarify that pediatric and adult primary care providers
should be combined to create the oversample.
Section C. Survey Protocol Development
• Information added to indicate that secret shoppers may present as an
associate or a representative of a new patient, such as a family member.
Section E. Collecting and Documenting Survey Results
• New disposition code added for use when secret shopper is asked to consult
an online scheduling system or a patient portal to obtain appointment
availability.
• Information added indicating that more than one contact attempt could be
made for the same provider in the same business day if the contact attempts
occurred in different parts of the business day.
Section G. Submitting Survey Results Data to CMS
• Information added noting that all QHP issuers must submit appointment wait
time surveys to CMS.
• The contact email address for data submission has been updated to
[email protected].
Appendix A. Required Secret Shopper Survey Results
• Information added to the file format requirements of data submissions from
issuers.
• Information added to indicate a reason that a phone number was invalid in
results reported (disconnected, not associated with surveyed provider).
Appendix B. Valid Survey Disposition Codes
• Information added in “instructions for reporting” column for disposition codes
“B”,”C”, “K”, and “T” regarding how to format additional information in the
“General Notes” field of results reported
Appendix D. General Tips and Script Suggestions
•

This Appendix added to provide script workarounds and general tips with
best practices for conducting secret shopper calls.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Table of Contents
Technical Assistance and Contact Information ..................................................................................... iv
Background ............................................................................................................................................. 1
A. Identifying the Population to Be Surveyed ........................................................................................ 2
B. Sample Selection ................................................................................................................................ 6
C. Survey Protocol Development ......................................................................................................... 10
D. Survey Administration ..................................................................................................................... 10
E. Collecting and Documenting Survey Results Data .......................................................................... 11
F. Calculating Compliance Rate and Other Figures Based on Results................................................. 13
G. Submitting Survey Results Data to CMS ........................................................................................ 16
Appendix A. Required Secret Shopper Survey Results ........................................................................ 17
Appendix B. Valid Survey Disposition Codes ..................................................................................... 23
Appendix C. Key Terms ....................................................................................................................... 39
Appendix D. General Tips and Script Suggestions .............................................................................. 41

Tables

Table 1. FFE-established Appointment Wait Times ................................................................................ 1
Table 2. Primary Care Provider Specialty Types ..................................................................................... 3
Table 3. Behavioral Health Provider Specialty Types.............................................................................. 4
Table 4. Minimum Survey Sample Sizes ................................................................................................. 7
Table 5. Example Appointment Wait Time Calculation ........................................................................ 13

Figures

Figure 1. Overview of Survey Sampling Process ..................................................................................... 9

iii

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Technical Assistance and Contact Information
For questions regarding policies and procedures contained in the Appointment Wait Time Secret
Shopper Survey Technical Guidance, please contact the Marketplace Service Desk (MSD) via email at
[email protected] or via phone at 1-855-CMS-1515 (1-855-267-1515).
For questions regarding how Qualified Health Plan (QHP) issuers must submit survey data, please
email [email protected].

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Background
In the 2025 Final Letter to Issuers in the FFEs, published April 10, 2024, the Centers for Medicare &
Medicaid Services (CMS) stated that, beginning January 1, 2025, Qualified Health Plan (QHP) issuers,
including stand-alone dental plan (SADP) issuers, in the Federally-facilitated Exchanges (FFEs) are
required to meet appointment wait time standards established by the FFEs, reflected in Table 1. For the
2025 plan year, QHP issuers, including SADP issuers, must ensure that enrollees seeking an
appointment are able to schedule an appointment within the time frames below at least 90% of the
time. CMS is particularly concerned with the ability of new patients to schedule appointments with innetwork providers; more than half of enrollees on the FFEs newly enroll in QHPs or change their
enrollment to a new QHP each year, and these enrollees may need to seek care as a patient who is new
to a provider.
Table 1. FFE-established Appointment Wait Times
Provider specialty type

Appointments must be available within

Behavioral Health

10 business days

Primary Care (Routine)

15 business days

Specialty Care (Non-urgent)

30 business days

CMS requires medical QHP issuers offering QHPs in the FFEs to contract with a third-party entity to
administer secret shopper surveys to meet the appointment wait time standards. The third-party entity
that conducts the surveys must be a separate and distinct entity from the medical QHP issuer. For
example, the third-party entity and the issuer cannot be affiliated companies, and they cannot be
subsidiaries of the same parent company. To limit the burden on QHP issuers, CMS intends to require
that secret shopper surveys be conducted for a QHP issuer’s primary care (routine) and behavioral
health providers for plan year 2025. CMS expects to require secret shopper surveys to be administered
with respect to specialty care (non-urgent) providers in future plan years.
As SADP issuers would generally contract with specialty care (non-urgent) providers, SADP issuers
would not be required to contract with a third-party entity to conduct secret shopper surveys for the
2025 plan year.
To demonstrate compliance with these standards, medical QHP issuers must contract with a third-party
entity to conduct a secret shopper survey, with surveying beginning on or shortly after January 1 and
completed by May 31 of each plan year and report the results of the surveys to CMS as part of QHP
issuer compliance and monitoring activities. The third-party entity must conduct secret shopper
surveys while presenting as a new patient (i.e., a patient attending their first-ever clinical encounter
with a practitioner at the location being surveyed). CMS may require medical QHP issuers to provide
CMS with documentation underlying the results of those surveys, for CMS’s review. Medical QHP
issuers must retain relevant documentation related to the surveys in accordance with the broad record
retention policies set forth in 45 CFR 156.705.
A QHP issuer’s third-party entity would be required to administer secret shopper surveys to a survey
pool, provided to issuers by CMS, that includes a statistically valid representation of providers across
the QHP’s network that are accessible to consumers within the requisite time and distance standards in

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

the service area. The third-party entity must identify a statistically valid, minimum sample size for
each provider type.
Issuers that fail to have a third-party entity conduct the secret shopper survey, fail to report the results, or
report results that do not reflect compliance with the appointment wait time standards (i.e., by reporting
results that do not reflect that enrollees seeking an appointment are able to schedule an appointment
within the time frames above at least 90% of the time), based on only those providers that count toward
the issuer’s satisfaction of the time and distance requirements under § 156.230(a)(2)(i)(A), would need to
add more contracted providers to the network to achieve alignment with the standards, and/or would
need to come into alignment with the standards by contracting with a third-party entity to conduct the
surveys and reporting compliant results as appropriate.
In order to assess compliance with the appointment wait time standards, the third-party entities should
collect information from provider offices on the availability of both in-person and telehealth
appointments. The calculation of the 90% compliance rate for secret shopper surveys of appointment
wait times will be based on whichever appointment, in-person or telehealth, has the shortest wait time.
For the purpose of the appointment wait time standards, CMS defines telehealth as “professional
consultations, office visits, and office psychiatry services through brief communication technology-based
service/virtual check-in, remote evaluation of pre-recorded patient information, and inter-professional
internet consultation.” This aligns with how CMS defines telehealth broadly for network adequacy in the
2023 Final Letter to Issuers in the Federally-facilitated Exchanges (Chapter 2, section 3, iii. Telehealth
Services). We acknowledge that telehealth is an important option for some patients to access care, and
the data collected in this first year of reporting for appointment wait time compliance will be used to
inform decisions regarding how we will measure compliance in future years.
This document provides issuers and their contracted third-party entities with additional technical
guidance and secret shopper survey implementation requirements.

A. Identifying the Population to Be Surveyed
Each fall, CMS will provide issuers with a file, referred to subsequently as the “provider population
file,” that contains the population or “universe” of providers from which QHP issuers’ third-party
entities must draw a statistically valid, randomized oversample. The provider population file will be
derived by CMS from provider data submitted by issuers on the “Network Adequacy Provider” tab in
the Network Adequacy template submitted during the annual QHP certification process. The provider
population file will include providers, from the list submitted by the issuer, that have been validated
and approved by CMS to count toward the issuer’s satisfaction of the time and distance standards for
the corresponding specialty types (i.e., primary care (routine) and behavioral health). Please note: The
provider population file will not include facility provider types (e.g., Outpatient Behavioral Health
Clinics), plan IDs, or phone numbers. QHP issuers are responsible for sharing provider phone numbers
with their contracted third-party entities. QHP issuers are permitted to share more than one phone
number per provider, if applicable and available. CMS will create the provider population file after the
final round of QHP certification data submission is complete.
The provider population file that CMS will generate for each issuer will include the following data
fields for each individual provider on the issuer’s submitted Network Adequacy template that CMS has
validated and approved during the QHP certification review process for the corresponding
specialty types:
2

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

1. Issuer Identification Number
2. Network Identification Number
3. Servicing Specialty Group (Outpatient Clinical Behavioral Health/Primary Care Provider)
4. National Provider Identifier (NPI)
5. Provider Name
6. Street Address
7. Street Address 2
8. City
9. State
10. ZIP Code
11. County Name
12. Specialty Code
13. Specialty
CMS defines primary care (routine) and behavioral health care providers for the purpose of assessing
issuer compliance with appointment wait time standards to include providers with the taxonomy codes
listed in Tables 2 and 3, which contain taxonomy codes that correspond to each provider type as listed
in the Network Adequacy template. Only providers included in the provider population file may
be surveyed.
Table 2. Primary Care Provider Specialty Types
Primary Care Provider Specialty Types for Primary Care (Routine) Category
for Appointment Wait Time Standards
NUCC
taxonomy code

Specialty type codes and
descriptions

NUCC display name

207Q00000X

Family Medicine (002)

Family Medicine Physician

207QA0000X

Family Medicine (002)

Adolescent Medicine (Family
Medicine) Physician

207QA0505X

Family Medicine (002)

Adult Medicine Physician

207QB0002X

Family Medicine (002)

Obesity Medicine (Family
Medicine) Physician

208D00000X

General Practice (001)

General Practice Physician

207QG0300X

Geriatrics (004)

Geriatric Medicine (Family
Medicine) Physician

207RG0300X

Geriatrics (004)

Geriatric Medicine (Internal
Medicine) Physician

207R00000X

Internal Medicine (003)

Internal Medicine Physician

207RA0000X

Internal Medicine (003)

Adolescent Medicine (Internal
Medicine) Physician

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Primary Care Provider Specialty Types for Primary Care (Routine) Category
for Appointment Wait Time Standards
NUCC
taxonomy code

Specialty type codes and
descriptions

NUCC display name

207RB0002X

Internal Medicine (003)

Obesity Medicine (Internal
Medicine) Physician

363LA2200X

Primary Care - Advanced
Registered Nurse Practitioner
(006)

Adult Health Nurse Practitioner

363LF0000X

Primary Care - Advanced
Registered Nurse Practitioner
(006)

Family Nurse Practitioner

363LP2300X

Primary Care - Advanced
Registered Nurse Practitioner
(006)

Primary Care Nurse Practitioner

363A00000X

Primary Care - Physician Assistant Physician Assistant
(005)

363AM0700X

Primary Care - Physician Assistant Medical Physician Assistant
(005)

208000000X

Primary Care – Pediatric (101)

Pediatrics Physician

2080A0000X

Primary Care – Pediatric (101)

Pediatric Adolescent Medicine
Physician

Note. NUCC = National Uniform Claim Committee.

Table 3. Behavioral Health Provider Specialty Types
Behavioral Health Provider Specialty Types for Behavioral Health Category
for Appointment Wait Time Standards
NUCC taxonomy code

Specialty type codes and
description

NUCC display name

101YA0400X

Addiction (Substance Use
Disorder) Counselor (106)

Addiction (Substance Use Disorder)
Counselor

207LA0401X

Addiction Medicine Physician
(800)

Addiction Medicine (Anesthesiology)
Physician

207QA0401X

Addiction Medicine Physician
(800)

Addiction Medicine (Family Medicine)
Physician

207RA0401X

Addiction Medicine Physician
(800)

Addiction Medicine (Internal Medicine)
Physician

2083A0300X

Addiction Medicine Physician
(800)

Addiction Medicine (Preventive
Medicine) Physician

103K00000X

Behavioral Analyst (801)

Behavioral Analyst

363LP0808X

Behavioral Health - Advanced
Practice Registered Nurse (108)

Psychiatric/Mental Health Nurse
Practitioner

364SP0808X

Behavioral Health - Advanced
Practice Registered Nurse (108)

Psychiatric/Mental Health Clinical
Nurse Specialist

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Behavioral Health Provider Specialty Types for Behavioral Health Category
for Appointment Wait Time Standards
Specialty type codes and
description

NUCC taxonomy code

NUCC display name

101Y00000X

Counselor (Mental Health and
Professional) (107)

Counselor

101YM0800X

Counselor (Mental Health and
Professional) (107)

Mental Health Counselor

101YP2500X

Counselor (Mental Health and
Professional) (107)

Professional Counselor

106H00000X

Marriage and Family Therapist
(105)

Marriage & Family Therapist

103T00000X

Psychologist (103)

Psychologist

103TA0400X

Psychologist (103)

Addiction (Substance Use Disorder)
Psychologist

103TA0700X

Psychologist (103)

Adult Development & Aging
Psychologist

103TB0200X

Psychologist (103)

Cognitive & Behavioral Psychologist

103TC0700X

Psychologist (103)

Clinical Psychologist

103TC1900X

Psychologist (103)

Counseling Psychologist

103TC2200X

Psychologist (103)

Clinical Child & Adolescent
Psychologist

103TE1100X

Psychologist (103)

Exercise & Sports Psychologist

103TF0000X

Psychologist (103)

Family Psychologist

103TF0200X

Psychologist (103)

Forensic Psychologist

103TH0004X

Psychologist (103)

Health Psychologist

103TH0100X

Psychologist (103)

Health Service Psychologist

103TM1800X

Psychologist (103)

Intellectual & Developmental
Disabilities Psychologist

103TP0016X

Psychologist (103)

Prescribing (Medical) Psychologist

103TP0814X

Psychologist (103)

Psychoanalysis Psychologist

103TP2701X

Psychologist (103)

Group Psychotherapy Psychologist

103TR0400X

Psychologist (103)

Rehabilitation Psychologist

103TS0200X

Psychologist (103)

School Psychologist

104100000X

Social Worker (102)

Social Worker

1041C0700X

Social Worker (102)

Clinical Social Worker

1041S0200X

Social Worker (102)

School Social Worker

Note. NUCC = National Uniform Claim Committee.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

B. Sample Selection
A QHP issuer’s third-party entity must survey a minimum number of providers of each provider specialty
type included in the provider population file (i.e., primary care (routine) and behavioral health). The
minimum number of providers that must be surveyed, based on the total number of providers of a specific
specialty type in an issuer’s QHP provider network, has been identified by CMS to produce maximum
confidence limits of +/- 5% for an expected compliance rate of 90% with a 95% confidence level for the
issuer’s QHP provider network per service area. Individual providers are identified by unique National
Provider Identifier (NPI). Please refer to Table 4 to determine the minimum sample size to be surveyed
for each provider type from each of an issuer’s QHP provider networks.
The required sampling approach is summarized in the following steps and illustrated in Figure 1:
Overview of Survey Sampling Process.
1. Select the Oversample: CMS acknowledges that network adequacy provider data submitted
by an issuer in the Network Adequacy template during the certification process are a point-intime representation of an issuer’s QHP provider networks. Thus, to account for providers that
are nonresponsive or ineligible when surveys are conducted, a QHP issuer’s third-party entity
must select a statistically valid random oversample of providers from the provider population
file that is 50% larger than the minimum sample sizes identified in Table 4. QHP issuers’ thirdparty entities should round-up to the nearest whole number when calculating the size of the
oversample population. If there are not enough providers in an issuer’s provider population file
to draw a full oversample that is 50% larger than the required minimum sample, then the thirdparty entity must create a reserve sample from the remaining providers that were not included
in the minimum sample. Pediatric primary care providers and adult primary care providers will
be delineated in the provider population file. QHP issuers’ third-party entities must combine all
the taxonomies within these categories to create the oversample of primary care providers for a
network. Pediatric or adult primary care providers should not be sampled separately.
2. Draw the Minimum Sample: For each issuer’s QHP provider network, the QHP issuer’s
third-party entity must draw, from the oversample population, a random, statistically valid,
minimum sample size for surveying of each provider type (i.e., primary care (routine) and
behavioral health).
3. As Applicable, Utilize the Reserve Sample: If the minimum sample includes ineligible or
nonresponsive providers, replacements must be drawn from the reserve sample. If the reserve
sample is exhausted, and if additional providers remain in the provider population file, QHP
issuers’ third-party entities must add additional providers of that same provider type to the
oversample via random selection. The QHP issuer’s third-party entity must continue to add
providers to the oversample using a random selection process until either the required
minimum number of providers per QHP provider network/service area are surveyed or all
providers within the QHP provider network/service area for the applicable provider type have
been surveyed.
For example, if an issuer’s QHP provider network contains 108 individual primary care providers, as
identified by unique NPI, then the minimum sample size that must be surveyed for the primary care
provider type for that issuer’s QHP provider network is 62. The QHP issuer’s third-party entity must
select one randomized oversample of 93 primary care (routine) providers (consisting of the minimum
sample of 62 plus a reserve sample of 31).
6

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Table 4. Minimum Survey Sample Sizes
Number of
specialty
category
providers in
issuer’s QHP
provider
network
1

Minimum
sample size to
be surveyed
1

Number of
specialty
category
providers in
issuer’s QHP
provider
network
26–30

Minimum
sample size to
be surveyed
25

Number of
specialty
category
providers in
issuer’s QHP
provider
network
251–260

Minimum
sample size to
be surveyed
91

31–35

261–270

36–40

271–280

41–45

281–290

46–50

291–300

51–60

301–325

61–70

326–350

100

71–80

351–375

102

81–90

376–400

103

91–100

401–425

105

101–110

426–450

106

111–120

451–475

108

121–130

476–500

109

131–140

501–600

113

141–150

601–700

116

151–160

701–800

119

161–170

801–900

120

171–180

901–1,000

122

181–190

1,001–2,000

130

191–200

2,001–3,000

133

201–210

3,001–4,000

134

211–220

4,001–5,000

135

221–230

5,001–10,000

137

231–240

10,001–15,000

138

241–250

15,001–50,000

138

A provider is ineligible to be surveyed (and therefore must be replaced by a randomly selected
provider from the reserve sample) if:
1. At the time of the call, the provider is no longer contracted with the issuer as an in-network
provider.
2. At the time of the call, the provider is not practicing and classified as a primary care provider
specialty type or behavioral health provider specialty type according to the specialty
types/descriptions listed in Tables 2 and 3.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

If a provider is determined to be ineligible upon the first contact attempt, no additional contact
attempts should be made, and the provider must be replaced by a randomly selected provider of the
same specialty type from the reserve sample.
Additionally, if a provider is nonresponsive, they must be replaced by a randomly selected provider
from the reserve sample. A provider is defined as nonresponsive if any of the following dispositions
are documented for each of three contact attempts associated with the same provider:
1. The provider was nonresponsive: The phone rang for 5 minutes without an answer.
2. The provider was nonresponsive: The phone number is not valid (for example, the number is
disconnected, or the phone number is not associated with the provider). If a provider’s phone
number is not valid, the secret shopper should not make two additional contact attempts using
that same invalid phone number before replacing the provider with another provider from the
reserve sample. The secret shopper may use more than one phone number to complete a
maximum of three contact attempts per provider.
3. The provider was nonresponsive: A voicemail greeting was received during regular business
hours, not during a designated lunch hour, and the voicemail system was not part of an
appointment scheduling procedure.
4. The provider was nonresponsive: The secret shopper was placed on hold by the provider’s
office for more than 15 minutes. Third-party entities’ secret shoppers should engage with
interactive voice response (IVR) systems; however, they are not expected to navigate a phone
tree for more than 20 minutes, inclusive of the 15 minutes they may be placed on hold.
5. The provider was nonresponsive: The call could not be completed due to communication
issue(s) such as poor call connectivity.
If a QHP issuer has multiple, identical provider networks (for example, the same provider network is
associated with several plan variations offered by the same issuer), then only one statistically valid,
randomized oversample of each provider type must be drawn and used for surveying the minimum
required number of providers of each provider type. The oversample can be used for all identical
provider networks.
If a QHP has multiple, nonidentical provider networks, then a statistically valid, randomized
oversample population of each provider type must be drawn from the provider population file for each
issuer’s QHP provider network. If the same provider is included in the oversample population drawn
for more than one issuer’s QHP provider networks and a survey result is obtained from that provider,
then that result can be reported and applied for that same provider for each sample in which the
provider appears.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Figure 1. Overview of Survey Sampling Process

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

C. Survey Protocol Development
QHP issuers must work with their third-party entities to develop and finalize a secret shopper survey
protocol that reasonably approximates an enrollee’s experience as a new patient attempting to obtain
appointment availability from a provider. The protocol must be appropriately designed to obtain an
offer for a first available appointment and adhere to the following minimum requirements:
1. The third-party entity must conduct secret shopper surveys while presenting as a new patient
(i.e., a patient attending their first-ever clinical encounter with a practitioner at the location
being surveyed). This may include posing as a representative or associate of a new patient
(i.e., a patient or a family member of a patient). The survey script must be designed such that
the secret shopper confirms whether the provider specifically offers primary care (routine)
and/or behavioral health services at the location being surveyed.
2. The survey script must be designed such that the secret shopper ascertains whether the
provider offers both in-person appointments and telehealth appointments, offers only
telehealth appointments, or offers only in-person appointments. If a provider offers both inperson and telehealth appointments for new patients, and the first (i.e., soonest) available
appointment offered is a telehealth appointment, then the secret shopper must also obtain
appointment availability information for the next available in-person appointment.
Alternatively, if a provider offers both in-person and telehealth appointments for new patients,
and the first (i.e., soonest) available appointment offered is an in-person appointment, then the
secret shopper must also obtain appointment availability information for the next available
telehealth appointment.
3. The survey protocol must be designed such that the secret shopper does not actually accept an
offered appointment (therefore precluding actual patients from being able to access that
appointment). For example, once an appointment is offered to the secret shopper, the secret
shopper can say that they need to consult their calendar before confirming and then end the call.
QHP issuers and their third-party entities may consider a protocol that uses simulated enrollee profiles
(consisting of, for example, plan identification numbers and/or other simulated identifying
information) if the QHP issuer and the QHP issuer’s third-party entity determine this approach is
feasible and/or believe this approach would increase the likelihood that providers would provide
appointment availability information to secret shoppers. Neither simulated enrollee profiles nor survey
scripts used by secret shoppers should convey the existence of an urgent medical need.
QHP issuers and their third-party entities may refer to the example script responses or “workarounds”
and general tips contained in Appendix D when developing their secret shopper survey scripts and
protocols. These examples may be useful when secret shoppers encounter certain scenarios, such as
being asked by a provider’s office for an insurance identification number, patient employer, or other
identifying information.

D. Survey Administration
Secret shopper calls must begin on or shortly after January 1 and must be completed by May 31 of the
plan year. All outgoing secret shopper calls must be conducted between 8:00 a.m. and 5:00 p.m. in the
providers’ respective time zones.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

The secret shopper may not contact a provider’s office more than three times to obtain appointment
availability information. For example, if the first contact attempt for a provider fails due to the provider
being nonresponsive as defined in the disposition codes enumerated in Appendix B, the secret shopper
may attempt to call that same provider’s office two additional times, for a maximum of three
contact attempts.

E. Collecting and Documenting Survey Results Data
QHP issuers must instruct their third-party entities to document secret shopper survey results for each
provider surveyed and/or contacted to be surveyed using the fields, definitions, and disposition codes
as described in Appendices A and B. Additionally, QHP issuers’ third-party entities must adhere to the
following guidelines when documenting and reporting survey results:
Scenario

Response to Scenario

If a surveyed provider indicates that the
date and time of the first available
appointment depends upon whether the
appointment is conducted in person or via
telehealth, the secret shopper must …

Document the first available appointment in the results reported,
use the first available appointment to calculate the actual wait
time length associated with the contact attempt (documented in
the “Time_Elapsed” field in the results reported), and indicate the
appointment format according to Appendix A.

If the surveyed provider indicates they
have an appointment available at a later
time and date than with an equivalent
provider and the secret shopper is offered
an appointment with such equivalent
provider, the secret shopper must …

Document the first available appointment offered with the
equivalent provider in the results reported according to
Appendices A and B.
An equivalent provider is defined as another in-network provider
of the same specialty type within the same practice location.

If a provider reports they are not currently
Ask whether there is a waiting list and/or estimated period during
accepting new patients, the secret shopper which the provider is not accepting new patients. For example,
the secret shopper may ask a follow-up question such as “Do
must …
you know when Dr. Smith will start accepting new patients
again?” and/or “Is there a waiting list for new patient
appointments, and if so, how long does it typically take to get off
the waiting list?”
If the provider/provider’s office does not know when the provider
will start accepting new patients and/or is unable to provide
estimated information about a waiting list or future appointment
availability, the secret shopper must document the appointment
wait time as “N/A” in the “Time_Elapsed” field in the results
reported, indicate that the provider does not have an
appointment available within the applicable standard in the
“Compliance_Indicator” field in the results reported, and note the
appropriate disposition code(s) in the results reported.
If the provider/provider’s office clearly indicates, for example, that
the provider will start accepting new patients again in 3 months,
the secret shopper must document the appointment wait time (in
the “Time_Elapsed” field in the results reported) by calculating
the number of business days from the date of the call that
resulted in an appointment offered and 3 months out from that
date, indicate that the provider does not have an appointment
available within the applicable standard, and note the appropriate
disposition code(s) in the results reported.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Scenario

Response to Scenario

Ask the provider to provide the date that a patient walking in at
If the provider reports that patients are
served on a walk-in or same-day basis, the the time of the call would be seen. Document that day's date as
the date of the soonest available appointment offered and enter
secret shopper must …
“0” in the “Time_Elapsed” field in the results reported.
If the provider is not scheduling
appointments at the time of the survey
because they are temporarily unavailable
due to vacation, sabbatical, parental leave,
personal issue, etc., the secret shopper
must …

Ask whether there is an estimated future point in time wherein
the provider will again be able to offer new patient appointments.
For example, the secret shopper may ask a follow-up question
such as, “Do you know when Dr. Smith will be back from
vacation and when her first available appointment would be?” If
the provider/provider’s office does not know when the provider
will again offer new patient appointments and/or is unable to
provide estimated information, the secret shopper must
document the appointment wait time as “N/A” in the
“Time_Elapsed” field in the results reported, indicate that the
provider does not have an appointment available within the
applicable standard (in the “Compliance_Indicator” field in results
reported), and note the appropriate disposition code(s) in the
results reported.
If the provider/provider’s office clearly indicates, for example, that
the provider will be offering new patient appointments in 3 weeks,
the secret shopper must document the appointment wait time (in
the “Time_Elapsed” field in the results reported) by calculating
the number of business days from the date of the call that
resulted in an appointment offered and 3 weeks out from that
date, indicate that the provider does not have an appointment
available within the applicable standard, and note the appropriate
disposition reason code(s) in the results reported.

If the secret shopper encounters
communication difficulties when calling a
provider, such as poor call quality and/or
not being able to hear the call recipient, the
secret shopper must …

End the call and document the appropriate disposition code
associated with the contact attempt (G). The secret shopper
must wait until at least the next part of the same business day
before initiating the next contact attempt for the same provider.
For example, if the first contact attempt occurred at 9:30 a.m.,
the secret shopper must not initiate the second contact attempt
until at least 1:00 p.m. or later on the same business day
(accounting for a lunch hour of 12:00 p.m. to 1:00 p.m.).
Similarly, if the first contact attempt occurred at 2:00 p.m., the
secret shopper must not initiate the second contact attempt for
the same provider until at least 8:00 a.m. or later on the next
business day.

If, as part of scheduling procedure for new
patient appointments, a provider requires
the secret shopper to leave a voicemail
with identifying information in order to
receive a call back and initiate scheduling,
the secret shopper must …

Not leave a voicemail and document the appropriate disposition
code associated with the contact attempt (Q). The secret
shopper must document the appointment wait time as “N/A” in
the “Time_Elapsed” field in the results reported and mark “N/A” in
the “Compliance Indicator” field in the results reported.
The secret shopper must not initiate remaining contact attempts
and must exclude the provider from the compliance rate
calculation and replace the provider with a randomly selected
provider of the same specialty type (primary care or behavioral
health) from the reserve sample.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Scenario

Response to Scenario

If, as part of scheduling procedure for new
patient appointments, a provider requires
the secret shopper to consult an online
scheduling tool and/or use an online
patient portal to view appointment
availability, the secret shopper must …

Not view an online scheduling tool and must document the
appropriate disposition code associated with the contact attempt
(U). The secret shopper must document the appointment wait
time as “N/A” in the “Time_Elapsed” field in the results reported
and mark “N/A” in the “Compliance Indicator” field in the results
reported.
The secret shopper must not initiate remaining contact attempts
and must exclude the provider from the compliance rate
calculation and replace the provider with a randomly selected
provider of the same specialty type (primary care or behavioral
health) from the reserve sample.

Appointment wait times must be calculated (and documented in the “Time_Elapsed” field in the results
reported) by counting the number of full business days between the date of the call during which a
request for an appointment was successfully executed and the date of the first available appointment.
When calculating appointment wait time length in business days, QHP issuers’ third-party entities
must adhere to the following rules:
•

Start counting the number of days from the next following day.

•

Exclude Federal holidays and weekends (Saturdays and Sundays) because these are not
considered business days.

•

When calculating appointment wait time length in business days, exclude the first day (e.g., the
day of the request) and count the last day.

Table 5. Example Appointment Wait Time Calculation
Sun.

Sun.

Mon.

Tue.

Wed.

Thur.

Fri.

(0)
Call

(1)

(2)

(3)

(4)

Mon.

Tue.

Wed.

Thur.

Fri.

Federal
Holiday

(5)
First available
appointment

Sat.

F. Calculating Compliance Rate and Other Figures Based on Results
QHP issuers’ third-party entities must calculate and report the following:
1. The compliance rate for the wait time standard for each provider type, for each issuer’s QHP
provider network according to the following formula:
Numerator: Number of providers that offered appointments within the time elapsed standard
as reflected by provider survey results with any of the following disposition codes: A, B, J, K,
S, T and accompanying Compliance_Indicator of “Y.” For disposition code definitions, see
Appendix B.
Denominator: Total number of eligible providers surveyed.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Compliance Rate Calculation Example:
Issuer Beta’s provider population file indicates the issuer’s single QHP provider network contains
525 primary care (routine) providers. Therefore, the QHP issuer’s third-party entity must randomly
select an oversample of 170 primary care (routine) providers from the provider population file to
include the minimum sample to be surveyed (113) plus a reserve sample of an additional 57 primary
care (routine) providers. The QHP issuer’s third-party entity surveys 113 primary care (routine)
providers via secret shopper protocol. No providers in the minimum sample were ineligible or
nonresponsive. Thus, no providers in the minimum sample were replaced with providers from the
reserve sample.
The survey results (expressed by combinations of disposition codes and compliance indicators per
provider surveyed) are as follows:
• 64 providers offered an appointment within the maximum time elapsed standard (Disposition
Code A, Compliance_Indicator Y).
• 5 surveyed providers offered an appointment with an equivalent provider that was not within the
time elapsed standard (Disposition Code B, Compliance_Indicator N).
• 5 surveyed providers offered an appointment with an equivalent provider within the time
elapsed standard (Disposition Code B, Compliance_Indicator Y).
• 5 providers were temporarily unavailable due to vacation, parental leave, personal issue, illness,
etc.; an appointment was not offered with an equivalent provider; and estimated availability
and/or waitlist information was provided. The availability and/or waitlist information provided
is not within the time elapsed standard (Disposition Code T, Compliance_Indicator N).
• 1 provider was not currently accepting new patients, and an appointment was not offered with
an equivalent provider (Disposition Code H, Compliance Indicator N).
• 8 providers were currently not accepting new patients, an appointment was not offered with an
equivalent provider, and estimated availability and/or waitlist information was provided. The
availability and/or waitlist information provided was within the time elapsed standard
(Disposition Code J, Compliance_Indicator Y).
• 25 providers were not currently accepting new patients; appointments were offered with
equivalent providers, and the appointments offered were within the time elapsed standard
(Disposition Code K, Compliance_Indicator Y).
Compliance Rate Numerator: 64 (Disposition Code A, Compliance_Indicator Y) + 5 (Disposition
Code B, Compliance_Indicator Y) + 8 (Disposition Code J, Compliance_Indicator Y) + 25
(Disposition Code K, Compliance_Indicator Y) = 102
Compliance Rate Denominator: 113
Compliance Rate = (102/113) × 100 = 90.2%
2. The percentage of nonresponsive and ineligible providers, according to the following formula:
Numerator: Number of providers that were deemed nonresponsive or ineligible as classified
by any of the following disposition codes: C, D, E, F, G, L, M as call outcomes documented
for each of the three contact attempts (if three contact attempts were required to be made) and
Compliance_Indicator of “N/A.”
Denominator: Total number of providers contacted.
14

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Percentage of Nonresponsive and Ineligible Providers Calculation Example:
Issuer Delta’s provider population file indicates the issuer’s single QHP provider network contains
249 behavioral health providers. Therefore, the QHP issuer’s third-party entity must randomly
select an oversample of 135 behavioral health providers from the provider population file to include
the minimum sample (90) plus a reserve sample of an additional 45 behavioral health providers.
The survey results (expressed by combinations of disposition codes and compliance indicators per
provider surveyed) are as follows:
•

25 providers offered an appointment within the time elapsed standard (Disposition
Code A, Compliance_Indicator Y).

•

25 providers were ineligible to be surveyed because they were no longer contracted with
the issuer as an in-network provider at the time of the survey call (Contact 1 Disposition
Code L, Compliance_Indicator N/A)

•

10 providers were nonresponsive because their phone numbers were not valid (Contact 1
Disposition Code C and Compliance_Indicator N/A).

•

10 providers offered an appointment with an equivalent provider within the time elapsed
standard (Disposition Code B, Compliance_Indicator Y).

•

50 providers offered an appointment within the time elapsed standard (Disposition
Code A, Compliance_Indicator Y)

•

5 providers offered an appointment that was not within the time elapsed standard
(Disposition Code A, Compliance Indicator_N).
– Percentage of nonresponsive and ineligible providers numerator: 25 (Contact 1
Disposition Code L, Compliance_Indicator N/A) + 10 (Contact 1 Disposition Code C,
Compliance_Indicator N/A) = 35
– Percentage of nonresponsive and ineligible providers denominator: 25 + 25 + 10 + 10 +
50 + 5 = 125
– Percentage of nonresponsive and ineligible providers: 35/125 × 100 = 28%
– Compliance rate numerator: 25 (Disposition Code A, Compliance_Indicator Y) + 10
(Disposition Code B, Compliance_Indicator Y) + 50 (Disposition Code A,
Compliance_Indicator Y) = 85
– Compliance rate denominator: 25 + 10 + 50 + 5 = 90
– Compliance rate: 85/90 × 100 = 94.4%

The compliance rate numerator, denominator, and result must be calculated and reported for each
appointment wait time standard for each QHP provider network. For example, if an issuer has two,
nonidentical QHP provider networks, the QHP issuer’s third-party entity must calculate (and the QHP
issuer must report) four compliance rates identified by QHP provider network ID: one compliance rate
for the primary care (routine) appointment wait time standard for each of the two QHP provider
networks and one compliance rate for the behavioral health appointment wait time standard for each of
the two QHP provider networks.
Similarly, the numerator, denominator and result for the percentage of nonresponsive and ineligible
providers must be calculated and reported for each QHP provider network. For example, if a QHP
15

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

issuer has two, nonidentical QHP provider networks, the QHP issuer’s third-party entity must calculate
(and the QHP issuer must report) two percentages of nonresponsive and ineligible providers, identified
by QHP provider network ID.

G. Submitting Survey Results Data to CMS
QHP issuers must submit secret shopper survey results (according to Appendix A) for each QHP
provider network serving enrollees in the service area to CMS during QHP issuer compliance and
monitoring activities. All QHP issuers must submit their secret shopper survey policy and/or process
documentation along with the secret shopper survey results. For example, issuers must submit thirdparty entity contracts in addition to their appointment wait time survey results. As part of this process,
the issuer must submit the data and documentation from the completed secret shopper surveys through
a secure file submission portal (hereinafter referred to as “the Portal”). Valid survey results and
accompanying calculations are due to CMS by the second Friday of June each year. Each issuer’s
unique link to the Portal will be provided by a CMS contractor within the data request distributed to
the QHP issuer. The QHP issuer must provide at least one (and no more than two) accurate email
addresses for specific points of contact who will be given access to CMS requests through the Portal.
CMS will provide additional instructions for the file formats and file-naming conventions within the
data submission request submitted to the QHP issuer by email or through the Portal. Additional
submission details and requirements that will be listed in these instructions are outlined below.
1. Issuers will be sent a unique link to access the Portal.
2. The Portal has a series of requirements that the issuer must follow. To start, the issuer will use
their unique link to access the Portal to begin each submission session. The Portal allows a
maximum file size of 20 GB per file; however, multiple files can be uploaded per submission
session. Additionally, each file name must not exceed 25 characters. Files may be rejected if
the file naming convention is not followed. To better facilitate both data entry and review, files
must be zipped by category for each requested item. The following file types will be denied:
EXE, MSI, and BIN.
3. Issuers will be asked to validate all information before submission to confirm that the
documents submitted are accurate, complete, and readable.
For further questions on how QHP issuers must submit survey data, please email
[email protected].

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Appendix A. Required Secret Shopper Survey Results
QHP issuers’ third-party entities must collect and document secret shopper survey results for each
provider surveyed/contacted to be surveyed using the fields, definitions, and disposition codes
described in this appendix and Appendix B. The required file formats for secret shopper survey results
submission are Excel (.xlsx) or CSV. Compliance rate fields and percentage of nonresponsive and
ineligible providers calculation fields may be listed in a tab or file that is separate from provider-level
data fields.
Column Field

Field Code

Definition/Instructions

National Provider
Identifier (NPI)

NPI

Enter the provider’s NPI

Provider Name (First
Name, Last Name)

Prov_Name

Enter the provider’s First and Last Name

Specialty Type

Spec_Type

Enter the applicable Specialty Type Code
of the Outpatient Clinical Behavioral
Health or Primary Care practitioner from
the list below:
• Outpatient Clinical Behavioral
Health Specialty Type Codes
include 102, 103, 105, 106, 107,
108, 800, and 801
• Primary Care Specialty Type Codes
include 001, 002, 003, 004, 005,
006, 101

Street Address 1

Street_Add_1

Enter the provider’s street address.

Street Address 2

Street_Add_2

Enter the additional street address for the
provider, if applicable.

City

Enter the city of the provider’s office.

State

Enter the state of the provider’s office.

County

Enter the county of the provider’s office.

Zip

Enter the ZIP code of the provider’s
office.

Network ID

Net_ID

Enter the QHP provider network ID in
which the provider is contracted.

Contact 1 Provider
Phone Number

Prov_Phone_1

Enter the phone number that the QHP
issuer’s third-party entity’s secret shopper
used to contact the provider for contact
attempt 1.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Column Field

Field Code

Definition/Instructions

Contact 1 Date

Contact_1_Date

Enter the date of the first contact attempt,
formatted as MM/DD/YYYY.
If the first contact attempt resulted in a
valid survey disposition, record the
disposition code in the “Contact1
Disposition” field.
If more than one contact attempt was
made, record the additional contact
attempt dates, times, and disposition
codes in the appropriate fields.
If the provider was determined to be
ineligible after the first contact attempt
(because the provider was no longer
contracted with the issuer as an innetwork provider or the provider is not
classified as either a primary care
(routine) provider or behavioral health
provider), no additional contact attempts
should be made.
If the provider was nonresponsive
because the phone number was invalid
(the phone number was disconnected
and/or the provider is not associated with
the phone number), no additional contact
attempts should be made.

Contact 1 Time

Contact_1_Time

Enter “AM” for contact attempts
conducted between the hours of
8:00 a.m. and 11:59 a.m. Enter “PM” for
contact attempts conducted between the
hours of 12:00 p.m. and 5:00 p.m.

Contact 1 Disposition
Code

Contact_1_Disposition

Enter the disposition code for Contact 1.

Contact 2 Provider
Phone Number

Prov_Phone_2

Enter the phone number that the QHP
issuer’s third-party entity’s secret shopper
used to contact the provider for contact
attempt 2.

Contact 2 Date

Contact_2_Date

If more than one contact attempt was
made, record the date of the second
contact attempt, formatted as
MM/DD/YYYY. If the first contact attempt
resulted in a valid survey disposition
code, this field does not need to be
reported.

Contact 2 Time

Contact_2_Time

Enter “AM” for contact attempts
conducted between the hours of
8:00 a.m. and 11:59 a.m. Enter “PM” for
contact attempts conducted between the
hours of 12:00 p.m. and 5:00 p.m.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Column Field

Field Code

Definition/Instructions

Contact 2 Disposition
Code

Contact_2_Disposition

Enter the disposition code for Contact 2.

Contact 3 Date

Contact_3_Date

If more than two contact attempts were
made, record the date of the third and
final contact attempt, formatted as
MM/DD/YYYY. If the first or second
contact attempt(s) resulted in a valid
survey disposition code, this field does
not need to be reported.

Contact 3 Provider
Phone Number

Prov_Phone_3

Enter the phone number that the QHP
issuer’s third-party entity’s secret shopper
used to contact the provider for contact
attempt 3.

Contact 3 Time

Contact_3_Time

Enter “AM” for contact attempts
conducted between the hours of
8:00 a.m. and 11:59 a.m. Enter “PM” for
contact attempts conducted between the
hours of 12:00 p.m. and 5:00 p.m.

Contact 3 Disposition
Code

Contact_3_Disposition

Enter the disposition code for Contact 3.

Date of First Available
(i.e., Soonest)
Appointment Offered

End_Date

If the valid survey disposition code for the
surveyed provider is one of the following:
A, B, J, K, T, U and appointment
availability information was obtained
and/or an appointment was offered by or
on behalf of either the surveyed provider
or an equivalent provider, record the date
of the first available (i.e., soonest)
appointment offered, formatted as
MM/DD/YYYY.

Format of First (i.e.,
Soonest) Available
Appointment Offered

End_Format

If the first available appointment offered
by or on behalf of either the surveyed
provider or an equivalent provider is a
telehealth appointment, enter “T” in this
field. If the first available (i.e., soonest)
available appointment offered by or on
behalf of either the surveyed provider or
an equivalent provider is an in-person
appointment, enter “P” in this field.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Column Field

Field Code

Definition/Instructions

Type(s) of
Appointments Offered
by Provider Indicator

Types_Offered

The secret shopper must ascertain
whether or not the provider offers both inperson and telehealth appointments for
new patients, solely telehealth
appointments for new patients, or solely
in-person appointments for new patients
and must document this information using
one of the following entries:
TO = Provider offers only telehealth
appointments for new patients and does
not offer in-person appointments.
PO = Provider offers only in-person
appointments for new patients and does
not offer telehealth appointments.
B = Provider offers both in-person and
telehealth appointments for new patients.

Date of Next Available
Appointment Offered

Next_Avail_Date

If a provider offers both in-person and
telehealth appointments for new patients,
and the first (i.e., soonest) available
appointment offered is a telehealth
appointment, then the secret shopper
must obtain appointment availability
information for the next available inperson appointment and record the date
of the next available in-person
appointment in this field.
Alternatively, if a provider offers both inperson and telehealth appointments for
new patients, and the first (i.e., soonest)
available appointment offered is an inperson appointment, then the secret
shopper must obtain appointment
availability information for the next
available telehealth appointment and
record the date of the next available
telehealth appointment in this field.

Time Elapsed Between
Date of Call That
Resulted in
Appointment Offered
and the First Available
(i.e., Soonest)
Appointment Offered

Time_Elapsed

Enter the number of business days
(excluding weekends and Federal
holidays) between the date of the call that
resulted in an appointment offered and
the date of the first (i.e., soonest)
available appointment offered. Enter only
whole numbers in this field.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Column Field

Field Code

Definition/Instructions

Was appointment
offered within the time
elapsed standard?

Compliance_Indicator

If the time elapsed (in full business days)
between the date of the call that resulted
in an appointment offered and the date
and time of the first (i.e., soonest)
available appointment offered is equal to
or less than the minimum time elapsed
standard, enter “Y.” If the value exceeds
the minimum time elapsed standard,
enter “N.”
If the provider was nonresponsive or
ineligible to be surveyed, enter “N/A.”

General Notes

Comments

This is a free text field for the QHP
issuer’s third-party entity to provide
additional information about the contact
attempts and/or notable findings. See
“instructions for reporting” column in
“Appendix B. Valid Survey Disposition
Codes” for specific instructions on how to
format the provision of additional
information in this field as required by
certain disposition codes.

Compliance Rate
Numerator

Compliance_Rate_Numerator

Number of providers that offered
appointments within the time elapsed
standard as reflected by provider survey
results with disposition codes: A, B, J, K,
T, U and accompanying
Compliance_Indicator of “Y.” For a
definition of each of these codes, please
see Appendix B.

Compliance Rate
Denominator

Compliance_Rate_Denominator

Total number of eligible providers
surveyed,

Compliance Rate1

Compliance_Rate

Result of dividing the Compliance_Rate
Numerator by the
Compliance_Rate_Denominator.

Percentage of
Nonresponsive and
Ineligible Providers
Numerator

Nonresponse_Numerator

Number of providers that were deemed
nonresponsive or ineligible as reflected
by provider survey results with disposition
codes: C, D, E, F, G, L, M as call
outcomes documented for each of the
three contact attempts (if three contact
attempts were required to be made) and
accompanying Compliance_Indicator of
“N/A.”

Percentage of
Nonresponsive and
Ineligible Providers
Denominator

Nonresponse_Denominator

Total number of providers contacted.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Column Field
Percentage of
Nonresponsive and
Ineligible Providers2

Field Code
Percent_Nonresponsive

Definition/Instructions
Result of dividing the
Nonresponse_Numerator by the
Nonresponse_Denominator.

1
As described above, QHP issuers’ third-party entities must calculate the compliance rate for each appointment wait time standard for each
issuer’s QHP provider network. For example, if a QHP issuer has two, nonidentical QHP provider networks, the QHP issuer’s third-party
entity must calculate (and the QHP issuer must report) four compliance rates, identified by QHP provider network ID: one compliance rate
for the primary care (routine) appointment wait time standard for each of the two QHP provider networks and one compliance rate for the
behavioral health appointment wait time standard for each of the two QHP provider networks.
2
As described above, QHP issuers’ third-party entities must calculate the numerator, denominator, and the result of dividing the percentage
of nonresponsive and ineligible providers numerator by the percentage of nonresponsive and ineligible providers denominator for the
percentage of nonresponsive and ineligible providers for each QHP provider network. For example, if an issuer has two, nonidentical QHP
provider networks, the QHP issuer’s third-party entity must calculate (and the QHP issuer must report) two percentages of nonresponsive
and ineligible providers, identified by QHP provider network ID.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Appendix B. Valid Survey Disposition Codes

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Appointment availability
information obtained (i.e.,
an appointment was
offered) with the surveyed
provider.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Calculate and record the
appointment wait time length and
record the number of business days
in the Time_Elapsed field.
For the “Compliance_Indicator”
field: If the time elapsed (in full
business days) between the date of
the call that resulted in an
appointment offered and the date
and time of the first available (i.e.,
soonest) appointment offered is
equal to or less than the minimum
time elapsed standard, enter “Y.” If
the value exceeds the minimum
time elapsed standard, enter “N.”

Include the provider in
the numerator if an
appointment is available
within time elapsed
standard (as reflected by
a Compliance_Indicator
of “Y.”)

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Do not include the
provider in the numerator
if an appointment is not
available within time
elapsed standard (as
reflected by a
Compliance_Indicator
of “N.”)
Include the provider in
the denominator
regardless of
Compliance_Indicator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Appointment availability
information obtained (i.e.,
an appointment was
offered) from or on behalf
of an equivalent provider,
not the surveyed provider.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Record the equivalent provider’s
name and specialty type (either
“PCP” for primary care provider or
“BH” for behavioral health provider)
in the “General Notes” field. The
format for entry into the “General
Notes” field is “Contact Attempt # Disposition Code - Text Directed
For The Disposition Code”. For
example: “Contact1-B-BH-Provider
Name”.
Calculate and record the
appointment wait time length and
record the number of business days
in the Time_Elapsed field.
For the “Compliance_Indicator”
field: If the time elapsed (in full
business days) between the date of
the call that resulted in an
appointment offered and the date
and time of the first available (i.e.,
soonest) appointment offered is
equal to or less than the minimum
time elapsed standard, enter “Y.” If
the value exceeds the minimum
time elapsed standard, enter “N.”

Include the provider in
the numerator if an
appointment is available
within the time elapsed
standard (as reflected by
a Compliance_Indicator
of “Y.”)

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider was
nonresponsive: Phone
number not valid (for
example, the phone
number is disconnected
and/or the phone number
is not associated with the
provider).

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
If a provider’s phone number is not
valid, the secret shopper should not
make two additional contact
attempts using the same invalid
phone number. However, the secret
shopper may complete additional
contact attempts, not to exceed
three, using different phone
numbers.
In the “General Notes” field, record
why the phone number was invalid.
For example, if the phone number
was disconnected or no longer in
service, document “phone number
disconnected” in the General Notes
field for the contact attempt. If the
phone number was not associated
with the provider, document “phone
number not associated with
provider” in the General Notes field
for the contact attempt. The format
for entry into the “General Notes”
field is “Contact Attempt # Disposition Code - Text Directed
For The Disposition Code”. For
example: "Contact1-C-phone
number disconnected”.

Exclude from the
calculation; do not
include the provider in
either the numerator or
the denominator.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation
Include the
provider in the
numerator and
denominator.

Replace with
provider from
reserve sample?
Yes, replace the
provider after the
first contact
attempt.

For the Compliance_Indicator field:
Enter “N/A.”

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider was
nonresponsive: Provider’s
phone rang for 5 minutes
without answer.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Determine whether all three contact
attempts resulted in nonresponsive
and ineligible disposition codes.
• If that is the case, for the
“Compliance_Indicator” field,
enter “N/A.”
• If that is not the case, follow the
instructions for reporting
associated with the disposition
code that resulted from the
responsive and eligible contact
attempt.

Determine whether all
three contact attempts
resulted in
nonresponsive and
ineligible disposition
codes.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation
Determine
whether all three
contact attempts
resulted in
nonresponsive
and ineligible
disposition codes.

Replace with
provider from
reserve sample?
Determine whether
all three contact
attempts resulted
in nonresponsive
and ineligible
disposition codes.

• If that is the
• If that is the case,
• If that is the
case, yes,
exclude the provider
case, include
replace the
from compliance
the provider in
provider.
calculation; do not
the numerator
include the provider in
• If that is not the
and
the numerator or
case, no, do not
denominator.
denominator.
replace the
• If that is not the case, • If that is not the
provider.
follow the instructions
case, do not
for compliance rate
include the
calculation for the
provider in the
disposition code
numerator and
associated with the
include in the
responsive or eligible
denominator.
contact attempt to
determine whether the
provider should be
included in the
numerator and
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider was
nonresponsive: Voicemail
greeting received during
regular business hours,
not during a designated
lunch hour, and the
voicemail greeting was
not part of an appointment
scheduling procedure.

Determine whether all
three contact attempts
resulted in
nonresponsive and
ineligible disposition
codes.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Determine
whether all three
contact attempts
resulted in
nonresponsive
disposition codes.

Determine whether
all three contact
attempts resulted
in nonresponsive
and ineligible
disposition codes.

• If that is the case,
• If that is the
• If that is the
exclude from
case, include
case, yes,
compliance
the provider in
replace the
calculation; do not
the numerator
provider.
include the provider in
and
• If that is not the
the numerator or
denominator.
case, no, do not
denominator.
• If that is not the
replace the
• If that is not the case,
case, do not
provider.
follow the instructions
include the
for compliance rate
provider in the
calculation for the
numerator and
disposition code
include the
associated with the
provider in the
responsive or eligible
denominator.
contact attempt to
determine whether the
provider should be
included in the
numerator and
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider was
nonresponsive: secret
shopper was placed on
hold for more than 15
minutes or secret shopper
was navigating a phone
tree/IVR for 20 minutes,
inclusive of the 15
minutes they may be
placed on hold.

Determine whether all
three contact attempts
resulted in
nonresponsive and
ineligible disposition
codes.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Determine
whether all three
contact attempts
resulted in
nonresponsive
disposition codes.

Determine whether
all three contact
attempts resulted
in nonresponsive
and ineligible
disposition codes.

• If that is the
• If that is the
• If that is the case,
case, include
case, yes,
exclude from
the provider in
replace the
compliance
the numerator
provider.
calculation; do not
and
include the provider in
• If that is not the
denominator.
the numerator or
case, no, do not
denominator.
• If that is not the
replace the
• If that is not the case,
case, do not
provider.
follow the instructions
include the
for compliance rate
provider in the
calculation for the
numerator and
disposition code
include the
associated with the
provider in the
responsive or eligible
denominator.
contact attempt to
determine whether the
provider should be
included in the
numerator and
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider was
nonresponsive: Call could
not be completed due to
communication issue(s)
such as poor call
connectivity.

Provider is not currently
accepting new patients;
appointment not offered
with equivalent provider.

Determine whether all
three contact attempts
resulted in
nonresponsive and
ineligible disposition
codes.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Determine
whether all three
contact attempts
resulted in
nonresponsive
disposition codes.

Determine whether
all three contact
attempts resulted
in nonresponsive
and ineligible
disposition codes.

• If that is the case, for the
“Compliance_Indicator” field,
enter “N/A.”
• If that is not the case, follow the
instructions for reporting
associated with the disposition
code that resulted from the
responsive and eligible contact
attempt.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field. Record the
appointment wait time length as
“N/A” in the “Time_Elapsed” field.
For the “Compliance_Indicator”
field: Enter “N.”

Do not include the
provider in the
numerator.
Include the provider in
the denominator.

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Code

Code definition

Instructions for
compliance rate
calculation

Instructions for reporting

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Provider is not currently
accepting new patients,
appointment not offered
with equivalent provider,
and no estimated
availability and/or waitlist
information provided.

Do not include the
provider in the
numerator.
Include the provider in
the denominator.

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Provider is not currently
accepting new patients,
appointment not offered
with equivalent provider,
and estimated availability
and/or waitlist information
was provided.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Calculate and record the
appointment wait time length and
record the number of business days
in the Time_Elapsed field.
If the time elapsed (in full business
days) between the date of the call
that resulted in an appointment
offered and the date and time of the
first available (i.e., soonest)
appointment offered is equal to or
less than the minimum time
elapsed standard, enter “Y.” If the
value exceeds the minimum time
elapsed standard, enter “N.”

Include the provider in
the numerator if an
appointment is available
within the time elapsed
standard (as reflected by
a Compliance_Indicator
of “Y.”)
Do not include the
provider in the numerator
if an appointment is not
available within time
elapsed standard (as
reflected by a
Compliance_Indicator
of “N.”)
Include the provider in
the denominator
regardless of
Compliance_Indicator.

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Code
K

Code definition
Provider is not currently
accepting new patients;
appointment was offered
with equivalent provider.

Instructions for
compliance rate
calculation

Instructions for reporting
Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Record the equivalent provider’s
name and specialty type (either
“PCP” for primary care (routine)
provider or “BH” for behavioral
health provider) in the “General
Notes” field. The format for entry
into the “General Notes” field is
“Contact Attempt # - Disposition
Code - Text Directed For The
Disposition Code”. For example:
“Contact1-K-PCP Provider
Name”. Calculate and record the
appointment wait time length and
record the number of business days
in the Time_Elapsed field.
If the time elapsed (in full business
days) between the date of the call
that resulted in an appointment
offered and the date and time of the
first available (i.e., soonest)
appointment offered is equal to or
less than the minimum time
elapsed standard, enter “Y.” If the
value exceeds the minimum time
elapsed standard, enter “N.”

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Code

Code definition

Instructions for reporting

Provider is ineligible:
Provider is not contracted
with QHP issuer as innetwork provider and/or is
not accepting the secret
shopper’s insurance at
the time of the call.

Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field. The secret
shopper should not make additional
contact attempts. Results fields
pertaining to additional contact
attempts do not need to be reported
or populated.
For the Compliance_Indicator field:
Enter “N/A.”
If possible, enter additional
descriptive information about the
provider in the “Comments” field.

Exclude from the
calculation; do not
include the provider in
either the numerator or
denominator.

Include the
provider in the
numerator and
denominator.

Yes, replace the
provider after the
contact attempt
that resulted in this
disposition code.

Provider is ineligible: At
the time of the call, the
provider is not classified
as a primary care provider
specialty type or
behavioral health provider
specialty type according
to the specialty
types/descriptions
contained in Tables 2 and
3.

Exclude from the
calculation; do not
include the provider in
either the numerator or
denominator.

Include the
provider in the
numerator and
denominator.

Yes, replace the
provider after the
contact attempt
that resulted in this
disposition code.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Code

Code definition

Instructions for
compliance rate
calculation

Instructions for reporting

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Referral required as
prerequisite in order to
offer an appointment.

Exclude from the
calculation; do not
include the provider in
either the numerator or
the denominator.

Do not include the
provider in the
numerator.
Include the
provider in the
denominator.

Yes, replace the
provider after the
contact attempt
that resulted in this
disposition code.

Provider’s office required
more information than
secret shopper was able
to reasonably provide in
order to offer appointment
availability.

Exclude from the
calculation; do not
include the provider in
either the numerator or
the denominator.

Do not include the
provider in the
numerator.
Include the
provider in the
denominator.

Yes, replace the
provider after the
contact attempt
that resulted in this
disposition code.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Other: None of the
provided disposition
codes describe the call
outcome and no
appointment availability
information was obtained.

Determine whether all
three contact attempts
resulted in
nonresponsive and
ineligible disposition
codes.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Determine
whether all three
contact attempts
resulted in
nonresponsive
disposition codes.

Determine whether
all three contact
attempts resulted
in nonresponsive
and ineligible
disposition codes.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Code

Code definition

Instructions for reporting

Provider’s office requires
secret shopper to leave a
voicemail with specific
details to receive a call
back from the office to
proceed with scheduling.

Exclude from the
calculation; do not
include the provider in
either the numerator or
denominator.

Do not include the
provider in the
numerator.
Include the
provider in the
denominator.

Provider is temporarily
unavailable due to
vacation, parental leave,
personal issue, illness,
etc. Appointment not
offered with equivalent
provider; no estimated
availability and/or waitlist
information provided.

Do not include the
provider in the
numerator.
Include the provider in
the denominator.

Do not include the No
provider in the
numerator.
Include the
provider in the
denominator.

Yes, replace this
provider after the
contact attempt
that resulted in this
disposition code.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider is temporarily
unavailable due to
vacation, parental leave,
personal issue, illness,
etc. Appointment not
offered with equivalent
provider; estimated
appointment availability
and/or waitlist information
was provided.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Do not include the No.
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Code
T

Code definition
Provider is temporarily
unavailable due to
vacation, parental leave,
personal issue, illness,
etc., appointment was
offered with equivalent
provider.

Instructions for
compliance rate
calculation

Instructions for reporting
Document this disposition code in
the corresponding contact attempt’s
“Disposition Code” field.
Record the equivalent provider’s
name and specialty type (either
“PCP” for primary care (routine)
provider or “BH” for behavioral
health provider) in the “General
Notes” field. The format for entry
into the “General Notes” field is
“Contact Attempt # - Disposition
Code - Text Directed For The
Disposition Code”. For example:
“Contact1-T-BH-Provider Name”.
Calculate and record the
appointment wait time length and
record the number of business days
in the Time_Elapsed field.
If the time elapsed (in full business
days) between the date of the call
that resulted in an appointment
offered and the date and time of the
first available (i.e., soonest)
appointment offered is equal to or
less than the minimum time
elapsed standard, enter “Y.” If the
value exceeds the minimum time
elapsed standard, enter “N.”

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation

Replace with
provider from
reserve sample?

Do not include the No
provider in the
numerator.
Include the
provider in the
denominator.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Instructions for
compliance rate
calculation

Code

Code definition

Instructions for reporting

Provider’s office requires
secret shopper to consult
an online scheduling tool
and/or use an online
patient portal to view
appointment availability.

Exclude from the
calculation; do not
include the provider in
either the numerator or
denominator.

Instructions for
percentage of
nonresponsive
and ineligible
providers
calculation
Do not include the
provider in the
numerator.
Include the
provider in the
denominator.

Replace with
provider from
reserve sample?
Yes, replace this
provider after the
contact attempt
that resulted in this
disposition code.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Appendix C. Key Terms
The terms listed in this section are defined for the purpose of implementing the appointment wait time
secret shopper surveys. These definitions are not meant to establish a new meaning for these terms
outside of this technical guidance document.
•

Equivalent provider. An equivalent provider is defined as another in-network provider
practicing and classified as the same specialty type within the same practice location and/or
serving enrollees in the same service area.

•

Facility provider types. For the purposes of this guidance document, a facility provider type is
not an individual provider and may include, but is not limited to, the following: Federally
Qualified Health Centers (FQHCs); outpatient health programs/facilities operated by Indian
tribes; tribal organizations; programs operated by urban Indian organizations; Ryan White
HIV/AIDS program provider facilities; Title X family planning clinics and Title X “look-alike”
family planning clinics; Indian Health Service (IHS) provider facilities, Indian tribal facilities,
tribal organization facilities, and urban Indian organization facilities; disproportionate share
hospitals (DSH); DSH-eligible hospitals; children’s hospitals; rural referral centers; sole
community hospitals; freestanding cancer centers; critical access hospitals; sexually transmitted
disease (STD) clinics; tuberculosis (TB) clinics; hemophilia treatment centers; black lung
clinics; ambulatory surgical centers; skilled nursing facilities; long-term care hospitals; rural
health centers; and ambulatory surgical centers. The provider population file will not include
facility provider types.

•

Ineligible provider. A provider who is ineligible to be surveyed (and therefore must be
replaced by a randomly selected provider from the reserve sample) if, at the time of the call, the
provider is: (a) no longer contracted with the issuer as an in-network provider and/or (b) the
provider is not practicing and classified as a primary care provider specialty type or behavioral
health provider specialty type according to the specialty types/descriptions listed in Tables 2
and 3.

•

Nonresponsive provider. A provider is defined as nonresponsive if any of the following
dispositions are documented for each of three contact attempts associated with the same
provider: (a) The provider’s phone rang for 5 minutes without an answer; (b) The provider’s
phone number is not valid (for example, the number is disconnected or the phone number is not
associated with the provider). If a provider’s phone number is not valid, the secret shopper
should not make two additional contact attempts before replacing the provider with another
provider from the reserve sample; (c) A voicemail greeting was received during regular
business hours, not during a designated lunch hour, and the voicemail greeting was not part of
an appointment scheduling procedure; (d) The secret shopper was placed on hold for more than
15 minutes; or (e) the call could not be completed due to communication issue(s) such as poor
call connectivity.

•

Oversample. A statistically valid, randomly selected population of providers that is 50% above
the minimum sample that must be surveyed based on QHP provider network size. This
oversample consists of a minimum sample to be surveyed and a reserve sample. QHP issuers’
third-party entities will draw from the reserve sample to replace providers that are
nonresponsive or ineligible.

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

•

Provider population file. A file that CMS will provide to issuers in fall 2024 containing the
population or “universe” of providers from which to draw a randomized oversample (consisting
of minimum sample to be surveyed and a reserve sample). The population of providers
contained in this file will be derived from provider data submitted by issuers on the “Network
Adequacy Provider” tab in their Network Adequacy template, submitted during the QHP
certification process. The provider population file will include submitted providers that CMS
has validated and approved to count toward the issuer’s satisfaction of the time and distance
standards for the corresponding specialty types.

•

Surveyed provider. An individual, eligible and responsive provider contacted by a secret
shopper to be surveyed.

•

Telehealth. For this purpose, CMS defines telehealth as “professional consultations, office
visits, and office psychiatry services through brief communication technology-based
service/virtual check-in, remote evaluation of pre-recorded patient information, and interprofessional internet consultation.”

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Appendix D. General Tips and Script Suggestions
General Tips:
•

If third-party entities’ secret shoppers are asked to provide information they could not
reasonably know, they can say that they do not know and can offer to bring the information to
the office when they come in for an appointment. If they are required to provide information
before appointment availability can be provided, third-party entities’ secret shoppers may
indicate that they can call back with the information and ask what is the soonest day and time
they can get an appointment.

•

If third-party entities’ secret shoppers are successful in obtaining a date and time for the next
available appointment, secret shoppers should be sure the appointment is not being held for
the secret shopper before getting off the phone.

•

If appointment schedulers’/providers’ offices find it challenging to provide information about
the earliest available appointment to the secret shopper without the secret shopper stating that
they have one or more specific medical conditions, third-party entities’ secret shoppers may
consider asking what the soonest available appointment would be for a general visit. For
example, secret shoppers may consider adding that they are new to the area and are trying to
get established with a provider.

•

If QHP issuers develop survey call scripts with their third-party entities that reference medical
conditions, the survey scripts should appropriately match the secret shopper or simulated
family member (e.g., gender, sex, age, race) in the survey protocol.

•

If third-party entities’ secret shoppers are asked about a patient preference for a male or female
doctor with an equivalent provider, third-party entities’ secret shoppers are encouraged to
indicate that they have no preference.

Script Suggestions:
The following is a list of questions and suggested responses that third-party entities’ secret shoppers
may consider using:
Appt Scheduler: I need your social security number in order to make an appointment.
Secret shopper: I never give that out over the phone. I will bring it in when I come.
Alternate: I don’t know it by heart and don’t have it with me. I can bring it when I come in.
Appt Scheduler: I need your insurance number in order to make an appointment.
Secret shopper: I don’t have it with me. I’ll bring it into the appointment with me.
Alternate: I can find it later and call back, but can you just tell me what day and time you can give me
an appointment for? I’m trying to work out my schedule.
Alternate: I understand that you will need to check my insurance first before you can firm this up, but
can you tell me when the soonest appointment is that you could schedule me for? I need to plan
childcare well in advance.
Appt Scheduler: I don’t find you in the insurance system.
Secret shopper: I am newly moved to the state, and they told me it could take a few days to show up in
the system.
41

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

Appt Scheduler: We’re busy right now—somebody will call you back later.
Secret shopper: Actually, I’m on someone else’s phone—can you give me a time when I can call you
back?
Alternate: I can’t take calls unless I’m on break. What would be a good time I can try back?
Appt Scheduler: Who is your employer—where is your insurance coming from?
Secret Shopper: My insurance is through my spouse. We just moved here and now I’m not going to be
able to remember the exact name of their new firm. It’s something-or-other Systems. My spouse is a
sales representative.
If asked for an address for spouse’s place of work:
Secret shopper: (He’s /She’s) a sales representative and doesn’t really have a fixed office.
Alternate: I don’t actually know the address. But I can find it and bring it in.
If asked for a phone number for spouse:
Secret shopper: (He/She) only has (his/her) work cell and I can’t give out that number. Can I have
(him/her) call you later?
Alternate: (He/She) just got a new phone and I don’t know the number by heart yet.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of
information unless it displays a valid OMB control number. The valid OMB control number for this information
collection is 0938-1415. This information collection requires QHP issuers, including SADP issuers, to ensure
access to a sufficient number and geographic distribution of essential community providers (ECPs), where
available, that serve predominantly low-income, medically underserved individuals. The time required to
complete this information collection is estimated to average less than 1,458 hours per response, including the
time to review instructions, search existing data resources, gather the data needed, to review and complete the
information collection. This information collection is mandatory under 2702(c) of the Public Health Service Act
and will be kept private in accordance with regulations at 45 CFR 155.260, Privacy and Security of Personally
Identifiable Information. If you have comments concerning the accuracy of the time estimate(s) or suggestions
for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Mail Stop C4-26-05, Baltimore, Maryland 21244-1850 and [email protected], Attention:
Information Collections Clearance Officer.
42

Appointment Wait Time Secret Shopper Survey Technical Guidance
for QHP Issuers in the FFEs

File Type	application/pdf
File Title	Appointment Wait Time Secret Shopper Survey Technical Guidance for Qualified Health Plan (QHP) Issuers in the Federally-facilita
Subject	appointment wait times, provider population, provider specialty types, sample, oversample, survey protocol, survey script, surve
Author	Centers for Medicare & Medicaid Services
File Modified	2024-12-10
File Created	2024-09-12