Form CMS-10896 CCSQ Support Central Live Chat Virtual Agent Survey

Request for Approval under the “Generic Clearance for the Center for Clinical Standards and Quality IT Product and Support Teams” (OMB Control Number: 0938-1397)

TITLE OF INFORMATION COLLECTION: CCSQ Support Central Live Chat and Virtual Agent Enhancement Survey

PURPOSE OF COLLECTION:

The data collection assesses the necessity and potential benefits of enhancing the Live Chat/Virtual Agent feature within the CCSQ Support Central website. We aim to gather customer insights to develop human-centered enhancements for the Live Chat/Virtual Agent feature, improving the customer experience by upgrading to a more efficient self-service customer journey that focuses on personalized support.

The insights gathered will be instrumental in enhancing the user experience (UX), streamlining the customer to CCSQ Service Center Live Agent interaction, optimizing the virtual agent conversational flow, and improving communication efficiency with CCSQ Support Central customers. The data collected will be utilized by CMS staff and Agency contractors. The data collection will be designed to be useful while keeping public participation minimally burdensome, aligning with the requirements of the Paperwork Reduction Act.

TYPE OF COLLECTION: (Check one)

[ ] Card Sorting [ ] Cognitive Testing

[ ] Field Studies [ ] First Click Tests

[ ] Focus Groups [ ] Participatory Design

[X] Survey [ ] Tree Testing

[ ] User Interviews [ ] Usability Testing

[ ] Other: _______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: __John “Jay” Vancil - [email protected] ____

To assist review, please provide answers to the following question:

PERSONALLY IDENTIFIABLE INFORMATION

1. Is personally identifiable information (PII) collected? [X] Yes [] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [X] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

GIFTS OR PAYMENTS

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [X] No

If Yes, describe:

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
CCSQ CMS Service Center Customers (Individuals) – Survey	2,000	0.083 hours (5 minutes)	167 hours
Totals

FEDERAL COST

The estimated annual cost to the Federal government is __N/A____

There is no additional cost for this survey implementation, as it is part of normal contractual duties and work prioritized by CMS stakeholders.

ESTIMATED PARTICIPANT COST BURDEN

Form	Total Burden Hours	Average Hourly Wage Rate*	Total Cost Burden
Online Survey	167	** $29.76	$4,970

*The May 2022 National Employment and Wage Estimates reported by the Bureau of Labor Statistics indicate an average hourly wage of $29.76 across the 50 U.S. states and the District of Columbia. The national average has been used to estimate the wages of survey respondents.

** Based on the mean hourly wage for all occupations, code 00-0000.

ACTIVITY DETAILS

1. How will you collect the information? (Check all that apply)

[X] Web-based or other forms of social media

[ ] Telephone

[] In-person

[] Mail

[] Other, Explain.

2. Will interviewers or facilitators be used? [ ] Yes [X] No

3. Who will you collect the information from?

The survey targets various groups within the CMS QualityNet community, such as CMS employees, federal contractors, clinicians, hospital staff, providers, and vendors, all of whom are integral users of the CCSQ Service Center. Their input is crucial for us to enhance our digital offerings and the overall customer experience with CMS tools and services.

To gather a broad range of perspectives, we plan to recruit participants by emailing individuals from our Human-Centered Design (HCD) Participation list in CMS CCSQ ServiceNow who have agreed to email outreach and research participation. This approach ensures we can collect feedback from customers who have opted in to HCD research efforts and are aware of the Live Chat/Virtual Agent feature.

Customers will only need to complete the survey once, and the survey platform will notify them if they attempt submitting another survey.

Participation in the survey is voluntary and at the customer's discretion.

Describe the people you will interact with or collecting information from and why the group is appropriate for the program / service to connect with. Please provide a description of how you plan to identify your potential group of respondents and if only a sample will be solicited for feedback, how you will select them (e.g., anyone who provided an email address to a call center representative, a representative sample of administrators who downloaded a report in May 2021, intercept interviews at a particular field office, a list of customers, e.g., a CRM database that has contact information, to reach out to that defines the universe of potential respondents and have a sampling plan for selecting from this universe). Attach a copy of your sampling plan if applicable.

4. How will you ask a respondent to provide this information?

We intend to distribute the survey to the CCSQ community by leveraging the QualityNet Mailer tool for email outreach. An email will be sent to our established mailing list, detailing the purpose of the data collection and including a link for those interested in taking the survey. For site awareness, we plan to promote the survey in a top banner on the CCSQ Support Central website. Additionally, we will promote the survey link in the CCSQ Service Center's quarterly newsletter. By employing multiple outreach strategies, we aim to engage our customer base effectively while ensuring that we collect only one response per participant.

For example, after an inquiry is submitted online, the final screen will present the opportunity to provide feedback by presenting a link to a feedback form / an actual feedback form.

5. What will the activity look like?

The survey will be on the CMS CCSQ SurveyMonkey platform, where all customer responses will be securely stored. This survey uses a technique known as "split logic" or "branching logic" to tailor the set of questions based on the respondent's answer to the first question on the survey. The purpose of utilizing the split logic in this survey ensures that the questions are relevant to the respondent's experience and knowledge, increasing the likelihood of meaningful responses. It should take approximately five minutes or less to complete the survey.

Additionally, it will keep respondents engaged by avoiding questions that don't apply to them, thereby reducing survey fatigue and dropout rates. We aim to gather specific insights from different segments of the customer base, which can inform more nuanced strategies for customer engagement, support, and education. By dividing the survey based on the initial awareness question, we can gather targeted insights into how aware and satisfied their customers are with specific features, and what barriers exist for those who are not aware. This can guide improvements in feature promotion, user experience, and overall customer satisfaction.

Here's a more detailed explanation of the process:

1. Survey Paths Based on Initial Answer

Depending on how the respondent answers the first question, they are directed to one of two distinct sets of questions, each set representing a different persona.

2. Persona A: Customers Aware of the Live Chat/Virtual Agent Feature

Path Trigger: The respondent answers "Yes" to the initial question.

Survey Content: This survey version is designed for customers who are already aware of the Live Chat/Virtual Agent feature. It contains about eleven (11) questions. The questions are designed to understand the respondent's experience and satisfaction with the feature, gather feedback on its usability, effectiveness, and possibly explore new capabilities and context of its use. The question types include satisfaction ratings, multiple choice questions, and open-ended text responses.

3. Persona B: Customers Unaware of the Feature

Path Trigger: The respondent answers "No" to the initial question.

Survey Content: This version is for customers who were not aware of the Live Chat/Virtual Agent feature before the survey. It includes about ten (10) questions. The questions for this group focus on understanding the respondent's current use of support channels, their potential openness to using such a feature, and what features or information could increase their likelihood of using it in the future. Like Persona A, the question types include satisfaction ratings, multiple-choice options, and open-ended responses but are framed within the context of their current knowledge and use of support options.

Describe the information collection activity – e.g., what happens when a person agrees to participate? Will facilitators or interviewers be used? What is the format of the interview/focus group? If a survey, describe the overall survey layout/length/other details. If User Testing, what actions will you observe / how will you have respondents interact with a product you need feedback on. 

6. Please provide your question list.

Persona A - CCSQ Support Central Live Chat Virtual Agent Survey Questions

Introduction: On behalf of the CMS CCSQ Service Center, we thank you for participating in this survey research effort! This survey aims to collect your valuable perspectives on improving the Live Chat/Virtual Agent functionality on the CCSQ Support Central website. Your feedback is highly valued as we continuously work towards improving our self-service offerings, and it will significantly influence our efforts to better serve your needs. 

The Virtual Agent within the Live Chat feature on the CCSQ Support Central is known as CARL (Chat and Resource Line). This Virtual Agent acts as an intelligent routing assistant collecting a customer’s name, authenticating their identity and routing them to the appropriate CCSQ Service Center Representative for a Live Chat experience based on the customer’s program or line of business.

In alignment with the Paperwork Reduction Act (PRA), the approved Office of Management and Budget (OMB) control number for this data collection is XXXX-XXX

1. Before this survey, have you ever used the Live Chat/Virtual Agent feature available on the CCSQ Support Central website? 

• Yes

• No 

 

2. Which program are you affiliated with? 

• EPCS - Electronic Prescribing for Controlled Substances 

• EQRS - End Stage Renal Disease Quality Reporting System 

• HQR - Hospital Quality Reporting 

• iQIES/QIES - Quality Improvement/Internet Quality Improvement & Evaluation System 

• QPP - Quality Payment Program 

• CCSQ Services and Operations Support - QualityNet IT Services, HARP, CCSQ Atlassian, ServiceNow & Slack 

• Other [Please Specify] 

3. How would you describe your first experience with the Live Chat and Virtual Agent CARL feature? 

• Very easy and straightforward 

• Somewhat easy 

• Neutral 

• Somewhat difficult 

• Very difficult 

 

4. When encountering issues or needing assistance, do you find that most of these situations:  

• Requires a direct phone conversation with a Service Center Agent.

• Could be effectively resolved through self-service options such as using a Live Chat/Virtual Agent Feature.

 

5. If you could improve one feature of the CCSQ Support Central Live Chat Virtual Agent experience, what would it be and why? 

• (open text field) 

 

6. Are there any specific features or functionalities from other virtual agents that you would like to see in CARL? 

• Advanced search capabilities 

• Answering FAQs 

• Use the Live Chat/Virtual Agent CARL feature on other CMS QualityNet Program websites (e.g., QPP, HQR, EQRS, iQIES/QIES) 

• Other (Please specify) 

7. How valuable would it be for CARL to suggest articles or resources in response to your queries? 

• Extremely valuable 

• Very valuable 

• Moderately valuable 

• Slightly valuable 

• Not at all valuable 

 

8. Imagine a Virtual Agent that can seamlessly fetch and integrate information from various QualityNet program websites (QPP, iQIES/QIES, EQRS, HQR, etc.) to answer your queries. Would this feature be useful for you? 

• Yes

• No

• Not Sure

 

9. If accessing the Live Chat/Virtual Agent service on the CCSQ Support Central website required you to log in through an authenticated portal, would this impact your likelihood of using the feature? 

• Yes 

• No 

• Unsure 
   

 

10. Under what circumstances do you choose to call the CCSQ Service Center instead of using the Live Chat/Virtual Agent CARL feature? 

• [open-text field] 

 

11. Would you be open to further outreach from us for additional feedback or participation in future customer research? 

• Yes (Please provide your name and email address) 

• No 

Closing Survey: 

Thank You for Completing Our Survey! 

We appreciate the time you have taken to provide your valuable insights. Your feedback is essential in helping us improve the customer experience and better meet your needs. Should you have any additional feedback or questions, please feel free to reach out to the CCSQ Service Center Human-Centered Design (HCD) team at [email protected] 

-----------------------------------------------------------------------------------------------------------------


Persona B - CCSQ Support Central Live Chat Virtual Agent Survey Questions

Introduction: On behalf of the CMS CCSQ Service Center, we thank you for participating in this survey research effort! This survey aims to collect your valuable perspectives on improving the Live Chat Virtual Agent functionality on the CCSQ Support Central website. Your feedback is highly valued as we continuously work towards improving our self-service offerings, and it will significantly influence our efforts to better serve your needs. 

The Virtual Agent within the Live Chat feature on the CCSQ Support Central is known as CARL (Chat and Resource Line). This Virtual Agent acts as an intelligent routing assistant collecting a customer’s name, authenticating their identity and routing them to the appropriate CCSQ Service Center Representative for a Live Chat experience based on the customer’s program or line of business.

In alignment with the Paperwork Reduction Act (PRA), the approved Office of Management and Budget (OMB) control number for this data collection is XXXX-XXX.  

1. Before this survey, have you ever used the Live Chat/Virtual Agent feature available on the CCSQ Support Central website? 

• Yes 

• No 

2. Which program are you affiliated with? 

• EPCS - Electronic Prescribing for Controlled Substances 

• EQRS - End Stage Renal Disease Quality Reporting System 

• HQR - Hospital Quality Reporting 

• iQIES/QIES - Quality Improvement/Internet Quality Improvement & Evaluation System 

• QPP - Quality Payment Program 

• CCSQ Services and Operations Support - QualityNet IT Services, HARP, CCSQ Atlassian, ServiceNow & Slack 

• Other [Please Specify] 

 

3. What has been your primary method of seeking support? 

• Call the CCSQ Service Center to speak with an agent

• Send an email to the CCSQ Service Center

  • Create a Ticket on the CCSQ Support Central website

• Other (Please specify) 

 

4. For which of the following issues would you most likely consider using a Live Chat/Virtual Agent to help resolve: (Select all that may apply) 

• Program Questions - For queries about measurement specifications, website content, file format requirements, or program details. 

• Policy Questions - For questions on data collection procedures, data types, final rule, or legislature about the program. 

• Technical Questions - If you’re experiencing difficulties with data submission, error messages, or website functionality. 

• Account Management Questions - For issues related to account roles, account registration, or password recovery. 

• Other - For general support or you require other forms of assistance. 

 

5. Imagine a Virtual Agent that can seamlessly fetch and integrate information from various QualityNet program websites (QPP, iQIES/QIES, EQRS, HQR, etc.) to answer your queries. How useful would this feature be for you? 

• Extremely Useful 

• Very Useful 

• Moderately Useful 

• Slightly Useful 

• Not Useful at all 

 

6. Are there any specific features or functionalities from other virtual agents that you would like to experience?

• Advanced search capabilities 

• More comprehensive FAQs 

• Using a Live Chat/Virtual Agent feature on other CMS QualityNet Program websites (e.g., QPP, HQR, EQRS, iQIES/QIES) 

• Other (Please specify) 

 

 

7. What concerns, if any, do you have about using a Virtual Agent for your support needs? (Select all that may apply) 

• Clarity of responses 

• Privacy of information 

• Response time 

• Accuracy of information 

• Other (Please specify) 

 

8. If accessing the Live Chat/Virtual Agent feature on the CCSQ Support Central website required you to log in through an authenticated portal, would this impact your likelihood of using the feature?

• Yes 

• No 

• Unsure 
   
   

9. Under what circumstances would you choose to call the CCSQ Service Center instead of using a live chat/virtual agent feature?

• (open text field) 

  

 

10. Would you be open to further outreach from us for additional feedback or participation in future customer research? 

• Yes (Please provide your name and email address) 

• No 

 

Closing Survey: 

Thank You for Completing Our Survey! 

We sincerely appreciate the time and effort you've dedicated to sharing your insights with us. Your feedback plays a crucial role in enabling us to enhance the customer experience and more effectively address your needs. 

We're excited to invite you to explore new self-service features available on the CCSQ Support Central website, including the Live Chat Virtual Agent and the option to Schedule A Call. These tools have been designed to offer you a more seamless and efficient support experience, catering to a wide range of inquiries and support needs. 

Should you have any further feedback, questions, or require assistance, please don't hesitate to contact the CCSQ Service Center Human-Centered Design (HCD) team at: [email protected]. 

Please make sure that all instruments, instructions, and scripts are submitted with the request.

7. When will the activity happen?

The timeframe for the data collection will be from the end of April until early June 2024.

We intend to close the survey in June 2024, or once we reach 2,000 responses, whichever comes earlier.

The survey will remain available in alignment with the timing of the overall clearance, or close prior, should we obtain the responses we need.

Describe the time frame or number of events that will occur (e.g., We will conduct focus groups on May 13,14, 15; We plan to conduct customer intercept interviews over the course of the Summer at the field offices identified in response to #2 based on scheduling logistics concluding by Sept. 10; or This survey will remain on our website in alignment with the timing of the overall clearance.)

Instructions for completing Request for Approval under the “Generic Clearance for the Center for Clinical Standards and Quality IT Product and Support Teams”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request (e.g. Comment card for soliciting feedback on xxxx).

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive, and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g., fill out a survey or participate in a focus group).

Burden: Provide the Annual burden hours: Multiply the number of responses and the participation time and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

ACTIVITY DETAILS: Complete each section as described.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide a description of how you plan to identify your potential group of respondents and how you will select them.

Submit all instruments, instructions, and scripts are submitted with the request