Form CMS-10398 (#44) CMS-10398 (#44) Customer Satisfaction Evaluation of Form CMS-416 Web-Bas

40398 – Oral Health Initiative MATHEMATICA POLICY RESEARCH

Task 2 –Evaluation of Form CMS-416 Web-based Training Modules

As part of the Option Year 1 scope of work for the Oral Health Initiative project, Mathematica is developing a brief web-based evaluation tool for users of the Form CMS-416 Dental Data Reporting web-based training developed under the base year of the contract. The training is designed to assist state Medicaid staff in improving the quality of the data they report on the Form CMS-416, and consists of six brief modules that users can take individually or together, and in any order.

The goal of the evaluation tool is to solicit feedback from the target audience of users of the training (state Medicaid staff) to identify potential modifications and improvements to the modules, which would be implemented later in Option Year 1. The evaluation tool has two thematic sections: 1) participant background; and 2) participant satisfaction and feedback.

The tool is presented in the following sections of this brief, with notes regarding Mathematica’s approach to the sections of the tool.

Introductory Language for Respondents

The Centers for Medicare & Medicaid Services (CMS) is conducting this survey to evaluate the usefulness of this web-based training on Form CMS-416 Dental Data Reporting, produced as part of the Oral Health Initiative. CMS will use your feedback to identify areas for improvement of the training to better support state Medicaid staff in reporting standardized and complete data on the Form CMS-416.

The survey should take 10-15 minutes to complete. While responses will be confidential and reported in the aggregate, we may be interested in contacting you in the future to gain a better understanding of your feedback. At the end of the survey, you will have the opportunity to provide your email address if you are willing to participate in additional follow-up.

If you have any questions about the survey or need technical support, please contact [identify point of contact for assistance].

We appreciate your time in offering input on the Form CMS-416 Dental Data Reporting web-based training!

Section 1: Your Background

1. Please indicate your agency, academic or organizational affiliation: [multiple choice, select one]

1. State or territory Medicaid/CHIP agency

2. Contractor to a state or territory Medicaid/CHIP agency

3. Medicaid/CHIP Managed Care Organization

4. State or territory public health agency

5. Federal agency

6. Contractor to a federal agency

7. Stakeholder organization (for example, child advocacy organization)

8. Provider

9. Academic institution

10. Other [write in, 100 characters]

2. Please indicate the state or territory in which you perform work related to oral healthcare for Medicaid/CHIP populations: [multiple choice, select one]

[50 states, territories and District of Columbia and include “national” and “other”, select one]

3. How did you learn about the training? [multiple choice, select one]

1. Colleague

2. CMS Oral Health Technical Advisory Group (OTAG) call

3. CMS website

4. CMS learning opportunity (e.g., webinar, learning collaborative)

5. Other [write in, 100 characters]

4. Which of the modules have you completed? If you do not recognize one of the modules or are not sure whether you have completed it, you can follow the link to view the training.  [multiple choice, select as many as apply]

1. Module 1: Overview of the Early and Period Screening, Diagnostic, and Treatment (EPSDT) Benefit

2. Module 2: Form CMS-416 Overview

3. Module 3: Form CMS-416 Specifications—Lines 1a and 1b

4. Module 4: Form CMS-416 Specifications—Lines 12a through 12e

5. Module 5: Form CMS-416 Specifications—Lines 12f and 12g

6. Module 6: Using Form CMS-416 Dental Data

5. Thinking about the training modules overall, please indicate your level of agreement with the statements below: [select one per statement]¹

	Strongly Agree	Agree	Disagree	Strongly Disagree
The objectives of the training were clearly defined.
The content was organized and easy to follow.
This training will be useful in my work.
The learning objectives of the training were met.
I would recommend this training to a colleague.

Section 2: Feedback on the training²

The following questions are about Module 2: Form CMS-416 Overview,³ which you indicated that you have completed.

1. Module 2 covered the following content:⁴

• Logistics of Form CMS-416 submission.

• Sources of data used in Form CMS-416 reporting.

• Assessing and improving data quality

Thinking about the content of Module 2, please indicate your level of agreement with the statements below: [select one per statement]⁵

	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree	Not Applicable
The material presented was accurate.
The content aligned with existing guidance.
The material was relevant to my agency/organization or myself.
The information was sequenced logically (from basic to specialized, simple to complex).
The exercises enhanced my understanding of the material.
The additional resources provided were comprehensive and useful.

2. The learning objectives for Module 2 are to:⁶

• Know how to submit the Form CMS-416 and how to get help.

• Understand the partners and processes involved with collecting and reporting Form CMS-416 data.

• Be familiar with the general principles of data quality.

Overall, how well does the content of the module support these learning objectives? [multiple choice, select one]

1. Not at all well

2. Not very well

3. Somewhat well

4. Very well

5. Extremely well

3. Below is a selection of a few of the slides that were included in Module 2.⁷ Recall that the overall objective of the modules is to help Medicaid/CHIP staff improve the quality of oral health data submitted on the CMS Form-416. Please rank these slides in order of most helpful in accomplishing that objective to least helpful in accomplishing that objective, where 1 is most helpful and 4 is least helpful:

[ ]

[ ]

[ ]

[ ]

4. You identified the following slide as the least helpful among the options presented:

[Insert graphic that respondent marked as “4” from the previous question]

Why did you make that selection? Please indicate specific features or concepts you feel should be included or changed to make this slide more helpful.

[free text field, up to 500 characters]

5. Thank you for your feedback! Do you have any other comments, questions, or suggestions about Module 2?

[free text field, up to 500 characters]

Section 3: Closing

1. Thank you for your feedback! If you have any additional comments or suggestions about how to improve the training, please enter it below:

[free text field, up to 500 characters]

2. Is this your first time completing an evaluation of the effectiveness of any of the web-based training modules on Form CMS-416 Dental Data Reporting?

[Yes/No, select one]

3. May we contact you with follow-up questions regarding your answers to this survey? If so, please enter your email address:

[free text field with check for valid email address structure, up to 100 characters]

We appreciate your time! For questions about this survey or the CMS Oral Health Initiative, please contact [identify appropriate point of contact].

PRA Disclosure Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0938-1148 (CMS-10398, #44). The time required to complete this information collection is estimated to average 15 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

1 For each statement where the respondent selects strongly disagree, a text box will appear asking for additional feedback and/or explanation.

2 Note that the survey will be structured such that the respondent will be asked to answer questions regarding one of the modules that they indicated they had completed in Section 1, Question 2. The survey will randomly select one of those modules per respondent until a predetermined quota of responses per module has been achieved (for example, once 15 respondents have answered questions pertaining to Module 3, the survey will cease to prompt respondents to answer questions about that module, generating questions about other modules instead).

3 For the purposes of this iteration, we will reference Module 2; however, the same questions will apply to other modules.

4 Note that these bullets will be modified based on the “content covered” section listed on slide 2 of each module.

5 For each statement where the respondent selects strongly disagree, a text box will appear asking for additional feedback and/or explanation.

6 Note that these bullets will be modified based on the “learning objectives” section listed on slide 2 of each module.

7 We will ask this question for a selection of slides from each module. The slides we propose to ask respondents about are:

• Module 1: Slides 3, 6, and 9 [for this question, we’ll only ask respondents to rank from 1 to 3]

• Module 3: Slides 4, 8, 12, and 17

• Module 4: Slides 5, 7, 9, and 22

• Module 5: Slides 5, 7, 12, and 19

• Module 6: Slides 5, 6, 7, and 14