CMS-10754 Observed Testing Protocol

Observed Testing Protocol (March 7, 2022)

INTRODUCTION AND PURPOSE OF THE INTERVIEW

Thank you for agreeing to participate in the interview session today. I’d like to briefly review the purpose and the confidentiality provisions that were described in the email we sent you.

As you know, CMS is required by Congress to develop a data collection system to collect data on cost, revenue, utilization, and other information from ground ambulance providers and suppliers (which we collectively call “ground ambulance organizations”). This information will be used to assess the relationship between costs and Medicare payment rates for ground ambulance services. In the future, a representative sample of ground ambulance organizations will be required to report this information or face a 10 percent reduction in payment.   

RAND is working under contract with CMS to support the development and implementation of this system. We asked a small number of ambulance organizations, including yours, to walk through the programmed instrument with us so we can learn from your experience. The feedback you provide today will help us determine if the instrument questions are relevant and easy to understand in a way that is consistent with what we intended, if the tool is presented in a way that enables people to easily find what they are looking for, and if the aids are relevant and useful to organizations. , The information we gather from these interviews will be used by RAND to inform its recommendations to CMS on how to improve the instrument.

During our session today, we will request that you screen share with us so we observe how you interact with the online programmed instrument. As you move through, we will ask you to “think aloud” to tell us your reactions and how you are answering each item. We will also chime in periodically with questions to better understand your process and how you were interpreting the items as they relate to your organization. We will be taking notes throughout

CONSENT/CONFIDENTIALITY PROVISIONS

• Taking part in this interview is completely voluntary.

• You can decline to answer any question asked in the course of the session, or stop at any point.

• We will not disclose your identity outside of the research team, and we will not link your responses to you in any reports that may result from our interview. 

• No one outside of the research team will have direct access to the information you provide.

Do you have any questions? (Yes/No)

Do you agree to participate? (Yes/No)

We will be taking notes throughout the session and we will not be recording today. We will destroy the notes after we create a summary of findings for CMS.

NOTE ON TIMING:

Before we dive in, I’d like to cover some logistics around timing. We want to be respectful of your time. I’m going to do my best to keep moving forward at a pace to get through all of the questions within that time. If you have additional feedback you’d like to share that we can’t get to, we’re happy to receive this by email or to set up another call to discuss.

Specific Notes for Interviewer:

1. Please do not help the respondent fill out the information. They might ask questions, but we are interested in understanding how they would handle the question were we not immediately there. Avoid correcting responses you suspect are incorrect. Probing to more fully understand the response and reasoning is appropriate. Note throughout and document whenever respondent asks for help or confirmation.

2. Make notes of how/if/when users access auxiliary information such as taking time to read instructions, using pop-up screens, hovering over definitions that relate to the functionality of the programmed instrument.

   1. We are especially interested in times when respondents are not using these facilitators throughout the instrument.

3. The answers to Sections 2 (Organizational Characteristics) and 3 (Service Area) of the instrument control the questions the organization sees later on. Make a note of whether respondents find any issues with what they selected in terms of what appeared later in the instrument, e.g., does the skip logic work as intended for this organization.

1. GENERAL QUESTIONS

I’d like to ask you a few questions about the process for completing the instrument.

• Other than yourself, was anyone else at your organization or at other organizations involved in preparing for today’s session?

  • If yes, can you tell us their job categories or which organization they work with?

• About how long did you take to prepare for today’s session?

2. SECTION BY SECTION REVIEW

Dashboard

1. Tell me about your experience with the Overview page. What information did you review? How helpful did you find the instructions??

   1. Did you read the instructions? What, if anything, did you learn from them?

      1. [Interviewer note]: Note if the respondent spent time on this page.

2. Was it clear how to launch the survey from the Dashboard page?

   1. [Interviewer note]: Note which button they pick first, the one next to the specific section or the one at the bottom.

Section 1: General Survey Instructions

3. In your own words can you tell me what information was conveyed in the General Instructions?

   1. Would you download the printable instrument from here? Why or why not?

Section 2: Organizational Characteristics

1. Thinking back over this section (feel free to click back through the screens), other than what we’ve already talked about, were there any specific questions that were particularly confusing or difficult to fill out in this section?

   1. PROBE: Which ones and why? Do you have any suggestions on how to make the question clearer? How about the shared services questions?

2. [Interviewer notes]:

   1. Did respondents use the pop-up definition for full list of Medicare provider of services?

   2. Are respondents using the ‘other’ categories appropriately? E.g., I see that you used “other” at this item, can you tell me why you used that category instead of one of the other ones?

   3. Did the respondent confirm the definition of paramedic intercept before answering?

Section 3: Service Area

1. These questions ask first about a primary service area then a secondary service area. In your own words, what is a primary service area vs. a secondary one? How did you choose which zip codes belong in each?

2. Is this a distinction that makes sense for your organization? Why or why not?

3. [Interviewer notes]:

   1. How did the respondent enter zip codes? Manually one by one? By county? By state? Copy/paste?

   2. Did respondent catch that zip codes are requested separately between the primary and secondary service areas?

Section 4: Emergency Response Time

1. What definition of response time does your organization use? Were you able to report data in the way you wanted here? Why or why not?

Section 5: Ground Ambulance Service Volume

1. There are a lot of definitions in this section. Did you read through these? Why or why not?

2. Were there any definitions that you felt did not work for your organization?

3. Other than what we’ve already talked about, were there any specific questions that were particularly confusing or difficult to fill out in this section?

   1. PROBE: Which ones and why? Do you have any suggestions on how to make the question clearer?

Section 6: Service Mix

4. There are also a lot of definitions in this section, too. Did you read through these? Why or why not?

1. Did you feel like they made sense for your organization?

2. Were the percentages in this section ones your systems (or billing company) already report to you?

   1. If no, where did you go to find the information or how did you calculate them?

Section 7: Labor Costs

1. What were your takeaways from the instructions screen? Please tell me in your own words what information was conveyed here.

   1. [Interviewer Note: We are looking to see if they have a general understanding on what to report, e.g. that staff are reported only once, staff are reported to the response staff if they have any response duties, etc.]

2. This section includes instructions on how to categorize your staff. Were you unsure about how to categorize any of your staff? If so, please tell me more about that – what made you unsure?

   1. Are there any major labor categories in your ambulance organization that were not included in the instrument? If so, can you tell us about them?

3. Tell me a little about how you determined total compensation amounts including benefits.

   1. What are you including in your total compensation calculation?

      1. [Probe: benefits, paid time off, etc.]

   2. Is total compensation something you already calculate? .

   3. Did you struggle with this activity for any (or all) positions or did this feel straightforward? [If a struggle] What would have helped you?

4. [If the respondent says they employ a medical director]: Why did you choose to report medical director information in this section rather than subsequent sections?

5. [Here or in an adjacent section based on flow:] Can you click on the dashboard. Now please return to the section you were working on?

6. [Interviewer notes]:

   1. Did respondent check instructions?

   2. Did the respondent incorrectly respond to the volunteer labor question in Section 2?

   3. When asked, was respondent able to get back easily to the section they were working on?

Section 8: Facilities Costs

1. Are there any facilities that you didn’t list here? Why?

2. Did you report the allocated amount or total amount for these questions?

   1. [Interviewer note: respondents could report total square footage with an allocation factor or report the square footage attributable to ambulance with no allocation factor]

3. [If applicable] How did you allocate facility costs(if applicable)? How did you come up with this allocation approach?

4. [Interviewer notes]:

   1. Did the respondent hover over the column definitions?

Section 9: Vehicle Costs

1. Are there any vehicles that you didn’t list here? Why?

2. Did you report the allocated amount or total amount for these questions?

3. [If applicable] Did you use the same allocation approach as in previous sections? If no, what did you use for this section? How did you come up with this allocation approach?

4. [Interviewer notes]:

   1. Did the respondent hover over the column definitions?

Section 10: Equipment, Consumable and Supply Costs

1. Is there any equipment or supply cost that you didn’t list here? Why?

2. Did you report the allocated amount or total amount for these questions?

3. [If applicable] Did you use the same allocation approach as previous sections? If no, what did you use for this section? How did you come up with this allocation approach?

4. [Interviewer notes]:

   1. Did the respondent hover over the column definitions?

Section 11: Other Costs

1. I know there were a lot of costs listed in this section. Can you think of anything that we missed? Does your organization have costs that are not represented in this data collection instrument?

2. Were there any specific costs where you weren’t sure whether they belonged in this section or a prior section? [prompt] For example any contracted services?

3. Did you report an allocated or total amount for these questions?

4. [If applicable] Did you use the same allocation approach as previous sections? If no, what did you use for this section and why?

5. [Interviewer notes]:

   1. Did the respondent hover over the column definitions?

      1. Were the definitions clear?

   2. Did respondent report something here that should have been reported elsewhere? Probe as needed.

Section 12: Total Cost

1. [Interviewer notes]:

   1. Ambulance only organizations should have the total costs add up throughout the instrument. Is there a difference, and if so, why? Probe as needed.

Section 13: Revenues

1. This section asks for reporting revenue by payer. How was this exercise for you? Did you feel like it was straightforward or did you struggle to respond?

   1. Did you include non-transport revenue from these payers? If so, why not in the other revenue section?

2. Did you have to reach out to a billing company to report data for this section?

3. How does the reported revenue compare to the number from your annual financial reporting? Is it the same or different?

   1. . If it is different, why?

   2. [Probe: are bad debt and charity care included (e.g. include amounts that you billed but hadn’t collected)?]

4. [Interviewer notes]:

   1. Ask why they have reported data for specific fields if there is a concern about placement and/or if respondent seemed to have any difficulty.

Wrap up

1. Is there anything you think is missing from the instrument?

2. Did you have any other overall comments regarding the functionality of the programmed instrument?

3. Either before or during this session, did you view or use the Quick Reference Guide?

4. What about the Frequently Asked Questions (FAQ)? If yes, any thoughts or feedback?

5. Did you view or download the Printable Data Collection Instrument? If yes, any thoughts or feedback? Did you print it and use it to collect information?

Thanks very much for your time…

9