Businesses (Vendors)

APPENDIX A-3a

DETAILED SITE VISIT INTERVIEW PROTOCOL:
STATE CHILD NUTRITION AGENCY STAFF

This page has been left blank for double-sided copying.

Detailed Site Visit Interview Protocol: State Child Nutrition Agency Staff¹

Evaluation of Demonstrations of Direct Certification with Medicaid for Free and Reduced-Price Meals (DCM-F/RP)

State:	Date:
Interviewee(s)/position(s):	Start time:
	End time:
Permission to record:	Interviewer:

Introduction

The purpose of this interview is to gather information about [State’s] demonstration of direct certification of students for free or reduced-price meals using Medicaid data. This new demonstration is known by the acronym DCM-F/RP [for States that participated in the previous demonstration, add: to distinguish it from the earlier DCM demonstration].

The information that we collect in this interview will be used to describe the experiences of States participating in the demonstration. Because each State’s project is unique, the description of a particular State’s experiences could identify that State. We will not use your name in our reports, but your identity might be inferred from the identity of your State and the nature of the information that you provide. If there is something that you want to say in confidence that you would not otherwise mention, let us know and we’ll use it to enhance our understanding, but we will keep the details private. Although participation in the data collection is a requirement of the demonstration, your individual participation in this interview is voluntary, and you do not have to answer any questions you don’t want to answer.

The interview will last approximately one hour. With your permission, we’d like to make an audio recording of the interview to fill in gaps in our notes. Do I have your permission to record the interview?

[If yes:] Thank you.

[If no:] You have my assurance that we will keep anything private that you wish. If you’d prefer, we will not cite anything that you say verbatim from the recording. Wait for response; if yes, record: Thank you. If no: That’s OK. Just bear with us/me as I take detailed notes.

Overview

Before we discuss the details of your direct certification process, I’d like to learn more about your role in [CN agency] and [State’s] direct certification efforts more broadly.

1. What is your position here at [CN agency]?

2. What is your role in the DCM-F/RP process?

[Moderator note: Because we’ll be interviewing almost everyone who plays a key role, the responses to this question will collectively summarize the DCM-F/RP process. Later sections of the protocols delve into the details of many activities, but if a response here mentions aspects not fully covered by later questions, probe for details here.]

1. [After each person in the interview describes his or her role:] Are there any aspects of the process conducted by your agency that you aren’t involved in? [If so:]

   1. Tell me about that part of the process.

   2. Who conducts that step of the process? [If someone not on the site visit schedule (or who was scheduled but unable to attend the interview):] Could I contact [that person] by telephone later to learn more about [his/her] role?

2. [If any key steps (planning the transition to DCM-F/RP, coordinating with the Medicaid agency, conducting the match[in central-matching states], providing match results or data and technical assistance to districts) haven’t been mentioned, ask:] What about [steps]? Who is involved in that part of the process? [If they’re not on the site visit schedule:] Could I contact them by telephone later to learn about their role?

3. Besides SNAP, does [State] directly certify students by using data from other public assistance programs?

   1. TANF (Temporary Assistance for Needy Families)?

   2. FDPIR (Food Distribution Program on Indian Reservations)?

   3. Foster children?

   4. Homeless/migrant children?

   5. Other programs?

Planning and preparation

4. How did you plan and prepare for DCM-F/RP?

   1. [Be sure the response includes both:] What preparations were made within your agency? What coordination with other agencies was needed during the planning phase?

   2. What concerns or challenges did you encounter when planning and preparing for DCM-F/RP?

   3. [California, Florida, and Massachusetts, ask:] How did participation in the first demonstration help you prepare for DCM-F/RP?

   4. What are some of the key changes you made to your approach for DCM-F/RP?

5. In what ways has the implementation of DCM-F/RP changed the direct certification process in [State]?

   1. Does the direct certification process this year differ from last year’s process in any other ways (not related to DCM-F/RP)?

Assessing eligibility

[Moderator note: No need to ask questions that have been fully answered in response to Question 2 above.]

As you described, to conduct the DCM-F/RP match, you must first receive a list of eligible students from [agency]. These students must meet two criteria: (1) they must receive Medicaid, and (2) they must have a family income below 130 percent of the Federal Poverty Level (FPL) to qualify for free meals or a family income between 130 and 185 percent of the FPL to qualify for reduced-price meals.

6. How did you work with your Medicaid agency to identify the correct children at these two income levels?

   1. [Probe:] What specifications or guidance did you provide to your Medicaid agency to help them identify the correct children?

7. What steps did you take before the first DCM-F/RP match to ensure that the list of eligible students was accurate?

   1. To what extent was your agency involved in the testing of the first DCM-F/RP file?
      [Moderator note:] Request copies of any materials provided to/by the Medicaid agency detailing the verification, monitoring, or testing process.

8. Is there any ongoing monitoring or testing of the list each time it is produced, or do you consider the initial testing sufficient? [If yes:] Please explain what monitoring or testing is done.

9. How is (are) the list(s) of eligible children provided to your agency?

   1. How are students eligible for free meals differentiated from those eligible for reduced-price meals?

   2. How often is the list provided to your agency?

   3. Are SNAP data provided in the same way? If not, please explain.

[Probe:] Are data provided by the same agency or different agencies?
[Probe:] Are SNAP and Medicaid cases maintained in the same file or separate files? [If same file, ask:] How are they differentiated?

10. What challenges have you encountered in obtaining the Medicaid data? In ensuring that the children in the Medicaid data are accurately identified as eligible?

    1. How did you overcome these challenges or, if ongoing, how do you plan to do so? [Probe:] For example, did you implement changes after some data elements from Medicaid did not fully meet your needs?

11. Are there any limitations to the Medicaid data that make direct certification more difficult than it could be? [Probe:] For example, are there certain fields that are difficult to interpret or often missing? [If so:] Do these same limitations apply to the SNAP data?

    1. Does your agency have different data standardization rules than [Medicaid agency]? If so, how do they differ and for what variables?
       [Probe:] For instance, are hyphenated names recorded differently in Medicaid than in other data sets such as SNAP or TANF?

    2. Are any data fields truncated?

    3. Do you use any information system(s) to add data (e.g., a state student ID) to the Medicaid file? If so, please describe.

Matching the Medicaid and student data

Now I’d like to discuss what happens after you receive the Medicaid file.

[Moderator note: No need to ask questions that have been fully answered in response to Question 2 above. Respondents may jump around when discussing their matching process due to the many components involved. Try using the DCM-F/RP flow chart you drafted from the State’s application to guide this part of the discussion, and be prepared to jump between questions in this section. Make sure that you address all gaps in the flow chart.]

12. Can you describe the next step in the DCM-F/RP process in [State] after you receive the list of eligible children?

    1. What databases and information system(s) are involved in the DCM-F/RP match?
       [Probe:] What database are the lists stored in? Do you use a statewide student database to facilitate the DCM-F/RP match? Does your State have statewide student IDs?

[Central-level matching, ask 12b–e:]

2. Do you use an in-house or third-party matching system?

3. Do you have to reformat the Medicaid data in any way before matching?
   [Probe:] Do you have to change variables to conform to specific formats or values? For instance, limiting “sex” to “M” or “F” or recording “date of birth” as “MM/DD/YYYY?” [If so:] Do you have to make similar changes to the SNAP data before DC-SNAP matching?

4. Do districts upload enrollment data, or does [State] maintain enrollment lists?
   [If the latter, ask:] When are enrollment lists updated?

5. Do private schools use different matching processes than public districts?

[Local-level matching, ask 12f–g:]

6. How do most districts conduct their DCM-F/RP match?

   1. What point-of-sale systems do your districts commonly use to conduct matching?

   2. How did point-of-sale vendors prepare for the DCM-F/RP match, if at all?

7. How much does the matching process vary across districts in [State]?

   1. Do you provide guidance regarding the data elements or matching algorithms that districts should use? Mandatory rules or suggestions?

   2. Do private schools use different matching processes than public districts?

   3. Do large districts use different matching processes than small or medium-sized districts?

1. [Central-level matching, ask:] What data elements are used to conduct the match?

   1. Do you use probabilistic matching (i.e., algorithms that allow inexact matches between data fields)? [If yes:] What is the minimum score required for a match?

   2. Are partial matches provided to districts? If so, what results in a positive versus partial match?

   3. Are nonmatches provided?

   4. Does [State] use phonetic matching, string matching, date-of-birth accommodation, or other approaches to improve matching success? If so, please describe in detail.

   5. [Moderator note: Ask if State have a reference guide that details the matching algorithms and if you may obtain a copy of it.]

2. To what extent do the direct certification results the State provides differ from the certification outcomes in the districts?

3. How [does State/do districts] ensure that students who could be directly certified through both Medicaid and another program (e.g., SNAP, TANF, or FDPIR) are recorded as certified based on the other program?

   1. If a student is directly certified by Medicaid and later enrolls in SNAP, do [you/your districts] change him or her to DC-SNAP? Please explain the process.

   2. How easy or difficult was it for your [State/districts] to ensure that matches with other program data take priority over Medicaid matches?

      1. What updates, if any, were required to your data systems?

      2. Are duplicates included in the files? For instance, if a child receives both SNAP and Medicaid, does he or she appear only once in the file(s)?

      3. Is there a flag in the direct certification file(s) noting what type of match it is?

4. How [does State/do districts] extend eligibility to other children that are in the same household as a child directly certified through Medicaid?

Match schedule

17. [Central-level matching, ask:] How often does [State] conduct a Medicaid match?

    1. How much time did [State] staff need to complete the first DCM-F/RP match, starting from when you received the first DCM-F/RP eligibility file to when you distributed the list of [matches/eligible students] to districts? How much time was needed for subsequent matches?

[Local-level matching, ask:] How often are lists of eligible children who receive Medicaid made available to districts?

18. How often do most districts [download their matches/conduct matching]?

    1. Are there any State requirements, or may districts set their own schedule?

    2. Is DCM-F/RP conducted on the same schedule as that used for direct certification with other programs?

19. Does the DCM-F/RP schedule differ from the timing of direct certification with other programs?

Dissemination

20. How are [matches/lists of eligible children] made available to districts?

    1. Is the list of [matches/eligible children] combined with the other direct certification files, or is it provided separately?

    2. What is the geographic scope of the file? Do districts receive a list of [matches/eligible children] for the entire State, several counties, a group of ZIP codes, or some other area?

21. How does your State record DCM-F/RP results?

    1. Did your State or districts have to make any changes to their systems to report a free or reduced-price match using Medicaid data? Please describe the changes.

    2. Has DCM-F/RP required any changes in your process for reporting data to FNS?

Outcomes

Let’s discuss some of the early outcomes you’ve seen thus far.

22. Think about the results of DCM-F/RP matching. Approximately how many children were on the DCM-F/RP lists provided to you by [agency]? Roughly what proportion of these cases was successfully matched?

    1. How does this compare to the proportion of SNAP cases successfully matched from the list of SNAP recipients?

1. [Probe, if applicable:] What are some reasons that you think the DCM-F/RP match was more/less successful than the DC-SNAP match?

23. Has DCM-F/RP matching success varied by student or household characteristics?

    1. Are there any differences by:

1. Race/ethnicity?

2. Family/household size?

3. Family/household composition?

4. Families/households with name differences?

5. Split households (refers to children who live in two separate households if divorced parents split custody)?

6. Grade level?

7. Medicaid eligibility category (e.g., children who qualify for Medicaid based on Modified Adjusted Gross Income (MAGI) versus other children, such as those with disabilities, who qualify based on non-MAGI income)?

8. Other student or household characteristics?

24. Has DCM-F/RP matching success varied by district characteristics?

    1. Are there any differences by:

1. Urban/rural?

2. Large/small?

3. Public/private?

4. Districts with a high versus low percentage of students certified for free or reduced price meals (or other economic indicators)?

5. Levels of DC-SNAP as a percentage of total free certifications?

6. [Local-level matching, ask:] Local data system characteristics?

7. Other district characteristics?

25. Did your participation in DCM-F/RP affect schools’ or districts’ eligibility for, and reimbursement rates under, the Community Eligibility Provision (CEP)?

    1. Did you expect that it would?

    2. Has implementing DCM-F/RP made CEP reporting procedures more/less difficult? Do you conduct direct certification for CEP every year or less frequently?

Challenges and resolutions

Now I’d like you to think about any difficulties you had in implementing the demonstration.

26. What challenges have you encountered in implementing DCM-F/RP?
    [Probe:] Any other serious challenges (e.g., communication, technical assistance, planning/preparation, data-sharing agreements, file creation, matching, file distribution, etc.)?

    1. [If not clear, ask:]Are these challenges unique to DCM-F/RP?

    2. What problems have these challenges caused?

    3. To what extent have you been able to resolve each of these challenges? How?

27. What challenges have your districts encountered in implementing DCM-F/RP?
    [Probe:] What issues do districts most often contact you about regarding DCM-F/RP?

    1. Have these challenges varied by district characteristics?

1. Are there any challenges common or unique to large versus small or medium-sized districts?

2. Are there any challenges common or unique to public versus private districts?

1. What consequences have these challenges caused?

2. To what extent have districts been able to resolve each of these challenges? How?

1. How much additional time and resources have been needed—at the State and district levels—to implement DCM-F/RP (over and above the efforts for conducting direct certification with other programs)?

   1. Which activities associated with DCM-F/RP have been the most time-consuming? The least time-consuming?

   2. Has DCM-F/RP saved [agency] time in any way? Has it saved districts time?

      1. [Probe:] Has it reduced the number of paper applications districts receive?

      2. [Probe:] Has it reduced time districts spend on verification?

3. [For states that conducted their first DCM-F/RP match after the beginning of the school year, probe:] Do you think it would have saved your agency or districts [more] time if the match had been conducted at the beginning of the school year? Please explain.

1. Which activities associated with DCM-F/RP have been the most difficult? The least difficult?

Best practices and lessons learned

We’re interested in what you’ve learned from your experience thus far, as one of the first States to implement DCM-F/RP.

29. What aspects of your State’s direct certification processes helped facilitate DCM-F/RP?

    1. Are there specific features of [your/your districts’] systems that helped facilitate the match?

    2. Are there other activities [State] conducted that helped facilitate implementation?

30. What aspects of your State’s [or districts’] direct certification processes made implementation more difficult?

    1. Are there specific features of your [or your districts’] systems that hindered implementation?

31. What technical assistance provided by FNS or the Centers for Medicare and Medicaid Services (CMS) did you find most helpful?

    1. In what areas could you have used more help or guidance from FNS or CMS?

32. What changes, if any, have been planned to improve the matching success of DCM‑F/RP?

    1. Are there other changes you think [State/districts] could make to improve matching success?

1. [Probe:] Are there any changes to your [and your districts’] systems, databases, or matching algorithms that could improve your matching success?

2. [Probe:] [Has/have] your [State/districts] explored phonetic matching, probabilistic matching, string matching, date-of-birth accommodation, or other approaches that are designed to improve matching success rates? What are the advantages and disadvantages of these approaches?

Closing

33. Is there anything else you would like to add about your experience in implementing DCM-F/RP so far?

34. As you know, we’ll also be meeting with staff in [two/four] districts in your State to learn about their roles in the process. In what ways are the districts we’re visiting typical of others in [State] in terms of student population, matching process, technological sophistication, etc.?

    1. In what ways are they different from others in [State]?

Thank you for meeting with us today. We’ll be contacting you [primary respondent for each agency] again in [month] to conduct a follow-up telephone interview to reflect on your experiences near the end of this DCM-F/RP school year.

1 In States where staff of a child nutrition agency vendor play a key role in the DCM-F/RP process, this protocol would be tailored for use in interviewing those respondents.