50305 DCM FRP_OMB PART B Final

SUPPORTING STATEMENT - PART B for

OMB Control Number 0584-NEW:

Evaluation of the Direct Certification with Medicaid for
Free and Reduced-Priced (DCM-F/RP) Meals Demonstrations

Conor McGovern

Research Analyst

COR/OPS

USDA, Food and Nutrition Service

3101 Park Center Drive

Alexandria, Virginia 22302

CONTENTS

PART B. COLLECTION OF INFORMATION USING STATISTICAL METHODS 1

B.1. Respondent Universe and Sampling Methods 1

B.2. Procedures for the Collection of Information 9

B.3. Methods to Maximize Response Rates and Deal with Non-Response 16

B.4. Test of Procedures or Methods to Be Undertaken 18

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 19

APPENDIX A: QUALITATIVE DATA COLLECTION MATERIALS

A.1. SITE VISIT LETTER

A.2. SITE VISIT SCHEDULING CALL PROTOCOL

A-2.a. SITE VISIT PREPARATION: STATE CHILD NUTRITION AGENCY

A-2.b. SITE VISIT PREPARATION: STATE MEDICAID AGENCY

A-2.c. SITE VISIT PREPARATION: DISTRICT

A-2.d. VENDOR TELEPHONE INTERVIEW EMAIL

A-3.a. Detailed site visit interview protocol: state child nutrition agency staff

A-3.b. Detailed site visit interview protocol: state Medicaid agency staff

A-3.c. Detailed site visit interview protocol: district staff

A-4. Site visit observation protocol

A.5. Site visit follow up telephone interview email

A.6.a. Detailed Follow-up interview protocol: state child nutrition agency staff

A.6.b. Detailed Follow-up interview protocol: state medicaid agency staff

A.6.c. Detailed Follow-up interview protocol: district staff

APPENDIX B: ADMINISTRATIVE RECORDS REQUEST

B-1. ADMINISTRATIVE RECORDS REQUEST EMAILS

B.2. ADMINISTRATIVE RECORDS REQUEST

APPENDIX C: STATE COST DATA COLLECTION MATERIALS

C-1. STATE COST LOG EMAILS

C-2. INSTRUCTIONS FOR STATE COST DATA TRACKING LOGS

C.3.A. STATE CHILD NUTRITION AGENCY COST LOG

C.3.B. STATE MEDICAID AGENCY COST LOG

C.4. SEMI-STRUCTURED STATE COST LOG CLARIFICATION
APPENDIX D: CONFIDENTIALITY PLEDGES

APPENDIX E: ANNUAL BURDEN ESTIMATE

APPENDIX F: PUBLIC COMMENTS AND RESPONSES

F1a: PUBLIC COMMENT #1

F1b: FNS RESPONSE TO PUBLIC COMMENT #1

F2a: PUBLIC COMMENT #2

F2b: FNS RESPONSE TO PUBLIC COMMENT #2

F3a: PUBLIC COMMENT #3

F3b: FNS RESPONSE TO PUBLIC COMMENT #3

APPENDIX G: NASS COMMENTS AND RESPONSES

APPENDIX H: RELEVANT DOCUMENTS

H.1. Section 104 of the Child Nutrition and WIC Reauthorization Act of 2004

H.2. Section 103 of the Healthy, Hunger-Free Kids Act of 2010

TABLES

B.1.1 Respondent Universe and Sample Size 2

B.1.2 Districts Included in Qualitative Data Collection Subsample 6

B.1.3 Sample Sizes and Expected Response Rates, by Research Activity 9

B.3.1 Expected Response Rates and Methods to Maximize Response Rates, by Research Activity 17

B.5.1 Individuals Consulted 19

PART B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Beginning in SY 2016–2017, FNS initiated a new demonstration of Direct Certification with Medicaid for Free and Reduced-Price (DCM-F/RP) Meals, under the administrative pilot authority in Section 18(c) of the Richard B. Russell National School Lunch Act (NSLA). This demonstration permits the selected States to use data from Medicaid files to identify students eligible to receive free or reduced-price meals and directly certify them at that level. FNS seeks approval to conduct data collection as part of the Evaluation of the DCM-F/RP Demonstration.

The demonstration was awarded based on a competitive RFA process. The Cohort 1 (SY 2016-17) States were chosen in the first round of the competitive RFA process from the pool of States that applied. A subsequent competitive RFA process was used to select Cohort 2 (SY 2017-18) States. States were chosen based on applications in which they demonstrated that they could successfully comply with all requirements and carry out the demonstration project, including providing all necessary data for the required evaluation. Direct certification with Medicaid data is currently only carried out through demonstration projects, including the original DCM demonstrations and the current DCM-F/RP demonstrations, which will be evaluated in Year 2 with this information collection.

The respondent universe for the Evaluation of the DCM-F/RP Demonstration includes the 15 State Child Nutrition Agencies, 15 State Medicaid Agencies, and 3 other State Agencies involved in the DCM-F/RP process in the 15 States that are conducting DCM-F/RP; the 7,218 school districts within these 15 States; and 4 vendors involved in the DCM-F/RP process (Table B.1.).

Table B.1.1. Respondent Universe and Sample Size

Respondent type	Respondent universe	Sample size
State Child Nutrition Agenciesa	15	15
State Medicaid Agenciesa	15	15
Other State Agencies	3	3
School Districtsb	7,218	34c
Vendors	4	4

^a Our burden estimates assume an average of three respondents within each Child Nutrition and Medicaid agency, but the sampling unit and unit of analysis is the agency (or the district), not the individual, and response rates will be computed at the agency level.

^b The respondent universe for school districts was estimated using the number of districts in each sample state that submitted FNS-742 data for SY 2015-2016. Our burden estimates assume an average of three respondents within each district, but the sampling unit and unit of analysis is the district, not the individual, and response rates will be computed at the district level.

^c Although qualitative data will be collected from a sample of 34 districts, all 7,218 districts in the demonstration will be included in the administrative records data provided by State Child Nutrition Agencies.

Summary of Prior Related Activity

The DCM-F/RP demonstration builds on an earlier demonstration of DCM for free meals (OMB Control Number 0584-0586 The Evaluation of Demonstrations of NSLP/SBP Direct Certification of Children Receiving Medicaid Benefits, discontinued 4/30/2016), which began in SY 2012–2013. Under the DCM demonstration, FNS authorized seven States to directly certify students for free meals using income data available through Medicaid. In some demonstration States, districts were randomly assigned to conduct DCM or to a control group; in others, the demonstration was implemented statewide. The evaluations of this demonstration examined the effects of DCM on participation, costs, and other outcomes. Data collection activities included administrative records, cost logs completed by State Child Nutrition and Medicaid Agency staff, a web survey on district costs, and telephone interviews on challenges encountered by States and districts. Findings from the evaluation were published by FNS in 2015 and 2016.¹ The response rates were 100 percent on the collection of administrative records, cost logs, and telephone interviews; and 80 percent or higher for all rounds of the survey in each State.

Data collection for the DCM-F/RP demonstration began with a pretest in seven States in SY 2016–2017. A previous OMB submission (Generic OMB clearance number 0584-0606 – Pretest for Evaluation of Direct Certification with Medicaid Demonstrations; clearance date 12/19/2016) covered the pretest. The current package requests clearance for data collection activities in SY 2017–2018. The pretest year of data collection includes staff from State Child Nutrition Agencies, Medicaid Agencies, and 16 school districts in seven States.² Similar to the proposed information collection, staff are expected to participate in multiple data collection activities, including site visits, follow-up interviews, and, for some, the completion of cost tracking logs and administrative records requests. The data collection is in progress, so response rates have not yet been computed. In addition, pretest respondents will participate in debriefings to provide input into improvements to the data collection instruments and processes.

Selecting States

The sample for the DCM-F/RP evaluation will include 15 States.³ FNS solicited applications from States to participate in DCM-F/RP, and purposively selected these States from among those that applied to begin conducting DCM-F/RP. Therefore, the sample will include all States participating in the demonstration but will not be representative of any larger set of States.

Different components of the DCM-F/RP study address different research questions, and the data collection activities and samples of respondents differ. We will collect four key types of data:

1. On-site interviews and observations (Appendices A-3a through A-3c and A-4);

2. Site visit follow-up telephone interviews (Appendices A-6a through A-6c);

3. Administrative data on certification and participation (Appendix B-2); and

4. State-level cost data (Appendices C-3a and C-3b).

Identifying School Districts for Site Visits and Follow-Up Telephone Interviews

Some analyses—those based on administrative records from the States—will include all districts in each State. Other analyses—those requiring data collection from districts—will focus on a subsample of districts. The subsample will include 2 districts in each State that conducts direct certification matching centrally at the State level, with the exception of California.⁴ Because California will transition from operating the demonstration in a small subset of districts (14) in SY 2016–2017 to the entire State in SY 2017–2018, we will collect qualitative data from 4 districts in that State—2 that began in SY 2016–2017 and 2 that will begin the following year. We will also include 4 districts in the State that does not conduct central matching. The total number of districts in the subsample used for the qualitative analyses will be 34 (Table B.1.2). This district subsample will be selected purposively and will not be representative of any broader set of districts. We will use information from FNS-742 School Food Authority (SFA) Verification Collection Report (approved under OMB # 0584-0594 Food Program Reporting System, expiration date September 30, 2019) and the USDA Economic Research Service Urban-Rural Continuum Codes to assess diversity along key characteristics, including public and private schools, and ranging in size, urbanicity, and levels of F/RP meal certification. We will also solicit input from State Child Nutrition Agency staff on other characteristics, such as local data systems and procedures. FNS will ask the State Agencies to reach out to the districts to explain the study requirements and encourage them to participate.

Table B.1.2. Districts Included in Qualitative Data Collection Subsample

	Number of states	Number of districts per state	Total number of districts
California	1	4	4
Virginia	1	4	4
Other DCM-F/RP States	13	2	26
Total	15	N/A	34

Note: This assumes that the seven States that began conducting DCM-F/RP in SY 2016–2017 continue their current matching process (in which all but Virginia conduct matching centrally) and that all eight States beginning the demonstration in SY 2017–2018 conduct matching centrally that year.

Next, we provide specific information on the respondent universe for each of the four study components.

1. Sample for the site visit activities.

1. Sample for the site visit activities. We will conduct site visits in each of the 15 States to learn about the demonstration implementation process. During the visits, we will interview the staff at State Agencies, selected districts and vendors that play key roles in the DCM-F/RP process. The sample for these on-site interviews will include 45 State Child Nutrition Agency staff (3 per State), 45 State Medicaid Agency staff (3 per State), 3 staff from other State Agencies that play key roles in the DCM-F/RP process (1 in each of 3 states), 102 district staff (3 per district), and 4 staff from State Child Nutrition and Medicaid Agencies vendors that play key roles in the direct certification process (1 for each of 4 vendors). The interview participants will be the staff most knowledgeable about the demonstration, as identified by the main contact at each State agency or district.

The site visits will also include observations of key activities in the process. The sample for these on-site observations will be a subsample of the staff involved in interviews, including 30 State Child Nutrition Agency staff (2 per State), 30 Medicaid Agency staff (2 per State), and 64 district staff (2 per district). The observations will be led by the staff responsible for completing that step of the process.

Based on prior experience with similar studies and respondent universes, we expect to achieve 100 percent response from these State, school district, and vendor staff. To maximize response rates and ensure the highest quality data possible, State Child Nutrition Agency staff will contact districts selected for site visits. Senior members of the evaluation team will serve as the site visit leads and will work with our main State contacts to ensure that the relevant staff at each State Child Nutrition and Medicaid Agency are included in the visits and encouraged to participate.

2. Sample for the follow-up telephone interviews. We will conduct telephone interviews with respondents at the State Child Nutrition and Medicaid Agencies and districts included in site visits. The sample for these telephone interviews includes one staff person from each agency or district visited, for a total of 15 State Child Nutrition Agency staff, 15 State Medicaid Agency staff, and 34 district staff. Based on prior experience with similar studies, we expect to achieve 100 percent response from these State Child Nutrition Agency staff, 93 percent from State Medicaid Agency staff, and 91 percent from school district staff. We will contact respondents who are slow in responding to interview requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, State Agency or FNS staff may contact any reluctant respondents to underscore the importance of study participation.

3. Sample for the administrative data on certification and participation. We will collect from State Child Nutrition Agencies district-level administrative data on certification and participation (meals served) for each district in the State. Where available, we will also gather information on DCM-F/RP match results and the certification status, method, and basis for matched students before that match. We will also collect statewide administrative data from each State on each district’s total number of reimbursable lunches and breakfasts served, by reimbursement category. The sample for this administrative data request will include 15 State Child Nutrition Agency staff. Based on prior experience with similar studies, we expect to achieve 100 percent response from these States. To ensure this high response rate, we will track respondents who have provided administrative records, and send email reminders to those who have not. We will contact respondents who are slow in responding to administrative records requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, FNS staff may contact any reluctant respondents to underscore the requirements of study participation.

4. Samples for the State-level cost data. We will collect data on administrative costs incurred by staff of State Child Nutrition Agencies, Medicaid Agencies, and other State Agencies that play key roles in the DCM-F/RP process using a customized Excel-based cost log and clarification calls. We will ask respondents to report only costs that are in addition to costs for direct certification activities that were taking place before the DCM-F/RP demonstration. The sample for these cost logs includes 15 State Child Nutrition Agency staff, 15 State Medicaid Agency staff, and 3 other State Agency staff. Based on experience from the first DCM demonstration, we expect to achieve 100 percent response from State Child Nutrition Agency staff and other State Agency staff and 93 percent from State Medicaid Agency staff. In total, a 99 percent response rate is anticipated. To ensure this high response rate, we will track respondents who have completed cost logs, and send email reminders to those who have not. We will contact respondents who are slow in responding to cost log requests to answer their questions and remind them of the importance of the demonstration evaluation. If necessary, FNS staff may contact the most reluctant respondents to underscore the requirements of study participation.

Table B.1.3 presents sample sizes and expected response rates for each research activity.

Table B.1.3. Sample Sizes and Expected Response Rates, by Research Activity

		Sample size			Expected number of participants
Respondents	Research activity	Agencies/ districts	Individuals	Expected response rate	Agencies/ districts	Individuals
State Child Nutrition Agency Staff	On-Site Interviews	15	45	100	15	45
	On-Site Observations	15	30	100	15	30
	Site Visit Follow-Up Telephone Interview	15	15	100	15	15
	Administrative Records Request	15	15	100	15	15
	State Cost Data Collection Tracking Logs	15	15	100	15	15
State Medicaid Agency Staff	On-Site Interviews	15	45	100	15	45
	On-Site Observations	15	30	100	15	30
	Site Visit Follow-Up Telephone Interview	15	15	93	14	14
	State Cost Data Collection Tracking Logs	15	15	93	14	14
Other State Agency Staff	On-Site Interviews	3	3	100	3	3
	State Cost Data Collection Tracking Logs	3	3	100	3	3
District Staff	On-Site Interviews	34	102	100	34	102
	On-Site Observations	34	68	100	34	68
	Site Visit Follow-Up Telephone Interview	34	34	91	31	31
Vendor Staff	On-Site or Telephone Interviewsa	4	4	100	4	4
Total Number of Responses		553	553	99	545	545
Total Respondent Sample		71	199	100	71	199

Note: State Child Nutrition Agency, State Medicaid Agency, other State Agency, and school district staff are all expected to participate in multiple data collection activities, including the site visits, follow-up interviews, and the completion of cost tracking logs and administrative records requests, where applicable. The “Total Number of Responses” row includes each relevant activity. Each individual is counted only once in the total respondent sample.

^a Vendor staff interviews may be conducted by telephone, if vendors are not located near the State agencies.

B.2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data Collection

In this section, we provide information on the specific data collection activities for each key DCM-F/RP study component:

1. On-site interviews and observations. To document the DCM-F/RP processes, we will use in-depth case studies that trace the relevant direct-certification workflow step by step. The core of these studies will be in-person site visits in each participating State, during which we will interview program and technical staff at the State and district levels who are involved in implementing DCM-F/RP. We will first send introductory letters (Appendix A-1) and conduct scheduling calls (Appendix A-2) with State agency contacts to arrange the visits, then provide a site visit preparation document (Appendix A-2a through A2-c) containing the schedule and an overview of topics to be discussed and documents requested. Each site visit will last two or three days. During the visits, we will interview the staff at each State Agency, selected district, and vendors (where applicable) that play key roles in the DCM-F/RP process, and we will conduct observations of DCM-F/RP procedures. For example, we will ask Medicaid staff to show us how they identify eligible children within the Medicaid data, and ask Child Nutrition and district staff to show us how they conduct data matching. These observations (Appendix A-4) will ensure that site visitors have a complete understanding of how DCM-F/RP is conducted. Interviews (Appendices A-3a through A3c) will provide detailed descriptions of DCM-F/RP procedures at the State and district levels and specific changes needed to initiate DCM-F/RP. We will ask open-ended questions and follow up with adaptive probes based on the information provided by the respondent.

2. Site visit follow-up telephone interviews. We will also conduct telephone interviews (Appendices A-6a through A-3c) with respondents at the State Agencies and districts that we visited during site visits. The interviews, which will be preceded by a scheduling email (Appendix A-5), will take place near the end of the school year and provide updated information on how the DCM-F/RP demonstration has changed in each State since the site visits. Interviewers will ask about whether and how challenges identified during the site visit have been resolved and about any additional challenges that may have been identified since the site visits. In addition, the telephone interviews will provide information about whether the staff time and resources needed to conduct DCM-F/RP decreased as staff grew accustomed to the new procedures.

3. Administrative records request. We will collect district-level administrative data from State Child Nutrition Agencies for the DCM-F/RP implementation year SY 2017–2018 and, for States beginning the demonstration in SY 2017–2018, a baseline year before the demonstration.⁵ We will request the data through an email (Appendix B-1) detailing the information required (Appendix B-2). The administrative records will include data on both certification for school meal benefits and participation (meals served), for each district in the State, and the request will focus on data that States already collect, to the extent possible, as follows:

• Certification data. We will collect data on certification status, method, and basis, including data elements reported on form FNS-742, plus the numbers of students directly certified for free meals and for reduced-price meals based on Medicaid. For the States that conduct their first DCM-F/RP match by the last operating day of October 2017, we will collect data for that point and for the last operating day of October in the baseline year. For the remaining demonstration States, we will request the data elements as of the point in time just after their first DCM-F/RP match (or a point after a reasonable portion of districts have conducted or triggered their matches, in States where districts determine the timing of matching) and for the same date during the baseline year. Where available, we will also collect additional administrative data on DCM-F/RP match results and prior certification information.

• Participation data. We will collect from each State elements from the district-level data that are aggregated to complete form FNS-10, the Report of School Operations (approved under OMB # 0584-0594 Food Programs Reporting System, expiration date September 30, 2019).⁶ To assess participation, we will collect each district’s total numbers of reimbursable lunches and breakfasts served, by reimbursement category (free, reduced-price, paid) in each month during the demonstration and the same months during the baseline year. To facilitate analyses of Federal reimbursement costs, we will also request the numbers of meals reimbursed at the slightly higher “needs-based” NSLP rates or “severe-needs” SBP rates, and the number served in districts certified to receive an extra six cents per lunch served based on meeting updated nutrition standards and meal patterns.

We will collect these administrative data in two batches. The liaison for each State will contact the Child Nutrition Agency staff to first request the baseline data (in States that are new to the demonstration) and the certification data (for States that conducted their first DCM-F/RP match at the beginning of the school year). We will collect the data covering the first year of the DCM-F/RP demonstration when they are available. We expect most data to be provided in Excel files, but we will accept other formats. Because the data will not contain any personally identifiable information, the files can be submitted via email.

4. State cost data collection tracking logs. To analyze administrative costs related to implementing DCM-F/RP, we will collect data four times during the school year on resource expenditures from relevant State Child Nutrition Agency, Medicaid Agency, and other State Agency staff. These data will cover labor, infrastructure, software, and other costs. They will reflect additional costs States incurred to implement the new demonstration, beyond those associated with existing State efforts on direct certification.

To facilitate careful tracking, we will provide cost log templates (Appendices C-3a and C-3b) to States in Excel format at the beginning of each data collection period. Respondents will use the template to track and report monthly records of the time each staff member spends on various DCM-F/RP demonstration activities, and other direct costs incurred, such as amounts paid to outside contractors. We will also provide instructions for the template (Appendix C-2), describing the seven tabs, which include (1) a list of activities that might be involved in the DCM-F/RP process, tailored to Child Nutrition and Medicaid Agencies; (2) a log to enter monthly time for each staff type; (3) a table to enter salary information for each staff type; (4) a table to enter other direct costs; (5) a table to enter indirect costs; (6) a place to provide contact information; and (7) an optional tab for recording time weekly rather than monthly.

Liaisons from the evaluation team will collect the cost logs from each agency in their States and will include instructions with each data request, sent via email (Appendix C-1). After the completion of the first log, the liaison will conduct a clarification call (Appendix C-4) to ensure the provided data is accurately interpreted. The logs will be collected in four batches in the following months: (1) July through September, (2) October through December, (3) January through March, and (4) April through June. For States new to the demonstration in SY 2017–2018, the first log will also collect data on any costs incurred prior to July 2017. Logs will be collected the month following the end of the quarter, contingent upon OMB approval prior to that month, or else retroactively based on recall in the month following OMB approval.

Data collector training. Before data collection begins, liaisons from the evaluation team will attend a half-day training webinar. The webinar will provide background on the demonstration and detailed instruction in how to collect the different types of data described above. Data collectors are expected to have experience collecting similar data for direct certification studies, such as the previous DCM or DCM-F/RP studies.

Sampling and Analysis

1. Statistical methodology for stratification and sample selection

Quantitative analyses will be based on the full universe of states and districts participating in the DCM-F/RP demonstration. The only sampling will be to select districts for the qualitative data collection (site visits and follow-up telephone interviews). Districts will be selected purposively (1) to reflect variation along several relevant characteristics and (2) to facilitate logistics of site visits; the sample will not be statistically representative.

2. Estimation procedure

Qualitative analysis. To begin the analysis of qualitative data, we will draft an internal analytic memo after each site visit that will summarize findings from the in-depth interviews and observations. We will construct an analysis framework based on the relevant research questions and emergent themes from the analytic memos. We will use this framework to conduct a thorough analysis of all interview and observation notes and State documents, using either qualitative analysis software (such as NVivo) or detailed theme tables in Microsoft Excel. The analysis will identify patterns across different States and districts, focusing on matching procedures, challenges, and successes in implementing DCM-F/RP.

Quantitative analysis. We will use administrative records data to compute for each district measures of key certification, participation, and Federal reimbursement outcomes in the year before DCM-F/RP implementation and in the first year of DCM-F/RP. We will use quantitative pre/post district fixed-effects analysis to estimate the changes in certification that accompany DCM-F/RP implementation, controlling for measurable time-varying district characteristics (such as enrollment and local economic conditions) and all district characteristics that are fixed over time.

For other outcomes, we will use descriptive analyses. For example, we will analyze data on State administrative costs to estimate the total State-level administrative costs of the new demonstration, as well as breakdowns of these costs by agency, timing, and category (such as labor).

3. Degree of accuracy needed for the purpose described in the justification. There are no precision requirements for this evaluation. Quantitative analyses will be based on the full universe of states and districts participating in the demonstration, which will provide more accurate estimates than a smaller sample.

4. Unusual problems requiring specialized sampling procedures. There are no unusual problems requiring specialized sampling procedures.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden. This data collection request is to study one year of the DCM-F/RP demonstration; all data collection will take place over the course of that year.

B.3. Methods to Maximize Response Rates and Deal with Non-Response

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Anticipated response rates and methods to maximize them are shown in Table B.3.1, below. Response rate assumptions are based on experiences with similar studies such as the first DCM demonstration. In that study, we obtained response rates of 100 percent on the collection of administrative records and State cost data tracking logs with a sample size of six State. The data collection for SY 2016–2017 of the DCM-F/RP demonstration is in progress, so response rates have not yet been computed. The first DCM demonstration did not include site visits or follow-up telephone interviews. We expect that the planned methods of data collection will result in the accurate, reliable data needed for the planned analyses and modeling at acceptable response rates. The number of completed instruments will be the numerator in response rate calculations. A completed instrument will be defined as one in which all critical items for inclusion in the main analysis are complete and within valid ranges. To maximize response rates and ensure the highest quality data possible, we will take a multipronged approach:

• Site visits and follow-up interviews with respondents will be scheduled in advance in order to answer their questions and ensure that the data collection takes place at convenient times.

• We will track respondents who have provided administrative records and completed cost logs and interviews, and send email reminders to those who have not.

• We will contact respondents who are slow in responding to administrative records, cost log, and interview requests to answer their questions and remind them of the importance of the demonstration evaluation.

• As appropriate, State Child Nutrition staff or FNS staff may contact the most reluctant respondents to underscore the requirement of study participation.

• Staff conducting the on-site interviews and telephone interviews will be qualified, well-trained professional interviewers.

Table B.3.1. Expected Response Rates and Methods to Maximize Response Rates, by Research Activity

Respondents	Research Activity	Expected Response Rate	Methods to Maximize Response Rate
State Child Nutrition Agency Staff	On-Site Interviews	100	Scheduling calls
	On-Site Observations	100	Scheduling calls
	Site Visit Follow-Up Telephone Interview	100	Scheduling emails
	Administrative Records Request	100	Email and telephone reminders
	State Cost Data Collection Tracking Logs	100	Email and telephone reminders
State Medicaid Agency Staff	On-Site Interviews	100	Scheduling calls
	On-Site Observations	100	Scheduling calls
	Site Visit Follow-Up Telephone Interview	93	Scheduling emails
	State Cost Data Collection Tracking Logs	93	Email and telephone reminders
Other State Agency Staff	On-Site Interviews	100	Scheduling calls
	State Cost Data Collection Tracking Logs	100	Email and telephone reminders
District Staff	On-Site Interviews	100	Scheduling calls
	On-Site Observations	100	Scheduling calls
	Site Visit Follow-Up Telephone Interview	91	Scheduling emails
Vendor Staff	On-Site Interviews	100	Scheduling calls
Total Respondent Sample		100	193

B.4. Test of Procedures or Methods to Be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

FNS received approval for pre-testing the data collection procedures and instruments for this evaluation under Approved Generic OMB Clearance No. 0584-0606 (approved on December 19, 2016). This request is to obtain clearance to conduct research with State Child Nutrition and Medicaid Agency officials and school district staff to develop, test, and improve evaluation data collection instruments and methodologies. We are pre-testing the site visits, cost logs, and administrative records requests, as well as supporting materials (including written instructions) in seven States in SY 2016–2017. A representative of the Child Nutrition Agency in each State is providing input related to administrative records data collection, representatives of both Child Nutrition and Medicaid Agencies in each State are providing input related to cost logs data collection, and representatives of the same agencies and at least one district in each State are providing input related to qualitative data collection.

Respondents have generally indicated that the process of scheduling site visits was not very difficult, the relevant staff participated, and the questions were clear. They also indicated that although it was sometimes challenging to classify certain types of costs, instructions were clear and comprehensive and completing the cost logs was not very difficult. Additionally, respondents have told us that the administrative data request was clear. Based on input from pre-test respondents and experiences during the pre-test data collection year to date, we have made the following revisions to data collection instruments:

• Added materials to send to site visit contacts in each location before the site visit, including the agenda, a list of topics to be discussed, and a list of requested documents (Appendices A-2a through A-2c).

• Added a few questions or probes to site visit protocols and the cost log clarification call protocol.

• Revised text in the State administrative cost data collection materials to refer to “Medicaid eligibility agency” rather than “Medicaid agency” because in some States the agency that operates the Medicaid program differs from the agency that determines eligibility. The latter agency is typically most involved in direct certification.

• Removed both the contact information tab and the option to provide initials rather than staff positions or job titles from cost logs, to address concerns from States about personally identifiable information.

• Added two new activity categories to the cost logs.

• Added a data element to the administrative data request form to ensure that States that cannot separate TANF, FDPIR, and other programs besides SNAP and Medicaid provide a total for those other programs.

In addition, we have reflected pretest experiences in the burden estimates shown in Appendix E.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Mathematica and FNS staff, as well as staff from Insight Policy Research, were consulted on statistical aspects of the design (see Table B.5.1). The same staff are proposed to be responsible for the collection and analysis of the study’s data.

Table B.5.1. Individuals Consulted

Mathematica Staff
Lara Hulsey	Project Director	609-936-2778
Joshua Leftin	Researcher	202-250-3531
Andrew Gothro	Researcher	202-250-3569
Daniela Golinelli	Senior Statistician	202-838-3597
Quinn Moore	Senior Researcher	609-945-6592
USDA Staff
Conor McGovern	FNS Project Officer (Research Analyst-COR/OPS/FNS)	703-457-7740
John Endahl	Senior Program Analyst/OPS/FNS	703-305-2127
Vivian Lees	Senior Technical Advisor/PMOSD/CN/FNS Child Nutrition Division	703-305-2322
Jamie Blair-Walker	Operational Support Branch, Child Nutrition Division	703-457-7751
Doug Kilburg	Mathematical Statistician, NASS	202-720-3777
Insight Policy Research Staff
Brian Estes	Senior Researcher	703-504-9492

1 Lara Hulsey, Joshua Leftin, Anne Gordon, Claire Smither Wulsin, Nicholas Redel, Allen Schirm, Nicholas Beyler, Brian Estes, and Carole Trippe. “Year 2 Impacts of Using Medicaid Data to Directly Certify Students for Free School Meals.” Final report submitted to the U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support. Princeton, NJ: Mathematica Policy Research, June 2016.

2 We originally assumed that the pretest would include 18 districts, but that was based on an assumption about the number of States in the pretest that conducted only local-level matching. When we later learned that an additional state conducted central-level matching, we reduced the number of districts included in the site visit to that state accordingly.

3 Seven of these States—California, Florida, Massachusetts, Nebraska, Utah, Virginia, and West Virginia—began conducting DCM-F/RP in SY 2016–2017, and eight States—Connecticut, Indiana, Iowa, Michigan, Nevada, Texas, Washington, and Wisconsin—will begin in SY 2017–2018.

4 With central matching, the State has primary responsibility for developing and maintaining the direct certification system.

5 The States that began the demonstration in SY 2016–2017 will have provided baseline data as part of the pre-test.

6 These data elements are not available at the district level in data that are already available.