DC-M_OMB PART B (5-21-13).clean

Supporting Statement for the Evaluation of Demonstrations of NSLP/SBP Direct Certification of Children Receiving Medicaid
Benefits

Part B

May 22, 2013

Allison Magness

TABLE OF CONTENTS

Part B. Collection of Information Using Statistical Methods 1

B.1. Respondent Universe and Sampling Methods 1

1. Sample for the Access Evaluation 4

2. Samples for the Participation and Cost Evaluation 5

3. Sample for the Match Validation Substudy 8

B.2. Procedures for the Collection of Information 9

1. Access Evaluation 9

2. Participation and Cost Evaluation 12

3. Match Validation Substudy 17

B.3. Methods to Maximize Response Rates and Deal with Non-Response 18

B.4. Test of Procedures or Methods to be Undertaken 19

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 20

APPENDIX A: certification and participation data request

APPENDIX B: state cost survey tracking logs and instructions

APPENDIX C: state cost interview protocols

APPENDIX d: state cost survey correspondence

APPENDIX e: state informational letters to districts

appendix f: District cost survey AND PRELIMINARY WEB SHOTS

aPPENDIX G: DISTRICT cost survey correspondence

appendix H: challenge interview protocols

APPENDIX I: challenge interview correspondence

APPENDIX j: MATHEMATICA CONFIDENTIALITY pledge

APPENDIX k: NASS COMMENTS and responses

APPENDIX L: Public comments and responseS

APPENDIX M: Memorandum of understanding

TABLES

B.1.1 Number of School Districts in the First Year of the DC-M Demonstration 3

B.1.2 Sample for Access Evaluation 4

B.1.3 Sample for Participation and Cost Evaluation in the First Year of the DC-M Demonstration 5

B.1.4 Respondent Universe, Samples, and Expected Response Rates for Participation and Cost Evaluation 6

B.1.5 Respondent Universe, Samples, and Expected Response Rates for MVS 8

B.2.1 Precision Estimates for Access Evaluation Distribution of Students Across NSLP Certification Categories 11

B.2.2 Precision Estimates for Cost Differences in First Year of the Participation and Cost Evaluation 17

B.5.1 Individuals Consulted 20

Part B. Collection of Information Using Statistical Methods

B.1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Selecting States

The Healthy, Hunger-Free Kids Act of 2010 (HHFKA) authorized demonstrations of direct certification with Medicaid (DC-M) beginning in SY 2012-2013. The Food and Nutrition Service (FNS) of the U.S. Department of Agriculture authorized demonstrations to be conducted by States in a subset of their school districts (DC-M1) or statewide (DC-M2). The HHFKA legislation stipulates that school districts selected for the demonstration in DC-M1 States in SY 2012-2013 collectively must include no more than 2.5 percent of all students certified for free and reduced-price meals in the nation, or approximately 688,000 certified students. This limit is raised to 5.0 percent for SY 2013-2014.

FNS solicited applications from States to participate in the DC-M demonstrations, and purposively selected five States from among those that applied to begin conducting DC-M in SY 2012-2013.¹ A second application process will be used to select additional States to begin DC-M in SY 2013-2014.² For SY 2012-2013, FNS selected three DC-M1 States (Florida, Illinois, and New York) and two DC-M2 States (Kentucky and Pennsylvania).³ States applying for DC-M1 include in their applications a list of school districts to be randomly assigned to the demonstration or to a control group for the evaluation.

Identifying and Selecting School Districts for Demonstrating DC-M

In DC-M1 States, we randomly assigned school districts to the demonstration group—which will conduct DC-M—or the control group. Construction of a State’s sample frame for random assignment began with the list of school districts proposed by the State in its application for the DC-M demonstration. Districts on this list were excluded from the sample frame if they were (1) too large relative to the legislated limit of 688,000 students certified for free or reduced-price meals in school districts conducting DC-M in the DC-M1 states; (2) private or otherwise not regular, public school districts; (3) operating one of several special provisions under which meals are provided for free to all students and traditional certification procedures are not conducted each year; or (4) part of a separate FNS study of the Community Eligibility Option (CEO). We created matched pairs of school districts in the resulting sample frame in each State and randomly assigned one district in each pair to the demonstration group and one to the control group.⁴

In DC-M2 States, all school districts are in the demonstration, but not all are included in all components of the evaluation, as discussed below. The sample frame for the evaluation was restricted based on the same criteria used for the DC-M1 states.

Table B.1.1 provides details of the number of school districts in the sample frame and in the demonstration in each State in SY 2012-2013.

Table B.1.1. Number of School Districts in the First Year of the DC-M Demonstration

	Number of School Districts in
State	Universea	Sample Frame for Evaluationb		Demonstration	Control Group	Year 1 Data Collection
Florida	58	48		24	24	48
Illinois	1174	688		344	344	550c
New York	32	32		16	16	32
All DC-M1 States	1,264	768		384	384	630
Kentucky	174	122		174	0	30
Pennsylvania	881	547		881	0	30
All DC-M2 States	1,055	669		1,055	0	60
Total						690

^aThe universe is defined as the number of school districts included in the State’s application for DC-M.

^bThe number of school districts in the sample frame in DC-M2 States is less than the total number of DC-M2 demonstration districts due to exclusions discussed in the text.

^cAlthough data on all 688 demonstration and control districts in Illinois will be collected from the state, only 550 of these districts will be contacted for data collection.

The HHFKA specifies that the demonstration sample can increase to 5 percent of certified students nationwide in SY 2013-2014. To expand the sample, additional States or additional school districts in some of the original three DC-M1 States will be added to the demonstration. After any new States are selected, we will conduct random assignment and sample districts for the second year of the evaluation following procedures developed for the first year.

Different components of the DC-M study address different research questions, and the data collection activities and samples of respondents differ. The DC-M study includes an Access Evaluation, a Participation and Cost Evaluation, and a Match Validation Substudy (MVS). Next, we provide specific information on the respondent universe and sampling procedures (when applicable) for each of these study components.

1. Sample for the Access Evaluation

The Access Evaluation will identify the potential impact of DC-M on student’s access to free NSLP and SBP meals based on a retrospective match of administrative records.⁵ It will include all school districts selected for the first year demonstration in Illinois, Kentucky, and New York. In the two States that do not maintain administrative student enrollment records at the State level—Florida and Pennsylvania—the Access Evaluation will include a subset of districts selected for the demonstration. (All States maintain Medicaid enrollment data at the State—or, in New York, the city—level). For Florida and Pennsylvania, the Access Evaluation will include the three demonstration districts in the Participation and Cost evaluation with the largest numbers of students certified for free and reduced-price meals.⁶ Table B.1.2 shows the resulting sample for the Access Evaluation. We expect to achieve 100 percent response from these States and school districts.

Table B.1.2. Sample for Access Evaluation

State	Categorya	Total Number of Demonstration School Districts	Number of School Districts in Access Evaluation	Number of respondents for Medicaid enrollment data collection	Number of respondents for student enrollment data collection
Florida	DC-M1	24	3	1 (State agency)	3 (districts)
Illinois	DC-M1	344	344b	1 (State agency)	1 (State agency)
New York	DC-M1	16	16b	1 (city agency)	1 (city agency)
Kentucky	DC-M2	174	174b	1 (State agency)	1 (State agency)
Pennsylvania	DC-M2	881	3	1 (State agency)	3 (districts)

^a DC-M1 States conduct DC-M in a subset of their school districts, and DC-M2 states conduct DC-M statewide.

^b In three states (Illinois, New York, and Kentucky) a single respondent will provide student enrollment data for multiple school districts. This is not possible in Florida and Pennsylvania due to limitations of State data systems.

2. Samples for the Participation and Cost Evaluation

The Participation and Cost Evaluation will estimate the effect of DC-M on participation and on certification and meal costs for each of two school years, SY 2012-2013 and SY 2013-2014.⁷ For the first year of the evaluation, we will include all demonstration and control school districts in Florida, and New York and, due to budget constraints, a randomly selected subsample of 275 district pairs in Illinois. In each of the two DC-M2 States (Kentucky and Pennsylvania), we selected a random sample of 30 school districts using probability proportional to size (pps) sampling, where the size measure used for sample selection was district enrollment. Table B.1.3 provides details of the number of school districts in each State.

Table B.1.3. Sample for Participation and Cost Evaluation in the First Year of the DC-M Demonstration

		Number of School Districts in:
					Year 1 Analysis
State	Category	Sample Framea		Demonstration	Demonstration	Control	Demonstration and Control
Florida	DC-M1	48		24	24	24	48
Illinois	DC-M1	688		344	275	275	550
New York	DC-M1	32		16	16	16	32
All DC-M1 States		768		384	315	315	630
Kentucky	DC-M2	122		174	30	0	30
Pennsylvania	DC-M2	547		881	30	0	30
All DC-M2 States		669		1,055	60	0	60
All States		1,437		1,439	375	315	690

^a The number of school districts in the DC-M2 State sample frames is less than the total number of DC-M2 demonstration districts because districts implementing a special operating provision, private districts, and certain nonregular public school districts were removed prior to sampling, as discussed in the text.

When the number of demonstration States expands in SY 2013-2014, we plan to again include in the Participation and Cost Evaluation all demonstration and control districts in the DC-M1 States and a random sample of approximately 30 school districts in each DC-M2 State. We anticipate that the total number of school districts in the SY 2013-2014 sample will be approximately 1,200, including all the districts from the SY 2012-2013 sample.

Participation and Cost Evaluation Data Collection Respondent Samples. For the Participation and Cost Evaluation, we will conduct telephone surveys about certification costs with State Child Nutrition and Medicaid Agencies, a web survey about certification costs with districts, and telephone interviews with State and district staff about the challenges encountered in implementing DC-M. These data collection activities are described in detail in Section B.2. Here, we describe the selection of respondent samples for each activity. Table B.1.4 presents sample sizes, response frequencies, and expected response rates for the cost surveys and challenge interviews for each of the two years of the Participation and Cost Evaluation.

Table B.1.4. Respondent Universe, Samples, and Expected Response Rates for Participation and Cost Evaluation

Respondent	Data Collection	Universe	Initial Sample	Frequency of Responses per Year	Expected Response Rate	Target Completed Cases
SY 2012-2013
State CN and Medicaid Directors	Challenge Interviews	10	9	2	100%	18
State CN Directors	State CN Cost Interviews	5	5	3	100%	15
State Medicaid Directors	State Medicaid Cost Interviews	5	5	3	100%	15
SFA Directors and Business Managers	District Cost Survey	1,318 a (for 690 districtsb)	1,318	1	80%	1,054
SY 2013-2014c
State CN and Medicaid Directors	Challenge Interviews	18	18	2	100%	36
State CN and Medicaid Directors	State Cost Interviews	18	18	3	100%	54
SFA Directors	Challenge Interviews	approximately 1,200	30	2	90%	54
SFA Directors and Business Managers	District Cost Survey	approximately 2,400 (for 1,200 districts)	2,400	5	80%	9,600

^a Nine pretest respondents will also complete the district cost survey, but are not included in the sampling.

^b A single pair of respondents will provide the data for all 32 community districts in New York City. Thus, in total, there will be 659 SFA directors and 659 business managers asked to respond for 690 districts.

^c States have not yet been selected for year 2 of the demonstration, so the universe numbers are estimated.

Cost surveys. We will conduct cost interviews with the Child Nutrition Directors and the Medicaid Directors in all five States to obtain information on the costs associated with the separate activities conducted by each agency in SY 2012-2013 and SY 2013-2014. State cost survey tracking logs are presented in Appendix B, and protocols for the follow-up state cost interviews are presented in Appendix C. Each director (or designee) will be interviewed three times during the school year.

We will conduct a district cost survey (Appendix F) on the web to collect information from school districts on certification costs in SY 2012-2013. This survey will begin in summer 2013, after OMB approval has been granted. At that time, all districts in the Participation and Cost Evaluation will be asked to complete the survey, and we have assumed that there will be two respondents—the School Food Authority (SFA) director and business manager—for each school district.⁸ For SY 2013-2014, we will collect certification cost data five times during the school year, with collection occurring shortly after such costs have been incurred.

Challenge interviews. In SY 2012-2013, we will conduct challenge interviews (Appendix H) with one or two State-level staff per State, for a total of nine respondents. Respondents will be selected in discussions with each State, but are likely to include State Child Nutrition Directors and, in some States, State Medicaid agency staff. For SY 2013-2014 challenge interviews, we will interview two State-level staff in each study State, for a total of 18 State-level interviewees (based on the assumption that nine States will be in the study in SY 2013-2014). In addition, we will select a subsample of the demonstration districts in the cost data collection for challenge interviews. The sampling of school districts for these interviews will be done purposively, and district interviews will be conducted only in States where DC-M matching is conducted at the school district level. We will interview staff from an average of 6 districts in each of these States, for a total of 30 school districts (based on the assumption that district-level DC-M matching will be conducted in five of the States in the study in SY 2013-2014).

3. Sample for the Match Validation Substudy

The Match Validation Substudy (MVS) will independently validate DC-M matches of student enrollment and Medicaid data in selected demonstration districts. The substudy will include 12 districts across three States, including one State that has a Statewide Student Information System (SSIS) and two that do not. We plan to select the States in early summer 2013, immediately after the SY 2013-2014 DC-M States are selected. In consultation with FNS, we will select two States that began conducting DC-M in SY 2012-2013 (one that has an SSIS and one that does not) and one State that will begin conducting DC-M in SY 2013-2014.

After selecting the States, we will select 12 school districts: 2 in the State with an SSIS and 5 in each of the two States without an SSIS. We will select these districts randomly, after stratifying to ensure variation on key dimensions to be determined in consultation with FNS. Potential stratifying variables include district size and the percent of students eligible for free or reduced price meals. We expect to achieve 100 percent response from these selected school districts for this substudy. Table B.1.5 displays the MVS sample.

Table B.1.5. Respondent Universe, Samples, and Expected Response Rates for MVS

Respondent	Data Collection	Universe	Initial Sample	Frequency of Responses per Year	Expected Response Rate	Target Completed Cases
State Medicaid Directors	Medicaid enrollment data	9	3	1	100%	3
State CN Directors	School enrollment data and match results	9	1	2	100%	2
SFA Directors and Data Managers	School enrollment data and match results	approximately 2,400 (in 1,200 districts)a	20	2	100%	40

^a States have not yet been selected for year 2 of the demonstration, so this number is an estimate.

B.2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

In this section, we provide information on the specific data collection activities for each key DC-M study component: the Access Evaluation, the Participation and Cost Evaluation, and the MVS.

1. Access Evaluation

The Access Evaluation analysis will primarily involve the collection of data from two sources: (1) State Medicaid enrollment files and (2) student enrollment files.

• Medicaid Data. We have requested Medicaid enrollment records for every child up to age 19 who was enrolled in Medicaid for at least one month between July 2011 and March 2012. We have requested two types of variables: (1) individual identifiers used to match the Medicaid data with student enrollment data and (2) eligibility, enrollment, and income data used to determine direct certification status. We are collecting these data from the Medicaid agency in each of the five States.

• Student Enrollment Data. Student enrollment files will include one record for every student enrolled in the school district. We have requested the actual student enrollment files used by the State or district for direct certification at three points in time: (1) the first month in which matching is conducted, (2) the match conducted in or closest to October 2011, and (3) the match conducted in or closest to January 2012. Where direct certification files are unavailable, however, we requested files showing student enrollment at the start of each of those months. The data elements requested from State/local agencies fall into two categories: (1) identifying information to match students with children in Medicaid records and (2) certification status for free, reduced-price, or paid meals. We will collect these data from a single child nutrition agency in each of three States (Illinois, Kentucky, and New York) and directly from three school districts each in Florida and Pennsylvania.

For the Access Evaluation, we will match students in these two types of data files using different matching algorithms at three levels of stringency, and assess whether Medicaid income data indicate eligibility for free or reduced-price meals under DC-M. Analyses will compare the distribution of certification status determined through this simulation with actual certification status in SY 2011–2012. Separate simulations will show the likely impact of DC-M under different matching algorithms and different policies.

Table B.2.1 provides an estimated breakdown of the number and percent of students across certification categories, separately for each state, the combined sample of school districts from DC-M1 States, the combined sample from DC-M2 States, and all States. The 95 percent confidence interval half widths are also provided for each percent estimate in the table. In Illinois, Kentucky, and the combined DC-M1 sample, all confidence interval half widths are below the target of three percentage points. Due to the number of school districts available for sampling or data collection budget constraints, the precision target could not be met for the other States.

Table B.2.1. Precision Estimates for Access Evaluation Distribution of Students Across NSLP Certification Categories

State (number of districts)	SNAP	FDPIR	TANF	Other	Categorically Eligible	Household Application	Total Free	Reduced-Price	Paid	Total
Florida (3)	16,615 9% 12.2	1,846 1% NA	16,615 9% 12.2	1,846 1% NA	11,077 6% 10.1	29,538 16% 15.6	77,538 42% 21.0	12,923 7% 10.8	94,153 51% 21.2	184,614 100%
Illinois (344)	55,395 9% 0.8	6,155 1% NA	55,395 9% 0.8	6,155 1% NA	36,930 6% 0.7	98,480 16% 1.1	258,510 42% 1.4	43,085 7% 0.7	313,905 51% 1.5	615,500 100%
New York (16)	15,022 9% 3.8	1,669 1% NA	15,022 9% 3.8	1,669 1% NA	10,015 6% 3.2	26,707 16% 4.9	70,105 42% 6.6	11,684 7% 3.4	85,127 51% 6.7	166,916 100%
DC-M1 Total	87,033 19% 1.1	9,670 1% NA	87,033 9% 1.1	9,670 1% NA	58,022 6% 0.9	154,725 16% 1.4	406,153 42% 1.9	67,692 7% 1.0	493,185 51% 1.9	967,030 100%
Kentucky (174)	61,504 9% 0.1	6,834 1% NA	61,504 9% 0.1	6,384 1% NA	41,003 6% 0.1	109,341 16% 0.1	287,020 42% 0.1	47,837 7% 0.1	348,524 51% 0.1	683,381 100%
Pennsylvania (3)	3,971 9% 11.5	441 1% NA	3,971 9% 11.5	441 1% NA	2,647 6% 9.5	7,059 16% 14.7	18,531 42% 19.8	3,088 7% 10.2	22,502 51% 20.0	44,121 100%
DC-M2 Total	65,475 9% 9.5	7,275 1% NA	65,475 9% 9.5	7,275 1% NA	43,650 6% 7.9	116,400 16% 12.2	305,551 42% 16.4	50,925 7% 8.5	371,026 51% 16.6	727,502 100%
DC-M1 and DC-M2 Total	152,508 9% 5.4	16,945 1% NA	152,508 9% 5.4	16,945 1% NA	101,672 6% 4.5	271,125 16% 6.9	711,703 42% 9.3	118,617 7% 4.8	864,211 51% 9.4	1,694,532 100%

Notes: It is assumed that certification data will be obtained for all school districts included in the study.

The three values in each cell are the estimated number of students, the estimated percentage of students, and the 95% confidence interval (CI) half width for the estimated percentage of students.

The total count in each row (last column of each row) is based on 742 enrollment data for school districts in the study, and that count is distributed across categories (SNAP, FDPIR, etc.) according to national percentages from the 742 data.

The cells highlighted in blue indicate that the precision target (95% CI half width of +/- 3 percentage points or less) is met. The target does not pertain to the very small cells highlighted in gray.

An intraclass correlation coefficient (ICC) of 0.15 is used for all CI half width computations (based on preliminary calculations using CCD data). All computations include finite population corrections (FPCs) to capture the effects on precision of the generally high sampling rate in each demonstration state. Some school districts were excluded from the sampling frames prior to sampling and are not counted in the FPC calculations. These exclusions limit the ability to generalize to the entire state (or beyond). The exclusions that most affect generalizability were imposed by the states or were necessary so as to not violate the limitation on the number of FRPL students that could be included in the demonstration districts in the DC-M1 states.

95% CI half widths in the DC-M1 Total, DC-M2 Total, and DC-M1 and DC-M2 Total rows are computed using weighted state-level variances where the weights are based on the number of school districts in the state sampling frames. These aggregations are not necessarily generalizable because of both the aforementioned exclusions of districts and the nonrandom selection of states.

2. Participation and Cost Evaluation

For the Participation and Cost Evaluation, we will collect four key types of data: (1) administrative data on participation and certification, (2) State-level cost data, (3) district-level cost data, and (4) data on DC-M challenges. Administrative participation data will include information on certification, as well as on participation in the NSLP, SBP, NSLP Afterschool Snack Program (ASP), and Special Milk Program (SMP). Cost data will include information from States and school districts on start-up costs and ongoing certification costs, as well as published information to be used (in conjunction with participation data) to assess meal costs and reimbursements. We will collect data on administrative costs incurred at the State level through interviews with State Child Nutrition Directors and Medicaid Directors. At the district, we will collect data on local-level certification costs through a web survey of SFA directors and business managers (Appendix F). Data on challenges to DC-M implementation will be collected through semi-structured interviews (protocols presented in Appendix H) with State agency and district staff. Sample sizes for each data collection activity were presented in Section B.1 (see Table B.1.4).

a. Participation and Certification Data

We will collect administrative data on certification and meal participation for each school district in either the demonstration or control group in a DC-M1 State and for each sampled district in a DC-M2 State. We will collect these data for both demonstration school years. We also will request the information for SY 2011–2012 to (1) help improve the precision of our estimates of the impacts of DC-M on certification and participation and (2) facilitate pre-post comparisons.⁹ Key district-level data to be collected for each period fall into two broad categories: (1) information on enrolled students by certification status and basis for certification; and (2) monthly participation information for the NSLP, SBP, ASP, and SMP.

We will work with each demonstration State to collect these district-level data, which States typically collect from school districts for administrative reporting. The reference date for the certification data will be the last operating day in October; because districts must report certification statistics to FNS on Form 742 as of that date, we expect these data to be readily available. Likewise, we expect participation data to be readily available because such data are reported by school districts to obtain reimbursement for foods served. We will contact each State to request the certification and participation data required and will provide any additional technical assistance required in transmitting the data files to Mathematica.

b. State Cost Data

For the State-level cost data collection, State agency contacts will be sent tracking logs (Appendix B) to monitor hours spent on activities related to DC-M, and then interviewed three times during the school year so we can develop a more detailed understanding of the costs described and ascertain if some costs have been missed (State cost interview protocols presented in Appendix C). Prior to the data collection, we will send each State agency contact an email and letter (Appendix D) about the study. We will also explain the cost data collection during one of the monthly conference calls FNS holds with representatives of each demonstration State, including the tracking logs and the follow-up interviews..

In SY 2012-2013 and SY 2013-2014, Mathematica will introduce State agency staff to the study in July. We will encourage them to keep monthly or weekly logs of DC-M activities, staff involved in the activity, and estimated hours. We will ask them to complete logs for July through March. Logs will be collected quarterly, and the interviews will be conducted early in the next month. The first follow-up interview will be in November, covering July through September; the second in February, covering October through December; and the third will be in April, covering January through March.

c. District Cost Data

At the school district level, the data collection on costs will include the start-up costs of DC-M and the ongoing costs related both to direct certification and to certification using household applications. To collect the cost data at the district level, we will administer a web-based survey (Appendix F) to the district respondents. We will ask our contact at each State Child Nutrition Agency to identify the best respondent for the district cost survey in each school district in the sample and to initiate contact with districts. We will provide draft letters/emails (Appendix E) for State agency staff to use to inform school districts of the upcoming data collection and instruct them to participate. We will then send each district designee a letter (Appendix G) with an overview of the study (presented in Appendix E). and asking them to serve as the main respondent for the survey, or to designate someone else and provide their contact information.

The district cost survey will ask the respondent to identify which staff conducted specific certification activities. For each staff member, the respondent will record the total time spent on certification activities and the person’s direct and indirect labor rates. The survey will also ask about other categories of costs related to certification and about school district characteristics and direct certification procedures.

The sample sizes and timing of the survey will differ by year:

• In the first year of the study, 690 school districts will be included in the survey.¹⁰ In July 2013, we will ask district staff to provide information on costs for the previous year, SY 2012-2013. We will ask respondents to report on certification activities during the initial part of the school year (August-October 2012), and then we will ask them to report for a typical month during the rest of the school year. This initial survey is expected to take about two hours, including preparation time (for example, to look up salary rates).

• In the second year, there will be approximately 1,200 school districts. Beginning in September 2013, we will ask districts to provide cost information bimonthly for the two previous calendar months. These subsequent monthly web surveys, including the tracking necessary to report on the costs, are expected to take 1.5 hours per district. We will request this information five times during SY 2013-2014.

d. Challenges Data

In both SY 2012-2013 and SY 2013-2014, we plan to conduct two rounds of semi-structured telephone interviews with respondents to learn about their challenges with DC-M (challenge interview protocols are presented in Appendix H). The first round of interviews will take place in September or October and will focus on challenges experienced in the initial DC-M match at the beginning of, or shortly before, the school year. The second round will be conducted in February, to identify challenges with subsequent matching. All interviews will be conducted by telephone, and each call is expected to take approximately one hour to complete. All respondents will be reminded that their responses will be kept private (Mathematica’s confidentiality pledge is presented in Appendix J).

State interviews. We will first send a letter, by email (Appendix I), to the primary State contact for the study. The letter will explain the purpose of the interviews. Two State offices are likely to be involved: (1) the State Child Nutrition Agency and (2) the State Medicaid Agency. We will conduct a separate interview with each of the two agencies in SY 2013-2014. In SY 2012-2013, we will interview only nine individuals, including at least one from each State. Staff in each agency will be given the opportunity to determine who is most suitable for participation in the interview.

The email message will briefly explain the purpose of the study and describe the general nature of the interview questions. An interviewer will follow up with each person by telephone and attempt to schedule the interview. After scheduling the interview, the interviewer will send the research topics to the respondent so that he or she is prepared for the interview and also can invite additional staff to join the call if there are questions that would best be answered by those staff members.

District interviews. District interviews will follow the same approach as State interviews. We will conduct only one interview per school district in the sample, typically with the director of the school meals programs. However, as with the State interviews, the primary respondent may chose to invite additional school district staff members to participate in the interview.

Analysis and Precision

Analyses for the Participation and Cost Evaluation will include the following:

• Impact estimates. We will estimate the impact of DC-M on (1) participation in the NSLP and SBP and (2) Federal meal costs and State and local administrative and implementation costs over two years of demonstrations. To estimate the impacts, we will compare the participation and cost outcomes of each demonstration school district in DC-M1 States with the outcomes of its matched control group district, aggregating these differences across school districts to generate State-level estimates of the impacts of DC-M. For DC-M2 States, we will compare within-State changes from before to after DC-M was implemented.

• National cost projections. The results of the impacts analysis will be used to develop national projections of the impact of DC-M, assuming national implementation of DC-M in the future. Projections will be based on assumptions about (1) how to generalize the participation and cost results to other States and (2) how impacts evolve over time.

• Challenges. We will conduct a qualitative analysis of the challenges that States and school districts face when implementing DC-M. To identify key challenges, we will code the interview data using NVivo, and examine themes in the coded data.

Our precision calculations focus on the precision of estimating the percentage difference in average certification costs per school district between demonstration and control districts in SY 2012-2013. Under the null hypothesis of no difference in costs, the 95 percent confidence interval for the impact estimate has a half width of 2.7 percentage points in the full sample of districts from all four DC-M1 States (Table B.2.2). For individual States, the 95 percent confidence interval for the impact estimate has half-widths ranging from 2.9 percentage points in Illinois to 12.0 in New York. These calculations reflect the substantial precisions gains from conducting random assignment within matched pairs of districts that are already expected to be similar on many of the characteristics that might induce variation in costs.

Table B.2.2. Precision Estimates for Cost Differences in First Year of the Participation and Cost Evaluation

State	Number of School Districts	Number of School District Pairs	90% CI Half Width	95% CI Half Width
Florida	48	24	8.2	9.8
Illinois	550	275	2.4	2.9
New York	32	16	10.1	12.0
DC-M1 Total	630	315	2.3	2.7

Notes: The precision calculations for states assume a within-pair standard deviation of 0.10 and a between-pair standard deviation of 0.20. These assumptions are based on calculations using preliminary SY 2011-12 administrative (Form 742) data. It is assumed that certification data will be obtained for all school districts included in the study.

The precision calculations for the DCM1 Total row are computed using weighted state-level variances where the weights are based on the number of school districts in the state sampling frames. These aggregations are not necessarily generalizable because of exclusions of districts and the nonrandom selection of states.

3. Match Validation Substudy

The MVS will collect data for SY 2013-2014 to independently validate matches made in 12 selected demonstration districts. Data will include the individual-level Medicaid and student enrollment files used in the direct certification process at the beginning of SY 2013-2014 as well as a file indicating the outcome of DC-M. We will also collect detailed information on the rules and algorithms used for the matching.

We will collect Medicaid data from State agency staff in all three States in the MVS. Most other data as well as information on matching rules will be collected from the entity that conducts the match: the State agency or the school district. In addition, in the States in which districts conduct matching, we will ask State-level staff about any statewide guidelines they expect all school districts to follow.

The data collection required for the substudy will be described in the MOU (Appendix M) signed between Mathematica and each State when it joins the demonstration. In summer 2013, we will contact each State to let it know which school districts have been selected for the substudy, to request the State-provided data, and, in the two States where data will be collected directly from districts, to request contact information for district staff. We will also provide a letter for these States to send to the districts informing them of the study and instructing them to provide the required data to Mathematica.

Mathematica staff will work with State and school district data managers to address any questions and provide technical assistance, as needed. State and districts will transmit their data to Mathematica using a secure file transfer protocol (FTP) site. Each State or district will have a unique password to access the site. To ensure that the files provided to Mathematica are the same files States and districts use for direct certification, we will collect each file when it is created, rather than waiting until the completion of the matching process to collect all of the data.

The MVS will include two key sets of analyses. First, for each school district in the MVS, the results of the matching conducted by the district (or its State) will be compared with the matching results obtained by Mathematica when using the same matching process, variables, and algorithms that were used by the district (or the State). Second, the results of the matching conducted by each district (or its State) will be compared with the results from matching using algorithms for each of three levels of match stringency, with separate results presented for each of the three levels.

B.3. Methods to Maximize Response Rates and Deal with Non-Response

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Anticipated response rates are shown in Table B.1.3 (see Section B.1). Response rate assumptions are based on our experiences with similar studies. To maximize response rates and ensure the highest quality data possible, we will take a multi-pronged approach:

• To gain attention and legitimize the demonstration, a letter from each State Child Nutrition director will be sent to each selected school district to build support for the study and encourage their full cooperation. The letter will describe the importance of the study, privacy protections, general information about the study, and instructions on whom to contact with questions or for additional information.

• Web administration allows respondents to complete the survey when they wish. We will design the survey so that it can be entered and exited as convenient.

• We will track which respondents have completed the cost instruments and send email reminders to those who have not.

• We will contact respondents who are slow in completing their forms to answer their questions, remind them of the importance of the demonstration evaluation, and help them complete their forms, if necessary.

• As appropriate, State Child Nutrition staff or FNS staff may contact the most reluctant respondents to underscore the requirement of study participation.

• Staff conducting the challenge interviews and the cost data collection follow-up interviews will be qualified, well-trained professional interviewers. Project-specific training will emphasize achieving high response rates by focusing on the privacy protections that respondents can be assured of, and by using refusal conversion techniques.

• Our expectation based on the experience of similar studies is that the planned methods of data collection will result in accurate and reliable data necessary for planned analyses and modeling at acceptable response rates. The number of completed instruments will be the numerator in response rate calculations. A completed instrument will be defined as one in which all critical items for inclusion in the main analysis are complete and within valid ranges.

B.4. Test of Procedures or Methods to be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Mathematica will pretest the district cost survey instrument designed for the web and any supporting materials (including written instructions and/or definitions) with up to 9 pretest respondents before data collection. After the final items have been developed and approved, we will program the web survey and conduct usability testing to ensure it is user-friendly and retains its format across various browsers. Additional testing to check the web survey will be conducted by a web instrument testing team. Errors identified will be documented, corrected, and rechecked. We will perform this process iteratively throughout the development of the instrument.

Additionally, the State-level cost and challenges data collection in SY 2012-2013 will allow us to refine our procedures and protocols in preparation for the more extensive data collection in SY 2013-2014.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Mathematica and FNS staff were consulted on statistical aspects of the design (see Table B.5.1). The same staff will be responsible for the collection and analysis of the study’s data. Michael Jacobsen from National Agricultural Statistics Service was also consulted.

Table B.5.1. Individuals Consulted

Mathematica Staff
Allen Schirm	Project Director	202-484-4686
Lara Hulsey	Deputy Project Director/Senior Researcher	609-936-2778
Nicholas Beyler	Statistician	202-250-3539
Frank Yoon	Statistician	202-554-7518
John Hall	Senior Statistician	609-275-2357
John Deke	Senior Researcher	609-275-2230
Sheng Wang	Statistician	609-936-2799
Anne Gordon	Cost Survey Task Leader/Senior Researcher	609-275-2318
Kimberly Smith	Researcher	609-945-3354
Nancy Cole	Senior Researcher	617-674-8353
FNS Staff
Allison Magness	FNS Project Officer	703-305-2098
Melissa Abelev	Branch Chief, Special Nutrition Evaluation Branch, Office of Research and Analysis	703-305-2209
Vivian Lees	Branch Chief, State Systems Support Branch, Child Nutrition Division	703-305-2322
NASS Staff
Michael Jacobsen	NASS Methods Reviewer	202-690-8639

1 A sixth State, Alaska, was initially selected but later withdrew from the demonstration.

2 Although the States have not yet been selected, the number of districts has been estimated (based on the legislated limit on the number of free and reduced-price students) and included in the burden calculations. Estimates assume four additional States will be selected in year 2.

3 In New York, only New York City is participating in the demonstration. Public schools under the authority of the New York City Department of Education are divided among 32 community districts. These 32 community districts will be randomly assigned to demonstration and control groups.

4 Districts were paired based on a set of variables including the percentage of students eligible for free or reduced-price meals, the percentage of eligible students that were certified based on an application, the overall school lunch participation rate, and a weighted average of the percentage of free meals served, the percentage of reduced-price meals served, and the percentage of full-price meals served, where the weights are the per-meal reimbursement rates.

5 We are not requesting clearance for the Access Evaluation. Each related data collection activity involves fewer than ten individuals and will have been completed before the package is submitted to OMB. Thus, no instruments for the Access Evaluation are included in the appendices. We mention the Access Evaluation here only for completeness, because it is part of the same study as the components for which we are requesting clearance and will inform these later components.

6 Sampling for the Participation and Cost Evaluation is described below.

7 The data collected for the Participation and Cost evaluation will also be used to examine the conditions that would make the use of a socioeconomic survey (SES) to establish Federal reimbursement claiming rates a cost-effective alternative to standard certification and reimbursement procedures (with or without DC-M).

8 A single pair of respondents will provide the data for all 32 community districts in New York City. Thus, in total, there will be 1,334 respondents asked to respond across 698 districts.

9 We obtained some SY 2011–2012 data from States during the sample frame development process and will not need to request such data again.

10 This number includes all 32 community districts in New York City. However, a single pair of respondents will provide the data for all 32 community districts. Thus, in total, there will be 1318 respondents (659 SFA directors and 659 business managers) asked to report on 690 districts.