0584-NEW Direct Ver Eval Study Part B REV

Part B
Collection of Information Employing Statistical Methods

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The following selections have been or will be made in support of the Direct Verification Evaluation Study:

1. Selection of case study States

2. Selection of LEAs within States:

1. Direct Verification Sample - To provide data on verification activities in SY2007-08

2. Nonresponse Sample - To provide retrospective data on verification nonresponse in SY2006-07

Independent samples of LEAs will be selected for (a) and (b) because the outcome measures require different measures of size for the selection process. The Nonresponse Sample requires LEAs that did not effectively use direct verification with Medicaid (DV-M) in SY2006-07. In LEAs that used DV-M, household nonresponse to verification occurred after directly verified applications were identified. Therefore, Tennessee and Washington are excluded from the Nonresponse Sample because the implementation of direct verification with Medicaid (DV-M) in these States in SY2006-07 was sufficiently widespread and effective that too few LEAs could be included.

The potential respondent universes and selection methods for each of the above samples are described below.

Selection of Case-Study States

FNS recruited seven States for this study: Georgia, Indiana, Oregon, South Carolina, Tennessee, Washington, and Wisconsin.

FNS sent a letter on June 30, 2005 to all State Child Nutrition (CN) Agencies, requesting voluntary participation in a pilot study. Five States volunteered for the study: Arizona, Indiana, South Carolina, Tennessee, and Washington. Arizona subsequently declined to participate in the pilot study. FNS recruited Oregon in August 2006, because Oregon had expressed interest in the study. As noted in section A.1, four States implemented DV-M in SY2006-07, although valid estimates of DV-M effectiveness were obtained from only two States.

In preparation for an expanded study of DV-M in SY2007-08, FNS contacted 14 States in April/May 2007 to request their participation. To date, Georgia and Wisconsin have volunteered to join the study.

Exhibit B1-1 presents characteristics of the selected states: geographic location, NSLP free and reduced-price applications, NSLP verification samples, and Medicaid income-eligibility limits for school-age children.

Exhibit B-1 Characteristics of Selected States
		NSLP Characteristics in Public LEAs					Medicaid / SCHIP income eligibility limit for school-age childrenb
State	Region	Number of LEAs	Applications approved for NSLP meals	Applications sampled for verification	Nonrespondents to verification requestsa
Georgia	Southeast	160	333,146	7,983	1,961		100 / 235
Indiana	Midwest	317	173,264	9,606	1,638		150 / 200
Oregon	West	177	68,433	2,208	569		100 / 185
South Carolina	Southeast	85	141,626	4,382	1,684		150 / na
Tennessee	Southeast	138	152,092	3,932	1,363		100 / na
Washington*	West	265	NA	3,222	894		200 / 250
Wisconsin	Midwest	429	104,638	2,998	493		200 / na
a The number of nonrespondents is the number of households (applications) not responding to verification requests. b Medicaid/SCHIP income eligibility limits are expressed as a percentage of the federal poverty level. “na” indicates no separate SCHIP program. * Washington data needed for sampling were incomplete at the time this package was prepared and are indicated with “NA” in the table.
Sources: USDA, Food and Nutrition Service. Verification Summary Reports, SY2006-07; USDA Survey of State Medicaid Agencies, 2005.

The selected States are located in three of the seven USDA regions (Mountain Plains, Northeast, Southwest, and Mid-Atlantic are not represented). The States represent a range of size from 68,433 NSLP applications in Oregon to 333,146 in Georgia (for all U.S. States the mean is 211,000 and the median is 124,500).

The selected States have a range of maximum Medicaid/SCHIP income eligibility limits. The Medicaid limit in Tennessee (100% FPL) is below the income eligibility limit for NSLP-free meals (130% FPL); the Medicaid/SCHIP limit in South Carolina is above the limit for NSLP-free (130% FPL) and below the limit for NSLP reduce-price (RP) meals (185% FPL); and the remaining four States have a maximum Medicaid/SCHIP income eligibility limit at or above the limit for NSLP-RP.¹

Sample Frame of LEAs Within States

The study requires probability samples of LEAs in each participating State. The sample frame is constructed from administrative data collected annually from LEAs by USDA Food and Nutrition Service (FNS), through the Verification Summary Report (OMB No. 0584–0026). The VSR collects the following data items from each LEA:

• Number of schools operating the NSLP

• Number of students and number of approved applications subject to verification, in each category of eligibility

• Number of students and applications selected for verification, in each category of eligibility and each disposition status

VSR data from SY2006-07 are used as the basis for the sample frame for this study.

LEA Sample #1 – Direct Verification Sample – Collection of SY2007-08 information

This sample is designed to provide information to estimate three outcome measures at the State level:

The percentage of applications (in the verification samples) that are directly verified with Medicaid data:

• Among all applications

• Among applications for NSLP-free

• Among applications for NSLP-RP

An independent sample of LEAs is selected for each State from the sample frame of all public LEAs. Measures of size are taken from the SY2006-07 VSR. An appropriate measure of size is the number of applications verified in SY2006–07.

The use of applications as the basis for outcome measures and measures of size reflects FNS guidance. The eligibility of all children listed on an application is verified when Medicaid data verify the eligibility of one child listed on the application.²

The precision required for a statewide estimate is a 95% confidence interval (CI) whose half-width is .03 (i.e., 3 percentage points—all the outcome measures in the study are proportions). This requirement applies to the overall rate of directly verified applications. The other two outcome measures are considered subgroup estimates, for which the required precision is a 95% CI whose half-width is .05. The actual sample sizes depend on the values that we assume for the underlying percentages (the worst-case assumption of 50% is unnecessarily conservative). The assumed percentages for each State are based on the Medicaid income eligibility limit in the State and the results from the SY2006-07 pilot study results in Tennessee and Washington, given the Medicaid income eligibility limits in those States. These values are shown in Exhibit B-2.

For each State, a few LEAs with the largest numbers of applications are designated as self-representing, and are automatically in the sample. A sample of the remaining school districts is then selected with probability proportional to size (PPS). The basic objective is to select a number of LEAs that would yield a sufficient sample of applications selected for verification.

Exhibit B-3 provides characteristics of the sampling frame, including the total number of public LEAs, number of LEAs designated as self-representing, and average size of verification samples. Exhibit B-4 provides characteristics of the Direct Verification sample of LEAs selected for the study. The sample of districts ranges from 14 in Georgia to 40 in Indiana, reflecting differences in the average number of applications sampled per LEA.

Exhibit B-2 Information About Outcome Measures for Sample Size Calculations
	GA	IN	OR	SC	TN	WA	WI
Medicaid/SCHIP income-eligibility limita (percent of poverty level)	100	200	185	150	100	250	200
Estimate of the percentage of applications directly verified with Medicaidb
Among all applications	10	18	18	15	10	18	18
Among free applications	14	18	18	18	14	18	18
Among RP applications	3	19	19	11	3	19	19
a The GA Medicaid limit is shown because SCHIP data will be not used for direct verification. b TN and WA estimates are among LEAs using direct verification with Medicaid in the SY2006-07 pilot study. Estimates for other States are interpolated from TN and WA estimates according to the Medicaid income eligibility limit of the State relative to the maximum income limit for NSLP (185% FPL).

Exhibit B-3 Characteristics of the Sampling Frame of LEAs
	GA	IN	OR	SC	TN	WA*	WI
All LEAs	160	317	177	85	138	265	426
Self-representing LEAs	2	8	5	3	3	NA	6
Avg # applications in verification samples
Self-representing LEAs	722	137	97	262	530	NA	177
Non-self-representing LEAs	43	13	9	42	18	NA	5

* Washington data needed for sampling were incomplete at the time this package was prepared and are indicated with “NA” in the table.

Exhibit B-4 Characteristics of Sample #1 – Direct Verification Sample – Collection of SY2007-08 Information
	GA	IN	OR	SC	TN	WA*	WI
Number of LEAs
Self-representing LEAs	2	8	5	3	3	NA	6
Districts in PPS stratum	12	32	32	18	13	NA	38
Total LEAs	14	40	37	21	16	NA	44
Expected sample size of applications
In self-representing LEAs	1444	1095	483	787	1591	NA	1063
In PPS stratum	1953	934	717	1298	569	NA	475
Total	3397	2029	1200	2085	2160	NA	1538

* Washington data needed for sampling were incomplete at the time this package was prepared and are indicated with “NA” in the table.

Exhibit B-5 Characteristics of Sample #2 – Nonresponse Sample – Collection of SY2006-07 Information
	GA	IN	OR	SC	WI
Number of LEAs
Self-representing LEAs	3	8	12	2	12
LEAs in PPS stratum	9	25	23	12	40
Total LEAs	12	33	35	14	52
Expected sample size of households that did not respond to verification requests
In self-representing LEAs	586	477	264	355	226
In PPS stratum	428	306	133	475	88
Total	1014	783	397	830	314

Note: Tennessee and Washington are not included in Sample #2 because those States successfully implemented direct verification in SY2006-07.

LEA Sample #2 –Nonresponse Sample – Collection of SY2006-07 information

This sample will provide retrospective administrative data needed to estimate the following outcome measure:

• The percentage of applications (in the verification samples) that are directly verified with Medicaid data (on a retrospective basis), among households that did not respond to verification requests in SY2006-07.

An independent sample of LEAs will be selected for each of five States from the sample frame of all public LEAs with any nonrespondents. It is possible that some LEAs will be selected for both samples #1 and #2. Tennessee and Washington are not included in sample #2 because of the substantial proportions of applications that were directly verified with Medicaid in SY2006-07, so that nonresponse is net of directly verified applications.

Measures of size are taken from the SY2006-07 VSR. An appropriate measure of size is the number of applications sampled for verification with no household response.

The outcome measure is considered a subgroup estimate, for which the required precision is a 95% confidence interval (CI) whose half-width is .05 (i.e., 5 percentage points—the outcome measure is a proportion). The actual sample sizes depend on the values that we assume for the underlying percentages (the worst-case assumption of 50% is unnecessarily conservative). The assumed percentage of directly verified nonrespondents has an upper bound equal to the assumed percentage directly verified among all applications sampled for verification. The directly verified percent of nonrespondents may be lower than the directly verified percent of the overall verification sample, because nonrespondents are less likely to be eligible for NSLP benefits.³ The upper bound provides a conservative measure for the purpose of sample size calculations (required sample sizes grow as proportions increase toward .50). Thus, the assumed percentages presented in Exhibit B-2 are used for sample size calculations, albeit with a lower required level of precision.

Exhibit B-5 provides characteristics of the Nonresponse Sample of districts selected for the study. The number of selected districts ranges from 12 in Georgia to 38 in Oregon.

Across all States, 43 LEAs are selected for both sample #1 and sample #2. The total numbers of unique LEAs selected for the study are: 18 in Georgia, 64 in Indiana, 112 in Oregon, 28 in South Carolina, and 16 in Tennessee.

B.2 Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data collection procedures have been developed for each of the data collection activities.

State Agency Interviews

The State Agency interviews conducted in November/December 2007 will collect information about direct verification implementation. State agencies will be contacted via e-mail to schedule an interview; this e-mail will include the interview topic guide and suggested times for the interview. The interview will be scheduled no earlier than one week after the State receives the topic guide. Interviews will be conducted by telephone, and Abt Associates will request permission to tape the interview, to ensure that all responses are documented accurately.

LEA Sample #1 – Direct Verification Sample (SY2007-08 Information) – Survey and Administrative Data

LEAs will be recruited to participate in the study during August 2007. Abt Associates will send a surface mailing to selected LEAs in August 2007. The mailing will include a summary of the study; explanation of the importance of collecting information from the respondent; a pamphlet summarizing results from the first year of the pilot; and a summary and schedule of data collection activities (LEA brochure). This mailing will include a pre-paid postcard and the request that LEAs confirm receipt of the mailing and willingness to participate in the study by checking responses on the postcard and returning the postcard.

Data collection forms will be sent to LEAs after OMB clearance is obtained, in October 2007. These forms will be sent via surface mail and via e-mail:

• The surface mailing will include a copy of the LEA brochure (previously sent in August), data collection forms, an envelope marked “confidential” for enclosing photocopies of NSLP applications, and a Federal Express mailing envelope with the contactor’s address and billing information pre-filled.

• The e-mail will contain the data collection forms suitable for printing, a link to the Web survey site, and password for gaining access to that site.

LEA responses will be due November 30. Abt Associates will send reminder e-mails to selected LEAs on November 26 and December 3. Beginning on December 15, Abt Associates will telephone all nonrespondents to encourage response.

LEA Sample #2 – Nonresponse Sample (SY2006-07 Information) – Administrative Data

Data collection procedures for this sample of LEAs will be analogous to the procedures for LEA sample #1, except this sample will provide administrative data and no survey responses.

• LEAs will be recruited to participate in the study during August 2007, using the same procedures used for LEA sample #1.

• Data collection requests will be sent to LEAs via surface mailing after OMB clearance is obtained, in October 2007. The surface mailing will include a copy of the LEA brochure (previously sent in August), an envelope marked “confidential” for enclosing photocopies of NSLP applications, and a Federal Express mailing envelope with the contactor’s address and billing information pre-filled.

• An e-mail will be sent to respondents two days after the surface mailing to notify them of the mailing and request that they notify Abt Associates if the mailing is not received.

• LEA responses will be due November 30. Abt Associates will send reminder e-mails to selected LEAs on November 26 and December 3. Beginning on December 15, Abt Associates will telephone all nonrespondents to encourage response.

State Administrative Data

As discussed in Section A, the contractor will obtain files of FS/TANF and Medicaid children from State agencies. The files will include child names and identifiers (corresponding to the data on the NSLP application). The Medicaid data will include family size and income data, needed to determine NSLP eligibility level. These files will be used for the verification nonresponse analysis. LEAs in sample #2 will provide copies of NSLP applications for households that failed to respond to verification requests in SY2006-07, and the contractor will match information for children on these applications to the FS, TANF, and Medicaid data to determine rates of direct verification for “verification nonresponders.”

State administrative data will be obtained through data sharing agreements with each of the States participating in the study.

LEA Telephone Interviews

LEA interviews will be conducted in December 2007 with LEAs from sample #1. These interviews will be conducted as forums. Abt Associates will designate times for two forums per State and invite LEAs to sign up to participate in a forum. The invitation will be sent to all selected LEAs during the first week of December, as part of a reminder that the data collection materials were due November 30. The invitation will include a list of broad topics for discussion. LEAs will be asked to sign-up for a forum via return e-mail. If more than eight LEAs in a State volunteer to participate, an additional forum will be scheduled.

LEAs volunteering for forum participation will be given a toll-free number for dialing in to the forum. Abt Associates will moderate the forum by opening up each topic and encouraging participation by all LEAs. Abt Associates will request permission to tape the forum, to ensure that all responses are documented accurately.

B.3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

The expected response rate for the Direct Verification Sample is 80 percent. The expected response rate for the Nonresponse Sample is 75 percent. The Nonresponse Sample is expected to yield a lower response rate because this sample is asked to provide retrospective data that may not be easily accessible.

The projected 80 percent response rate for participation in the Direct Verification Survey is based on the SY2006-07 pilot study, which recruited 121 LEAs in four States and achieved a 70 percent response rate. The SY2006-07 pilot relied on State CN directors to recruit selected LEAs, and limited effort was spent converting initial refusals. For the SY2007-08 evaluation, the FNS contractor, Abt Associates, will recruit LEAs and follow-up to convert refusals. In addition, collection of administrative data has been simplified to reduce burden on LEAs.

Abt Associates will use the following procedures to maximize response rates for this data collection:

• State CN Agencies will be contacted, by e-mail, to notify them of the LEAs selected for the two samples; LEAs selected for both samples will be highlighted.

• State CN Agencies will be asked to notify the selected LEAs, by e-mail, that Abt Associates Inc. will contact them to request their participation in the study, and that the State CN Agency encourages them to participate.

• Abt Associates will send a surface mailing to selected LEAs in August 2007. The mailing will include a summary of the study; explanation of the importance of collecting information from the respondent; a pamphlet summarizing results from the first year of the pilot; and a summary and schedule of data collection activities. This mailing will include a pre-paid postcard and the request that LEAs confirm receipt of the mailing and willingness to participate in the study by checking responses on the postcard and returning the postcard.

• After three weeks, Abt Associates will contact nonresponding LEAs to confirm receipt of the package and willingness to participate in the study. Multiple attempts will be made, via telephone and electronic mail.

• All respondents will be informed that their participation in the study is voluntary and that their responses will be treated as confidential.

• When potential respondents initially refuse to participate, the task leader for data collection will call the potential respondent to elicit support for and cooperation with the study. The importance of that person’s responses to the success of the study will be stressed, and assurances of complete confidentiality will be given.

From the contractor’s past experience, most local agencies agree to participate after a personal contact is made and data collection requirements are clearly explained. Remaining refusals tend to be agencies experiencing staffing changes, audits, or other external pressures that make participation particularly burdensome. Thus, it is expected that the contractor’s follow-up efforts will be instrumental in achieving the survey’s planned response rate of 80 percent.

B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The Direct Verification Report and Time and Cost Report were tested during the first year of the pilot study. State CN agencies recruited LEAs to completed data collection forms and provided photocopies of NSLP applications in the SY2006-07.

B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The sampling plan for the Direct Verification Evaluation Study was designed by Abt Associates Inc., the contractor chosen by FNS. Individuals at Abt Associates who contributed to the statistical aspects of the research design and who are responsible for the analyses include the project director, Nancy Cole, and the senior sampling statistician, David Hoaglin. These staff members work at the contractor's headquarters at 55 Wheeler Street, Cambridge, Massachusetts 02138, and can be reached by telephone at (617) 492-7100.

References

Burghardt, J., Silva, T., and Hulsey, L. “Case Study of National School Lunch Program Verification Outcomes in Large Metropolitan School Districts” (CN-04-AV3). Alexandria, VA: USDA, Food and Nutrition Service, OANE. Project Officer: Paul Strasberg. April 2004. Contract # GS-10F-00502.

Cole, Nancy and Christopher Logan (2007a). Computer Matching in the National School Lunch Program: 2005. Volume 1:Final Report (CN-05-PDM). Alexandria, VA: USDA, Food and Nutrition Service, OANE. Project Officer, Jenny Genser. February 2007. Contract # 43-3198-3-3718.

Cole, Nancy and Christopher Logan (2007b). Direct Verification Pilot Study: First Year Report. (CN-07-DV). Alexandria, VA: USDA, Food and Nutrition Service, OANE. Project Officer, Sheku Kamara. April 2007. Contract # AG-3198-D-06-0060.

1 Nationwide, only Tennessee has a Medicaid income eligibility limit below the NSLP-free cutoff; four States have a maximum Medicaid/SCHIP limit above 130% FPL and below 185% FPL; and 42 States have a maximum Medicaid/SCHIP limit at or above 185% FPL. Source: Cole and Logan (2007a); 47 States responded to the Survey of State Medicaid Agencies.

2 USDA Policy Memo SP-32-2006, “Clarification of Direct Verification,” August 31, 2006.

3 Burghardt et al. conducted a case study of NSLP verification outcomes in 21 school districts across 7 metropolitan areas. They found that only 68.8 percent of nonresponder cases were eligible for free or reduced-price benefits at the time they were interviewed to evaluate eligibility status.

27