SUPPORTING STATEMENT - PART B for
OMB Control Number 0584-[NEW]:
Evaluation of the Independent Review of Applications Process
Jinee Burdg
Office of Policy Support
Food and Nutrition Service
US Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302
Phone: 703-305-2744
E-mail: [email protected]
05/17/2018
Table of Contents
B.1 Respondent Universe and Sampling Methods 4
B.2 Procedures for the Collection of Information 7
B.3 Methods to Maximize the Response Rates and to Deal with Nonresponse 10
B.4 Test of Procedures or Methods to be Undertaken 12
B.5 Individuals Consulted on Statistical Aspects & Individuals Collecting and/or Analyzing Data 14
Tables
Appendices
Appendix A: Study Materials
A1. Richard B. Russell National School Lunch Act (Section 22(b)(3))
A2. 7 CFR 245.11
A3. FNS-874 Form
A4. Richard B. Russell National School Lunch Act (Section 28(c))
A5. Statement of Confidentiality and Nondisclosure
A6. IRA OMB Burden Table 1
A7. Westat IRB Approval Letter
Appendix B: Cognitive Testing Scheduling Request and Protocol
B1. Recruitment Email for Cognitive Test of Survey
B2. Recruitment Email for Cognitive Test of Interview Guide
B3. Cognitive Testing Protocol
B4. IRA Pretest Results
Appendix C: Study Notification Letters from FNS
C1. Study Notification Email from FNS to Regional Offices
C2. Study Notification Email from FNS to State Child Nutrition Directors
Appendix D: State Director Survey
D1. Email with Link to State Director Survey
D2. Reminder Email to Complete State Director Survey
D3. Telephone Script for Nonrespondent State Directors
D4. Evaluation of the Independent Review Process: State Director Survey
D5. State Director Survey Web Version
D6. Thank You Email to State Directors
Appendix E: LEA Interviews and Collection of Household Applications
E1. Email to Notify State Agencies of Selected LEAs
E2. State Agency Email to Selected LEAs (Interview-Only LEAs)
E3. State Agency Email to Selected LEAs (Interview and Application LEAs)
E4. Email to Schedule LEA Telephone Interviews and Frequently Asked Questions for LEA Directors
E5. Reminder Email to Schedule LEA Telephone Interviews
E6. Confirmation Email for LEA Telephone Interviews
E7. Evaluation of the Independent Review of Applications Process LEA Interview Guide
E8. Thank You Email to LEAs for Interviews (Interview-Only LEAs)
E9. Email to Schedule LEA Calls on Household Applications (Interview and Application LEAs)
E10. Reminder Email to Schedule LEA Calls on Household Applications
E11. Confirmation Email for LEA Calls on Household Applications
E12. Guide for Telephone Calls on Collection of Household Applications
E13. Email to LEAs with Procedures to Submit Household Applications
E14. Thank You Email to LEAs for Household Applications (Interview and Application LEAs)
Appendix F: Public Comments
F1. Public Comment 1
F2. Public Comment 2
Appendix G: Response to Public Comments
G1. FNS Response to Jean Public
G2. FNS Response to SNA (Harvey)
G3. FNS Response to SNA (Montague)
Appendix H: Comment from NASS
Appendix I: Response to NASS Comment
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
The respondent universe for this study includes all State agencies that administer the National School Lunch Program (NSLP) and all local educational agencies (LEAs) that conducted the independent review of applications (IRA), as reported on the FNS-874 Form. The number of LEAs that conduct IRA varies from year to year. In the three years that data have been reported, the number of LEAs ranges from approximately 375 to more than 700 each year. All 51 State Child Nutrition Directors that administer NSLP will be included in the study (U.S. territories will be excluded). We are including all State Directors to ensure we have a complete picture of State implementation of IRA, which can vary based on a variety of factors that are not readily captured in a subsample (e.g., State size, staffing, training methods, emphasis placed on the requirement, number of error-prone LEAs, etc.). Of the LEAs that conducted IRA, the respondent universe will contain 35 LEA Directors. The respondent universe will include 60 LEA key staff from the LEAs where the Director agrees to participate.
Sampling Methods and Response Rates
This study uses quantitative (i.e., survey and review of household applications) and qualitative methodologies (i.e., survey and in-depth interviews) to address the research questions. All 51 State Directors will be included in the sample to complete the State Director Survey (Appendix D4), with none refusing participation. Based on FNS experience with other studies, State Directors of Child Nutrition typically cooperate with such requests, especially when the request is made by FNS and with sufficient advance notice, as we are planning to do. FNS will send a Study Notification Email from FNS to State Child Nutrition Directors (Appendix C2) to all States participating in the NSLP, informing States of the study purpose, activities and expected timeframes. When the State Director Survey is ready to be launched, we will send the Email with Link to State Director Survey (Appendix D1) to ask State Directors to complete the online survey. That email will contain a unique PIN and instructions.
We will use purposive sampling to select a diverse group of LEAs that conducted an IRA in SY 2016-2017 in order to provide the richest descriptive information through the in-depth interviews; diverse LEA characteristics will also be important for the quantitative data analysis of household applications. We will use the approximately 550 LEAs that reported on the FNS-874 in SY 2016-2017 as the sampling universe from which to select the LEA sample. A primary consideration in LEA selection will be whether the LEA was required to conduct IRA based on Criteria 1 (10% or higher certification error on AR) or Criteria 2 (other factors). We will obtain information on the selection criteria used for each LEA through the State Director Survey. Within the two categories (Criteria 1 vs. Criteria 2), we will then seek variation in the following characteristics:
Reported Changes
LEAs with zero reported changes on the FNS-874 for SY 2016-2017
LEAs with reported changes on the FNS-874 for SY 2016-2017
Continuation vs. Non-Continuation of IRA Process
LEAs conducting IRA only in SY 2016-2017
LEAs conducting IRA in both SY 2016-2017 and SY 2017-2018
State Policies and Procedures for IRA (from State Director Survey results)
Data sources used to discontinue IRA (Administrative Review (AR) results vs. FNS-874 vs. other)
Categories used within Criteria 2 (lower thresholds of AR error, new LEAs, changes in key LEA personnel, new systems, other)
Other policies or procedures of interest
Size of LEA
Number of applications
Student enrollment or similar measure
Type of LEA
Public
Private/Charter
Geography
Across multiple States and FNS regions
We will seek to maintain that same diversity when we select a purposive subsample of 20 LEAs from which to collect and analyze applications for two non-consecutive school years. We will exclude from consideration for the subsample any LEAs that reported less than 50 applications subject to IRA, per the FNS-874. This will allow us to conduct more meaningful analyses while still examining smaller LEAs.
The expected response from the universe of 35 LEAs and 60 key staff is 30 LEA Directors and 60 LEA key staff for qualitative interviews, and 20 LEA Directors and 40 LEA key staff for the application submission. We anticipate one or two LEAs will be replaced based on additional information the State Directors provide upon notification (e.g., the State agency is conducting AR for that LEA during the timeframe for study data collection) and a few others may refuse to participate upon being contacted about the study. State agency support of the study will help achieve the desired response rate. We do not expect any LEA key staff to be nonrespondents once the LEA Director agrees to participate, because we anticipate the LEA Director will only identify key staff to participate in interviews after taking into account staff availability and expertise. Finally, the study team will contact respondents to schedule the telephone interviews well in advance, and maintain flexibility to accommodate the availability of respondents to help ensure their participation.
Once we select LEAs to participate in the study, we will send an Email to Notify State Agencies of Selected LEAs (Appendix E1) to ask State agencies to email the LEAs and notify them of their selection into the study and encourage their participation. We will provide State agencies with two emails they can use for this purpose: the State Agency Email to Selected LEAs (Interview-Only LEAs) (Appendix E2), and the State Agency Email to Selected LEAs (Interview and Application LEAs) (Appendix E3). We will follow the emails from the State agencies to LEA Directors with an Email to Schedule LEA Telephone Interviews (Appendix E4). This email will notify them again of their selection into the study for interviews only or for both interviews and collection of household applications, and request availability for the telephone interview.
Table B1-1 shows the response rates for each data collection activity in the study, and the overall response rate for the study.
Respondent |
Universe |
Sample Size |
Respondents |
Response rate (%) |
State Director Survey |
51 |
51 |
51 |
100 |
LEA Interviews (Directors) |
550 |
35 |
30 |
86 |
LEA Interviews (Key staff) |
1100 |
60 |
60 |
100 |
Household applications (Directors) |
35 |
20 |
20 |
100 |
Household applications (Key staff) |
70 |
40 |
40 |
100 |
Overall response rate |
|
|
|
97 |
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
As discussed in section B.1, we will collect quantitative descriptive data and qualitative information from all State agencies that administer NSLP through the online survey. We will collect qualitative information from a purposive sample of LEAs through the in-depth interviews. For these data collection activities, no statistical methodology for stratification and sample selection or estimation procedures are needed because we are collecting data from all State Directors and only a purposive sample of LEAs that conducted IRA. Therefore, the data collected from LEAs will provide rich information about the IRA process, implementation and challenges, but it will not be nationally representative. We will use the following procedures to collect these data:
FNS and then the study team will inform State Child Nutrition Directors of the study purpose, activities and expected timeframes, and ask the State Director to complete the online survey (Appendices C2 and D1). We will send reminder emails (Appendix D2) and call nonrespondent State Directors (Appendix D3) to obtain all responses within the two-month timeframe for survey completion. Two months is expected to be sufficient to accommodate State Director schedules.
We will notify the relevant State Directors of selected LEAs and ask them to encourage LEA participation by sending emails to the LEAs (Appendices E1, E2, E3).
We will then notify selected LEA Directors, including whether they are participating only in interviews or both interviews and the collection of household application, and ask them to schedule the telephone interview (Appendix E4). This correspondence will also indicate that the directors may include key LEA staff with appropriate expertise to join the interview. The qualitative telephone interviews will not exceed 90 minutes, and will be audio-recorded and led by a professionally trained researcher using the interview guide (Appendix E7). Following the completion of each LEA interview, we will send the 10 interview-only LEAs a thank you email (Appendix E8).
For the second phase of LEA data collection, after LEA interviews, we will ask a purposive subsample of LEAs (20) to submit household applications for two, non-consecutive school years. Collecting applications for two non-consecutive school years instead of three consecutive years minimizes burden on LEAs while still allowing us to analyze the sustainability of changes/improvements in the application approval process. No statistical methodology for stratification and sample selection or estimation procedures are needed to select the 20 LEAs. As with the LEAs selected for in-depth interviews, the 20 LEAs selected to provide applications (from the 30 LEAs interviewed) will be purposively selected to represent a variety of characteristic, but will not be a nationally representative sample. To further reduce burden on the selected LEAs, we will use statistical sampling procedures to collect the household applications from these LEAs, as follows:
Where the number of applications for a sampled LEA is relatively small (i.e., 100 or less), we will request that LEAs submit all applications for the two specified school years.
For LEAs with a larger total number of applications (i.e., more than 100), we will provide detailed procedures to LEAs to select a random sample of the applications to submit. We will determine the sample size for each LEA using a minimum detectable difference (MDD) in the percentage of applications in error.
Table B2-1 shows the sample size required to obtain MDD for different population sizes (i.e., the total number of applications reviewed by the LEA). Given estimated error rates around 10 percent (per Criteria 1 of IRA), changes of 2 or 3 percentage points are likely to be meaningful, suggesting that we want to set the MDD relatively low; we will set the MDD at 5 percent. The shaded cells show combinations of sampling sizes by population size that achieve at least 5 percent MDD.
Table B.2-1. Minimum Detectable Difference in Error Rates
Sample size |
Total number of applications reviewed by LEA |
||||||||
50 |
100 |
150 |
200 |
250 |
500 |
1,000 |
5,000 |
10,000 |
|
43 |
0.049 |
0.099 |
0.111 |
0.116 |
0.119 |
0.125 |
0.128 |
0.131 |
0.131 |
75 |
- |
0.049 |
0.070 |
0.078 |
0.082 |
0.091 |
0.095 |
0.098 |
0.098 |
100 |
- |
0.000 |
0.049 |
0.060 |
0.066 |
0.076 |
0.081 |
0.084 |
0.084 |
120 |
- |
- |
0.035 |
0.049 |
0.056 |
0.067 |
0.073 |
0.076 |
0.077 |
185 |
- |
- |
- |
0.017 |
0.032 |
0.049 |
0.056 |
0.061 |
0.062 |
230 |
- |
- |
- |
- |
0.016 |
0.041 |
0.049 |
0.054 |
0.055 |
280 |
- |
- |
- |
- |
- |
0.033 |
0.043 |
0.049 |
0.050 |
We will review procedures for sampling during a phone call with each LEA, and summarize the phone discussion in a follow-up email to LEAs (Appendices E12 and E13). If necessary, we will have follow up calls if questions arise.
There are no unusual problems requiring specialized sampling procedures. In addition, we are only conducting each data collection activity (State survey, LEA interviews, collection of household applications) once for each respondent. For those LEAs selected for in-depth interviews and to provide household applications, we are minimizing confusion and spreading out the burden by handling each data collection activity at a discrete time.
Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
The expected response rate for State Directors is 100 percent. Notification from FNS to State Directors will help ensure all State Directors respond. In addition, we expect to provide sufficient time--approximately two months--for State Directors to complete the online State Director Survey, which will alleviate any workload or scheduling conflicts. State Directors will also be able to delegate completion of portions of the survey to key staff within the agency, as needed. State Directors of Child Nutrition Programs typically respond to these types of requests, especially when given sufficient time to respond, as well as reminders.
For LEAs, we will enlist the support of State Directors to encourage their participation. Specifically, in the letter we send to State Directors informing them of the LEAs selected to participate, we will attach a letter for State Directors to send to LEAs to encourage their participation (Appendices E2 and E3). LEAs will also receive the Email to Schedule LEA Telephone Interviews (Appendix E4), which reiterates the purpose of the study and their role, and encourages their participation. We expect to replace some of the LEA Directors initially selected to accommodate scheduling conflicts and other issues which may arise, which is reflected in the initial sampling universe of 35 LEA Directors, of which 30 LEA Directors are expected to participate in interviews, and 20 are expected to provide a sample of household applications. The expected response rate for LEAs of 86 percent is reasonable given that Section 28 of the Richard B. Russell National School Lunch Act (42 U.S.C. 1769i) requires organizations participating in the School Meal Programs to cooperate with FNS studies and evaluations. In addition, State agencies support and encourage LEA participation; only when an LEA has significant scheduling conflicts (e.g., State Administrative Review during the same time period) or other extenuating circumstances (e.g., LEA Director on medical leave) do they refuse to participate.
LEA selection will be based on a purposive sample selected from LEAs that conducted IRA. The purposive sample will represent LEAs of a variety of characteristics related to IRA (see section B.1). We will conduct qualitative interviews in order to collect more detailed information than would be possible through a web survey. The information collected through the interviews and review of applications for these LEAs will provide important insights for FNS into the apparent discrepancies between the LEAs identified to conduct IRA and the results reported on the FNS-874 Local Educational Agency Second Review of Applications. Specifically, a key criterion for identifying LEAs to conduct IRA is 10 percent or more documented errors in applications (on State Administrative Review), yet few LEAs report changes to applications when they conduct IRA. Understanding why this might occur will help FNS assess the effectiveness of IRA in reducing errors. Depending on the results of the study, FNS may consider changes to the criteria States must use to select LEAs for IRA, to the reporting process or timelines, or to the data elements that are reported.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
We conducted pre-test interviews of the State Director Survey and the LEA Interview Guide using the Cognitive Testing Protocol and recruitment materials (Appendices B1, B2, and B3). The State Director Survey was pretested with staff from two State agencies, the Colorado Department of Education and the Arkansas Department of Education. It was also tested with two FNS regional office staff, one of whom previously worked at the State level, who had valuable insight into the workings of all States in their region and could provide a broad perspective on the survey instrument and questions that might be problematic. The LEA Interview Guide was tested with three LEAs that had conducted IRA, each in a different State (Michigan, Utah, and Wisconsin). The LEAs were selected to achieve diversity in size, geographic location, school type (public, private, charter), the number and types of errors flagged in the FNS-874 Local Educational Agency Second Review of Applications data from School Year 2016-17, and whether the LEA conducted an IRA in one school year or more than one school year.
The survey and the interview guide were tested to ensure that the respondents interpreted the questions as intended and could easily respond, that the interview guide was easy for the interviewer to administer, and to verify the burden estimates. In both cases, trained interviewers reviewed the instruments question by question with the respondents, observed and documented any issues that arose for either respondents or interviewers, and discussed any points of difficulty with respondents.
Following the pretests, the interviewer and an analyst reviewed their notes from each interview and produced a list of themes and patterns within the interview data. In particular, staff focused on problems and issues with the instruments, including areas where the respondents demonstrated confusion, hesitation, uncertainty, and/or discomfort. Staff discussed the results of the analysis to validate the findings and confirm recommendations. Themes and patterns were organized, evaluated, synthesized, and summarized into report form.
Pretest results are summarized in the IRA Pre-test Memo (Appendix B4). Findings and recommendations from the cognitive testing were used to refine the survey and interview guide. Specifically, several response options were added or revised on the State Director Survey, including responses related to State training topics, notification procedures, typical LEA questions on IRA, oversight, and State follow-up procedures. In addition, we added open text fields to capture more detailed information on edit checks in State reporting systems and information on how State agencies use the FNS-874 Local Educational Agency Second Review of Applications to improve program administration. For the LEA interview guide, we expanded some probes based on LEA responses during the cognitive interview; separated several questions that initially had multiple parts to help ensure respondents provide input on all aspects of the question; and provided additional interviewer instructions about framing questions appropriately based on the individual LEA circumstances and reporting results. Cognitive testing verified the initial burden estimates for the survey and interviews.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The proposed protocol, draft survey, and interview guide were developed and reviewed extensively by FNS and Westat staff. FNS staff and Westat will participate in the analysis of the data, as well as development of reports. In addition, Audra Zakzeski, Mathematical Statistician with the National Agricultural Statistical Service’s Methodology Division reviewed Part A and Part B of this OMB clearance package and provided comments (Appendix H).
Name |
Affiliation |
Title |
Contact information |
Melissa Rothstein |
Westat |
Senior Study Director |
301-315-5975 |
Laurie May |
Westat |
Vice President |
301-517-8068 |
Thea Zimmerman |
Westat |
Senior Study Director |
240-314-2413 |
Lindsay Giesen |
Westat |
Study Director |
240-453-5693 |
Chris Manglitz
|
Westat
|
Statistician |
301-294-4460
|
Jinee Burdg |
USDA/FNS |
Social Science Policy Analyst |
703- 305-2744 |
Audra Zakzeski |
NASS |
Mathematical Statistician, Methodology Division |
202-690-8637 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for OMB No |
| Author | USDA |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |