SUPPORTING STATEMENT - PART A for
OMB Control Number 0584-[NEW]:
Evaluation of the Independent Review of Applications Process
Jinee Burdg
Office of Policy Support
Food and Nutrition Service
US Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302
Phone: 703-305-2744
E-mail: [email protected]
05/17/2018
Table of Contents
A1. Circumstances that make the collection of information necessary. 4
A2. Purpose and Use of the Information. 7
A3. Use of information technology and burden reduction. 12
A4. Efforts to identify duplication. 13
A5. Impacts on small businesses or other small entities. 13
A6. Consequences of collecting the information less frequently. 14
A7. Special circumstances relating to the Guidelines of 5 CFR 1320.5. 14
A8. Comments to the Federal Register Notice and efforts for consultation. 15
A9. Explain any decisions to provide any payment or gift to respondents. 17
A10. Assurances of confidentiality provided to respondents. 17
A11. Justification for any questions of a sensitive nature. 18
A12. Estimates of the hour burden of the collection of information. 19
A13. Estimates of other total annual cost burden. 27
A14. Provide estimates of annualized cost to the Federal government. 27
A15. Explanation of program changes or adjustments. 28
A16. Plans for tabulation, and publication and project time schedule. 28
A17. Displaying the OMB Approval Expiration Date. 30
A18. Exceptions to the certification statement identified in Item 19. 30
Tables
Appendices
Appendix A: Study Materials
A1. Richard B. Russell National School Lunch Act (Section 22(b)(3))
A2. 7 CFR 245.11
A3. FNS-874 Form
A4. Richard B. Russell National School Lunch Act (Section 28(c))
A5. Statement of Confidentiality and Nondisclosure
A6. IRA OMB Burden Table 1
A7. Westat IRB Approval Letter
Appendix B: Cognitive Testing Scheduling Request and Protocol
B1. Recruitment Email for Cognitive Test of Survey
B2. Recruitment Email for Cognitive Test of Interview Guide
B3. Cognitive Testing Protocol
B4. IRA Pretest Results
Appendix C: Study Notification Letters from FNS
C1. Study Notification Email from FNS to Regional Offices
C2. Study Notification Email from FNS to State Child Nutrition Directors
Appendix D: State Director Survey
D1. Email with Link to State Director Survey
D2. Reminder Email to Complete State Director Survey
D3. Telephone Script for Nonrespondent State Directors
D4. Evaluation of the Independent Review Process: State Director Survey
D5. State Director Survey Web Version
D6. Thank You Email to State Directors
Appendix E: LEA Interviews and Collection of Household Applications
E1. Email to Notify State Agencies of Selected LEAs
E2. State Agency Email to Selected LEAs (Interview-Only LEAs)
E3. State Agency Email to Selected LEAs (Interview and Application LEAs)
E4. Email to Schedule LEA Telephone Interviews and Frequently Asked Questions
E5. Reminder Email to Schedule LEA Telephone Interviews
E6. Confirmation Email for LEA Telephone Interviews
E7. Evaluation of the Independent Review of Applications Process LEA Interview Guide
E8. Thank You Email to LEAs for Interviews (Interview-Only LEAs)
E9. Email to Schedule LEA Calls on Household Applications (Interview and Application LEAs)
E10. Reminder Email to Schedule LEA Calls on Household Applications
E11. Confirmation Email for LEA Calls on Household Applications
E12. Guide for Telephone Calls on Collection of Household Applications
E13. Email to LEAs with Procedures to Submit Household Applications
E14. Thank You Email to LEAs for Household Applications (Interview and Application LEAs)
Appendix F: Public Comments
F1. Public Comment 1
F2. Public Comment 2
Appendix G: Response to Public Comments
G1. FNS Response to Jean Public
G2. FNS Response to SNA (Harvey)
G3. FNS Response to SNA (Montague)
Appendix H: Comment from NASS
Appendix I: Response to NASS Comment
Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
This is a new information collection request. The U.S. Department of Agriculture (USDA), Food and Nutrition Service (FNS), is conducting this study to provide key information about the processes and overall effectiveness of the Independent Review of Applications requirement, which is intended to reduce administrative certification error in the National School Lunch Program (NLSP) and School Breakfast Program (SBP).
The NSLP and SBP are federally funded meal programs operating in public and nonprofit private schools and residential childcare institutions. Federal cash reimbursement is provided to most local educational agencies (LEAs) based on certifying students as eligible for free, reduced price, or paid meal benefits. LEAs certify students through household applications or by direct certification through a household’s participation in a federal means-tested program such as the Supplemental Nutrition Assistance Program. There were more than 30 million students in over 100,000 public and nonprofit private schools receiving school lunches through the NSLP every school day in Fiscal Year (FY) 2016, with approximately 22 million receiving meals free or at a reduced price.1 That same FY, the SBP served 14.6 million students daily, with 12.4 million of these students receiving free or reduced-price meals.2 Together, these programs received approximately $16.5 billion in federal funds in FY 2016.3
Program integrity is a long-standing issue of concern for the NSLP and SBP. The Access, Participation, Eligibility and Certification study (OMB #0584-0530, NSLP/SBP Access, Participation, Eligibility, and Certification Study, discontinued 3/31/2008), showed 3.75 percent4 administrative certification error in the NSLP and SBP in School Year (SY) 2005-2006. For the same school year, FNS found 3.0 percent administrative certification error through its oversight activities as part of Administrative Review, referred to as the Regional Office Review of Applications (RORA). For household applications, administrative certification error occurs when the LEA incorrectly certifies a student as free, reduced price, or paid based on the information provided by the household on the application.
To decrease administrative certification error, the Healthy, Hunger-Free Kids Act of 2010 (HHFKA, P.L. 111-296) amended Section 22(b)(3) of the Richard B. Russell National School Lunch Act (42 U.S.C 1769c (b)) (Appendix A1) to require an "independent review of applications" (IRA) for certain LEAs. Under the independent review provision, LEAs that demonstrate high levels of, or a high risk for, administrative error associated with certification, verification and other school meal program administrative processes are required to conduct a second review of the eligibility determinations made via household applications. The second reviewer, who must be someone other than the original determining official, re-examines the information provided by the household on the application to determine if all required information was provided and the correct eligibility determination was made.
IRA was first implemented in SY 2014-2015. Program regulations at 7 CFR 245.11 (Appendix A2) provide two criteria for State agencies to annually identify LEAs that must conduct IRA:
Criteria 1: All LEAs with 10 percent or more of certification/benefit issuances in error, as determined by the State agency during an administrative review (AR); and
Criteria 2: LEAs that the State agency considers at risk for administrative certification error but were not selected under Criteria 1.
State agencies use data from the AR to identify LEAs that meet Criteria 1. Under Criteria 2, State agencies have significant discretion to identify other LEAs at risk for certification error. This may include, for example, LEAs with certification/benefit issuance errors between 5 and 10 percent, and LEAs new to NSLP or SBP, with new administrative staff or a new electronic system. LEAs required to conduct the IRA must continue each year until the LEA demonstrates to the State agency that no more than 5 percent of reviewed applications required a change in eligibility determination.
To obtain data on the results of the IRA process, FNS requires State agencies to submit an annual report via the FNS-874 Form, Local Educational Agency Second Review of Applications (Appendix A3) (OMB #0584-0594, Food Programs Reporting System (FPRS), expiration date 9/30/19). The FNS-874 provides detailed information, by LEA, on the results of the IRA process, including the number of applications reviewed; the number of applications with changed eligibility determinations based on the second review; information on the types of changes (e.g., free to reduced price); and reasons for changes (e.g., gross income calculation error, incomplete application error).
FNS-874 data for SY 2014-2015 and SY 2015-2016 show that relatively few LEAs subject to IRA reported any changes in initial certification decisions as a result of IRA. This was unexpected given that the primary criterion for identifying LEAs to complete the IRA process is a demonstrated certification/benefit issuance error rate of 10 percent or more on AR. FNS is pursuing this Evaluation of the Independent Review Process study to provide information about the IRA process at the State and LEA levels, its results, and its overall effectiveness in reducing administrative certification error. FNS has contracted with Westat to carry out this scope of work.
For studies of the Child Nutrition Programs, including NSLP and SBP, Section 28(c) of the Richard B. Russell National School Lunch Act (42 U.S.C. 1769i), the authorizing statute for the NSLP (Appendix A4), requires entities participating in the programs authorized under the Richard B. Russell National School Lunch Act and the Child Nutrition Act of 1966 (42 U.S.C. 1771), to cooperate in the conduct of evaluations and studies.
Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate how the agency has actually used the information received from the current collection.
The primary purpose of this voluntary one-time data collection is to provide a description of the IRA process at the State and LEA levels, its results, and its overall effectiveness in reducing administrative certification error. The key research objectives relate to assessing how State agencies and LEAs implement and report on IRA. Evaluation of the processes and effectiveness in reducing administrative certification error will culminate in a set of recommended practices for improving the process. There is currently no other effort that can address the research objectives of the proposed study. The information obtained from the study will help FNS disseminate best practices and determine if changes may be needed to the IRA requirement or associated reporting. Specifically, under the study we will:
Collect data and describe the process and policies surrounding the second, independent review of applications at the State and LEA levels.
For a subsample of LEAs selected under Objective #1, collect data and conduct a review and analysis of household applications for two nonconsecutive school years (SY 2016-2017 and SY 2018-2019) to describe administrative certification errors for household applications.
Assess the effectiveness of the second, independent review of applications process and the FNS-874 reporting requirements, and provide recommendations for best practices.
The study will collect data from 51 State agency Directors5 of Child Nutrition (including Washington, D.C.), and from a purposive sample of LEA Directors who have completed the IRA process, possibly assisted by other key staff. Primary data collection will involve one web-based State Director Survey, and a telephone interview with 30 LEA Directors, who may be assisted by one or two key staff. Twenty of the 30 LEAs interviewed will be purposively selected to submit household applications for two, non-consecutive school years (SY 2016-2017 and SY 2018-2019) for review and analysis by the study team. Table A.2-1 lists the data collection instruments, the source of the information, and the key information gathered by the instrument.
Table A.2-1. Instrument specification
Source |
Instruments/mode |
Key information |
State CN Directors |
|
|
LEA Directors |
|
|
Results of the study will be made available to the public via the FNS website, and State Directors and LEAs that participate in the study will be notified and provided a link to the online results. In the following sections, we describe the data being collected, from whom and how we will collect it, and the recruitment procedures. More detailed information on sampling and data collection is available in Part B of this supporting statement.
State Agency Processes for IRA
To collect information on the processes and procedures that State agencies use for IRA, we will ask all State Child Nutrition Directors to complete the web-based State Director Survey (Appendix D4). The survey will collect information on identification and notification of LEAs; training and technical assistance; State monitoring of the IRA process; reporting; additional State policies and procedures; and challenges. Internally, FNS will send a Study Notification Email from FNS to Regional Offices (Appendix C1) with information about the study purpose, objectives, activities and expected timeframes. FNS will also send a Study Notification Email from FNS to State Child Nutrition Directors (Appendix C2) to all State agencies participating in the NSLP, informing them of the study purpose, objectives, activities and expected timeframes. When the State Director Survey is ready to be launched, within two weeks of OMB approval, we will send the Email with Link to State Director Survey (Appendix D1) to ask State Directors to complete the online survey and provide a PIN and instructions. We will send a reminder email to nonrespondents every two weeks, using the Reminder Email to Complete State Director Survey (Appendix D2), and follow up by phone if needed at the end of the two-month period using the Telephone Script for Nonrespondent State Directors (Appendix D3). When we reach a nonrespondent State Director by phone, we will ask that they complete the online survey as soon as possible. Following completion of the survey, we will send a Thank You Email to State Directors (Appendix D6).
LEA Processes for IRA
Once the State Director Survey is closed out and initial data review and analysis are completed, we will select a sample of LEAs and conduct in-depth telephone interviews to collect information on the processes and procedures LEAs use for IRA. The LEA Interview Guide (Appendix E7) contains open-ended questions and probes about LEA notification; selection of the second review official; step-by-step processes and timeframes for IRA; results and reporting of IRA; certification process changes resulting from IRA; and challenges. We will send an Email to Notify State Agencies of Selected LEAs (Appendix E1), to ask State agencies in affected States to encourage LEA participation by sending emails to the LEAs. We will provide State agencies with two emails they can use for this purpose: the State Agency Email to Selected LEAs (Interview-Only LEAs) (Appendix E2), and the State Agency Email to Selected LEAs (Interview and Application LEAs) (Appendix E3).
One week later, we will send LEA Directors of selected LEAs an Email to Schedule LEA Telephone Interviews and Frequently Asked Questions (Appendix E4). . This email will notify them of their selection into the study and whether they will be asked to participate only in interviews, or in interviews and in collection of household applications, and request availability for the telephone interview. If needed, we will send a Reminder Email to Schedule LEA Telephone Interviews (Appendix E5) one week later. Once scheduled, we will send a Confirmation Email for LEA Telephone Interviews (Appendix E6) that includes the date/time of the call and a toll-free number.
In-depth interviews with LEA Directors and their key staff will not exceed 90 minutes. All interviews will be led by a professionally trained researcher who will use the LEA Interview Guide (Appendix E7) for the duration of each interview. Prior to beginning the interview, participants will verbally consent to the interview, using language found at the start of the LEA Interview Guide. With the participants’ consent, we will audio-record LEA interviews, and verbatim notes will be compiled. During the interview we will ask LEA Directors to send us any written policies and procedures we discuss during the interview. Following the completion of each LEA interview, we will send the Thank You Email to LEAs for Interviews (Interview-Only LEAs) (Appendix E8) to the 10 LEAs that will only participate in interviews.
Household Applications
For the 20 LEAs that are also being asked to provide a sample of household applications for review and analysis by the study team, we will send the Email to Schedule LEA Calls on Household Applications (Interview and Application LEAs) (Appendix E9) upon completion of the LEA interview. This email will thank the 20 LEAs for their participation in the LEA interview, and request their availability for a phone call to answer questions and discuss the process for collecting the household applications. If needed, we will send a Reminder Email to Schedule LEA Calls on Household Applications (Appendix E10) one week after the initial request. Once scheduled, we will send a Confirmation Email for LEA Calls on Household Applications (Appendix E11).
During the phone calls with LEAs in advance of collection of household applications, we will review the procedures for LEAs to sample and submit household applications to the study team, and answer any LEA questions. (See section B2 of this supporting statement for discussion of sampling procedures). We will use the Guide for Telephone Calls on Collection of Household Applications (Appendix E12) to conduct these calls, which includes the specific school years for which applications are being requested; sampling procedures to select applications to send to the study team; and methods to submit the applications to the study team. We will ask LEAs to send the files electronically through a secure FTP server or mail them using a secure package tracking service. The method ultimately used will depend on the capability and comfort level of staff at each LEA. In cases where neither electronic nor mail submission is feasible due to technology, policy, or other unforeseen challenges, we will travel on-site to the LEA and collect the applications.
After each call, we will send an Email to LEAs with Procedures to Submit Household Applications (Appendix E13), to summarize the points covered in the call. If necessary, we will repeat these procedures and have a second call with each LEA to answer questions that arise as they are sampling and preparing applications for submission. After receipt of the applications from LEAs, we will send the Thank You Email to LEAs for Household Applications (Interview and Application LEAs) (Appendix E14).
Pretesting
In preparation for study launch, we pre-tested the State Director Survey and LEA Interview Guide to ensure that: (1) respondents interpret the questions as intended and can easily respond; and (2) for the interview guide, interviewers can easily administer the instrument. We conducted pre-test interviews by telephone among eight respondents: two FNS Regional Office staff, two State Child Nutrition Directors, one State Child Nutrition Associate Director, and three LEA Directors. Interviews lasted no more than 90 minutes. Feedback from the pre-test interviews helped to refine questions that respondents found confusing, repetitive or ambiguous, as well as questions that interviewers found challenging to administer. The pretest interview protocols and recruitment materials may be found in Appendix B.
Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.
FNS is committed to complying with the E-Government Act of 2002 to promote the use of technology. Respondent burden will be reduced through the use of information technology for data collection to the extent possible. The State Director Survey (Appendix D4) will be web-based. Screen shots of the web survey may be found in Appendix D5. We expect that all 51 State agencies will complete the survey electronically. We will also collect the household applications from LEAs electronically via a secure FTP server, for those LEAs that are able to provide them in this manner. Out of a total of 1,192 responses for this collection, we estimate that 172 responses (14.4 percent) will be submitted electronically.
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Question 2.
There is no similar information collection. We have made every effort to avoid duplication. Through careful review of the data requirements, we have determined that no current data are similar to that proposed for collection in this study. Further, this study does not ask respondents to report data that they have already reported to FNS on the FNS-874. No data exist on the processes in place to conduct the IRA.
If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden.
We expect that up to half of the 35 LEAs that we will select and recruit for interviews will qualify as small entities6. Although smaller LEAs are involved in this data collection effort, they deliver the same program benefits and perform the same function as any other LEA. For all respondents, we will limit requested or required information to the minimum required for the intended use. Out of the total 204 respondents for this collection (which also includes the non-respondents), we estimate that 45 (22 percent) will be from small entities, representing 35 LEAs.
Describe the consequence to Federal program or policy activities if the collection is not conducted, or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
This is a voluntary one-time data collection activity. To minimize confusion and help spread out the burden, the LEAs asked to participate in interviews and to provide household applications will be contacted at two discrete times for these activities. If we do not conduct this study at this time, FNS will not have information on the issues, challenges and burden of the IRA process or on the effectiveness of the IRA. The data obtained through this study will help FNS determine best practices for conducting the IRA.
Explain any special circumstances that would cause an information collection to be conducted in a manner:
Requiring respondents to report information to the agency more often than quarterly;
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Requiring respondents to submit more than an original and two copies of any document;
Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
There are no special circumstances relating to the Guidelines of 5 CFR 1320.5. We are conducting the collection of information in a manner consistent with the guidelines in 5 CFR 1320.5.
If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8 (d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.
Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.
Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years even if the collection of information activity is the same as in prior years. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.
In accordance with 5 CFR 1320.8(d), FNS published a notice on February 6, 2018, in the Federal Register, Volume 83, Number 25, pages 5235-5239, and provided a 60-day period for public comments. The public comment period ended on April 9, 2018. FNS received a total of two comments, as provided in Appendices F1 through F2. Appendices G1 and G2 include FNS’s responses to the comments.
Commenters acknowledged the importance of evaluating the IRA process. One commenter expressed concerns about program integrity and the need to protect taxpayer dollars while providing healthy meals. Commenters stressed the importance of including LEAs that have conducted the IRA process and the need to provide training to LEAs required to conduct the IRA process. They also remarked upon the importance of including school nutrition directors in the evaluation of the IRA process, and wanted to ensure that contacts with LEAs be made at a time in the school year that is not overly burdensome and providing respondents adequate time to reply.
In responding to the comments, FNS assured commenters that we are equally concerned with program integrity and the importance of ensuring nutritious meals for all students at reasonable cost to taxpayers. For commenters concerned about including LEAs required to complete the IRA process and the importance of including school nutrition directors, FNS provided details about the proposed data collection plan, which states that we will sample LEAs who have conducted the IRA process, and that the school nutrition director is the primary point of contact for the evaluation study. FNS responded to commenters expressing the need for additional training to conduct the IRA that we will inquire about any training provided to those conducting the IRA. Finally, FNS assured commenters concerned about the added burden to LEA staff that we plan to contact no more than 30 LEAs nationwide, we will not schedule interviews at the beginning of the school year, and the data collection period is sufficiently long to allow some flexibility for individual LEAs.
In addition to the public input on the Notice, the study team identified two Child Nutrition Program experts to review and provide input on the study. Zoe Neuberger, Senior Policy Analyst, Center on Budget and Policy Priorities, 202-408-1080, and Barbara E. Martin, Private Trainer and Consultant, 856-722-8066. Consultations about the research design, sample design, data sources and needs, and study reports occurred during the study’s planning and design phase, and will continue throughout the study. An additional consultant was Audra Zakzeski, Mathematical Statistician with the National Agricultural Statistical Service’s Methodology Division, 202-690-8637 (Appendices H and I).
Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
There will be no incentives provided to respondents in this study.
Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
Study participants will be subject to safeguards as provided by the Privacy Act of 1974 (5 USC §552a), which requires the safeguarding of individuals against invasion of privacy; these safeguards will have been documented in an informed consent form found at the start of the survey and interview guide (Appendices D4 and E7). In addition, all Westat project staff signed a confidentiality and nondisclosure agreement (Appendix A5). We will safeguard the privacy and security of electronic and hard copy data during the data collection and processing period following the system of record notice (SORN) titled FNS-8 USDA/FNS Studies and Reports, published in the Federal Register on April 25, 1991 (56 FR 19078). We will not link identifying information to participants’ responses and will conduct analyses on data sets that include only respondent ID numbers. Survey responses will be submitted on a secure web site managed by Westat. Respondents will securely transmit all household application data to Westat via secure fax, FTP site, or prepaid sealed mailings; Westat stores data in locked file cabinets or password-protected computers, made accessible only to Westat project staff. We will destroy names and phone numbers within 12 months after the end of the collection and processing period (approximately 03/2020). Westat’s Institutional Review Board (IRB) is the organization of record overseeing all human subjects’ activities for the study. A copy of the IRB approval letter is in Appendix A7.
This study does not collect any personally identifiable information nor do any of the forms require a Privacy Act Statement.
Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
In general, questions on the State Director Survey (Appendix D4) and topics covered in the LEA Interview Guide (Appendix E7) are not considered sensitive. Participants can choose to skip any question, or to discontinue participation in the study with no penalties. The majority of questions required for the interviews were pre-tested (Appendix B3) and no participants expressed unwillingness to answer the questions.
The household applications include sensitive information on household members, federal program benefits received by the household, race/ethnicity and income questions, and the student’s sex. The information is completed by families outside of the study to determine eligibility for the NSLP/SBP meal benefits. (Collection of household applications is approved under the burden for 7 CFR 245, (OMB# 0584-0026, 7 CFR 245Determining Eligibility for Free and Reduced Price Meals and Free Milk in Schools, expiration date 03/31/2020)). All information collected for the study will be kept private and will not be shared with others not involved in the study. In addition, LEAs will securely transmit all household application data to Westat via secure fax, FTP site, or prepaid sealed mailings; Westat stores data in locked file cabinets or password-protected computers, made accessible only to Westat project staff. We will destroy names and phone numbers within 12 months after the end of the collection and processing period (approximately 03/2020). Household applications are being collected for this study because they are the only source of information for determining whether initial eligibility determinations and second reviews through IRA are conducted properly.
Provide estimates of the hour burden of the collection of information. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.
A. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.
B. Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.
FNS affected public for this data collection is across 50 States and Washington, D.C. FNS plans to send out the Study Notification Email from FNS to State Child Nutrition Directors (Appendix C2) to the 51 State-level CN Directors.
We will collect information about the IRA process and procedures from these various respondent types: 51 State CN Directors and 30 LEA Directors. With this submission, there are 204 respondents, 1,192 responses, and 433.2 annual burden hours. The burden table below (Table A.12-1) and Appendix A6 present the number of respondents, frequency of response, and annual hour burden to collect these data. All participant burden (regardless of title) includes time to read communication materials; the burden for State CN Directors includes time to complete the web survey; the burden for LEA Directors includes time to schedule a telephone interview, participate in the interview, participate in phone calls to discuss the sampling and collection of household applications, and collect and transmit the household applications. The burden table also includes the instrument pre-testing efforts conducted with two State CN Directors, one State CN Associate Director, and three LEA Directors.
State Child Nutrition Directors. The sample size of State Child Nutrition Directors is 51; FNS anticipates all 51 will fully participate.
LEA Directors. The sample size of Local Education Agency Directors is 35; FNS anticipates that 30 will complete the telephone interview; 20 will participate in the collection of household applications.
LEA Key Staff. The sample size of LEA key staff is 60; FNS anticipates that 60 will complete the telephone in-depth interview; 40 will participate in the collection of household applications.
The estimates of respondent cost are based on the burden estimates and use the U.S. Department of Labor, Bureau of Labor Statistics, May 2017 National Occupational and Wage Statistics. We used both Occupational Group (999200) State Government (excluding schools and hospitals) and Occupational Group (611000) Educational Services (including private, state, and local government schools) to estimate annualized costs for managers or directors at the State agencies and LEAs. We used the mean hourly wage for each job category to estimate annualized costs.
The hourly wage rate used for the State CN Director and State key staff is $44.89 (Occupation Code 11-9030, State Government-999200).7 The hourly wage rate used for the LEA director and key staff is $41.51 (Occupation Code 11-9039, Educational Services-611000).8 The State CN Director staff total 121.7 burden hours, which is $5,465.23. The LEA directors and key staff total 311.4 burden hours, which is $12,928.20 (see Appendix A6). The total estimated annualized cost is $18,393.50.
No respondents will keep records of data; therefore, we do not include burden hour estimates for recordkeeping.
Table A.12-1. Annual Burden Table
|
|
|
|
|
Responsive |
Non-Responsive |
|
||||||||
Respondent Category |
Type of respondents |
OMB Appendix Number |
Instruments |
Sample Sizea |
Number of respondents |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Number
of |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Grand Total Annual Burden Estimate (hours) |
State Child Nutrition (CN) Agency |
State Director
|
B1 |
Recruitment Email for Cognitive Test of Survey |
3 |
3 |
1 |
2 |
0.0501 |
0.1 |
0 |
0 |
0 |
0.000 |
0.0 |
0.1 |
B3 |
Cognitive Test of the State Director Survey |
3 |
3 |
1 |
3 |
1.5000 |
4.5 |
0 |
0 |
0 |
0.000 |
0.0 |
4.5 |
||
C2 |
Study Notification Email from FNS to State CN Directors |
51 |
51 |
1 |
51 |
0.0501 |
2.6 |
0 |
0 |
0 |
0.000 |
0.0 |
2.6 |
||
D1 |
Email with Link to State Director Survey |
51 |
51 |
1 |
51 |
0.0501 |
2.6 |
0 |
0 |
0 |
0.000 |
0.0 |
2.6 |
||
D2 |
Reminder Email to Complete State Director Survey #1 |
40 |
40 |
1 |
40 |
0.0167 |
0.7 |
0 |
0 |
0 |
0.000 |
0.0 |
0.7 |
||
D2 |
Reminder Email to Complete State Director Survey #2 |
30 |
30 |
1 |
30 |
0.0167 |
0.5 |
0 |
0 |
0 |
0.000 |
0.0 |
0.5 |
||
D2 |
Reminder Email to Complete State Director Survey #3 |
20 |
20 |
1 |
20 |
0.0167 |
0.3 |
0 |
0 |
0 |
0.000 |
0.0 |
0.3 |
||
D2 |
Reminder Email to Complete State Director Survey #4 |
10 |
10 |
1 |
10 |
0.0167 |
0.2 |
0 |
0 |
0 |
0.000 |
0.0 |
0.2 |
||
D3 |
Telephone Script for Nonrespondent State Directors |
5 |
5 |
1 |
5 |
0.0835 |
0.4 |
0 |
0 |
0 |
0.000 |
0.0 |
0.4 |
||
D4 |
State Director Survey |
51 |
51 |
1 |
51 |
1.0000 |
51.0 |
0 |
0 |
0 |
0.000 |
0.0 |
51.0 |
||
Table A.12-1. Annual Burden Table (continued)
|
|
|
|
|
Responsive |
Non-Responsive |
|
||||||||
Respondent Category |
Type of respondents |
OMB Appendix Number |
Instruments |
Sample Sizea |
Number of respondents |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Number
of |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Grand Total Annual Burden Estimate (hours) |
State Child Nutrition (CN) Agency |
State Director |
D6 |
Thank You Email to State Directors |
51 |
51 |
1 |
51 |
0.0167 |
0.9 |
0 |
0 |
0 |
0.000 |
0.0 |
0.9 |
E1 |
Email to Notify State Agencies of Selected LEAs |
25 |
25 |
1 |
25 |
0.0501 |
1.3 |
0 |
0 |
0 |
0.000 |
0.0 |
1.3 |
||
N/A |
State Agency Sends Email to Selected LEAs |
35 |
35 |
1 |
35 |
0.1670 |
5.8 |
0 |
0 |
0 |
0.000 |
0.0 |
5.8 |
||
State Key Staff |
D4 |
State Director Survey |
51 |
51 |
1 |
51 |
1.0000 |
51.0 |
0 |
0 |
0 |
0.000 |
0.0 |
51.0 |
|
State Government Sub-Total |
105 |
105 |
4.05 |
425 |
0.2865 |
121.7 |
0 |
0.00 |
0 |
0.0000 |
0.0 |
121.7 |
|||
Local Education Agency (LEA) |
LEA Director
|
B2 |
Recruitment Email for Cognitive Test of Interview Guide |
4 |
3 |
1 |
3 |
0.0501 |
0.2 |
1 |
1 |
1 |
0.1670 |
0.2 |
0.3 |
B3 |
Cognitive Test of the LEA Interview Guide |
3 |
3 |
1 |
3 |
1.5000 |
4.5 |
0 |
0 |
0 |
0.000 |
0.0 |
4.5 |
||
E2 / E3 |
State Agency Email to Selected LEAs |
35 |
35 |
1 |
30 |
0.0501 |
1.8 |
0 |
0 |
0 |
0.000 |
0.0 |
1.8 |
||
E4 |
Email to Schedule LEA Telephone Interviews and FAQs |
35 |
30 |
1 |
30 |
0.1670 |
5.0 |
5 |
1 |
5 |
0.1670 |
0.8 |
5.8 |
||
E5 |
Reminder Email to Schedule LEA Telephone Interviews |
20 |
20 |
1 |
20 |
0.0501 |
1.0 |
0 |
0 |
0 |
0.000 |
0.0 |
1.0 |
||
E6 |
Confirmation Email for LEA Telephone Interviews |
30 |
30 |
1 |
30 |
0.0167 |
0.5 |
0 |
0 |
0 |
0.000 |
0.0 |
0.5 |
||
Table A.12-1. Annual Burden Table (continued)
|
|
|
|
|
Responsive |
Non-Responsive |
|
||||||||
Respondent Category |
Type of respondents |
OMB Appendix Number |
Instruments |
Sample Sizea |
Number of respondents |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Number
of |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Grand Total Annual Burden Estimate (hours) |
Local Education Agency (LEA) |
LEA Director
|
E7 |
LEA Interview Guide (includes consent) |
30 |
30 |
1 |
30 |
1.5000 |
45.0 |
0 |
0 |
0 |
0.0000 |
0.0 |
45.0 |
E8 |
Thank You Email to LEAs for Interview (Interview-only LEAs) |
10 |
10 |
1 |
10 |
0.0167 |
0.2 |
0 |
0 |
0 |
0.0000 |
0.0 |
0.2 |
||
N/A |
LEA Director to send relevant IRA policy/procedure documents |
30 |
30 |
1 |
30 |
0.2500 |
7.5 |
0 |
0 |
0 |
0.0000 |
0.0 |
7.5 |
||
E9 |
Email to Schedule LEA Calls on Household Applications (Interview and Application LEAs) |
20 |
20 |
1 |
20 |
0.0835 |
1.7 |
0 |
0 |
0 |
0.0000 |
0.0 |
1.7 |
||
E10 |
Reminder Email to Schedule LEA Calls on Household Applications |
10 |
10 |
1 |
10 |
0.0501 |
0.5 |
0 |
0 |
0 |
0.000 |
0.0 |
0.5 |
||
E11 |
Confirmation Email for LEA Calls on Household Applications |
20 |
20 |
1 |
20 |
0.0167 |
0.3 |
0 |
0 |
0 |
0.000 |
0.0 |
0.3 |
||
E12 |
Guide for Telephone Calls on Collection of Household Applications |
20 |
20 |
2 |
40 |
0.5000 |
20.0 |
0 |
0 |
0 |
0.0000 |
0.0 |
20.0 |
||
E13 |
Email to LEAs with Procedures to Submit Household Applications |
20 |
20 |
1 |
20 |
0.0835 |
1.7 |
0 |
0 |
0 |
0.0000 |
0.0 |
1.7 |
||
E14 |
Thank You Email to LEAs for Household Applications (Interview and Application LEAs) |
20 |
20 |
1 |
20 |
0.0167 |
0.3 |
0 |
0 |
0 |
0.0000 |
0.0 |
0.3 |
||
Table A.12-1. Annual Burden Table (continued)
|
|
|
|
|
Responsive |
Non-Responsive |
|
||||||||
Respondent Category |
Type of respondents |
OMB Appendix Number |
Instruments |
Sample Sizea |
Number of respondents |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Number
of |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Grand Total Annual Burden Estimate (hours) |
Local Education Agency (LEA) |
LEA Key Staff |
E6 |
Confirmation Email for LEA Telephone Interviews |
60 |
60 |
1 |
60 |
0.0167 |
1.0 |
0 |
0 |
0 |
0.000 |
0.0 |
1.0 |
E7 |
LEA Interview (includes consent) |
60 |
60 |
1 |
60 |
1.5000 |
90.0 |
0 |
0 |
0 |
0.0000 |
0.0 |
90.0 |
||
E8 |
Thank You Email to LEAs for Interview (Interview-only LEAs) |
20 |
20 |
1 |
20 |
0.0167 |
0.3 |
0 |
0 |
0 |
0.0000 |
0.0 |
0.3 |
||
E9 |
Email to Schedule LEA Calls on Household Applications (Interview and Application LEAs) |
40 |
40 |
1 |
40 |
0.0835 |
3.3 |
0 |
0 |
0 |
0.0000 |
0.0 |
3.3 |
||
E10 |
Reminder Email to Schedule LEA Calls on Household Applications |
20 |
20 |
1 |
20 |
0.0501 |
1.0 |
0 |
0 |
0 |
0.000 |
0.0 |
1.0 |
||
E11 |
Confirmation Email for LEA Calls on Household Applications |
40 |
40 |
1 |
40 |
0.0167 |
0.7 |
0 |
0 |
0 |
0.000 |
0.0 |
0.7 |
||
E12 |
Guide for Telephone Calls on Collection of Household Applications |
40 |
40 |
2 |
80 |
0.5000 |
40.0 |
0 |
0 |
0 |
0.0000 |
0.0 |
40.0 |
||
E13 |
Email to LEAs with Procedures to Submit Household Applications |
40 |
40 |
1 |
40 |
0.0835 |
3.3 |
0 |
0 |
0 |
0.0000 |
0.0 |
3.3 |
||
Table A.12-1. Annual Burden Table (continued)
|
|
|
|
|
Responsive |
Non-Responsive |
|
||||||||
Respondent Category |
Type of respondents |
OMB Appendix Number |
Instruments |
Sample Sizea |
Number of respondents |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Number
of |
Frequency of response |
Total Annual responses |
Hours per response |
Annual burden (hours) |
Grand Total Annual Burden Estimate (hours) |
Local Education Agency (LEA) |
LEA Key Staff |
N/A |
Collection and Transmission of Sample of Household Applications |
40 |
40 |
1 |
40 |
2.0000 |
80.0 |
0 |
0 |
0 |
0.0000 |
0.0 |
80.0 |
E14 |
Thank You Email to LEAs for Household Applications (Interview and Application LEAs) |
40 |
40 |
1 |
40 |
0.0167 |
0.7 |
0 |
0 |
0 |
0.0000 |
0.0 |
0.7 |
||
Local Government Sub-Total |
99 |
93 |
8.18 |
761 |
0.4079 |
310.4 |
6 |
1.00 |
6 |
0.1670 |
1.0 |
311.4 |
|||
|
TOTAL |
|
|
204 |
198 |
5.99 |
1,186 |
0.3644 |
432.2 |
6 |
1.00 |
6 |
0.1670 |
1.0 |
433.2 |
Provide estimates of the total annual cost burden to respondents or recordkeepers resulting from the collection of information, (do not include the cost of any hour burden shown in questions 12 and 14). The cost estimates should be split into two components: (a) a total capital and start-up cost component annualized over its expected useful life; and (b) a total operation and maintenance and purchase of services component.
There are no capital/start-up or ongoing operation/maintenance costs associated with this information collection.
Provide estimates of annualized cost to the Federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.
The total annual cost to the Federal Government is $303,734.25, or $911,202.76 over the 36-month period of the contract. The largest cost to the Federal Government is to pay the contractor $850,585.00 to conduct this study and deliver reports and data files. The information collection also assumes a total of 416 hours of a Federal Employee’s time per year for a GS-13, Step 1 in the Washington, DC area, at $46.46 per hour for a total of $19,327.36 per year. The information collection also assumes a total of 16 hours of a Branch Chief's time per year: for a GS-14, Step 1 in Washington, DC area, at $54.91 per hour for a total of $878.56 per year. Federal employee pay rates are based on the Office of Personnel Management (OPM) salary table for 2018 for the Washington, DC, metro area locality (for the locality pay area of Washington-Baltimore-Arlington, DC-MD-VA-WV-PA).9
Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.
This is a new information request, which will add 433.2 total annual burden hours and 1,192 total annual responses to FNS burden inventory.
For collections of information whose results are planned to be published, outline plans for tabulation and publication.
The data will be analyzed using descriptive, bi-variate, and multivariate analysis. The findings will be synthesized and published in a technical final report form as well as a summary for the general public – both of which will be posted on the FNS website. The final report will address all research objectives. The data analyses will involve both a process evaluation and an outcome evaluation, and data will be tabulated for both. The data analysis will be conducted as follows:
Prepare analytic data files. The main data source for the outcome evaluation will be the roughly 6,000 school meal benefit applications. We will document the frequency of applications with correct and incorrect meal benefit determinations during each school year and for each LEA in the sample. Descriptive statistics will be generated to check for missing data, outliers, and inconsistent data patterns.
Tabulate data. Data tables will be specified for all research questions under the three study objectives: (1) Collect data and describe the process and policies surrounding the second, independent review of applications; (2) For a subsample of LEA’s selected in Objective 1, collect data and conduct a review and analysis of household applications for two (2) nonconsecutive school years (SY 16-17 and SY 18-19) to describe certification errors for household applications; and (3) Assess the effectiveness of the second, independent review of applications process and the FNS-874 reporting requirements and provide recommendations for best practices.
The process evaluation will utilize data gathered through qualitative interviews. The qualitative interviews will be analyzed using qualitative data analysis software such as NVivo; data will be coded, and sub-group specific as well as cross-cutting themes will be identified. We will present tables documenting both supporting and contradictory evidence for each theme.
For the household applications, we will tabulate the percent of applications with incorrect eligibility at each combination of error (free, reduced price, paid). In addition, we will determine and tabulate other types of administrative errors that occur such as certified applications with missing or incomplete data. From these tabulations, we will construct our primary outcome variable – an overall percent administrative certification error rate for each LEA and school year.
Results of the study will be made available to the public via the FNS website, and State Directors and LEAs that participate in the study will be notified and provided a link to the online results.
Table A.16-1 shows the data collection, analysis & coding, and reporting schedules for the final briefing and report.
Table A.16-1. Reporting Schedule
Activity |
Schedule |
|
Conduct State CN Director Survey |
1-9 |
Weeks after OMB approval |
Select Data Collectors |
12 |
weeks after OMB approval |
Train Data Collectors |
13 |
weeks after OMB approval |
Conduct LEA Data Collection |
21-66 |
weeks after OMB approval |
Analyze Data |
66-77 |
weeks after OMB approval |
Draft, Revised and Final Reports |
77-97 |
weeks after OMB approval |
Draft, Revised and Final Briefing Materials |
91-95 |
weeks after OMB approval |
Draft, Revised and Final Data files |
77-97 |
weeks after OMB approval |
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
All data collection instruments will display the OMB control number and expiration date.
Explain each exception to the certification statement identified in Item 19 of the OMB 83-I" Certification for Paperwork Reduction Act."
The agency is able to certify compliance with all provisions under Item 19 of OMB Form 83-i.
5 State agency Directors from each of the 50 States and the District of Columbia.
6 For these purposes, a small LEA is one with fewer than 500 enrolled students.
7 https://www.bls.gov/oes/current/naics4_999200.htm#11-0000
8 https://www.bls.gov/oes/current/naics3_611000.htm#11-0000
9 Office of Personnel Management, General Schedule, accessed March 16, 2018, at: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB.pdf. Hourly rates calculated as annual rate divided by 2,087, as directed by “Fact Sheet: How to Compute Rates of Pay at: https://www.opm.gov/policy-data-oversight/pay-leave/pay-administration/fact-sheets/how-to-compute-rates-of-pay/
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for OMB No |
| Author | USDA |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |