Download:
pdf |
pdf05/03/2016
TO:
Stephanie Tatham, OMB Desk Officer
Office of Information and Regulatory Affairs
Office of Management and Budget
THROUGH:
Ruth Brown, Information Collection
United States Department of Agriculture
Office of Chief Information Office
FROM:
Lynnette Thomas
Branch Chief, Planning and Regulatory Affairs Office
Food and Nutrition Service (FNS)
Re:
Request Approval to Perform Research Under Approved Generic OMB
Clearance No. 0584-0524
The USDA Food and Nutrition Service requests OMB approval for a formative research project
under generic clearance number 0584-0524. The proposed research will focus on the
communication between school districts and households in the process of verifying household
eligibility for free and reduced price school meal benefits under the National School Lunch
Program (NSLP) and the School Breakfast Program (SBP).
Local educational agencies (LEAs) are required by statute 42 U.S.C. 1758(b)(3)(D) to verify a
small sample of household applications approved for free or reduced price benefits each school
year. LEAs select their samples from approved applications on file as of October 1, and notify
households by letter that they have been selected for verification review. The letters instruct
households to return documentation in support of the type and amount of income that they
reported on their applications.
On review of the documentation submitted by households, LEAs either confirm or amend the
certification decisions reached at the time the applications were originally processed.
Households that fail to return supporting documentation lose their free or reduced-price school
meal benefits. A 2004 USDA case study found that many of the households that failed to
respond to LEA verification requests were, in fact, income eligible for the benefits that were
awarded to them at the time their applications were processed.
The goal of this research is to identify communication protocols that reduce the incidence of
household non-response to LEA verification requests. The project proposed here follows up on
a project conducted in school year 2015-2016, cleared under the same generic clearance. The
previous project varied the content, but not the timing, of LEA verification letters to
households.
This second proposed project will test the effectiveness of further changes in the content of
1
LEA letters, based on results of the 2015-2016 work, as well as changes in the timing of LEA
verification. The project will be led by the White House Social and Behavioral Sciences Team
in cooperation with FNS.
1. Title of Project
Test Modified Communication Protocols to Reduce Household Non-response in NSLP/SBP
Verification Process
2. Control Number
0584-0524, expiration date: 6/30/2016
3. Entities affected by this project
•
Local educational agencies (LEA)
The research team will recruit LEAs to participate in a protocol described below. The
total study will be made up of a geographically diverse selection of the largest LEAs in
terms of number of households contacted for verification. Each of these LEAs will use
the revised communication protocol’s timing and content for a randomly selected subset
of their verification sample households (the intervention group) while retaining their
previous process for the remaining households in the verification sample (the control
group). The intervention group will be selected by randomization at the household level.
Randomization will ensure that selection is fair and that the results of the intervention can
be rigorously evaluated. LEAs will use a single protocol that outlines action steps for
both the intervention and control households.
Participating LEAs will record the verification procedures that they followed and the
results of the process. Results will be shared with the research team. LEAs will report the
following data to the researchers:
o Dates registered automatically by the LEAs’ software indicating: 1
LEA initial contact with verification sample households
LEA receipt of household documentation for verification purposes
o Method of receipt for all contacts, i.e. the number of receipts by
email/mail/phone/in-person, etc.
o Name of any specialized software or vendor used during the verification process
o A copy of the final verification letter used
o Final copies of any other printed materials given to verification households
o Estimated person-hours dedicated to the verification process
o The ultimate outcome of the verification process (using the same measures
1
No new record keeping will be requested, but LEAs will be requested to share the dates that are automatically
recorded by their software system.
2
reported on the FNS-742 (OMB No. 0584-0026, Exp. 4/30/2016)) but split into
intervention and control household subtotals
•
Household applicants for free and reduced price school meals
Household beneficiaries are affected by this project, but no new information will be
collected from households as part of this project, nor will any households be contacted by
FNS or the research team. The obligation of a household to respond to LEA requests for
verification documentation is identical whether their household is part of the intervention
group or the control group.
It is worth noting that the intent of the research protocol is to make the verification
procedure more efficient and effective. Therefore, it is possible that the changes
implemented will reduce the overall time needed by households and LEAs to process
each household’s verification by using a more streamlined process and incentivizing
more families to respond to the initial contact without need for additional follow-up from
the LEA.
4. Research Objectives and Design
Objectives:
This project will test the effectiveness of limited changes in the content and timing of letters
sent by LEAs to households requesting documentation to support the households’ applications
for free or reduced price school meal benefits. The project will also test the effectiveness of
changes in LEAs' follow-up communication protocol with households. In both cases
effectiveness will be measured by:
•
•
The rate at which households respond to LEA communication with the requested
documentation
The time it takes households to respond to the LEAs' initial and follow-up requests 2
The ultimate goal of the project is to identify communication protocols that reduce the
household non-response rate. Households that fail to respond to the LEA request for
documentation lose their free or reduced price benefits. In the school year 2014-2015
verification process, 35.8 percent of households contacted by their LEAs failed to respond.
Among the nine largest LEAs, the rate of non-response was even worse at 52.2%. This is more
than four times higher than the percent of households that lose the benefits by responding to the
verification process and exceeding the income threshold. Research conducted by USDA in
2004 suggests that many of these households may be income-eligible for free or reduced-price
2
This will be based on the date registered automatically by the software system for when a particular household was
sent the verification request and when they were marked as verified after submission of documents. LEAs will not
need to employ any new record keeping system, but will be requested to export and share the dates registered by their
software system. Ideally this data will be exported at household level for matching and analysis.
3
meals. 3
The project will take place in the summer and fall of 2016, during the application and
verification process for school year 2016-2017.
Sample selection:
The research team will sample at least three LEAs but no more than nine with the largest
number of households selected for verification as reported in the most current available FNS742 dataset. 4 The LEAs will be included in the sample if they reported a response rate below
80% in SY 2015-2016 with a verification sample of at least 500 applications. Within each of
the LEAs, revised communication materials and protocols will be randomly assigned to one
half of the households selected for verification (intervention group) while the other half of the
households will continue to receive the same communications as normally would be used by
the LEAs (control group).
Because the project is focused on the NSLP/SBP verification process, the research team will
draw their samples from the subset of LEAs that will process traditional applications in school
year 2016-2017. The following LEAs will be dropped from the FNS-742 universe before
selecting the intervention and control group samples:
•
•
•
LEAs that operate district-wide under the Community Eligibility Provision as of school
year 2015-2016
LEAs that are expected to see significant decreases in household applications due to
increase in CEP participation in school year 2016-2017
LEAs that operate under NSLP/SBP Provision 2 or Provision 3 and will be a non-base
year in school year 2016-2017
On selection of LEAs for participation in the project, FNS will contact the appropriate FNS
Regional Offices and State Agencies by letter to inform them of the purpose of the project and
the nature of the data collection. FNS and the research team will then contact the State
agencies, working with them to secure the participation of the selected LEAs. After securing
the participation of LEAs, the research team will work with the LEAs and their vendors
providing verification sampling software to integrate the proposed changes to the
3
The 2004 USDA Case Study of National School Lunch Program Verification Outcomes in Large Metropolitan
School Districts found that just over half of households that failed to respond to verification requests in 21 large
metropolitan LEAs in the fall of 2002 were eligible for at least the level of benefits they were initially certified to
receive. Although the LEAs examined in the case study are not representative of all LEAs, and the makeup of the
households that are subject to verification review today is much different than was the case in 2002, the structure of
the verification process has not changed much over the years. For this reason, these findings continue to raise concern
that the verification process is a barrier to program access for some households.
4
The FNS-742 is an LEA-level form submitted annually by State agencies that administer the school meal programs.
It is one of the few forms with LEA level statistics, making it ideal for use in selecting LEA-level samples for FNS
research projects.
4
communication timing and content.
The research team will provide a written protocol to all LEAs and will hold an individual
phone calls with any LEA that requires assistance in implementing research protocol.
5. Number of participants / respondents
Proposed Sites
for Pilot
Number of
respondents
Number of intervention
households
Number of control
households
Participating LEAs 9 5
2800
2800
Notes Only the LEAs will be reporting the pilot evaluation data to the research team
while the households will be reporting the same information that would be
required regardless of the pilot project and therefore are not respondents. The
total number of households may change in accordance with verification sample
size requirements in the Senate’s 2016 Child Nutrition Reauthorization bill.
LEAs selected for participation will implement the following steps as part of the research
protocol for intervention households:
1. Use of a verification communication material designed by the research team that
incorporates insights from the behavioral sciences literature. The verification letter may
be customized by the LEA with input from the research team and FNS
2. Send verification letters to households on a continuous basis (as certification decisions
are made) along with letters notifying households of their certification for program
benefits. The research team will provide assistance on selecting and notifying households
for verification on a continuous basis
3. Follow a standard protocol for reminding households to return verification
documentation. This will include one or more contacts via letter, email, and phone
designed by the research team and sent to households at intervals specified by the
research team. The protocol may also include text messages and phone calls to
households
4. Acceptance of household documentation in multiple forms:
a. Photocopies delivered by mail (the current standard)
b. Original documents delivered by mail with expectation that the LEA will return
original documents to the household
c. Emailed pictures of documents
5. Provide documentation to the research team that records when each step in the research
protocol was implemented
6. Provide data to the research team on when notification letters were sent (regarding initial
acceptance), when verification requests were sent, and when responses were received
5
Depending upon our success in recruitment of LEAs for participation, the research team may be able to reduce the
number of participating LEAs to as few as three (3)
5
from households 6
6. Time Needed per Response
Respondent
Type
Local Education
Agencies
Software Vendors
LEA: Decline
participation
States Agencies
Time
Minutes Hours 7
Type of Response
Recruitment and follow-up discussions
with research team in conference call
and/or individually
Customization of verification letter
Review of instructions and
implementation of research protocol
1. Implement continuous
sampling/selection process (weekly)
2. Marginal time per letter prepared
Reminders to households (50%)
Processing documentation other than
paper copies (50%)
Document household response & date
Provide copy of verification letter to FNS
Prepare final report on household level
outcomes
Determine best method for automating
study protocol through software
90
1.5
120
2
60
1
15
0.25
1
5
0.017
0.083
3
0.05
2
10
0.033
0.167
30
0.5
120
2
Communication about recruitment
30
0.5
Communication regarding the project and
recruitment
60
1
6
As noted earlier, this will only be requested if the data can be automatically generated by the LEAs software system.
The LEA will not be requested to manual track dates of interaction with each household beyond their normal
procedures.
7
Hours shown rounded to nearest thousandth
6
7. Total Respondent Burden
(a)
Affected
Public
LEA:
Intervention
(f)
(e)
Number
(d)
Responses
of
Number of
per
Responses
(d ˟ e)
Respondents Respondent
(g)
Hours
per
Response
(h)
Total
burden
Hours 9
(f ˟ g)
(b)
Type of Response
(c)
Attachment
Recruitment and follow-up
discussions with research team in
conference call and/or individually
Attachment H
9
1
9
1.5
13.5
Customization of verification letter
Attachment A
9
1
9
2
18
Review of instructions and
implementation of intervention
protocol
Attachment D
(provides
instructions)
9
1
9
1
9
1. Implement continuous
sampling/selection process
(weekly) 10
Attachment
E 11
9
12
108
0.25
27
8
8
Table shows rounded hours. Unrounded hours used in calculation for Total burden hours (h)
Hours are rounded up to the a tenth of an hour. Unrounded hours are used for calculation of total burden hours.
10
Based on conversations with software vendors, the time burden upon LEAs may be more or less depending on how much of the process can be automated and
installed without additional time. The time may be less if one software provider services all included LEAs and can easily install the automated process.
11
This diagram provides a high level overview of the selection of applications into the “rolling verification” and “October 1” verification subsamples. Because
LEAs will rely on their software to select these subsamples, this diagram is mainly background reference for the LEAs. FNS and the SBST will use the same
diagram in more technical conversations with software vendors to discuss possible coding changes necessary to support the rolling verification process.
9
7
(a)
Affected
Public
(f)
(e)
Number
(d)
Responses
of
Number of
per
Responses
(d ˟ e)
Respondents Respondent
(g)
Hours
per
Response
(h)
Total
burden
Hours 9
(f ˟ g)
(b)
Type of Response
(c)
Attachment
2. Marginal time per letter prepared
Attachment A
9
311.1 12
2800
0.017
46.7
Reminders to households (50%) 13
Attachment B
&C
9
155.6 14
1400
0.083
116.7
Processing documentation other than
paper copies (50%) 15
Attachment D
(provides
instructions)
9
155.6 16
1400
0.05
70
Document household response &
date 17
Attachment F
9
622.2 18
5600
0.033
186.7
Provide copy of verification letter
to FNS
Attachment
D (provides
instructions)
9
1
9
0.167
1.5
12
8
Average responses from intervention households. Rounded for display, unrounded used in calculations.
Intervention households only.
14
Half of average responses from intervention households. Rounded for display, unrounded used in calculations.
15
Intervention households only.
16
Half of average responses from intervention households. Rounded for display, unrounded used in calculations.
17
This provides a maximum burden. Much of this documentation will be automated by the software in use and as a result the burden will likely be lower through
the use of technological automation.
18
Average responses from both intervention and control households. Rounded for display, unrounded used in calculations.
13
8
(a)
Affected
Public
(f)
(e)
Number
(d)
Responses
of
Number of
per
Responses
(d ˟ e)
Respondents Respondent
(g)
Hours
per
Response
(h)
Total
burden
Hours 9
(f ˟ g)
(b)
Type of Response
(c)
Attachment
Prepare final report on household
level outcomes
Attachment D
(provides
instructions)
9
1
9
0.5
4.5
Software
Vendors
Determine best method for
automating study protocol through
software
Attachment E
(rolling
verification
logic model) 19
9
1
9
2
18
LEA: Decline
participation
Communication about recruitment
Attachment H
4 20
1
9
0.5
2
State Agencies
Communication regarding the project
and recruitment
Attachment G
6 21
1
6
1
6
8
State & Local Government Sub-total
19
11378
502
Business Sub-total
9
9
18
19
FNS and the SBST will use the Attachment E diagram in conversations with the vendors that design and maintain the verification software used by the
project’s participating LEAs. The purpose of those conversations is to discuss how the software can support rolling verification and whether any coding changes
will be necessary.
20
Maximum number of LEAs estimated to decline participation.
21
Maximum number of State Agencies contacted: assumes one LEA will decline and thus another state will need to be contacted.
9
(a)
Affected
Public
(b)
Type of Response
(c)
Attachment
TOTAL
(f)
(e)
Number
(d)
Responses
of
Number of
per
Responses
(d ˟ e)
Respondents Respondent
28
10
11387
(g)
Hours
per
Response
8
(h)
Total
burden
Hours 9
(f ˟ g)
520
Methodology / Research Design
Statistical testing:
The minimum of three proposed LEAs would comprise an estimated sample of 5600
households receiving verification applications. 22 This sample size will enable the project to
detect a 3.1% change in the response rate with a Type I error rate of 5% and a Type II error rate
of 10%. 23 This means that the study will have a 90% chance of detecting a positive outcome of
a response rate of 57.8% for intervention households compared to 54.7% among the control
group households.
Intervention details: changes to letter
An alternative verification letter will be provided to all participating LEAs for intervention use.
22
Note that while the respondent time incorporates only the intervention households (2800) the full sample size
calculation includes the control group households.
23
This sample size calculation does not control for stratification. The calculation was based on the baseline levels of
response rates in the sample from the 2014-2015 school year. The expected change is informed by preliminary data
the 2015-2016 pilot results.
11
The letter template will be customized by the research team with LEA input. That template will
then be adapted by the LEA to each LEA’s context.
Intervention details: changes to timing
Currently, LEAs determine the verification sample in one of three ways:
1. Verification process 1: Standard
2. Verification process 2: Alternate 1
3. Verification process 3: Alternate 2
Typically, LEAs using the Standard verification process wait until at least 1 October to begin
verification. They determine the total number of applications and multiply by 0.03 to obtain the
total number of applications to verify. They also determine the total number of error-prone
applications (applications with incomes within $100 of the limit). They then select the
verification sample as the lesser of:
•
•
(Total applications)*0.03, selected from error prone applications.
3,000 error prone applications.
To use the 3% sample method continuously (rather than on or after 1 October), LEAs would
simply determine whether or not an individual application required verification at the time of
eligibility determination. There are a number of ways to do this. One way to do this would be to
establish a random order ahead of time, such that, for example, the first approved application
that is considered error prone is asked for verification, the second is not, the third is not, and so
on. This way, as soon as individuals are determined to be eligible, LEAs know whether the
household income must be verified. This would allow for a much shorter lag between
application and verification.
The research team will help LEAs to determine an exact procedure for conducting the
continuous sampling. For example, notifications and verification samples could be bundled and
sent out on a weekly basis.
Since LEAs are required to notify households of an eligibility determination—i.e. notify them
of a successful application—LEAs will include a request for verification at this time for
selected households. Collapsing the verification step and the notification step will reduce the
total communications burden on the LEAs and decrease the likelihood of household nonresponse.
Intervention details: Reminders
The research protocol includes extra reminders to households selected for verification. Currently,
LEAs must make one follow up attempt to contact households that do not respond. LEAs will
communicate with intervention households that do not respond to the initial verification request
through reminder letters, emails, phone calls, and/or texts. Letters and emails are the most
12
straightforward technology to use, and thus these are the modes of communication that we expect
all of the LEAs to be capable of utilizing. The research team will design the reminder template to
have a similar look and feel to the redesigned letter and will also create a communication
protocol around phone call reminders.
8. Federal Costs
FNS will offer to pay the costs of:
1. Stamped return envelopes included with the initial verification letters sent by
LEAs to intervention households
2. The cost of letters and/or postcards plus postage for up to two reminder mailings
by LEAs to intervention households
A high end estimate of this cost assumes that all intervention group households are sent
postage-paid return envelopes with their initial verification letters and are all contacted by mail
two additional times during the course of the intervention.
Total Federal Cost: $5,000
9. Confidentiality
Household level information
LEAs will collect no additional information from households through this project. All
household level information collected by LEAs as part of the certification and verification
processes will remain with the LEAs. LEAs will report only summary statistics on verification
outcomes to FNS and the research team.
10. List of Attachments
•
•
•
•
•
•
•
•
Attachment A: “Attachment A - Modified Verification Letter.docx”
Attachment B: “Attachment B – Reminder Letter.docx”
Attachment C: “Attachment C – Intervention-Reminder Call Script.docx”
Attachment D: “Attachment D – Protocol for Participating LEAs.docx”
Attachment E: “Attachment E - Rolling Assignment Procedure Logic Model.docx”
Attachment F: “Attachment F - Report Template for HH Contact & Responses.xlsx”
Attachment G: “Attachment G – State Recruitment Letter.docx”
Attachment H: “Attachment H – LEA Recruitment Letter.docx”
13
| File Type | application/pdf |
| Author | Wepplo, Timothy - FNS |
| File Modified | 2016-05-03 |
| File Created | 2016-05-03 |