USDA_Review of Child Nutrition OMB_SS Part B FinalX

SUPPORTING STATEMENT PART B for

OMB Control Number 0584-NEW

Review of Child Nutrition Data and Analysis for Program Management

October 5, 2016

Project Officer: Dennis Ranalli

Contracting Officer’s Representative

Food and Nutrition Service

U.S. Department of Agriculture

3101 Park Center Drive

Alexandria, Virginia 22302

Phone: 703-305-2149

Email: [email protected]

TABLE OF CONTENTS

List OF EXHIBITS 1

Part B: Collection of Information Employing Statistical Methods 1

B.1. Respondent Universe and Selection Methods 1

B.2. Procedures for the Collection of Information 4

B.3. Methods to Maximize Response Rates and the Issue of Non-Response 9

B.4. Tests of Procedures 13

B.5. Consultants 14

ATTACHMENT A: Statutes and Regulations

Attachment B: Study recruitment Materials and surveys

Attachment B.1: Advance notice to STATE DIRECTORS

ATTACHMENT b.1.1: STATE Study OVERVIEW

ATTACHMENT B.1.2: sTATE Study faq

ATTACHMENT B.1.3: STATE STUDY BROChURE

Attachment B.2: Advance notice to SFA DIRECTORS

ATTACHMENT B.2.1: sfa STUDY overview

ATTACHMENT B.2.2: sfa STUDY faq

ATTACHMENT b.2.3: sfa STUDY brochure

Attachment B.3: SFA SURVEY EMAIL INVITATION

Attachment B.4: STATE agency SURVEY EMAIL INVITATION

ATTACHMENT B.5: SFA Survey Email Reminder

ATTACHMENT B.6: STATE Survey Email Reminder

Attachment B.7: SFA SURVEY FOLLOW-UP TELEPHONE SCRIPT

Attachment B.8: STATE SURVEY FOLLOW-UP TELEPHONE SCRIPT

Attachment B.9: PAPER STATE SURVEY

Attachment B.10: PAPER SFA SURVEY

Attachment B.11: STATE SURVEY (web Screenshots)

Attachment B.12: SFA SURVEY (web SCREENSHOTS)

attachment C: Literature reviewed for duplicate effort

Attachment D: Feedback on Survey Development

Attachment D.1: EXTERNAL CONSULTANTS

Attachment D.2: cognitive interview testing report

ATTACHMENT E: Data Confidentiality Pledge

Attachment F: Public Comments

ATTACHMENT G: NASS COMMENTS

ATTACHMENT H: Sample Sizes, Estimated Burden, and Estimated Cost of Respondent Burden (Table)

ATTACHMENT I: METHOD TO ADJUST FOR NON-RESPONSE BIAS

List OF EXHIBITS

Table B.1: Number of Entities (SFAs) in the Sampling Universe……………………..………..3

Table B.2: Individuals Responsible for Statistical Aspects and Data Collection and Analysis…15

Part B: Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Selection Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

USDA’s Food and Nutrition Service (FNS) is requesting Office of Management and Budget’s (OMB) approval to conduct two data collection activities: 1) a survey of State Child Nutrition (CN) agencies responsible for administering the National School Lunch Program (NSLP) and the School Breakfast Program (SBP); and 2) a nationally representative survey of School Food Authorities (SFAs). Each of these is a new data collection activity. FNS will collect each of them only once. The survey of State CN agencies does not require sampling; it is a census. A census is required for State CN agencies because their MIS are unique and would not be properly represented through a sample. State CN agency MIS are often acquired at various points in time and customized to meet specific needs. State CN agencies typically use a combination of different systems. For these reasons, and given that there are only 51 respondents, a census will be used to collect this information instead of a survey. FNS will distribute a Web-based survey to SFA directors in all 50 State CN agencies and the District of Columbia. FNS expects a 100 percent response rate to the state survey.

The survey of SFAs will employ stratified random sampling techniques to select a nationally representative sample of SFAs. The sample will be representative in terms of the seven (7) FNS regions, and the number of students in an SFA. FNS classifies States into the following regions¹:

• Mid-Atlantic (Delaware, District of Columbia, Maryland, New Jersey, Pennsylvania, Virginia, West Virginia)

• Midwest (Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin)

• Mountain Plains (Colorado, Iowa, Kansas, Missouri, Montana, Nebraska, North Dakota, South Dakota, Utah, Wyoming)

• Northeast (Connecticut, Maine, Massachusetts, New Hampshire, New York, Rhode Island, Vermont)

• Southeast (Alabama, Florida, Georgia, Kentucky, Mississippi, North Carolina, South Carolina, Tennessee)

• Southwest (Arkansas, Louisiana, New Mexico, Oklahoma, Texas)

• Western (Alaska, Arizona, California, Hawaii, Idaho, Nevada, Oregon, Washington)

For this study, FNS defines SFA size as follows:

• Small – Less than 1,000 Students

• Medium – Between 1,000 and 5,000 Students

• Large – More than 5,000 Students.

FNS requires the sample to be sufficient to generate national estimates of population characteristics with the following levels of precision:

• All public SFAs: ±5.0% at the 95% level of confidence; and

• Sub-groups: between ±7.0% and 10.0% at the 95% level of confidence.

To achieve the specified levels of precision, the analytic samples must include completed surveys from a total 2,016 SFAs and 96 for each subgroup.

FNS has identified the following sub-groups:

• SFA size: Small, Medium, Large

• SFAs with at least one school that operates under the Community Eligibility Provision (CEP) or one of the other special provisions; and

• SFAs with no schools that operate under a special provision.

Table B.1: Number of Entities (SFAs) in the Sampling Universe

FNS Region	SFA Size
	Small	Medium	Large	Total
Mid-Atlantic	1,266	146	87	1,499
Midwest	3,488	218	106	3,812
Mountain Plains	2,161	73	80	2,314
Northeast	1,450	124	39	1,613
Southeast	844	163	180	1,187
Southwest	1,976	111	153	2,240
Western	1,668	199	250	2,117
Total	12,853	1,034	895	14,782

Source: Form FNS-742 VSR Database for SY2013-2014

Expected Response Rate: FNS anticipates an 80 percent response rate for the SFA web survey. To obtain the number of completed surveys to meet the precision requirements, an initial sample of 2,520 SFAs will be selected with 120 SFAs in each of the subgroups specified by FNS. Combined, FNS expects a response rate of 86.6 percent, approximately 2,227 completed responses out of the total sample 2,581(comprising 51 from State CN agencies and 2,176 fromSFAs) from both the State CN agencies census and the SFA survey.

To achieve this response rate, a series of pre-survey and timed follow-up activities have been planned. Section B.3 describes the proposed approach in detail. In summary, it comprises affective pre-survey recruitment and notifications of sampled SFAs and States, well-timed follow-up activities, and effective survey design.

B.2. Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

B.2.1 Statistical Methodology for Stratification and Sample Selection

A stratified random sample of SFAs will be selected using the three SFA sizes (described in B.1) and seven FNS regions, resulting in 21 strata. FNS does not require separate statistical estimates for each region. However, stratifying the sample by FNS region ensures that the sample will be distributed across the entire country. The initial sample of 2,520 SFAs will be allocated to each of the 21 strata in proportion to each stratum’s share of the total population of 14,782 SFAs. For example, there are 1,266 small SFAs in the Mid-Atlantic region, or 8.6 percent of the national total of 14,782 SFAs. Therefore, 8.6 percent of the total sample of 2,520 SFAs (213 SFAs) will be assigned to this stratum. The initial sample of SFAs in each of the three groups (small, medium and large) exceeds the minimum of 120 needed to meet the precision requirements.

The sample frame for this study will be the SFAs included in Form FNS-742, School Food Authority Verification Summary Report (VSR) Database, for SY2013-2014 (OMB No. 0584-0026 7 CFR Part 245 Determining Eligibility for Free and Reduced Price Meals and Free Milk in Schools, expiration date 10/31/19). In October of each school year, SFAs are required to verify the income reported on a sample of applications for free and reduced-price meals submitted by parents. The SFAs report the results of these verifications to their State CN agencies, which, in turn compile the information for all of the SFAs in the State and submit the compiled information to FNS on the FNS-742 by the following March 15. The VSR Database includes virtually all public SFAs and contains all of the information needed to select nationally representative samples of all public SFAs and public SFAs in each of the sub-groups identified by FNS. This study excludes private SFAs in order to ensure sample sizes are sufficient to meet the expected precision levels for SFA analyses at the national and subgroup levels.

A sampling weight will be assigned to each responding SFA in the sample. The base weight will be the inverse of the probability of selection of the responding SFA. The base weight will be adjusted for nonresponse to the survey. The sampling weights will ensure that the results will be nationally representative. This careful weighting process will eliminate most of the non-response bias. If the response rate falls below 80 percent, a separate non-response bias analysis will be conducted by comparing the effect on the survey estimates assuming various potential responses on the part of non-respondents. The information from the VSR Database will be used to construct the sampling frame. The sampling frame will be the basis on which we assess the extent to which survey nonresponse differs by SFA characteristics such as income level, SFA size, or region.

B.2.2 Measurement/Estimation Procedures

FNS is primarily interested in the data elements contained in SFA and State CN agency MIS that States currently are not required to report to FNS. Thus, researchers will use data from the survey to provide descriptive statistics of the MIS systems of SFAs and a regression analysis, which will examine factors associated with the adoption of MIS by SFAs/State CN agency. The data from the survey will include both continuous and categorical variables. For continuous variables, such as costs of developing an SFA/State CN agency’s electronic MIS, the analysis will include an examination of frequency distributions along with means, medians, and standard deviations. There will be separate descriptive analyses for each sub-group. The descriptive analyses will examine characteristics of MIS such as:

• The use of electronic MIS for administrative functions by SFAs;

• The number of administrative functions for which SFAs use existing MIS; and

• The types of administrative functions for which SFAs typically use MIS.

In addition to the characteristics of the MIS, FNS will also analyze the types and frequency of technical assistance, if any, that SFAs receive from their respective State CN agencies.

For SFAs, the descriptive analyses will test for differences among the following sub-groups:

• Size of SFA (Small, Medium, and Large); and

• SFAs that operate under the Community Eligibility Provision (CEP) or one of the other provisions (2 and 3).

The test for differences between sub-groups will use Chi-square (Χ²) tests for categorical variables and one-way ANOVA tests for continuous variables.

In addition to descriptive analyses, we will use multivariate regression models to produce a more detailed characterization of the relationships between adoption of MIS for the NSLP/SBP and SFA/State CN agency characteristics. Since most outcomes are categorical, logistic and generalized logistic regressions will be used to examine the determinants of various outcomes. The following logistic regression model will use pooled SFA survey data across all sub-groups:

For example, the dependent variable (Y) is equal to 1 if the SFA uses an MIS for any of its functions and equals zero otherwise. The independent variables for the regression model include:

• , which includes all available SFA and State CN agency characteristics from the survey data.

• , which includes identifiers for FNS region.

• , a mean zero disturbance term.

This analysis will be repeated for each SFA size group and use t-tests to determine if the estimated coefficient associated with each SFA is statistically significant. The results of these models will allow for an assessment of whether significant differences exist in the adoption of MIS across SFA and State characteristics and the direction of those differences. For example, results from the models will assess whether larger SFAs/CN agencies are more likely to use MIS that collect and maintain data elements beyond what is required, controlling for other State and SFA characteristics.

B.2.3. Degree of accuracy needed for the purpose described in the justification

FNS requires the sample to be sufficient to generate national estimates of population characteristics with the following levels of precision for SFAs:

• All public SFAs: ±5.0% at the 95% level of confidence; and

• Sub-groups: between ±7.0% and 10.0% at the 95% level of confidence.

To achieve the specified levels of precision, the analytic samples must include completed surveys from a total 2,016 SFAs and 96 for each sub-group. In addition, we will create separate estimates for SFAs with and without at least one school which is either a Provision 2/3 school or a CEP school. Together with the SFA size groups, we can, for example, examine if the data systems are different in SFAs with no special provisions compared to those under special provisions. Using data from the VSR Database, which includes both the number of schools operating under a special provision or CEP and the total enrollment in these schools, FNS will assess whether the type of data system used by an SFA is related to the proportion of schools operating under a special provision or CEP and/or the proportion of an SFA’s enrollment in such schools.

While the sample design does not include these two sub-groups (CEP and proportion of SFA enrollment in CEP schools) as stratifiers, the sampling design will include SFAs in these two groups approximately in proportion to their representation in the population of 14,782 SFAs. Since the vast majority of SFAs (90.6 percent) do not have any schools that are either a Provision 2/3 or a CEP school, the sample will include enough SFAs in this sub-group (no Provision 2/3 or CEP schools) to develop separate estimates for this sub-group that meet the precision requirements. While only 9.4 percent of SFAs have at least one school that is either a Provision 2/3 or a CEP school, even without stratifying the sample on these two sub-groups, the sample will include approximately 237 SFAs in the sub-group of SFAs that have at least one school that is either a Provision 2/3 or a CEP school. The sample will therefore include more than enough SFAs in this sub-group to meet the precision requirements.

B.2.4. Unusual problems requiring specialized sampling procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden

This data collection will be conducted by FNS once only, in the 2016-2017 school year.

B.3. Methods to Maximize Response Rates and the Issue of Non-Response

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Survey response rates are crucial for this project, as high response rates ensure the external validity of the study by allowing for accurate inferring of characteristics of SFAs and their adoption of specific MIS and related functionalities. For this study, the response rate is defined as the proportion of completed and partially completed Web survey interviews to the sample of eligible SFAs identified from the SY 2013-2014 FNS VSR Database. During the survey, a response is deemed partial when a respondent opens the survey and answers at least one question but has not completed the entire survey. FNS expects a response rate of 80 percent in the SFA survey, 100 percent in the State census, and a total response rate of 86.6 percent across both surveys. If the 86.6 percent response rate is not achieved, detailed non-response analysis and adjustments will be done using the procedures outlined in Attachment I.

We have developed a series of activities to ensure this response rate is attained. These activities include: 1) effective recruitment by developing detailed recruitment materials and seeking cooperation of State agencies that administer school meals and SFAs through FNS regional offices, 2) a detailed follow-up plan, and 3) an effective Web survey design. These three steps are described in detail below. FNS will continuously assess patterns of non-response to determine the extent to which survey non-response differs by SFA characteristics such as income level, SFA size, or region. Similarly, this will be done for State CN agencies.

1. Effective Recruitment

To ensure a high response rate to the Web survey, there is a comprehensive plan to obtain the cooperation of State agencies that administer school meals programs, and recruit and incentivize SFA administrators to participate. Support for the study will be demonstrated through FNS regions and States. The plan includes four (4) steps:

1. The contractor will notify FNS regional offices about the study and request that they notify State agencies that administer school meals programs and SFAs about the study and its data collection efforts. The notification will also request the regional office to appoint a study liaison.

2. The contractor will send a study information package to the selected regional liaisons. The package will include draft letters from the FNS regional offices to the State child nutrition agency directors informing them of the study and the nature of the data collection, and request their participation. The package will also include draft letters from the FNS regional office to the sampled SFAs.

3. The contractor will directly send the study information package to each State to request information on the most appropriate respondent to the survey. We will contact these persons to confirm the accuracy of contact information and make corrections when necessary.

4. The contractor will send the study information package to the sampled SFAs. The package will include letters from FNS requesting cooperation from the sampled SFAs. We will confirm contact information and then will contact sampled SFAs with study information. The information will also include a brochure describing functionality of the online survey system.

2. Follow-up

In addition to effective recruitment, the data collection will include a comprehensive follow-up strategy. Email survey invitations will be followed-up with three email reminders to non-responders. Purposely trained data collectors will also call non-respondents to remind them to complete the online survey and be available to answer any questions about the survey. There will be three waves of follow-up telephone calls to non-respondents. Telephone interviewers will be trained using a training manual specifically designed for this study. The training manual will cover every aspect of the call, including initial contact, overview of the survey, establishing rapport, overcoming soft refusals and other forms of resistance, and a glossary to assist data collectors and helpdesk personnel in explaining all FNS and child nutrition program-related terms in a consistent manner. In addition, the training manual will include comprehensive item-by-item specifications to allow data collectors to assist respondents who may have questions regarding how to answer specific survey items.

3. Effective Web Survey Design

This data collection will employ effective Web survey design strategies to ensure a high completion rate. These strategies--essentially efforts to decrease the workload of survey respondents--increase the perceived reward of completing the survey. The Web-based questionnaires have been designed with emphasis on ease-of-use and user-friendliness to ensure that respondents easily understand the questions, make accurate responses, and proceed smoothly to the end of the survey. Specifically, the Web surveys include clear descriptions of computer actions at point of use and the option to skip questions. In addition to incorporating these design features in the online questionnaire, each question has been crafted based on feedback from cognitive interviews to make it easily comprehensible and concise.

B.3.2 Bias due to Non-Contact and Non-Response

The contractor will continuously examine completed responses for the presence of non-response bias. Specifically, a non-response bias analysis will assess whether the survey sample is representative of the SFAs in the FNS regions. “Non-response bias” may result from a respondent’s failure to answer a particular question, because either the respondent refuses to answer or does not know the answer.

Non-response bias will be addressed by comparing characteristics of responders to non-responders using the VSR Database. This will involve determining the extent to which non-responders differ from responders by SFA characteristics such as income level, size, and region. Results of such bias analyses will increase understanding of the study sample and the population to which the study findings can be generalized. The results from non-response bias analysis will be used to determine generalizability of the results to all SFAs across the country. In order to generalize results, a sampling weight will be assigned to each responding SFA in the sample. The base weight will be the inverse of the probability of selection of the responding SFA. The base weight will be adjusted for non-response to the survey. Attachment I discusses the re-weighting method in detail. The sampling weights ensure that the results will be nationally representative.

B.4. Tests of Procedures

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Two activities were conducted to refine the data collection instruments –consultations with directors of State CN agencies and SFAs and cognitive pre-testing of draft survey instruments.

State CN Agencies and SFA Consultation. Before developing the survey instruments, three State CN agencies and six SFAs were consulted (a total of nine). States and SFAs were selected to represent the diverse scale and structure of data management systems known to exist across the country. The consultations comprised unstructured discussions and reviews of data dictionaries of child nutrition MIS. Through the interviews, the research team gained insights into the history and types of MIS used by States and SFAs; terminologies relating to child nutrition MIS; respondents’ knowledge of their systems; costs and sources of funds for acquisition and operation of MIS; industry trends; current MIS functions; available data elements and reporting activities; database structures and software architecture; and maintenance and management of existing systems.

Cognitive Pre-testing. The survey instruments were cognitively tested using a sample of nine participants, including both State and SFA representatives. The pretests were based on cognitive interviewing techniques, which test the survey content and ensure that instructions and wording of questions are clear and response options are adequate. The cognitive interviews also assessed the validity of the questions. The feedback from the cognitive tests were used to refine the survey instruments. As a result, the revised instruments are clearer, more easily understood, and more consistent in meaning. See Attachment D.2 for detailed information concerning the cognitive pre-test, and for changes resulting from the cognitive pretests.

B.5. Consultants

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

FNS has contracted with IMPAQ International, LLC and its subcontractor Kokopelli Associates to collect and analyze the information. Frederic Glantz of Kokopelli Associates developed the sampling procedures. Table B.2 identifies the individuals at these organizations who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation, Dennis Ranalli, will be responsible for receiving and approving all contract deliverables. His contact information is also included below. In addition, FNS consulted with Chunlin Dong from National Agricultural Statistics Service (NASS) for expert consultation about the availability of data, the design, level of burden, and clarity of instructions for this collection.

Table B.2: Individuals Responsible for Statistical Aspects and Data Collection and Analysis

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Madeleine Levin	Senior Research Associate (Data collection and analysis)	IMPAQ International 1101 Vermont Avenue, NW, 11th Floor Washington, DC 20005	(202) 774-1982
Frederic Glantz	President (Data collection and analysis)	Kokopelli Associates, LLC 365 Camino Sin Salida Santa Fe, NM 87501	(505) 983-0785
Ann Middleton	Senior Research Analyst (Data collection and analysis)	IMPAQ International, LLC 1333 Broadway, Suite 300 Oakland, CA 94612	(510) 597-2425
Samuel Kofi Ampaabeng	Research Associate (Data collection and analysis)	IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21044	(443) 259-5193
Mousumi Sarkar	Research Associate (Data collection and analysis)	IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21044	(443) 259-5195
Dennis Ranalli	Social Science Policy Analyst and COR	Office of Policy Support SNP Research and Analysis Division 3101 Park Center Drive Alexandria, VA 22302	(703) 305-2559
Chunlin Dong	Mathematical Statistician (Methods Division)	National Agricultural Statistics Service/USDA 1400 Independence Ave, SW Washington, DC 20250	(202) 720-8951

1 This list excludes American Samoa, Guam, and Mariana Islands (Western), Puerto Rico, Virgin Islands (Mid-Atlantic) because the focus of the study is on the 50 States and the District of Columbia.