Procurements Part B OMB Package Revised Final 11292018

Supporting Justification for OMB Clearance for the Study of School Food Authority (SFA) Procurement Practices

OMB Control Number 0584-NEW

Part B

November 29, 2018

Office of Policy Support
Food and Nutrition Service
United States Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302

Project Officer: Ashley Chaifetz
Telephone: 703-457-7741
Email: [email protected]

Table of Contents—Part B: Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods. 1

B.2 Procedures for the Collection of Information 6

B.3 Methods to Maximize the Response Rates and to Deal with Non-Response 17

B.4 Test of Procedures or Methods to Be Undertaken. 20

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data. 22

List of Tables

Table B1 Respondent Universe and Sample Sizes 4

Table B2 Statistical Precision and Minimum Detectable Differences for the Web Survey Sample 11

Table B3 Overview of Data Collection Activities 13

Table B4 SFAs Selected for Pretesting 21

Table B5 Individuals Consulted on Design, Data Collection, or Analysis 22

List of Attachments

Appendix A1 Email Notification from FNS to Regional Offices

Appendix A2 Email Notification from Regional Offices to State Child Nutrition Directors

Appendix A3 Email Notification to State Child Nutrition Directors

Appendix A4 Email Notification to School Food Authority Directors

Appendix B1 Study of School Food Authority (SFA) Procurement Practices: Frequently Asked Questions – Web Survey and In-Depth Interview

Appendix B2 Study of School Food Authority (SFA) Procurement Practices: Frequently Asked Questions – In-Depth Interview

Appendix C1.a Pre-Survey Notification Letter (Web Survey)

Appendix C1.b Survey Notification Email (Web Survey with Link)

Appendix C1.c Survey Reminder Email 1 (Web Survey with Link)

Appendix C1.d Survey Reminder Email 2 (Web Survey with Link)

Appendix C1.e SFA Director Telephone Reminder Script (Web survey)

Appendix C2.a Post-Survey Response Clarification Email (Web Survey)

Appendix C2.b Post-Survey Response Clarification Phone Call Script (Web Survey)

Appendix C3 Post-Survey Thank You Email (Web Survey)

Appendix C4.a Pre-Interview Notification Letter (In-Depth Interview)

Appendix C4.b Pre-Interview Scheduling Phone Call Script (In-Depth Interview)

Appendix C4.c Pre-Interview Reminder Email (In-Depth Interview)

Appendix C4.d Participant Confirmation Email (In-Depth Interview)

Appendix C5.a Post-Interview Response Clarification Email (In-Depth Interview)

Appendix C5.b Post-Interview Response Clarification Phone Call Script (In-Depth Interview)

Appendix D1 SFA Procurement Practices Web Survey

Appendix D1.a SFA Procurement Practices Web Survey (Web Version)

Appendix D2 SFA Procurement Practices In-Depth Interview Guide

Appendix E Thank You Email for Participation in Study

Appendix F1.a Federal Register 60-Day Notice Public Comment 1

Appendix F1.b Federal Register 60-Day Notice Public Comment 2

Appendix F1.c Federal Register 60-Day Notice Public Comment 3

Appendix F1.d Federal Register 60-Day Notice Public Comment 4

Appendix F1.e Federal Register 60-Day Notice Public Comment 5

Appendix F1.f Federal Register 60-Day Notice Public Comment 6

Appendix F1.g Federal Register 60-Day Notice Public Comment 7

Appendix F1.h Federal Register 60-Day Notice Public Comment 8

Appendix F1.i Federal Register 60-Day Notice Public Comment 9

Appendix F1.j Federal Register 60-Day Notice Public Comment 10

Appendix F1.k Federal Register 60-Day Notice Public Comment 11

Appendix F2.a Federal Register 60-Day Notice FNS' Response to Public Comment 1

Appendix F2.b Federal Register 60-Day Notice FNS' Response to Public Comment 2

Appendix F2.c Federal Register 60-Day Notice FNS' Response to Public Comment 3

Appendix F2.d Federal Register 60-Day Notice FNS' Response to Public Comment 4

Appendix F2.e Federal Register 60-Day Notice FNS' Response to Public Comment 5

Appendix F2.f Federal Register 60-Day Notice FNS' Response to Public Comment 6

Appendix F2.g Federal Register 60-Day Notice FNS' Response to Public Comment 7

Appendix F2.h Federal Register 60-Day Notice FNS' Response to Public Comment 8

Appendix F2.i Federal Register 60-Day Notice FNS' Response to Public Comment 9

Appendix F2.j Federal Register 60-Day Notice FNS' Response to Public Comment 10

Appendix F2.k Federal Register 60-Day Notice FNS' Response to Public Comment 11

Appendix G1 Estimates of Respondent Burden

Appendix G2 National Agricultural Statistics Service (NASS) Comments

Appendix G3 Response to National Agricultural Statistics Service (NASS) Comments

Appendix H1 Healthy Hunger-Free Kids Act Section 305

Appendix H2 Child Nutrition Act of 1966

Appendix H3 Richard B. Russell National School Lunch Act Amended Through February 2014 Section 28

Appendix H4      Richard B. Russell NSLP Act Amended through February 2014 Section 12

Appendix H5      7 CFR 210.21

Appendix H6      7 CFR 220.16

Appendix H7      7 CFR 225.17

Appendix H8      7 CFR 226.22

Appendix H9      2 CFR 200

Appendix I Procurement Practices – Final Memorandum on Pretest Results

Appendix J Data Collector Confidentiality Agreement

Appendix K Process of Data Collection

COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

This study involves two data collection components. First, a subsample of the Child Nutrition Program Operations Study II Year 2 (CN-OPS-II Year 2) (OMB Control Number 0584-0607, expiration date 07/31/2020) respondents will complete the School Food Authority (SFA) Procurement Practices Web Survey (web survey) (Appendix D1). Second, the SFA Procurement Practices In-Depth Interviews (IDIs) (Appendix D2) will be conducted with a subset of SFAs that complete the web survey. Described in more detail in section B.2, the sample for the web survey will be selected based on responses to the CN-OPS-II Year 2 survey procurement module and appropriately weighted to result in nationally representative estimates of SFAs that the completed the CN-OPS-II Year 2 procurement module.

As detailed in section B.2, the Study Team will explicitly stratify the sample of SFAs that completed the procurement module in the CN-OPS-II Year 2 survey into approximately eight strata (referred to as “models”), including four major models and four minor models.¹ The models were defined by two factors. The first factor is the model respondent size: major models consist of at least 15 percent of respondents and minor models consist of between 5 and 15 percent of respondents. The second factor is the classification of the respondents to dimensions that define the procurement process: Contracting, Suppliers, Management, Decision Makers, and State Monitoring.

The procurement models will be developed from the responses to the procurement practices module of the CN-OPS-II Year 2 SFA questionnaire. Questions from this module will be mapped to five dimensions that define procurement practices: Contracting, Suppliers, Management, Decision Makers, and State Monitoring. Respondents to the mapped questions will be categorized within each dimension to subgroups deemed important for analysis by subject matter experts. Because of the high degree of variability in classification across the dimensions, cluster analysis will be used to group respondents into initial models by their dimensions categories.² Classification and regression tree (CART) analysis will be used to identify potential models that can be collapsed into other models. The final models will be created after the models identified for collapsing in the CART analysis are collapsed into the other appropriate models.

Each stratum will represent a set of similar procurement methods (e.g., use of Food Service Management Companies [FSMCs] or Cooperative Purchasing Agreements [CPAs]) and policies (e.g., extent of procurement of local foods) identified from the CN-OPS-II Year 2 responses. The Study Team will also implicitly stratify (i.e., sort) the SFAs that completed the CN-OPS-II Year 2 survey procurement module by characteristics such as SFA size and urbanicity within the explicit strata.

The respondent universe and sample sizes for the web survey and IDIs are described in Table B1. The expected response rate for the entire collection is estimated at 80 percent. The approximate respondent universe of SFAs for this study consists of the 1,750 public SFAs expected to participate in the CN-OPS-II Year 2 survey.³ These SFAs operate in public school districts in the United States and outlying territories, and were required to submit form FNS-742 SFA Verification Collection Report Summary Data (OMB Control Number 0584-0594 Food Programs Reporting System (FPRS), expiration date 6/30/2019) to FNS in school year (SY) 2014–15. The CN-OPS-II Study respondent universe for all four years includes all SFAs that participated in the National School Lunch Program (NSLP) or the School Breakfast Program (SBP), with the following exceptions:

• SFAs that operate only in Residential Child Care Institutions that do not have daytime students

• SFAs that do not have students who are eligible for free/reduced-price (F/RP) lunch

• SFAs in some outlying territories that are not required to complete form FNS-742

• Private schools that participate in NSLP

Table B1. Respondent Universe and Sample Sizes^a

Instrument	Respondent Category	Strata	Respondent Universe	Initial Sample	Expected Response Rate	Target Completed Casesb
SFA Procurement Practices Web Survey (Appendix D1)	SFA directors (subsample of CN-OPS-II Year 2 respondents)	Major Model 1	263	125	80%	100
		Major Model 2	263	125	80%	100
		Major Model 3	262	125	80%	100
		Major Model 4	262	125	80%	100
		Minor Model 1	175	50	80%	40
		Minor Model 2	175	50	80%	40
		Minor Model 3	175	50	80%	40
		Minor Model 4	175	50	80%	40
		Total	1750	700	80%	560
SFA Procurement Practices In-Depth Interview Guide (Appendix D2)	SFA directors (subsample of SFA Procurement Practices Web Survey respondents)	Major Model 1	100	25	80%	20
		Major Model 2	100	25	80%	20
		Major Model 3	100	25	80%	20
		Major Model 4	100	25	80%	20
		Minor Model 1	40	7	80%	5
		Minor Model 2	40	7	80%	5
		Minor Model 3	40	6	80%	5
		Minor Model 4	40	6	80%	5
		Total	560	125	80%	100
Total Collection				825	80%	660

^a It is estimated that 1,750 SFAs will complete the CN-OPS-II Year 2 (OMB Control Number 0584-0607, expiration date 07/31/2020) procurement module. It is further estimated that each major model will be assigned 15 percent of these responding SFAs and each minor model will be assigned 10 percent of these responding SFAs, with each responding SFA being assigned to one and only one model. The exact number of major and minor procurement models created, and the exact number of SFAs assigned to each model, will depend on the procurement module data collected for CN-OPS-II Year 2.

^b The targeted completed cases are rounded to account for complete respondents.

Based on the Study Team’s prior experience with SFA surveys conducted for other studies, web survey and IDI response rates of 80 percent are expected and are required by the Office of Management and Budget (OMB). Additionally, by design, SFAs sampled for this study will have completed, approximately nine months earlier, the CN-OPS-II Year 2 survey; therefore, the sample is expected to include respondents who are likely to agree to the web survey and IDI because they have already shown a propensity to participate in FNS studies.⁴ It is important to note that this approach minimizes burden because the Study Team is utilizing the responses to the CN-OPS-II Year 2 study to inform which SFAs to select for the current study. The SFAs will not be asked to answer similar questions on the web survey, which reduces the overall burden placed on SFAs, yet still answers the current study’s research questions. Given the expected response rates of 80 percent, the Study Team will select enough SFAs overall (n = 700) to achieve 560 total completed web surveys (700 x .80 = 560). More specifically, the Study Team will select enough SFAs within each of the four major procurement model strata (n =125) to achieve 100 completed web surveys (125 x .80 = 100), and enough SFAs within each of the four minor procurement model strata (n = 50) to achieve 40 completed surveys (50 x .80 = 40).⁵

To ensure an 80 percent web survey response rate, the Study Team will follow a multistep process, beginning with notification of the study through well-established Food and Nutrition Service (FNS) communication channels, and then utilizing a user-friendly web interface to the survey, providing email and telephone support, and email and telephone reminders. Additionally, the web survey allows respondents to save and exit at any time and then return to complete the survey later. To maximize IDI response rates, respondents will be recruited to participate within four weeks of completing the web survey and will be provided with reminders and support by mail, telephone, and email to continue their engagement with the study. Additionally, the IDI will be scheduled at a time that is most convenient for the respondent.

In the event that an 80 percent response rate is not reached during the data collection period, the Study Team may extend the proposed data collection period. If the final web survey response rate drops below 80 percent, a nonresponse bias analysis will be conducted, and weighting adjustments to correct for potential biases will be performed.

For the IDI, a total of 125 SFAs that responded to the web survey will be purposefully selected to obtain 100 completed IDIs, given an 80 percent response rate. The Study Team anticipates 20 SFAs from each of the four major procurement models (a total of 80 SFAs), and five SFAs from each of the approximately four minor models (a total of 20 SFAs) will be interviewed. The IDIs will provide greater depth of information about procurement from a smaller sample of SFAs to supplement the web survey responses. As with the web survey, the Study Team may extend the time to conduct the IDIs and increase outreach efforts in the event that an 80 percent response rate is not reached during the data collection period. As noted above, the sample is expected to include respondents who are likely to agree to the IDI because they have already shown a propensity to participate in the study.

B.2 Describe the procedures for the collection of information including:

Statistical methodology for stratification and sample selection

The SFA is the unit of analysis for CN-OPS-II Year 2 (OMB Control Number 0584-0607, expiration date 07/31/2020), and likewise, the web survey and the IDIs. To ensure adequate representation of SFAs across multiple dimensions of procurement, the sampling frame for the web survey and the IDIs will be composed of all SFAs that responded to the CN-OPS-II Year 2 survey and completed the procurement module in that survey and, hence, can be placed into one of the procurement models.

Sample design for the web survey. SFAs will be selected from the sampling frame using a stratified sampling design with equal probability sampling of the SFAs within each stratum (i.e., model), where strata are defined by the procurement models; a set of eight models, consisting of four major and four minor procurement strategies, is anticipated. These procurement models will be created using cluster analysis. Stratification not only helps to ensure that adequate sample sizes are obtained for important analytic subgroups of interest (e.g., groupings by SFA size), but can also be effective in reducing the sampling errors of estimates that are correlated with SFA characteristics such as enrollment size.

The Study Team will also implicitly stratify by other characteristics such as SFA size and urbanicity. Within each stratum, SFAs will be randomly sorted and sequentially selected for implicit stratification. Sampling will be conducted so that the sample sizes will be large enough to meet the target number of completes given an expected response rate of 80 percent; random sorting allows the sample to be released in stages or waves, as needed, ensuring that the final released sample will be a random, representative sample of the full sample selected.

Table B1 above outlines the proposed sample design for the web survey, subject to the procurement models that will be created once the CN-OPS-II Year 2 data are available (as described above), and pending information on the associated number of SFAs assigned to each model from the CN-OPS-II Year 2 respondents. Similarly, the estimated population or universe counts by strata will be drawn from the CN-OPS-II Year 2 survey results and sampling frame data.

Sample design for the IDI. The sampling frame for the IDIs will consist of all SFAs that completed the web survey, and will be stratified by each procurement model as was done for the web survey, along with a set of factors that describe the characteristics of the SFAs in each model as identified in the web survey.⁶ Such characteristics will be identified on a flow basis using preliminary analysis of the web survey data as they are collected. As an example of one potential factor, many SFAs may use a major competitive model among producer cooperatives for which the SFA director is the primary decision maker with State or local oversight; however, among SFA users of such a model, variances may exist in the types of producer cooperatives they use as reported on the web survey. FNS may wish to discuss such variances with the Study Team during the IDIs, so such characteristics will be incorporated into the stratification of the IDI sample.

Estimation procedure

For estimation purposes, web survey sampling weights reflecting the overall probabilities of selection and differential nonresponse rates will be attached to each data record providing usable SFA data. Web survey weights will be created at the SFA level to account for the stratified sampling design and sample release and to adjust for survey nonresponse. The weights will also be calibrated to the population of CN-OPS-II Year 2 respondents that completed the procurement module in that survey so they can be used to produce nationally representative estimates. The Study Team will also check the final weights for outliers and trim as needed to ensure that no single SFA has too much influence on weight-based estimates. For example, larger and smaller SFAs within the same stratum will receive approximately the same weight, pending nonresponse and other adjustments.

The first step in the weighting process will be to assign a base weight to each sampled SFA. The base weight is equal to the reciprocal of the probability of selecting the SFA for the study, which will vary by sampling stratum under the proposed stratified sample design. Next, the base weights will be adjusted for nonresponse within cells consisting of SFAs that are expected to be homogeneous with respect to response propensity. To determine the appropriate adjustment cells, we will conduct a nonresponse bias analysis to identify characteristics of SFAs that are correlated with nonresponse. Within these cells, a weighted response rate will be computed and applied to the SFA base weights to obtain the corresponding nonresponse-adjusted weights.

Jackknife replication will be used to calculate the standard errors of the survey-based estimates in order to account for the complex features of the sample design. Using jackknife replication, subsamples or "replicates" will be created to preserve the basic features of the full sample design. A set of weights (i.e., “replicate weights”) will then be constructed for each jackknife replicate. Using the full sample weights and the replicate weights, estimates of any survey statistic can be calculated for the full sample and for each of the jackknife replicates. The variability of the replicate estimates is used to obtain the variance of the survey statistic. The replicate weights can be imported into variance estimation software (i.e., SAS, STATA) to calculate standard errors of the survey-based estimates. In addition to the replicate weights, stratum and unit codes will be created to allow for the calculation of standard errors using Taylor series approximations if desired; however, although replication and Taylor series methods generally produce similar results, jackknife replication has some advantages in reflecting statistical adjustments used in weighting, such as nonresponse, weight trimming, and post-stratification.⁷

Degree of accuracy needed for the purpose described in the justification

Web survey. Table B3 provides expected precision and minimum detectable differences (MDDs) for the web survey sample. The expected precision (confidence interval) and MMD calculations apply to the sampling frame, all SFAs that responded to the CN-OPS-II Year 2 survey and completed the procurement module in that survey. The sample sizes were selected to balance multiple objectives, including minimizing the burden some SFAs could face if asked to complete the CN-OPS-II Year 2 survey, the web survey, and IDI; conducting the survey within project resources; and providing a sufficient level of statistical precision to detect meaningful differences between model types, and for the SFA population as a whole, in terms of their procurement practices. The final expected sample of 560 completed web surveys will yield an overall level of statistical precision of plus or minus 4.3 percentage points, for a 95 percent confidence interval under conservative design assumptions.⁸

For any major procurement model, the completed sample will yield an overall level of statistical precision of plus or minus 10.3 percentage points, for a 95 percent two-tailed confidence interval. When comparing two of the major procurement models, the Study Team expects an MDD of 15.0 percentage points, suggesting that any estimated differences of 15.0 percent or larger will be statistically significant at the .05 level.⁹ The MDD for comparisons between two minor procurement models is estimated to be 24.0 percentage points. For any of the minor procurement models, the statistical precision is expected to be lower since smaller samples will be selected from the minor procurement model groups; however, overall, the precision levels should still be sufficient to obtain meaningful estimates and explore differences across and within major and minor procurement models. Moreover, the Study Team will also explore the results from the web survey qualitatively—focusing on what the data from the IDIs reveal. It should be noted that if a smaller number of minor models is identified, the Study Team will enlarge the sample sizes for the major models to provide a greater degree of precision for each major model type.

Table B2. Statistical Precision and Minimum Detectable Differences for the Web Survey Sample

Sample	Precision (95% Confidence Interval Half Width)	Minimum Detectable Difference (Comparing Two Procurement Models)
Overall	4.3	NA
Major Procurement Model	10.3	15.0
Minor Procurement Model	16.3	24.0
Table Note: Calculations assume a binary outcome with 50 percent responding in the affirmative (most conservative assumption) and a design effect due to the weighting of 1.1 (also a conservative assumption) to account for unequal weighting due to the stratified design and additional weighting adjustments. For a binary outcome with less than or greater than 50 percent responding in the affirmative, 95 percent confidence intervals and MDDs will likely be smaller.

In-Depth Interview. As noted above, web survey data will be used to identify which SFAs assigned to a given procurement model type have various characteristics for which further information about SFA practices is desired. Such characteristics will include the extent to which the SFA uses these procurement methods and the motivating factors and circumstances that influenced their choice. Once these factors are identified, the Study Team will stratify the IDI sampling frame and select a sample of SFAs in each stratum for IDI recruitment. While our objective is to follow a random process for the selecting of SFAs from each stratum, some selections may be purposive in nature, based on other considerations such as an SFA’s existing relationship with FNS or if there is a desire to explore, in more detail, SFAs with certain characteristics based on answers to specific questions on the web survey.¹⁰ In addition, we plan to create the sampling frame and conduct the sampling of the SFAs for IDIs on a flow basis while the web survey is being conducted. The creation of the sampling frame and the sampling will occur in batches, so SFAs will differ in their chance of selection for IDI participation. Overall, this process will ensure IDIs are conducted on SFAs with a range of procurement practices, including those that are of particular interest to FNS, while allowing the survey to achieve the sampling objectives. The sample selection process will also take into account demographic and geographic characteristics to ensure a representative sample across these domains. For example, the Study Team will ensure that SFAs from urban, suburban, and rural areas are included in the IDIs.

A sample of SFAs will be selected in each of the IDI strata during the flow of the selection process, and we will work the SFAs in random order, making replacements as needed to account for nonresponse.

Unusual problems requiring specialized sampling procedures

FNS does not anticipate any unusual problems requiring use of specialized sampling procedures.

Any use of periodic (less frequent than annual) data collection cycles to reduce burden

This is a one-time study; concern regarding the periodicity of data collection cycles is not applicable.

General data collection procedures

Table B2 summarizes the data collection plan. Both the web survey (Appendix D1) and the IDI (Appendix D2) will be administered with a subset of respondents (SFA directors) from the CN-OPS-II Year 2 sample.

Table B3. Overview of Data Collection Activities

Instrument	Respondent Category	Mode	Length	Frequency
SFA Procurement Practices Web Survey (Appendix D1)	SFA directors (subsample of CN-OPS-II Year 2 respondents)	Web	90 minutes	Once
SFA Procurement Practices In-Depth Interview Guide (Appendix D2)	SFA directors (subsample of SFA Procurement Practices Web Survey respondents)	Telephone	90 minutes	Once

SFA Procurement Practices Web Survey (Appendix D1). First, FNS will notify its Regional Office liaisons about the study via the Email Notification from FNS to Regional Offices (Appendix A1). Then, FNS Regional Office liaisons will notify SAs about the web survey via the Email Notification from Regional Offices to State Child Nutrition Directors (Appendix A2).

The Study Team will follow up by contacting the SAs via the Email Notification to State Child Nutrition Directors (Appendix A3). Next, the SAs will notify the sampled SFAs about the web survey via the Email Notification to School Food Authority Directors (Appendix A4). After the SFAs have been notified of the study by the SAs, the Study Team will mail a study package out to the selected SFA directors. The package will include the Study of School Food Authority (SFA) Procurement Practices: Frequently Asked Questions – Web Survey and In-Depth Interview (Appendix B1) and a Pre-Survey Notification Letter (Web Survey) (Appendix C1.a) that will explain the purpose of the study and provide instructions on how to access the web survey. Both documents will provide contact information for the study’s toll-free help line and email help desk. Within one week after the study package is sent, the Study Team will send SFAs the Survey Notification Email (Web Survey with Link) (Appendix C1.b), which will contain information similar to the study package, and the SFA’s unique link to the web survey.

The Study Team will send two reminder emails to SFAs who have not completed their web survey. Survey Reminder Email 1 (Web Survey with Link) (Appendix C1.c) will be sent out about two weeks after the initial invitation email and Survey Reminder Email 2 (Web Survey with Link) (Appendix C1.d) will be sent out about four weeks after the initial invitation email. The reminder emails will include survey information, a link to the web survey, contact information for the help desk, and a reminder of the web survey due date.

Toward the end of the data collection period, the Study Team will also follow up with nonrespondents by telephone using the SFA Director Telephone Reminder Script (Web Survey) (Appendix C1.e) to encourage web survey completion, answer any questions they may have, and offer an opportunity to complete the web survey over the telephone. If an SFA director elects to complete the web survey via telephone, a trained member of the Study Team will use the SFA Director Telephone Reminder Script (Web Survey) (Appendix C1.e) to greet the respondent and walk them through the web survey (Appendix D1), recording their survey responses using the SFA’s unique survey link.

Web survey responses will be reviewed as they are received. If additional clarifying information is needed, the Study Team will email respondents requesting clarification on particular questions using the Post-Survey Response Clarification Email (Web Survey) (Appendix C2.a). Should additional clarification be needed after receipt of a response to a clarification email, or if an SFA director does not respond to the email, a member of the Study Team will call the SFA director using the Post-Survey Response Clarification Phone Call Script (Web Survey) (Appendix C2.b) to obtain clarifying information. It is particularly important to ensure that all data collected on the web survey is understood, as this will inform the selection of the SFAs for participation in the IDIs. At the end of the web survey data collection, a Post-Survey Thank You Email (Web Survey) (Appendix C3) will be emailed to all participating SFAs. FNS expects a total of 560 SFAs to complete the web survey, an 80 percent response rate.

In-Depth Interviews. IDIs will be conducted with a total of 100 SFAs that participate in the SFA Procurement Practices Web Survey (Appendix D1). Selection of SFAs for the IDIs will be based on models of procurement practices to be developed from responses to the CN-OPS-II Year 2 survey. It is anticipated that the models will be based on elements of SFA procurement practices (e.g., use of an FSMC) and SFA characteristics (e.g., geographic region, size). SFAs will be recruited to participate in the IDI on a rolling basis. Within four weeks of an SFA’s completion of the web survey, a Pre-Interview Notification Letter (In-Depth Interview) (Appendix C4.a) will be emailed to the 125 SFAs that have been selected for an IDI. The Pre-Interview Notification Letter (In-Depth Interview) (Appendix C4.a) will be addressed to the SFA director, who will be asked either to serve as a point of contact for scheduling the interview, or to identify another person for this task. Next, one week after emailing the notification letter to selected SFAs, a trained interviewer from the Study Team will begin contacting SFA directors using the Pre-Interview Scheduling Phone Call Script (In-Depth Interview) (Appendix C4.b) to confirm receipt of the letter, answer any questions, and schedule an interview. About two weeks after emailing the initial notification letter, the interviewer will send a Pre-Interview Reminder Email (In-Depth Interview) (Appendix C4.c) to those SFA directors who have not scheduled their IDI. For the remaining SFA directors who have not scheduled their IDI, a second telephone call will be made using the Pre-Interview Scheduling Phone Call Script (In-Depth Interview) (Appendix C4.b) to once again attempt to schedule an interview. Within one week of the scheduled interview, the interviewer will send the Participant Confirmation Email (In-Depth Interview) (Appendix C4.d) to the SFA director to confirm the interview date and time.

IDIs will be conducted over the telephone using the semi-structured IDI (Appendix D2), and, with the permission of the respondent(s), will be recorded using a conference call interface with recording capability such as WebEx. The interviews are expected to average 90 minutes. If answers remain unclear after the IDI, the Study Team will follow up with respondents using the Post-Survey Response Clarification Email (In-Depth Interview) (Appendix C5.a). The Study Team will use the Post-Survey Response Clarification Phone Call Script (In-Depth Interview) (Appendix C5.b) for respondents that do not reply to the email. At the conclusion of the entire data collection, the Thank You Email for Participation in the Study (Appendix E) will be sent to all SFAs that participated in the IDI (approximately 100 SFAs, assuming an 80 percent response rate).

B.3 Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Overall response rate projections were presented earlier in Table B1. The Study Team estimates that 80 percent of the sampled SFAs will complete the web survey, and 80 percent of SFAs who are invited to participate in the IDI will complete the IDI when the Study Team uses the procedures detailed below, which have resulted in successfully achieving an 80 percent response rate with SFAs in previous surveys.

The Study Team will follow the procedures listed below in order to maximize response rates and address issues of nonresponse for the web survey:

• The letters to invite SFA directors to participate will emphasize the importance of this study and how the information will help FNS to better understand and address current policy issues related to Child Nutrition Program (CNP) operations.

• Current contact information will be used for all initial correspondence and updated as needed throughout the data collection period to facilitate communication between the SFA and the Study Team.

• Designated FNS Regional Office staff will serve as regional study liaisons and be kept closely informed of data collection progress so that they will be able to answer questions from SFAs and States and encourage participation.

• A toll-free number and study email address will be provided so that SFA directors can receive assistance with the web survey.

• The Study Team will send up to two email reminders (Appendix C1.c and C1.d) to SFA directors who have not yet completed the web survey.

• The Study Team will follow up once by telephone using the SFA Director Telephone Reminder Script (Web Survey) (Appendix C1.e) with all sampled SFA directors who do not complete the web survey within a specified period and urge them to complete it. At that point, if the SFA directors prefer to complete the web survey or remaining sections of the web survey over the telephone, a telephone interviewer will administer the web survey (Appendix D1) or remaining parts of the web survey over the telephone.

The following procedures will be used to maximize the completion rates and minimize nonresponse for the telephone-administered IDI:

• Conduct an interviewer training specific to this study to gain familiarity with interviewing and recruiting procedures, the interview questions and probes, and handling anticipated recruitment challenges such as working with/around gatekeepers to schedule the interviews

• Use a core group of trained interviewers with knowledge of SFA procurement practices, vocabulary, and processes, as well as experience with obtaining responses from SFA directors

• Use a core group of trained interviewers experienced with gaining cooperation from SFA directors, based on data collection in other FNS studies

• Use a core group of trained interviewers experienced with telephone interviews, particularly interviewers who have proven their ability to obtain cooperation from a high proportion of sample members

• Provide a toll-free number and email address for respondents to verify the study’s legitimacy or to ask other questions about the study

• Send respondents a Pre-Interview Notification Letter (In-Depth Interview) (Appendix C4.a), make up to two pre-interview scheduling phone calls (using Pre-Interview Scheduling Phone Call Script (In-Depth Interview), Appendix C4.b), and send respondents one Pre-Interview Reminder Email (In-Depth Interview) (Appendix C4.c) and one Participant Confirmation Email (In-Depth Interview) (Appendix C4.d).

If the final web survey response rate drops below 80 percent, a nonresponse bias analysis will be conducted, and weighting adjustments to correct for potential biases will be performed.

B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The web survey and IDI were pretested in May–August 2017. The Procurement Practices – Final Memorandum on Pretest Results can be found in Appendix I. The pretest instruments were fully tested. Pretest participants were debriefed by telephone and provided opportunities for general comments about both the web survey and the IDI instruments. Nine SFA directors from three States participated in the pretest.

Given that the Special Nutrition Program Operations Study Year 3 (SN-OPS Year 3) (OMB Control Number 0584-0562 Special Nutrition Program Operations Study (SN-OPS), discontinued date 04/30/2016) included questions about procurement, the Study Team chose a subset of SFAs, based on their SN-OPS Year 3 responses, to pretest the web survey and IDI instruments. The three pretest States came from FNS’s Southeast Region (SERO) because of ease of coordination, and because there were no expected differences across regions regarding vocabulary or procedure. From this region, the Study Team sampled from SFAs that indicated they either use an FSMC (SN-OPS Year 3 Question 4.1a = yes) or have a CPA (SN-OPS Year 3 Question 4.1b = yes) to manage their procurement of USDA Foods or commercial products. The list was further narrowed to only those SFAs that responded to SN-OPS Year 3 Question 4.3, which asked how the SFA monitors the execution of contracts or cooperative agreements. From the remaining SFAs, the Study Team selected the three States in SERO that had the most SFAs remaining in the list—South Carolina (20); Kentucky (19); and Florida (17). The Study Team then selected 25 SFAs that varied across six characteristics relevant to the study: (1) use of an FSMC and/or CPA; (2) SFA size and locale; (3) F/RP lunch rates; (4) SFA monitoring practices to oversee contracts or CPAs; (5) the use of advisory councils; and (6) participation in Farm to School activities during SY 2012–13. These characteristics allowed the SFA directors to respond to all survey and interview questions, therefore giving the Study Team the ability to pretest all survey and interview questions.

Table B4 lists characteristics of the nine SFAs whose SFA directors participated in the pretest. Purposefully, the modal SFAs by size (e.g., medium) and locale (e.g., rural) were the same for both the SN-OPS Year 3 sampling frame and the group selected for pretesting.

Table B4. SFAs Selected for Pretesting

SFA Name	SFA Sizea	SFA Localeb	FSMCc	CPAd
SFA #1	Very large	Suburban	Yes	No
SFA #2	Very large	City	No	Yes
SFA #3	Medium	Rural	Yes	No
SFA #4	Medium	Town	Missing	Yes
SFA #5	Medium	Rural	Yes	Yes
SFA #6	Large	Town	No	Yes
SFA #7	Small	Rural	No	Yes
SFA #8	Very Large	Suburban	Yes	Missing
SFA #9	Large	Suburban	Yes	No
Note: a. SFA Size: small (1–999 students); medium (1,000–4,999 students); large (5,000–24,999 students); and very large (25,000 or more students). Murdoch, J., Campbell, A., Condon, E., Fox, M. K., Harrison, R., Miller, M. . . . Shen, Y. (2016). Special Nutrition Program Operations Study (SN-OPS) SY 2013–14 report. U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support. Retrieved from https://fns-prod.azureedge.net/sites/default/files/ops/SNOPSYr3.pdf b. SFA Locale: city, suburban, town, and rural. Based on the NCES urban-centric locale codes that assess proximity to a principal city and urbanized area. https://nces.ed.gov/ccd/rural_locales.asp c. Source: SN-OPS Year 3, Question 4.1a: Does your SFA use a management company or have a cooperative purchasing agreement to manage the procurement of USDA Foods or commercial products? Management company __Yes __No d. Source: SN-OPS Year 3, Question 4.1b: Does your SFA use a management company or have a cooperative purchasing agreement to manage the procurement of USDA Foods or commercial products? Cooperative purchasing agreement __Yes __No

Pretest respondents were contacted by U.S. mail, email, and telephone. For both the web survey and IDI instruments, pretest respondents were asked about ease of comprehension (e.g., confusing wording or layout, failure to grasp concepts) and length of time to complete. Recruitment materials were also pretested for comprehension. All recruitment materials and instruments were revised to incorporate pretest results, including clarifying statements and questions regarding diction, removing repetitive questions, and adding definitions of key terms throughout the survey. Additional information can be found in Procurement Practices – Final Memorandum on Pretest Results (Appendix I).

B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Table B5 presents a summary of individuals consulted on study design, data collection, and/or analysis.

Table B5. Individuals Consulted on Design, Data Collection, or Analysis

Name	Title	Telephone Number	Email
Primary Contractor – 2M Research
Steven Garasky	Project Director	817-856-0876	[email protected]
Nicholas Beyler	Statistical Quality Assurance	202-796-3955	[email protected]
Molly Matthews-Ewald	Senior Public Health Research Analyst	817-856-0875	[email protected]
Michael Jacobsen	Survey Statistician	703-214-1189	[email protected]
Morgan Miller	Research Analyst	202-770-2094	[email protected]
Moyo Kimathi	Research Analyst	817-856-0868	[email protected]
Subcontractor – Mathematica Policy Research
Charlotte Cabili	Researcher	202-238-3322	[email protected]
Liana Washburn	Nutrition Research Analyst	202-250-3551	[email protected]
Food and Nutrition Service Staff
Ashley Chaifetz	Social Science Research Analyst, Project COR	703-457-7741	[email protected]
National Agricultural Statistics Service Staff
Prakash Adhikari	Mathematical Statistician	202-720-5467	[email protected]

1 The exact number of major and minor procurement models created will depend on the procurement module data collected for CN-OPS-II Year 2. There may be more or fewer than four major models and four minor models. If this is the case, the web survey sample design will be modified accordingly while retaining the overall goal of sampling 700 SFAs to obtain 560 completed surveys.

2 A cross-tabulation of the five dimensions was conducted prior to the cluster analysis to see if any models could be identified. No major or minor models were identified in this step, thereby necessitating the use of cluster analysis.

3 The respondent universe for the CN-OPS-II Study consisted of the 14,854 public SFAs with day students that participated in the NSLP or the SBP and submitted form FNS-742 SFA Verification Collection Report Summary Data (OMB Control Number 0584-0594, expiration date 6/30/2019) for SY 2014–15. From those 14,854 SFAs, the CN-OPS-II Study sampled 9,939 SFAs to support all four years of data collection, with approximately 2,188 SFAs included in the sample each year. Of these, approximately 1,750 SFAs are expected to respond each year (2,188 x 80% response rate). The sample of 700 SFAs to be included in the SFA Procurement Practices Study will be drawn from the 1,750 SFAs expected to participate in Year 2 of CN-OPS-II (conducted fall 2017).

4 A nonresponse analysis was conducted for the CN-OPS-II Year 2 data collection to determine the potential bias from nonresponse uncovered differences between respondents and nonrespondents for urbanicity. However, the proposed study focuses on responses to the procurement section, which could not be determined from the CN-OPS-II Year 2 nonrespondents.

5 Four major models x 100 completed web surveys + four minor models x 40 completed surveys = 400+160 = 560 total completed web surveys.

6 By design, we are expecting 100 web survey completes for each major procurement model strata and 40 web survey completes for each minor procurement model strata. If we have fewer completes in any of the models, we can oversample from other models to achieve the expected 100 total IDIs.

7 See Rust, K. F., & Rao, J. N. K. (1996). Variance estimation for complex surveys using replication techniques. Statistical Methods in Medical Research, 5, 283–310.

8 The precision calculations assume a binary outcome of 50 percent and a design effect due to weighting of 1.1 to account for unequal weighting due to the stratified design and weighting adjustments for nonresponse and other factors. These are conservative assumptions since outcomes with less or greater than 50 percent response would likely yield smaller confidence intervals (i.e., greater precision), and the design effect, due to weighting, may be smaller or larger than 1.1, also yielding different confidence intervals.

9 This MDD is a product of the target sample size of 560 SFAs for the web surveys which has been set according to the budget and burden constraints of the study. One way to improve the power (and reduce the MDD) is to collapse across procurement models (i.e., have fewer models) so that each model has more SFAs. Whether this can be done will depend on the CN-OPS-II Year 2 procurement module data obtained.

10 For example, there may be particular interest in learning more about SFAs that use competitive contracting with no State monitoring, in which case one or more SFAs with those characteristics will be selected for the IDIs.