Appendix I Procurement Practices - Final Memorandum on Pretest Results

The purpose of this study is to describe and assess the practices of School Food Authorities (SFAs) related to procuring food for five Child Nutrition (CN) programs: National School Lunch Program (NSLP); School Breakfast Program (SBP); Fresh Fruit and Vegetable Program (FFVP); Summer Food Service Program (SFSP); and Child and Adult Care Food Program (CACFP), and to better understand how SFAs make decisions that lead to these procurement practices. Study findings will describe similarities and differences in how SFAs procure goods and services and elucidate key issues and experiences of SFAs.

The SFA Procurement Practices study goes beyond previous studies that concentrated on single food service or CN programs (e.g., NSLP, SBP, or SFSP) or studies that focused on single procurement practices (e.g., use of Food Service Management Companies [FSMCs]) at the SFA level. Using a nationally representative sample of SFAs, this study is one of the first U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) studies of SFA procurement practices for CN meal programs to comprehensively examine FSMCs, group purchasing entities, recordkeeping, local purchases, and food purchase specifications.

Data for this study will be collected from SFAs through the SFA Procurement Practices Web Survey (web survey) and the Study of SFA Procurement Practices In-Depth Interview (IDI) Guide. 2M Research Services (2M) and Mathematica Policy Research (Mathematica) (collectively referred to as the Study Team) will address the research objectives and answer the associated research questions by deploying a mixed-methods design. Mixed-methods research involves collecting, analyzing, and integrating (or mixing) quantitative (web survey) and qualitative (IDI) research (and data) in a single study. The advantage of this form of research is that the combination of both qualitative and quantitative research provides a better understanding of a research problem or issue than either research approach could provide alone.

The web survey is designed to be completed by an SFA prior to an IDI; both instruments are designed to inform each other and respond to the study research questions. The web survey is a structured tool that informs the IDI, which has a more flexible structure so that relevant information from the web survey can be incorporated.

This report summarizes key findings from pretesting the web survey and IDI data collection instruments, and recommended changes to these instruments.

Pretest Overview

During May–August 2017, the Study Team pretested the web survey (Appendix A) and IDI (Appendix B). The SFA Procurement Practices Web Survey Debriefing Interview Guide used in pretesting the web survey is provided as Appendix C, while interviewer instructions and script for the IDI were included in the pretested IDI protocol. The purpose of the pretest was to understand respondent burden, assess instrument flow, and identify questions and terms needing clarification. Prior to the pretest, the Study Team received feedback on both data collection instruments from FNS reviewers; this feedback was incorporated into the instruments prior to the pretest. The updated data collection instruments that incorporate the recommended changes based on the pretest findings will be provided separately as Deliverable 3.5a (Final Data Collection Instruments).

Selection of pretest respondents

Given Office of Management and Budget (OMB) restrictions for pretesting, only nine SFA directors pretested a printed version of the web survey and the IDI. The nine SFA directors were from three States. The three pretest States came from the FNS Southeast Regional Office (SERO) to reduce burden, since only one Regional Office would need to be contacted. From SERO, the Study Team determined the number of SFAs that indicated they either use an FSMC (Special Nutrition Program Operations Study–Year 3 [SN-OPS Year 3]) Question 4.1a = yes) or have a cooperative purchasing agreement (SN-OPS Year 3 Question 4.1b = yes) to manage their procurement of USDA Foods or commercial products. The list was further narrowed to only those SFAs that responded to SN-OPS Year 3 Question 4.3, which asked how the SFA oversees the execution of contracts or cooperative agreements. From these remaining SFAs, the Study Team selected the three States in SERO that had the most SFAs, which were South Carolina (20), Kentucky (19), and Florida (17). These States also provided variation in State monitoring (SN-OPS Year 3 Question C.9).

In these three States, the Study Team selected 25 SFAs that varied across six characteristics: (1) use of an FSMC and/or cooperative purchasing agreement; (2) SFA school size and locale; (3) free and reduced price (F/RP) lunch rates; (4) SFA monitoring practices to oversee contracts or cooperative purchasing agreements; (5) the use of advisory councils; and (6) participation in Farm to School activities during the 2012–2013 school year. The Study Team identified 9 SFAs as primary candidates for pretesting; 16 SFAs served as potential substitutes if any of the original 9 SFAs declined to participate in the pretest. The 25 selected SFAs were approved by the study’s Contracting Officer’s Representative (COR) on April 17, 2017.

Conducting the pretests

2M worked with FNS and State CN directors to reach out to the directors of the nine SFAs identified as primary candidates for pretesting, informing them of the SFA Procurement Practices study and need for their cooperation to pretest the printed version of the web survey and participate in the IDI pretest. After SFAs were informed of the study by State CN directors, 2M mailed pretest SFAs packages containing an introduction letter describing the study and a printed copy of the web survey. 2M followed this package with a reminder email 2 days later. 2M asked participants to complete and return the printed survey either by email or express delivery (keeping a copy for themselves so that they could review their responses and any notes they made during the debriefing).

After completing the printed survey, the SFA directors were scheduled to complete a survey debriefing that was conducted by 2M over the telephone. The debriefing interview was designed to elicit respondent feedback and focused on asking respondents to identify questions and sections that were unclear, recommend changes to the instruments, and estimate completion times to assess burden. At the end of the survey debriefing, 2M scheduled the IDI pretest and debriefing with the same SFA directors. IDI pretesting was conducted by Mathematica over the telephone. In preparation for the IDI pretesting, 2M provided Mathematica with each pretest respondent’s completed paper survey and 2M’s survey debriefing interview notes. Similar to the web survey debriefing, the IDI debriefing was designed to elicit respondent feedback and focused on identifying questions and sections that were unclear, recommending changes, and estimating completion times to assess burden.

The Study Team collected nine completed web surveys and completed eight web survey debriefings, eight IDIs, and eight IDI debriefings. The Study Team was unable to schedule a web survey debriefing, IDI, and IDI debriefing with a SFA director who completed the web survey.

Web Survey Pretest Results
1. Survey length

Prior to pretesting, the Study Team estimated that the printed survey would take 60 minutes to complete. 2M collected self-reported survey completion times from pretest participants during the web survey debriefing interview; these times are presented in Table 1. Survey completion times varied significantly across the pretest SFAs, ranging from 30 minutes to 4 hours. Factors contributing to longer completion times included the need for multiple staff members to complete the survey; skip logic not being applicable (i.e., more survey items were relevant to the respondent); difficulty comprehending some questions; and needing additional time to look up answers.

Table 1. Burden Estimates for Printed Version of Web Survey

SFA	Self-Reported Time to Complete	Time Used for Burden Calculation (Minutes)
SFA-1	Total time 30 minutes for hard copy but not really timed; not even an hour.	30
SFA-2	I did it between different things; an estimate would be an hour; I went through it quickly; some of it seemed so redundant.	60
SFA-3	Probably an hour or two because so much really didn’t relate to me; in a school, you don’t ever get time to sit and work on something.	90*
SFA-4	. . . possibly 1 to 2 hours.	90
SFA-5	Over the course of a week; generally, 30–40 minutes intervals. Approximately 2.5 hours.	150
SFA-6	. . . probably about 3 hours.	180
SFA-7	. . . 3 or 4 hours; closer to 4 hours.	210*
SFA-8	Total time 4 hours.	240
SFA-9	Did not participate in debriefing.	N/A
Average		131
Median		120
Minimum		30
Maximum		240

*Midpoint was used when respondent provided a time range.

Key findings and recommended instrument changes

Primary concerns with the web survey, expressed by respondents during the debriefing, included survey length (i.e., burden) and that some questions were confusing and unclear. Respondents identified specific sections of the web survey and specific questions during the debriefing that they thought could either be eliminated to reduce redundancy and length, or revised to improve clarity. The Study Team revised the web survey with this information in mind, while also ensuring that survey questions continued to coordinate with and inform the IDI, and that all research questions were addressed by either the web survey or the IDI.

Strategies to reduce burden. Table 2 summarizes recommended changes to the web survey instrument to reduce burden. First, to eliminate redundant questions, the Study Team identified web survey and IDI questions that collected the same information. Questions in both instruments that collected the same information, but did not inform the other instrument, were maintained in one instrument and removed from the other. Pretest respondents also noted that certain questions were difficult to differentiate and appeared to be redundant. To address this, similar questions that collected slightly different information were formatted into a matrix to better differentiate the information being requested.

Next, the Study Team reduced burden and redundancies by formatting questions into “Select all that apply” lists. For instance, Section IV, Question 2, included 18 sub-questions requiring a “Yes/No” response for each item. These sub-questions were restructured into a list with the question rephrased to “Select all that apply.” The Study Team also applied this strategy to best practices questions which asked SFAs if they currently have best practices for a variety of procurement processes. A new section, Section VI, was created for asking SFAs to identify current best practices in a single place in the survey. Questions asking about best practices desired by the SFA were removed from the web survey, as they were also asked in the IDI.

Table 2. Web Survey - Strategies to Reduce Burden

Section and Question Number from Pretest Web Survey*	Summary of Respondent Comments and Strategies to Reduce Burden	Web Survey Revisions
Sec. I, Q. 9; Sec. II, Qs. 9–10	Web survey questions repeated in the IDI	Questions were removed from the web survey and maintained in the IDI.
Sec. I, Qs. 13–14, 16–17, 19–34; Sec. II, Qs. 11–13, 18–19, 21–22; Sec. IV, Qs. 2, 5–6; Sec. V, Qs. 1–4, 6–7, 5–13	Web survey questions identified as particularly redundant or repetitive of other survey questions	Questions were restructured into matrices and “select all that apply” lists, as possible.
Sec. I, Qs. 13–14, 16–17, 19–20, 23–34; Sec. II, Qs. 21–22	Questions related to best practices, when asked at the end of each discussed topic, were perceived as repetitive and redundant.	Questions asking about the SFA’s current best practices were combined into one matrix at the end of the survey in new Section VI. Questions asking about best practices the SFA desired to have were removed. The IDI will ask about desired best practices.

* Question numbering reflects the pretest version of the web survey provided in Appendix A.

Revisions to improve clarity. Table 3 summarizes recommended changes to the web survey instrument to improve question clarity. Pretest respondents recommended several changes to the introduction, questions, and how definitions of terms are provided. In particular, the introduction was revised to further encourage SFA directors to get help from staff for questions for which they did not know how to respond. Also, response options were added or clarified, questions were reworded, and skip patterns were modified. Pretest respondents also identified technical terms for which they wanted definitions to help to improve their understanding of the survey questions. The revised web survey instrument now provides definitions at the first appearance of these terms and indicates that hover definitions will be programmed for each instance of a term to allow respondents to be reminded of the definition thereafter.

Table 3. Web Survey – Revisions to Improve Clarity

Section and Question Number from Pretest Web Survey *	Summary of Respondent Comments and Strategies to Reduce Burden	Web Survey Revisions
Introduction	For many questions, respondents indicated that they were not familiar with or did not work with procurement topics specified in the survey. However, respondents reported that other staff members did provide, or could have provided, information to complete the survey.	The introduction was updated to further encourage SFA directors to complete the survey with support from other staff members and offices.
Sec. I, Q. 4 Sec. IV, Qs. 9, 11 Sec. V, Qs. 3, 4, 6, 7, 15	Respondents suggested additional response options to many process-related questions.	Additional response options were incorporated in these survey questions.
Sec. III, Q. 7	Respondents indicated that FSMCs often manage all procurement services (e.g., sourcing local foods) and therefore they could not respond to this question.	The question was rephrased to better align with the research question it sought to address by also referencing other procurement agents aside from FSMCs. The skip pattern was also revised so that SFAs that do not work with an FSMC or other procurement agents will not be asked this question.
Sec. I, Q. 10 Sec. IV, Qs. 3, 7, 8. 11 Sec. V., Qs. 1–4, 6, 7, 8–13	Some respondents had difficulty comprehending these questions.	These questions were reworded for clarity.
	Respondents requested definitions for technical procurement terms. Technical terms for procurement processes may differ per SFA. Therefore, having definitions for terms like “local bonding” ensure SFAs understand the questions and response options.	The Study Team will provide full definitions of technical terms in their first appearance in the web survey. A hover definition will be provided if the term appears in future instances. For review purposes, 2M highlighted defined terms in the revised data collection instruments.

* Question numbering reflects the pretest version of the web survey provided in Appendix A.

In-Depth Interview Pretest Results
1. Respondent burden

The pretest version of the IDI protocol is shown in Appendix B. Two approaches were used during pretesting. In the first approach, three interviews were conducted to completion to obtain burden estimates. After these interviews were completed, retrospective probing was used to gather feedback about the instrument. The other five pretests were cognitive interviews, where neutral probing was used to gather feedback on question content and understandability throughout the interview while they were fresh in the respondent’s mind. Because this process lengthened the interviews, burden estimates were not collected for the five pretests that used cognitive interviewing.

Table 4 provides completion times for each section of the IDI, along with total completion times, based on the three pretest interviews conducted to completion. As Table 4 shows, Section VI (Contract Management and Recordkeeping) took the longest to complete, while Section VII (Challenges and Best Practices) took the least amount of time. Sections III through VIII of the IDI contained the instrument questions, while Section I contained interviewer instructions and Section II contained the introduction to the interview. One of the three pretest respondents used cooperative purchasing in their SFA and the other two respondents used FSMCs. The questions about FSMCs and cooperative purchasing were embedded within each section. Therefore, all respondents answered all sections, but not all questions in each section.

Table 4. Average minutes per section and total time for in-depth interview

			IDI Section
SFA	FSMC	Cooperative Purchasing	III Decision-making and Strategy	IV Contract Types	V Solicitation Process	VI Contract Management and Recordkeeping	VII Training and Resources	VIII Challenges and Best Practices	Total Time (minutes)
1	No	Yes	18	15	24	40	12	24	136
2	Yes	No	10	9	17	27	7	14	87
3	Yes	No	11	12	13	23	4	9	75
Mean			13	12	18	30	8	16	99

Note: Section I of the IDI protocol is Interviewer Preparation and is not part of the interview. Section II is an Introduction. The questions begin with Section III, which is why Sections I and II are not shown in this table. Total times listed include time for Section II, Introduction, which took about 3 minutes on average.

The target time for completing an IDI is 60–90 minutes. As Table 4 shows, the average length of a pretest interview was 99 minutes, slightly higher than the maximum target time for an interview. Burden should be reduced where possible for the following reasons:

The respondent’s knowledge of procurement was a primary driver of interview length. The highest burden estimate came from a respondent with deep knowledge about procurement, and that interview took 58 minutes longer to complete than the shortest interview. The Study Team wants to ensure enough time during data collection to collect rich data from the respondents who are highly knowledgeable.
The pretest IDI protocol and web survey had a fairly high level of overlap in content. Reducing this overlap would likely decrease respondent fatigue, and thus increase the quality of responses, particularly those closer to the end of the IDI. As Table 4 shows, the longest section of the IDI was Section VI, Contract Management and Recordkeeping. The other sections in the IDI contain between 7 and 14 questions, but Section VI has 34 questions, although not all questions in the section would have been relevant to all respondents. Respondents who use either an FSMC or cooperative purchasing would be asked a minimum of 22 questions in this section. After completing Section VI, several respondents said they were tired and appeared to have more trouble concentrating on the questions at hand. For example, in response to a question in Section VIII, the respondent reported being “zoned out,” and the question had to be repeated. At the end of the IDI, another respondent remarked that “my brain is tired” because the IDI was so long. At least one respondent began to answer questions more slowly, with more pauses, during Sections VII and VIII, than they had in previous sections of the IDI.

To reduce overlap between the web survey and IDI and decrease respondent burden, the Study Team removed questions from the IDI protocol that were covered in the web survey. The Study Team also compared responses on the web survey to the IDI responses for these questions to assess similarity. Table 5 shows the questions that were removed from the IDI, along with questions from the pretest version of the web survey that cover the same content. These questions tended to either duplicate responses from the web survey, were confusing as evidenced by respondents asking for clarification or saying they did not understand a question, or were answered with little detail, such as a yes/no response.

Table 5. Questions removed from the IDI protocol

Revision Number	Section and Question Number from Pretest IDI Protocol	Rationale
1	III.3	Covered in web survey questions IV.2 and IV.3. This question generally confused respondents and did not generate new information beyond what was covered in the web survey.
2	IV.1	Covered in Question I.8. Respondents gave the same answers to the ones given in the web survey.
3	IV.6	Covered in Question I.7. Respondents gave the same answers to the ones given in the web survey.
4	V.1	Covered in Question I.4. Respondents gave the same answers to the ones given in the web survey.
5	V.4	Covered in web survey Section V. Respondents gave the same answers to the ones given in the web survey.
6	V.5	Covered in web survey Section V. Respondents gave the same answers to the ones given in the web survey.
7	V.6	Covered in web survey Section V. Respondents gave the same answers to the ones given in the web survey.
8	V.11	Covered in web survey Section V. Respondents gave yes/no answers with little or no additional detail.
9	VI.1	Covered in Question II.7. Respondents gave the same answers to the ones given in the web survey.
11	VI.26	Covered in Questions I.21 and I.22. Respondents gave the same answers to the ones given in the web survey.
12	VI.29	Covered in Question IV.9. Respondents gave the same answers to the ones given in the web survey.
13	VI.30	Differences in tracking practices can be analyzed using subgroup analysis of Question IV.9. This was not applicable to any pretest respondents; all respondents either used an FSMC for of their procurement or did not use an FSMC for procurement.

Changes to IDI protocol and questions

Revisions to items in the IDI protocol are shown in Table 6. These revisions include (1) changing wording to make questions clearer or elicit more accurate responses; (2) making some questions more open-ended to elicit more detailed responses; and (3) revising sensitive questions to make them more approachable.

Table 6. Questions revised in the IDI protocol

Revision Number	Revision	Section and Question Number from Pretest Protocol	Rationale
1	Changed question wording from “What internal decision-making processes does your SFA use to develop the procurement strategy or plan?” to "What is the process for making procurement-related decisions?"	III.4	Original question confused respondents; it is now more open-ended and has follow-up questions to elicit more detailed answers.
2	Changed question wording from “Is purchasing from local sources a priority for your SFA’s overall procurement goals?” to "How does local purchasing fit in with your SFA’s overall procurement goals?"	III.5	Levels of responsiveness to this question varied; it is now rephrased it to be less direct and more open-ended.
3	Changed question wording from “What data or other information does your SFA use to inform the overall procurement strategy or plan? [Probe: For example, does your SFA use meal participation, point-of-sale, or other types of data to help develop big-picture priorities or guiding principles around procurement?]” to "What data or other information does your SFA use when making procurement decisions? [Probe: For example, does your SFA use meal participation, point-of-sale, or other types of data to help develop big-picture priorities or guiding principles around procurement?]"	III.9	The phrase "procurement strategy or plan" did not resonate with respondents.
4	Changed question wording from “Does your SFA complete a price or cost analysis before publishing [solicitation type]?” to “Does your SFA analyze prices or costs before publishing [solicitation type]?”	IV.5	In the pretest, respondents who said they did not complete a price or cost analysis went on to describe activities that involved comparing prices, doing background research, etc. Question wording now encourages respondents to think broadly about the concept of a price or cost analysis.
5	Changed "specifications" to "provisions."	IV.8	Specifications must be outlined for most, if not all, products. Reworded to clarify that the question is about when SFAs include contract provisions for small, minority- or woman-owned, or local businesses.
6	Changed "specifications" to "provisions."	IV.9	Specifications must be outlined for most, if not all, products. Reworded to clarify that the question is about when SFAs include contract provisions for small, minority- or woman-owned, or local businesses.
7	Changed question wording from “Did the benefits your SFA hoped for in contracting with the FSMC(s) come to fruition? [Probe: If so, please describe how you have observed these benefits in practice.]” to “Thinking back to when you first started working with the FSMC(s), did the benefits your SFA initially hoped for in contracting with them come to fruition? [Probe: If so, please describe how you have observed these benefits in practice.]”	VI.7	Reworded to prompt respondents to reflect on the past instead of confining their answer to the benefits they are getting in the present.
8	Revised question from “Who provides internal oversight for decision-making staff at each stage of the procurement process? [Probe: Who monitors the evaluation of bids, and contract award decisions? Who provides oversight for the staff responsible for planning, drafting specifications, advertising solicitations, awarding contracts, and managing contracts?]” to "What is the staffing structure in your SFA and who provides oversight for each staff member?"	VI.23	Sensitivity of topic made respondents to pretest less forthcoming with answers. Question folded in to introductory questions on staffing structure and language toned down to make the subject matter seem less sensitive.
9	Changed question wording from “How are accounting errors identified and rectified?” to "When an accounting error is identified, what is the process for rectifying it?"	VI.34	Pretest responses to this question lacked detail. Identification of accounting errors is covered in the web survey in Question IV.10; this question revised to focus on the process for rectifying accounting errors so respondents could focus on that aspect.
10	Moved this from the first question in this section to the third.	VII.1	Respondents were sensitive to being asked about the State's shortcomings; moved the question from being first in the section to promote openness once they become comfortable with section’s subject matter.
11	Changed wording from “What technical assistance is available?” to “What technical assistance is available on the State level?”	VII.2	Reworded to remind respondents to answer about State-level technical assistance.
12	Changed question wording from “What type(s) of guidance would be most beneficial, and on which topic(s)?” to “Does your SFA desire [best practices/additional best practices] on this topic?”	VIII.14	To reduce overlap and respondent burden, respondents will be asked whether they want best practices in the IDI instead of being asked in the web survey. To thoroughly cover this topic, interviewers will ask about each best practice specified in the web survey individually.

In addition to modifications designed to improve the quality of data collected during the IDI, the Study Team also implemented structural changes to improve the utility of the IDI protocol itself. These changes are intended to help interviewers navigate the IDI protocol more efficiently. These changes make the protocol simpler, more streamlined, and easier to navigate, while retaining the necessary content and level of detail. Table 7 lists the IDI protocol sections from the pretest version and the updated version, and Table 8 contains a detailed list of structural changes to the IDI protocol and supporting rationales, organized by protocol section and item number. Some of these changes are described in more detail below.

Table 7. IDI protocol sections

Pretest IDI Protocol Section Number	Pretest IDI Protocol Section Name	Updated IDI Protocol Section Name
I	Interviewer Instructions	Procurement Structure
II	Introduction	Procurement Approach
III	Procurement Decision-Making and Strategy	Solicitation Process
IV	Contract Types	Contract Types
V	Solicitation Process	Working with FSMCs
VI	Contract Management and Recordkeeping	Cooperative Purchasing
VII	SFA Training and Resources	SFA Training and Resources
VIII	Challenges and Best Practices	Challenges and Best Practices

A significant finding from the pretest IDI was that Section VI, Contract Management and Recordkeeping, was both difficult for the interviewer to navigate and fatiguing for respondents. In response, the Study Team divided Section VI into four discrete, smaller sections. In the revised IDI protocol, there is a stand-alone section for SFAs who use one or more FSMCs, and a stand-alone section for SFAs who participate in one or more group purchasing arrangements. The interviewer can now skip an entire section if it is not relevant to the respondent. Remaining questions from Section VI (27, 28, 32, 33, and 34) were placed into either an introductory section (Section I, Procurement Structure, in the revised IDI protocol), or a catchall section on procurement approach (Section II in the revised IDI protocol).

The pretest interviewer would have benefited from knowing more about the basic structure of the respondent’s SFA from the beginning of the IDI (instead of relying on what could be learned from reviewing the web survey responses). In addition, it sometimes took respondents a while to open up to the interviewer, and the data collected at the beginning of the IDI could therefore be less detailed than they would have been if the respondent were more at ease. To address these two pretest findings, the Study Team moved some questions that ask for basic information about the respondent’s SFA to the beginning of Section III and renamed the section “Procurement Structure” to accurately reflect the content. The Study Team moved the section on solicitation methods (previously Section V) ahead of the section on contracts (previously Section IV). The Study Team made this change because we found that respondents were confused when they were asked about solicitation methods after answering questions about contracts. The Study Team adjusted the placement of several interview questions, as shown in Table 8.

The Study Team also made minor changes to improve the functionality of the IDI protocol. These changes include removing introductory text in Section VIII that pretest respondents did not find helpful during pretesting, and moving question VIII.13 from Section VIII (Challenges and Best Practices) to Section VII on training, where it will fit better with other questions on a similar topic and will likely elicit more detailed responses. Finally, the Study Team added six questions to the protocol that will be used to confirm information the SFA provided in the web survey. These are questions I.1, II.1, II.4, III.1, IV.1, and VII.1 in the updated IDI protocol. Revising the structure to be question-based streamlines the IDI protocol and reminds the interviewer to ask the respondent to confirm the information before proceeding with the IDI. This format is a more natural part of the interview process than the previous format of an interviewer instruction to confirm these responses.

Table 8. Changes to improve protocol organization

Revision Number	Revision	Pretest Instrument Section or Question Number	Rationale
1	Changed name from Procurement Decision-Making and Strategy to Procurement Structure	III	After some questions in this section were revised and others were moved to this section from Section VI, the name no longer reflected the content, so the section was renamed.
2	Moved Section V, Solicitation Methods, ahead of Section IV, Contracts	IV, V	Respondents were confused when they were asked about solicitation methods after answering questions about contracts.
3	Split content from Contract Management and Recordkeeping (VI) into one of four new sections: Section I, Procurement Structure; Section II, Procurement Approach; Section V, Working with FSMCs; or Section VI, Cooperative Purchasing	VI	Section VI was long and fatiguing for respondents, and the IDI protocol also required interviewers to read internal instructions throughout the time they were conducting the interview, and to navigate a number of different questions that are not relevant to the respondent in the moment. Having separate sections for FSMCs and cooperatives will allow interviewers to skip an entire section if it is not applicable to a respondent, instead of reading through a section and picking out the applicable questions.
4	Removed interviewer narration: “Now I am going to ask about different aspects of the procurement process. By this I am referring to the range of activities your SFA undertakes to procure goods and services. For example, think about assessing your procurement needs, drafting solicitations, evaluating bids/responses, managing the contract, keeping records, etc.”	VIII	IDI protocol was wordy and complicated, making it difficult to use. Pretesting revealed that this text was not useful, so it is now removed. Interviewers will be able to probe if respondent is not providing enough detail.
5	Moved questions to Section III of the revised IDI protocol, Procurement Structure	III.5, III.6, III.7, III.8, III.10, III.11, III.12, VI.27, VI.28, VI.32, VI.33, VI.34	Section VI, Contract Management and Recordkeeping, was overly burdensome. Some questions were moved from this section to others. Many fit best in Section III of the revised IDI protocol, Procurement Structure. The first section of questions in the pretest version of the IDI protocol, Section III, Procurement Decision-Making and Strategy, lacked introductory content, so interviewer lacked basic information about the SFA as the interview progressed. Section III was revised to focus on introductory content, and some questions from the pretest version were moved to the Procurement Structure section, where they fit better.
6	Combined the questions, “Why does your SFA choose to procure goods or services for programs together?” and “Why does your SFA choose to procure goods or services separately?”	III.11, III.12	IDI protocol was wordy and overly complicated. Questions were combined to streamline the protocol.
7	Combined question VI.21, “Why does your SFA choose not to use an FSMC?” with question VI.22 “Why doesn’t your SFA participate in any group purchasing entity?”	VI.21, VI.22	IDI protocol contained a distracting number of instructions for interviewer; questions were combined in response. Section VI was long and fatiguing for respondents, so content from this section was moved to one of four new sections. This question now fits best with other introductory questions on basic procurement structure.
8	Moved to Section III, Procurement Structure	VI.21, VI.22, VI.23	Pretest interviewer said she would have benefited from knowing more about the basic structure of the respondent’s SFA from the beginning of the IDI. She also noted that it sometimes took respondents a while to open up to the interviewer. First section in the protocol, Section III, was revised in response to include content on the SFA’s overall procurement structure.
9	Moved question from Section VIII, Challenges and Best Practices, to Section VII, Training and Resources	VIII.13	Question did not elicit much of a response; it was moved to a section with similar questions in an effort to generate more detailed responses.
10	Added new questions to Section III, Procurement Structure. These are Questions III.1, III.2, and III.3 in the revised protocol.	n/a	Pretest interviewer said she would have benefited from knowing more about the basic structure of the respondent’s SFA from the beginning of the IDI. She also noted that it sometimes took respondents a while to open up to the interviewer. In response, the first section in the protocol, Section III, was revised to include content on the SFA’s overall procurement structure.
11	Added new questions prompting interviewers to confirm SFA responses from the web survey before proceeding with the interview. These are Questions IV.1, IV.4, V.1, VI.1, VI.6, and VI.8.	n/a	Pretest interviewer was already asking respondents to confirm web survey data without prompting, and interviewers would benefit from simple prompts to do this.

Appendix A – Pretested Web Survey

SFA Procurement Practices Web Survey

You have been selected to participate in an important study. For the “Study of School Food Authority (SFA) Procurement Practices,” 2M Research Services (2M) is conducting a survey for the U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS). FNS has commissioned this study to describe and assess SFA practices related to procuring food, goods, and services for the following Child Nutrition programs: The National School Lunch Program (NSLP), School Breakfast Program (SBP), Fresh Fruit and Vegetable Program (FFVP), Summer Food Service Program (SFSP), and Child and Adult Care Food Program (CACFP). FNS is seeking to better understand how SFAs make decisions that shape their procurement practices.

The purpose of this survey is to gather information about your SFA’s procurement practices. This survey has five sections in regard to: (1) the types of solicitations and contracts your SFA uses; (2) your procurement methods; (3) your food sources, including, but not limited to, USDA Foods and food from local sources; (4) planning, decision-making and procurement monitoring; and (5) types of guidance and training used by your SFA. Survey findings will describe similarities and differences in how SFAs procure food, goods, and services, which will help FNS understand key issues and experiences of SFAs.

Please answer as openly and honestly as possible. Your answers will be kept private; individual names will not be used in any reports, and only aggregated results will be reported. Your participation in this survey will not affect your employment or your SFA’s participation in any Child Nutrition programs.

If you have any questions about the study or about completing this survey, please email Dr. Steven Garasky, 2M Project Director, at [email protected] or call him at 817-856-0876. Thank you for your participation in this important survey.

Section I. Solicitations and Contracts

In this section, we want to understand the ways your SFA procures food, goods, and services for Child Nutrition programs (i.e., NSLP, SBP, FFVP, SFSP, and CACFP) for school year (SY) 2016–2017. Please focus on solicitations that were newly developed, solicitations that were developed for contracts active in SY 2016–2017, newly-awarded contracts, and active contracts in SY 2016–2017. We are interested in contracts solicited for self-operating SFAs, as well as for Food Service Management Companies (FSMCs) and contracts procured under an intergovernmental/interagency agreement. Examples of goods and services include food, equipment, and supplies purchased for use in Child Nutrition programs, or personnel hired to prepare and serve food for Child Nutrition programs.

Please indicate the Child Nutrition program(s) in which your SFA participates.

Select all that apply:

NSLP
SBP
FFVP
SFSP
CACFP

How does your SFA define local for school foods?

Within a certain number of miles

(If checked, within how many miles does your SFA define local? ___________ miles)

Within your county
Within your State
Within your State and adjacent States
Within your FNS region
We do not have a set definition.

The next questions are about your competitive solicitation methods.

During SY 2016–2017, what competitive solicitation methods did your SFA use to procure food, goods, and services for Child Nutrition programs at your SFA’s schools?

Select all that apply:

Competitive sealed bids (i.e., an invitation for bids)
Competitive proposals (i.e., using a Request for Proposals process)
Micro-purchasing procedures
Small purchasing procedures
Noncompetitive proposals

What factors influence which competitive solicitation methods your SFA uses to procure food, goods, and services?

Select all that apply:

Administrative capacity for developing contracts
Amount of time available to plan the procurement
Brands that are available
Complexity of the purchase
Cost
Duration of the contract
Kind/variety of foods available
Pre-approval of vendor by USDA
Size of the purchase
Specific standards that need to be met for products
Specific standards that need to be met for the Child Nutrition programs
Other (Specify): ______________________________________________________
Other (Specify): ______________________________________________________
Other (Specify): ______________________________________________________

This next set of questions is about the types of contracts your SFA awarded through competitive solicitation methods for SY 2016–2017. SFAs may award different types of procurement contracts, including the following:

Fixed-price contracts provide a stated price that is fixed without any adjustment for the duration of the contract.
Fixed-price with economic price adjustment contracts provide for upward and downward adjustment of the stated contract price that is fixed on an economic indicator such as the Consumer Price Index.
Cost-reimbursable contracts reimburse the vendor for costs incurred, and they may (or may not) include a fixed fee.

Which type(s) of contract(s) did your SFA award for SY 2016–2017 for procured food, goods, and services?

Select all that apply:

Fixed-price contracts
Fixed-price with economic price adjustment contracts
Cost-reimbursable contracts (no fixed fee)
Cost-reimbursable with fixed-fee contracts
Other (Specify): ______________________________________________________

Forward contracts include an agreement between parties to buy a product at a specified price on a future date. For example, an SFA may contract with a farmer or producer to provide a commodity at a set price before it has been produced. Did your SFA use forward contracts in SY 2016–2017?

Do State or local regulations limit your SFA to using specific types of contracts (e.g. fixed price, cost-reimbursable, forward contracts)?

What factors influence which contract type your SFA awards for procuring food, goods, and services?

Select the top three factors:

Consistency in product
Quality of service
Type of service
Level of administrative burden and oversight required
Level of contract oversight required
Level of risk (other than type of pricing such as length of contract)
Previous experience
Small purchase and micro-purchasing thresholds
Type of pricing (fixed, flexible)
Other (Specify): ____________________

What benefits does your SFA experience by using each type of procurement contract?

[Note for pretesters: Please only select benefits for contract types you selected in Questions 5 and 6. For the web version of the survey (full study), only those contract types will be displayed.]

Select all that apply:

Contract Type	Greater consistency in product quality	Greater consistency in service quality	Greater consistency in product type	Greater consistency in service type	Less administrative burden and oversight required	Less contract oversight required	Less risk	Fixed pricing	Flexible pricing	Other (Specify): ___
Fixed-price
Fixed-price with economic price adjustment
Cost-reimbursable
Cost-reimbursable with fixed fee
Forward contracts
Other (specify): ______________

The next questions are about formal and informal procurement. SFAs utilize formal and informal procurement procedures as described in the table below:

Formal Procurement	(1) developing the solicitation, (2) publicly announcing/advertising the solicitation, (3) evaluating offers, (4) awarding the contract, and (5) managing the contract
Informal Procurement	(1) developing the solicitation, (2) identifying potential vendors to obtain three bids/offers, (3) evaluating responses, (4) awarding the contract, and (5) managing the contract

In addition to specifications for products and services, payment terms, and required contract provisions and conditions, what additional procurement contract provisions are integrated in your formal procurement for school foods?

Select all that apply:

Small business
Minority-owned business
Woman-owned business
Geographic preference
Other (Specify) _________________

In addition to specifications for products and services, payment terms, and required contract provisions and conditions, what additional procurement contract provisions are integrated in your informal procurement for school foods?

Select all that apply:

Small business
Minority-owned business
Woman-owned business
Geographic preference
Other (Specify) _________________

The next set of questions is about best practices your SFA may have regarding the procurement of food, goods, and services for your Child Nutrition programs. Best practices help your SFA understand what works well for implementing particular aspects of procurement. They may be available in various forms, such as “how-to” instructions or recommendations, templates, or lessons learned, and may have been developed by your SFA or State Agency. Best practices are internal standards that supplement Federal, State, and local procurement guidance and resources used to help your SFA comply with procurement regulations.

Does your State Agency provide contract language or prototypes for FSMC contracts with SFAs?

Yes
No (skip to Question 15)

Does your SFA have best practices for State Agency contract language or prototypes for FSMC contracts?

Does your SFA desire best practices (or additional best practices) from your State Agency on using contract language or prototypes for FSMC contracts?

Does your State Agency provide contract language or prototypes for SFA contracts with processors, including USDA Foods processors?

Yes
No (skip to Question 18)

Does your SFA have best practices on using State Agency contract language or prototypes for procurement contracts with processors, including USDA Foods processors?

Does your SFA desire best practices (or additional best practices) from your State Agency on using contract language or prototypes for procurement contracts with processors, including USDA Foods processors?

Does your State Agency provide contract language or prototypes for SFA contracts with broadline distributors? Broadline distributors (also known as general-line or full-line distributors) are a type of merchant grocery wholesaler company that handles a variety of food and supply products, and other food service materials.

Yes
No (skip to Question 21)

Does your SFA have best practices on using State Agency contract language or prototypes for procurement contracts with broadline distributors?

Does your SFA desire best practices (or additional best practices) from your State Agency on using contract language or prototypes for procurement contracts with broadline distributors?

The Buy American provision requires SFAs participating in the NSLP or SBP to buy agricultural commodities produced in the United States and to buy food products processed in the United States using agricultural commodities, such that at least 51 percent of the final processed product consists of domestically grown agricultural commodities.

Does your SFA use specific methods/processes to ensure that your SFA complies with the Buy American provision?

Yes
No
Don’t Know/Unsure

What methods/processes does your SFA use to ensure that your SFA complies with the Buy American provision?

Mark “Yes”, “No”, or “Don’t Know/Unsure” for each item:

Include Buy American or “domestic” clause in product specifications for food

Yes
No
Don’t Know/Unsure

Include Buy American or “domestic” clause in bid solicitations for food

Yes
No
Don’t Know/Unsure

Include Buy American or “domestic” clause in requests for proposals for food

Yes
No
Don’t Know/Unsure

Include Buy American or “domestic” clause in purchase orders for food

Yes
No
Don’t Know/Unsure

Include Buy American or “domestic” clause in other procurement documents for food

Yes
No
Don’t Know/Unsure

What methods/processes does your SFA use to ensure that your SFA complies with the Buy American provision?

Mark “Yes”, “No”, or “Don’t Know/Unsure” for each item:

Request suppliers provide certification of the origin of their food products

Yes
No
Don’t Know/Unsure

Request distributors provide certification of the origin of their food products

Yes
No
Don’t Know/Unsure

Have a process for a vendor to request permission to substitute a domestic food product with a non-domestic product

Yes
No
Don’t Know/Unsure

Require distributors provide documentation showing acceptable reasons for purchase of non-domestic food products

Yes
No
Don’t Know/Unsure

Examine food product packaging provided by suppliers and distributors

Yes
No
Don’t Know/Unsure

Inquire with the manufacturer about the origin of the food product

Yes
No
Don’t Know/Unsure

Inquire with the distributor about the origin of the food product

Yes
No
Don’t Know/Unsure

What methods/processes does your SFA use to ensure that your SFA complies with the Buy American provision?

Mark “Yes”, “No”, or “Don’t Know/Unsure” for each item:

Inquire with the supplier about the origin of the food product

Yes
No
Don’t Know/Unsure

Monitor contractor performance to ensure compliance with “domestic” foods

Don’t Know/Unsure

Other (Specify): _____________________________________________________________

Does your SFA have best practices for including the Buy American provision in contract solicitations?

Does your SFA desire best practices (or additional best practices) from your State Agency on including the Buy American provision in contract solicitations?

Does your SFA have best practices on including the Buy American provision in contracts?

Does your SFA desire best practices (or additional best practices) from your State Agency on including the Buy American provision in contracts?

The next couple of questions ask whether your SFA has best practices regarding the maximum length of procurement contracts.

Does your SFA have best practices on setting maximum durations in procurement contracts?

Does your SFA desire best practices (or additional best practices) from your State Agency on setting maximum durations on procurement contracts?

The next questions ask whether your SFA has best practices regarding methods for publicly announcing and otherwise issuing invitations for bids and requests for proposals for procurement contracts.

For what procurement contract solicitation methods does your SFA have best practices?

Select all that apply:

Publicly announcing/advertising invitations for bids
Publicly announcing/advertising requests for proposals
Direct solicitation from contractors for invitations for bids
Direct solicitation from contractors for requests for proposals
None

Does your SFA desire best practices (or additional best practices) from your State Agency on procurement contract solicitation methods?

Select all that apply:

Publicly announcing/advertising invitations for bids
Publicly announcing/advertising requests for proposals
Direct solicitation from contractors for invitations for bids
Direct solicitation from contractors for requests for proposals
None

The next questions ask whether your SFA has best practices regarding procurement contract product specifications, such as food specifications, product adherence to the meal pattern requirements in the nutrition standards for school meals, and product quality.

Does your SFA have best practices for including product specifications in procurement contracts?

Does your SFA desire best practices (or additional best practices) from your State Agency for including product specifications in procurement contracts?

The next two questions ask whether your SFA has best practices on local bonding requirements. SFAs are required to follow bonding requirements for procurement contracts that entail construction or facility/utility improvements and exceed the Federal small purchase threshold of $150,000. Bonding requirements refer to, at a minimum, (a) bid guarantees such that bidders will execute contractual documents in the specified time after the bid is accepted, (b) performance guarantees that ensure the contract performance is fulfilled, and (c) payment bonds that assure payment as required by law of all executed labor and material work.

Does your SFA have best practices on local bonding requirements?

Yes
No
Don’t Know/Unsure

Does your SFA desire best practices (or additional best practices) from your State Agency on local bonding requirements?

Section II. Procurement Methods

Questions in this section are about procurement management practices with respect to your SFA’s use of FSMCs and/or participation in group purchasing entities (cooperatives, agents, third-party entities). Questions at the end of the section ask about consolidating procurement across Child Nutrition programs.

In the following questions, we want to understand the ways that your SFA may be using an FSMC to procure food, goods, or services for your Child Nutrition programs for SY 2016–2017.

Did your SFA use an FSMC to procure food, goods, or services for your Child Nutrition programs for SY 2016–2017?

Yes
No (skip to Question 11)

How long has your SFA used an FSMC in any capacity?

_______________ (number of years)

Don’t Know/Unsure

Does your SFA currently contract with one FSMC or groups of FSMCs?

Only one FSMC (skip to Question 6)
Groups of FSMCs

How many FSMCs did your SFA contract with for SY 2016–2017?

________________ (number of FSMCs)

Of the NUMBER of FSMCs your SFA contracts with for SY 2016–2017, how many of those contracts include other SFAs?

________________ (number of FSMC contracts that include other SFAs)

Don’t Know/Unsure

Do(es) the FSMC(s) manage one, some, or all aspects of procurement for your SFA’s Child Nutrition programs? Please consider all aspects of procurement, such as contracting for meal preparation, program administration, procuring processing for bulk USDA Foods items, and ordering processed USDA Foods, as well as services such as menu planning and management and supervision of labor, when responding. For example, your SFA may contract with an FSMC for meal preparation, but not for procurement services.

Only one aspect
Some, but not all, aspects
All aspects

Why did your SFA decide to use the services of an FSMC?

Select all that apply:

FSMCs find year-round availability of key food products
FSMCs coordinate procurement of local foods with regular procurement
FSMCs provide a wider variety of local goods than vendors
FSMCs offer lower prices than vendors
FSMCs offer more consistent prices than vendors
My SFA lacks the kitchen equipment to process/prepare local foods
FSMCs help my SFA maintain compliance in procurement regulations and policies
FSMCs provide food, goods, and services on time and resolve delivery issues
FSMCs procure products that meet cafeteria specifications
FSMCs hire staff which reduces SFA personnel oversight and costs
My SFA lacks the personnel to administer program operations for break-even or positive cash flow
My SFA lacks the expertise to plan, prepare, and serve nutritious food and meals
My SFA lacks the ability to attract and retain qualified staff to administer and operate the program

My SFA’s previous experience with a FSMC
The FSMC was already in use when I began working for my SFA
Directed by district leadership
Other (Specify): ___________________________________________________________

How did your SFA learn about available FSMCs?

Select all that apply:

State Agency
Local Education Agency
School officials (e.g., board members, district administrators)
Issued a Request for Information
Advertised solicitation for FSMC services
Other SFAs
Approached by FSMC(s)
Advisory group
The FSMC was already in use when I began working for my SFA
Other (Specify): ___________________________________________________________

Did any FSMCs help transition your SFA to using an FSMC?

Yes
No (skip to Question 11)
Don’t Know/Unsure (skip to Question 11)

How did the FSMC(s) help transition your SFA to using an FSMC?

Select all that apply:

Assisted with maintaining and organizing contract records
Provided health certification and inspection for facilities used to prepare food and meals
Assisted with State Agency inspections
Supported maintenance of records (i.e., supported invoices, receipts, other evidence)
Wrote the FSMC contract
Provided extra training of staff
Replaced staff
The FSMC did not help with transition
No transition was needed (i.e., SFA used FSMC services in the past)
Other (Specify): __________________________________________________________

The following questions ask about whether your SFA combines the procurement of any food, goods, or services for any Child Nutrition programs. An example of combining would be to use the same procurement method (e.g. small purchases, micro-purchases, FSMCs) to procure milk for the NSLP and SBP programs.

Please indicate the Child Nutrition program(s) for which your SFA combines the procurement of any food.

Select all that apply:

NSLP
SBP
FFVP
SFSP
CACFP
Don’t Know/Unsure

Please indicate the Child Nutrition program(s) for which your SFA combines the procurement of any goods.

Select all that apply:

NSLP
SBP
FFVP
SFSP
CACFP
Don’t Know/Unsure

Please indicate the Child Nutrition program(s) for which your SFA combines the procurement of any services.

Select all that apply:

NSLP
SBP
FFVP
SFSP
CACFP
Don’t Know/Unsure

In the following questions, we want to understand the ways that your SFA may have used group purchasing entities (cooperatives, agents, third-party entities) to procure food, goods, and services for your Child Nutrition programs for SY 2016–2017.

Does your SFA participate in any group purchasing entities (cooperatives, agents, third-party entities) to procure food, supplies, or equipment for any Child Nutrition programs for SY 2016–2017?

Yes
- No (skip to Question 20)

Do any of these group purchasing entities procure local products for your SFA?

This next question asks about interstate group purchasing entities (cooperatives, agents, third-party entities). An interstate group purchasing entity is usually a group purchasing organization or third party entity that is not comprised of only SFAs. This entity could also be called a “cooperative,” and could be public, private, or a public-private partnership. SFAs may establish a procurement contract through an interstate group purchasing entity in which, for example, an SFA may “piggyback” onto an existing agreement with a State Agency, local government entity, or non-government entity in another State.

Does your SFA belong to an interstate group purchasing entity?

The next questions ask about your SFA’s participation in statewide group purchasing entities or programs. These include statewide group purchasing entities (including cooperatives) and programs administered by the State’s Office of Child Nutrition (OCN) and statewide cooperatives that are not administered by the State OCN.

Does your State OCN offer a statewide group purchasing entity or other statewide purchasing programs?

Yes
No (skip to Question 20)
Don’t Know/Unsure (skip to Question 20)

Does your SFA participate in these purchasing cooperatives or programs?

Yes
No
Don’t Know/Unsure

Does your SFA participate in any statewide purchasing cooperatives not administered by your State OCN?

Yes
No
Don’t Know/Unsure

In the next questions, we are interested in learning what procurement methods are used by your SFA to source products, especially local products, and services for your Child Nutrition programs for SY 2016–2017.

Please check boxes in the grid to indicate whether your SFA has sourced foods locally and/or non-locally for any of the product groups listed in the far-left column, through any of the procurement methods you identified earlier which are listed in the top row.

[Note for pretesters: Please only select the procurement methods that you selected in Section 1 Question 3 and Section II Questions 1 and 14. For the web version of the survey (full study), only these procurement methods will be displayed.]

Select all that apply:

		Procurement Methods
Food Component Groups	Source	Competitive Proposal Methods	Sealed Bid Methods	Small Purchases	Micro-Purchases	FSMCs	Group Purchasing Entities (cooperatives and agents that procure using Program and government-wide regulations)	Third Party Group Purchasing Entities (not bound by Program and government-wide regulations)
Fruits	Local
Fruits	Non-local
Vegetables & Vegetable Subgroups (legumes, leafy greens, etc.)	Local
	Non-local
Grains/Grain Products	Local
Grains/Grain Products	Non-local
Meats/Meat Alternatives and Seafood	Local
Meats/Meat Alternatives and Seafood	Non-local
Dairy/Fluid Milk	Local
Dairy/Fluid Milk	Non-local

Does your SFA have best practices on procuring local products?

Does your SFA desire best practices (or additional best practices) from your State Agency on procuring local products?

Please check boxes in the grid to indicate whether your SFA has sourced any of the foods, goods, or services listed in the far-left column through any of the procurement methods that are listed in the top row for Child Nutrition programs at your SFA’s schools for SY 2016–2017.

Select all that apply:

		Procurement Methods
Services	Competitive Proposal Methods	Sealed Bid Methods	Small Purchases	Micro-Purchases	FSMCs	Group Purchasing Entities (cooperatives and agents that procure using Program and government-wide regulations)	Third Party Group Purchasing Entities (not bound by Program and government-wide regulations)
A la carte meal service
After-school snacks
Menu planning
Ensuring compliance with procurement rules and regulations
Procuring equipment
Managing food preparation
Procuring foods
Procuring goods or services
Managing services for Child Nutrition programs
Delivering food for meal preparation
Delivering meals to sites
Preparing meals
Managing meal services at sites
Providing nutrition education
Providing on-site staff to support provided services
Procuring services to convert raw and/or bulk USDA Foods into ready-to-use end products
Procuring products processed from bulk USDA Foods
Program management
Managing and supervising labor
Product inventory
Product storage
Site cleanup after meals
Procuring vended meals
Storing vended meals
Managing safety and sanitation
Other (Specify): ___________

Section III. Sourcing Foods

This section asks about how your SFA sources foods for Child Nutrition programs, including USDA Foods and foods from local sources, in SY 2016–2017.

1. Does your SFA ever divert bulk USDA Foods for processing?

Does your SFA competitively procure services from manufacturers or processors to convert (process and/or package) raw and/or bulk USDA Foods into ready-to-use end products for Child Nutrition programs?

3. From which of the following sources does your SFA competitively procure USDA Foods bulk-purchased end products for Child Nutrition programs? By USDA Foods bulk-purchased end products, we mean raw and/or bulk USDA Foods that have been converted into ready-to-use end products.

Select all that apply:

Manufacturers or food processors
Distributors (for example, Sysco or US Foods)
Groups of farmers that sell products centrally, such as producer co-ops or food hubs
Suppliers contracted by the FSMC, cooperative purchasing group, and/or group buying organization
State-level processor
Meal vendor (for example, Revolution Foods)
Other food source supplier 1 (Specify)_________________________________________
Other food source supplier 2 (Specify)_________________________________________

4. Which value pass-through methods does your SFA use to obtain credit for the value of the USDA Foods in processed end-products?

Select all that apply:

Rebates
Net-off invoices
Fee-for-service
Direct discount
Don’t know/Unsure

The following questions ask about your SFA’s procurement of local foods and participation in Farm to School practices in SY 2016–2017.

Of your total food costs for Child Nutrition programs in SY 2016–2017, what percentage of costs do you estimate were spent on foods from local sources?

_______ percent

Don’t Know/Unsure

What is your SFA’s percentage goal for the amount spent on foods from local sources for your Child Nutrition programs?

_______ percent

Our SFA does not have a percentage goal
Don’t Know/Unsure

What food purchasing and promotion practices does your SFA’s FSMC use to support your SFA’s procurement from local sources?

Select all that apply:

Sourcing products that meet the SFA’s geographic preference, including from local farms
Sourcing products from school gardens
Collaborating with school garden coordinators
Promoting garden education in the cafeteria
Supporting school gardening in other ways (Specify): _________________

Other (Specify): __________________
Our FSMC does not provide this support
Our SFA does not work with an FSMC

Does your SFA participate in Farm to School practices such as serving local food products in school, conducting educational activities related to local foods, or creating and tending school gardens?

No
Don’t Know/Unsure

Section IV. SFA Planning and Monitoring

This section asks how your SFA makes decisions to determine its overall procurement strategy and monitors procurement contracts. You will also be asked for information about your SFA’s internal recordkeeping and accounting, FSMCs, and Buy American requirements for Child Nutrition programs for SY 2016–2017.

The following questions focus on the information your SFA uses for its procurement planning process and individuals that are involved with planning.

Does your SFA use internal recordkeeping systems that support forecasting and planning (e.g., tracking daily food sales) to make procurement decisions?

What information does your SFA use to determine its overall procurement strategy (that is, the big-picture priorities that shape your SFA’s approach to procurement)?

Mark “yes” or “no” for each item:

Food service staffing needs

Food service staffing costs

Food service equipment and supply needs

Food service equipment and supply costs

Food service operations

Program management/administration needs

What information does your SFA use to determine its overall procurement strategy (that is, the big-picture priorities that shape your SFA’s approach to procurement)?

Mark “yes” or “no” for each item:

Child Nutrition program participation rates

Menu items/variety

Local wellness policies and nutrition goals

Input from State Agency and/or State Distributing Agency

Input from other SFAs

Input from an advisory council

Input from the student body

Input from the community

Storage capacity

What information does your SFA use to determine its overall procurement strategy (that is, the big-picture priorities that shape your SFA’s approach to procurement)?

Mark “yes” or “no” for each item:

Regulatory meal pattern requirements

School board

Other

Yes (Specify): ________________________________________________________
No

Please rank the three most important sources of information your SFA uses to determine its overall procurement strategy by indicating the most important item with a “1,” the second most important with a “2,” and the third most important with a “3.”

[Note for pretesters: Please rank the sources of information for which you selected “Yes” in Section IV Question 2. For the web version of the survey (full study), only those sources of information will be displayed.]

______ Food service staffing needs

______ Food service staffing costs

______ Food service equipment and supply needs

______ Food service equipment and supply costs

______ Food service operations

______ Program management/administration needs

______ Child Nutrition program participation rates

______ Menu items/variety

______ Local wellness policies and nutrition goals

______ Input from State Agency and/or State Distributing Agency

______ Input from other SFAs

______ Input from an advisory council

______ Input from the student body

______ Input from the community

______ Storage capacity

______ Regulatory meal pattern requirements

______ School board

Please indicate all staff involved in procurement planning for your Child Nutrition programs during SY 2016–2017 by selecting the corresponding box in the list below.

Select all that apply:

SFA nutrition director
SFA food service director or manager
SFA head cook or kitchen/cafeteria manager
District business office or purchasing department official
District superintendent
District school board
State Agency Child Nutrition director or staff
FSMC liaison
Governing or advisory board
Sales broker
Group purchasing entity (cooperative, agent, or third-party service)
Cooperative that only includes Child Nutrition Program operators
Other (Specify): __________________________________________________________

The next two questions ask whether your SFA has best practices on the procurement process time frame, as steps in the procurement process may be subject to time pressures. Best practices are internal standards that differ from Federal, State, and local procurement guidance and resources used to help your SFA comply with procurement regulations.

For what steps in the procurement process does your SFA have best practices for managing procurement time frames?

Select all that apply:

Developing the solicitation
Advertising the solicitation
Evaluating offers
Awarding a contract
Managing the contract
None

For what steps in the procurement process would your SFA desire best practices (or additional best practices) from your State Agency for managing procurement time frames?

Select all that apply:

Developing the solicitation
Advertising the solicitation
Evaluating offers
Awarding a contract
Managing the contract
None

The following questions ask about your SFA’s monitoring processes for contract evaluation and awards.

What methods of oversight (records, documentation, and mechanisms) does your SFA use to monitor staff in their procurement-related evaluation decisions (e.g., bid reviews)?

Select all that apply:

FSMC monitoring forms
Cost records
Revenue records
Invoice records
Meal production records
Meal count records
Meal claim records
RFP or IFB
Email communications
Telephone communications
Site visits
Meetings
Procurement plan
Code of Conduct
Other (Specify): _________________________________________________________
Our SFA does not monitor staff in their procurement-related evaluation decisions

What methods of oversight (records, documentation, and mechanisms) does your SFA use to monitor staff in their contract award decisions?

Select all that apply:

RFP or IFB evaluation criteria and results

Meetings/negotiation phase, if applicable (RFP only)
Procurement plan
Code of Conduct

Other (Specify): _________________________________________________________
Our SFA does not monitor staff in their contract award decisions

The following questions ask about your SFA’s internal recordkeeping and accounting, contractor performance, and Buy American requirements.

What records or methods does your SFA use to track discounts, rebates, and credits for commercially purchased foods in cost-reimbursable contracts?

Select all that apply:

Electronic accounting system
Current and historical sales or velocity reports (a velocity report provides the SFA the quantity, the date of purchase, and other information about the purchase of food products from a vendor)
Invoices for expenses
Defined procedures in contract
Other (Specify): _________________________________________________________

How does your SFA identify accounting errors when tracking discounts, rebates, and credits for commercially purchased foods in cost-reimbursable contracts?

Select all that apply:

Using an electronic accounting system
Maintain receipt records in which the contractor identified costs
Review bills and invoices that individually identify discounts, rebates, and applicable credits
State Agency reviews the costs, discounts, rebates, and applicable credits maintained by the contractor
Local agency reviews the costs, discounts, rebates, and applicable credits maintained by the contractor
Review the methods contractors use to track discounts, rebates, and other applicable credits
Monitor revenues and expenses based on the accounting method of the contractor
Determine if the vendor is compliant with contractual disclosure method
Other (Specify): __________________________________________________________

How does your SFA monitor contractor performance after award? An example of performance would be compliance with Buy American requirements and documentation of exceptions.

Select all that apply:

Identify people within the SFA responsible for monitoring activities
Examine invoices and documentation provided by contractor
Review the average credit-purchase proportion (Value of Credit Reported/Value of Purchased Food)
Request certification identifying the percent of U.S. content in supplied products
Review applicable contractor health certification records

Our SFA does not monitor contractor performance after award

Section V. SFA Guidance and Training

This section contains questions about whether your SFA uses any guidance or templates when developing solicitations for procurement of food, goods, and services, as well as your thoughts on their usefulness. By guidance, we are referring to assistance or information provided by any State or local government entity. This section also includes questions about any trainings your State Agency and local education agency provide related to procurement. We begin with questions about guidance.

Does your State Agency or local education agency require your SFA to use FNS or State-provided guidance for developing procurement contract solicitations?

Does your State Agency or local education agency require your SFA to use guidance that the local education agency has provided for developing procurement contract solicitations?

What guidance has FNS or your State Agency provided to your SFA for developing procurement solicitations?

Select all that apply:

Institute of Child Nutrition online courses (e.g., “General Requirements and Responsibilities,” “How Procurement Works: Planning, Executing, and Administering a Competitive Procurement”)
Institute of Child Nutrition in-person trainings (e.g., “Procurement in the 21^st Century”)
Memos from your State Agency
Memos from FNS
Frequently Asked Questions (FAQs) from your State Agency
Frequently Asked Questions (FAQs) from FNS
Issue briefs
Manuals
State-sponsored training
USDA training (e.g., webinars, pamphlets, in-person training)
Other (Specify) __________________________________________________________
Neither FNS nor my State Agency has provided guidance for developing procurement solicitations (skip to Question 5)

What guidance provided by FNS or your State Agency for developing procurement solicitations has your SFA explicitly used?

[Note to pretesters: Please only consider the response options you selected in Section V Question 3. For the web version of the survey (full study), only those response options will be displayed.]

Select all that apply:

Institute of Child Nutrition online courses (e.g., “General Requirements and Responsibilities,” “How Procurement Works: Planning, Executing, and Administering a Competitive Procurement”)
Institute of Child Nutrition in-person trainings (e.g., “Procurement in the 21^st Century”)
Memos from your State Agency
Memos from FNS
Frequently Asked Questions (FAQs) from your State Agency
Frequently Asked Questions (FAQs) from FNS
Issue briefs
Manuals
State-sponsored training
USDA training (e.g., webinars, pamphlets, in-person training)
Other (Specify) _________________
My SFA has not used any guidance provided by FNS or my State Agency for developing procurement solicitations

Have any members of your SFA staff, including you, completed any of the following Institute of Child Nutrition online courses related to procurement contract solicitation?

Select all that apply:

Financial Management: A Course for School Nutrition Directors
Food Production Records
Inventory Management and Tracking
Quality in Value Added Products
USDA Foods: Department of Defense (DoD) Fresh Fruit and Vegetable Program - The Basics
USDA Foods: Life Cycle of USDA Foods
USDA State Agency Guidance to Procurement Topic 1: General Requirements and Responsibilities
USDA State Agency Guidance to Procurement Topic 2: How Procurement Works: Planning, Executing, and Administering a Competitive Procurement
USDA State Agency Guidance to Procurement Topic 3: Special Considerations
Procurement in the 21^st Century

None
Other (Specify) ___________________________________________________________
We have not completed any Institute of Child Nutrition online courses related to procurement contract solicitation

What kind of guidance has your local education agency provided to your SFA for developing procurement solicitations?

Select all that apply:

Memos
FAQs
Issue briefs
Manuals/handbooks
Local procurement requirements
Slide decks
Informational web pages
Other (Specify) _________________
My local education agency has not provided guidance for developing procurement solicitations (skip to Question 8)

What kind of guidance provided by your local education agency for developing procurement solicitations has your SFA explicitly used?

[Note to pretesters: Please only consider the response options that you selected in Section V Question 6. For the web version of the survey (full study), only those response options will be displayed.]

Select all that apply:

Memos
FAQs
Issue briefs
Manuals/handbooks
Local procurement requirements
Slide decks
Informational web pages
Other (Specify) __________________________________________________________
- My SFA has not used any guidance provided by my local education agency for developing procurement solicitations

The next questions ask about FNS-, State Agency-, and local education agency-provided templates.

Does your State Agency or local education agency require your SFA to use FNS- or State Agency-provided templates when developing procurement contract solicitations?

Does your State Agency or local education agency require your SFA to use templates that the local education agency has provided when developing procurement contract solicitations?

What templates have FNS or your State Agency provided to your SFA for developing procurement solicitations?

Select all that apply:

Solicitation templates for competitive sealed bids
Solicitation templates for RFPs
Solicitation templates for small purchases
Other (Specify) _________________
Neither FNS nor my State Agency has provided templates for developing procurement solicitations (skip to Question 12)

What templates provided by FNS or your State Agency for developing procurement solicitations has your SFA explicitly used?

[Note to pretesters: Please only consider the response options that you selected in Section V Question 10. For the web version of the survey (full study), only those response options will be displayed.]

Select all that apply:

Solicitation templates for competitive sealed bids
Solicitation templates for RFPs
Solicitation templates for small purchases
Other (Specify) ___________________________________________________________
- My SFA has not used any templates provided by FNS or my State Agency for developing procurement solicitations

What templates has your local education agency provided to your SFA for developing procurement solicitations?

Select all that apply:

Solicitation templates for competitive sealed bids
Solicitation templates for RFPs
Solicitation templates for small purchases
Other (Specify) ___________________________________________________________
My local education agency has not provided templates for developing procurement solicitations (skip to Question 14)

What templates provided by your local education agency for developing procurement solicitations has your SFA explicitly used?

[Note to pretesters: Please only consider the response options that you selected in Section V Question 12. For the web version of the survey (full study), only those response options will be displayed.]

Select all that apply:

Solicitation templates for competitive sealed bids
Solicitation templates for RFPs
Solicitation templates for small purchases
Other (Specify) _________________
- My SFA has not used any templates provided by my local education agency for developing procurement solicitations

Does your State Agency make training available to SFAs on procurement issues, including training provided by another party?

Yes
No (skip to Survey Section Verification Screen)

15. What training topics does your State Agency make available to your SFA about procurement?

Select all that apply:

Updates and changes to Federal policies and regulations
Updates and changes to State policies and regulations
Products and specifications
Solicitations and evaluations
Food and supply orders
Group purchasing efforts
Forecasting
Solicitations for contracts with FSMCs
Farm to School activities and initiatives
Procurement of local foods
Processors of USDA Foods
Developing solicitations and contracts
Services
Professional standards
Code of Conduct
Procurement procedures
Allowable procurement methods to use
Other (Specify): ________________________________________________________

SURVEY VERIFICATION SECTION

You are almost done! Please make sure you have completed all sections of the survey. Once you have verified that the survey is complete, please return it either in the prepaid envelope provided or by emailing a scanned copy to Dr. Steven Garasky, 2M Project Director, at [email protected]. Please remember to save a copy of the survey for the debriefing interview.

After we receive the completed survey, a member of the study team will contact you to schedule the survey debriefing interview. We’ll schedule the final phase of the pretesting, the in-depth interview, when we conduct the survey debriefing.

Thank you again for participating in the pretesting phase of this study! If you have any questions about the study, please email Dr. Garasky at [email protected] or call him at 817-856-0876.

Appendix B – Pretested In-Depth Interview Protocol

In-Depth Interview (IDI) Protocol

Study of SFA Procurement Practices

In-Depth Interview Guide

I. Interviewer Preparation

Review procurement model characteristics.
Review SFA sample characteristics such as size, percent free/reduced-price lunches, and geographic region
Review web-based procurement survey responses such as Child Nutrition programs operated, participation in Farm to School initiatives or other local procurement efforts, and individuals involved in the procurement process
Tailor protocol to reflect procurement model and other SFA characteristics and to incorporate web-based survey responses

II. Introduction (2 min)

Thanks for participating in the “Study of School Food Authority (SFA) Procurement Practices,” which we are conducting on behalf of FNS. As you may know, FNS has commissioned this study to describe and assess SFA practices related to procuring goods and services for five Child Nutrition programs: The National School Lunch Program, School Breakfast Program, Fresh Fruit and Vegetable Program, Summer Food Service Program, and Child and Adult Care Food Program. The study will also examine how SFAs make decisions that lead to these procurement practices. Findings will describe similarities and differences in how SFAs procure goods and services and help USDA-FNS understand key issues and experiences of SFAs.

This in-depth interview will last up to 90 minutes and builds on the responses your SFA provided in the web-based survey. Please answer as openly and honestly as possible. Your answers will not be linked with the names of any individuals or SFAs.

If respondent(s) previously agreed to a recording:

When we scheduled our interview, you indicated that I have your permission to make an audio recording of the interview to fill in gaps in our notes. The recording will be kept in a secure location and destroyed after the study is over. Can you confirm that I have your permission to record the interview?

III. Procurement Decision-Making and Strategy (15 min)

I’m going to start by asking you some “big picture” questions about your overall procurement strategy and related factors.

Who is responsible for making decisions at each stage of the procurement process? [Probe: Who assesses the goods or services needed, drafts specifications, advertises solicitations, awards contracts, and manages contracts?]
What is each person’s job title?
How would you summarize your SFA’s overall procurement strategy? [Probe: What are the big-picture priorities or guiding principles that shape your SFA’s approach to procurement?]
What internal decision-making processes does your SFA use to develop the procurement strategy or plan? [Probe: Interviewer, probe based on SFA responses to web survey questions IV.1 and IV.2.]
Is purchasing from local sources a priority for your SFA’s overall procurement goals?

For SFAs that do not purchase local products:

Please provide any reasons why your SFA doesn’t specifically purchase local products.
Does your SFA have any interest in learning how or starting to purchase local products?

For all SFAs:

Can you describe any barriers to purchasing local products?
What data or other information does your SFA use to inform the overall procurement strategy or plan? [Probe: For example, does your SFA use meal participation, point-of-sale, or other types of data to help develop big-picture priorities or guiding principles around procurement?]

Interviewer: List the SFA’s responses from the web survey on whether they procure goods and services together or separately.
If SFA procures goods and services together:

Please describe how you procure goods or services together. [Probe: Did you make the decision to procure goods or services in this way or was the decision made by someone in this role before you?]
Why does your SFA choose to procure goods or services for programs together?

If SFA procures goods or services separately:
Why does your SFA choose to procure goods or services separately?

IV. Contract Types (5 min)

Now I’m going to ask you about the different types of contracts your SFA uses and how they are created. I’ll begin by listing the contract types your SFA reported on the web-based survey.

Interviewer: List contract types SFA reported on web survey.

How does your SFA decide which contract type to use?

Interviewer: Cycle through questions 2, 3, and 4 for each contract type.

What are the benefits of [contract type]?
What are the challenges associated with [contract type]?
What resources does your SFA use when writing [contract type]? [Probe: Does your SFA use a template for (contract type)? Where is the resource or template from or how was it developed? Is it from a local, State, or Federal source?]

Interviewer: List contract types SFA reported not using, based on web survey. Ask Q5 for each contract type not used.

Why does your SFA choose to not use [contract type]? [Probe: What challenges are associated with (contract type not used)?]
Do State or local-level regulations play a role in choosing a contract type? If so, what is that role? [Probe: Do State or local regulations limit your SFA to using specific types of contracts? If so, how?]

Interviewer: Tell respondent(s) whether their SFA reported using forward contracts based on the web survey. Confirm their web survey response. If SFA confirms not using forward contracts, skip to Question 8.

Under what circumstances does your SFA use a forward contract? [Probe: Why does your SFA use a forward contract in this/these situation(s)? Are forward contracts used for particular items? If so, what items?]

Interviewer: List the types of specifications the SFA reported using in contracts from the web survey results.

Can you describe the process your SFA uses to determine what kinds of specifications are included in school food contracts?
Under what circumstances does your SFA use specifications? When don’t you use them?

[Probe: Is this decision related to the type of contract you are using or to the type of goods or services you are looking to purchase? Why do you use specifications in these instances?]

The survey indicated that your SFA specifies geographic preference in your contracts. How is this being done?

V. Solicitation Process (20 min)

This section of the interview is about your choice of solicitation methods and the process for developing solicitation documents.

Interviewer: List the solicitation methods indicated by respondent on the web survey. Make sure the responses to each question distinguish practices for each type of solicitation method.

What factors influence which competitive procurement method(s) your SFA uses? [Probe: Factors such as quantity of food to be purchased, specifications, or regulations]

Interviewer: Cycle through questions 2–10 for each solicitation method.

What process does your SFA use to create [solicitation type]?
How are regulations taken into consideration as your SFA develops [solicitation type]? [Probe: How are Federal regulations taken into consideration? How are State and local regulations taken into consideration?]
What resource(s) does your SFA use to create [solicitation type]? [Probe: For example, documents that provide guidance or expertise from another staff member; these could include templates, training materials, or example documents]
Where do these resources come from? [Probe: For example, resources may be provided by your SFA, the State, an FSMC, or another source.]
If your SFA uses a template for [solicitation type], how was it developed? Where did it originate? [Probe: where did you get the template? Who gave it to you?]
What are the advantages and disadvantages of using a template for [solicitation type]?
Does your SFA complete a price or cost analysis before publishing [solicitation type]?
Why or why not?
How is this analysis conducted?

Interviewer: After cycling through questions 2–10 for each solicitation method, proceed to question 11.

Are there requirements associated with using any of the templates or resources, such as local, State, or Federal procurement rules or regulations?

VI. Contract Management and Record Keeping (10–25 min)

Now I’ll ask you some questions about your SFA’s experiences managing contracts, including working with FSMCs and group purchasing entities. If your SFA has procurement contracts with multiple FSMCs or as part of multiple group purchasing entities, I’d also like to hear how your experiences differ by contract.

Interviewer: Read list of web survey responses for reasons for contracting with an FSMC. Skip to question 11 if SFA does not contract with an FSMC.

For SFAs that contract with one or more FSMC(s):

What initially prompted your SFA to begin contracting with the FSMC(s)? [Probe: In the web survey you responded that (cite reason(s) reported in web survey). Can you tell me more about that? Are there any other factors contributing to why your SFA contracts with an FSMC? What were the main reasons your SFA began contracting with FSMCs?]
When your SFA began contracting with the FSMC(s), to what degree was the FSMCs involved in the process? [Probe: Who facilitates the process for each contract? Do facilitator roles vary by contract?]
What is the process for establishing a contract with the FSMC(s)? [Probe: What elements of the process vary across contracts? What elements are the same? What about the process works well? What could be improved?]
If your SFA also does independent in-house procurement, how does the process for establishing contracts differ when working with the FSMC(s) versus independent procurement?
In general, what are the benefits of working with the FSMC(s)?
How does your SFA measure these benefits? [Does your SFA use observation as a measurement tool? If so, how?]
Did the benefits your SFA hoped for in contracting with the FSMC(s) come to fruition? [Probe: If so, please describe how you have observed these benefits in practice.]
What are the challenges of working with the FSMC(s)?
Are there things your SFA would like to change about how the FSMC(s) operate?
How would you describe your SFA’s overall level of satisfaction with the FSMC(s)?

Interviewer: Ask questions 11–19 for SFAs that participate in one or more group purchasing entities. If the SFA participates in multiple organizations, cycle through the questions for each one before proceeding in the protocol:

What type of group purchasing entity/entities does your SFA participate in?
How does this group purchasing entity work? [Probe: Who are the partners in the group purchasing entity? What role does each partner play? Is your group comprised solely of SFAs (cooperative), an agent for an SFA or cooperative, or is it a group with non-SFA members (third-party entities)? How many SFAs are in the group? Is the group local, regional, or national?]
How is the procurement process conducted for the group purchasing entity? [Probe: What role does your SFA play? What role do other SFAs play? Who is responsible for things such as communicating with vendors, scheduling deliveries, and making payments?]
What is the fee structure associated with the group purchasing entity?
Why did your SFA decide to participate in this group purchasing entity? If respondent was not involved in the initial decision, ask: Why did your SFA decide to continue participating in the group purchasing entity this school year?
Did your SFA seek out the arrangement or was your SFA approached by another party, such as another SFA or a third-party entity representative? [Probe: If you were approached, by whom? Can you describe this initial contact?]
When you joined the group purchasing entity, what benefits and challenges did your SFA expect?
Do the benefits and challenges you have experienced match those you initially expected?
Have there been any unexpected challenges or benefits?
Are there any benefits from group purchasing entity that your SFA is measuring? [Probe: Have there been any measurable financial benefits? How are you measuring these benefits?]

For SFAs that don’t use any FSMCs:

Why does your SFA choose not to use an FSMC? [Probe: Are there barriers preventing your SFA from contracting with an FSMC?]

For SFAs that don’t participate in group purchasing entity:

Why doesn’t your SFA participate in any group purchasing entity? [Probe: Are there barriers preventing your SFA from participating?]

Next, I’ll ask you about how your SFA monitors contractor performance and compliance with the Buy American provision, and conducts other recordkeeping activities. In your responses, please describe any differences between contract types. If your SFA does small purchasing and micro-purchasing that doesn’t require contracts, please include descriptions of how you track costs and keep records for them when applicable.

Interviewer: For SFAs that contract with an FSMC and also manage independent procurement contracts, ask respondent(s) to differentiate between practices that are applicable in all cases, FSMC contracts only, and independent contracts only.

Interviewer: For question 23, refer back to who was mentioned as key decision-making staff in Section III and probe accordingly. Ensure that respondent addresses who is responsible for monitoring the evaluation of bids and contract award decisions specifically.

Who provides internal oversight for decision-making staff at each stage of the procurement process? [Probe: Who monitors the evaluation of bids, and contract award decisions? Who provides oversight for the staff responsible for planning, drafting specifications, advertising solicitations, awarding contracts, and managing contracts?]
How does your SFA monitor contractor performance after a contract has been awarded? [Probe: How is performance monitored for independent contracts and FSMC contracts? What are the differences in how they are monitored?]

For SFAs that use at least one FSMC:

What is the process for improving FSMC performance, if necessary?

For all SFAs:
How does your SFA monitor compliance with the Buy American provision? [For SFAs that do independent procurement and work with an FSMC Probe: How does monitoring differ for between FSMC and independent contracts?]

Interviewer: If respondent needs to be reminded what the Buy American provision is, say, “The Buy American provision requires SFAs participating in the NSLP or SBP to buy agricultural commodities produced in the United States, and buy food products processed in the United States using agricultural commodities such that at least 51 percent of the final processed product consists of domestically grown agricultural commodities.”
How does your SFA document exceptions to the Buy American provision? [Probe: How would your SFA document an instance where non-domestic foods needed to be purchased? For SFAs that use FSMC(s): How do you ensure that the FSMC(s) you work with document these exceptions? Who is responsible for this?]
What internal recordkeeping systems does your SFA use to make procurement decisions? [Probe: Examples of recordkeeping systems include daily participation numbers or competitive food sales. How do you keep records for small purchases or micro-purchases that don’t require contracts? For SFAs that use FSMC(s): How do you work with your FSMC(s) on record keeping? What types of activities or systems is each party responsible for?]

The next questions are about how your SFA tracks discounts, rebates, and credits for commercial products when awarding a cost-reimbursable contract, and for USDA Foods when awarding either a cost-reimbursable or fixed-price contract.

How does your SFA track discounts, rebates, and credits for commercial products? [Probe based on web survey responses to question IV.9 For SFAs that use FSMCs: Please describe how your SFA interacts with the FSMC(s) in this tracking process. Who is responsible for different aspects of the tracking?]

For SFAs that hold FSMC contracts and independent contracts:

How does tracking differ between your FSMC contracts and other contracts?

For SFAs that hold FSMC contracts:

How does your SFA track discounts, rebates, and credits for USDA Foods? [Probe: What recordkeeping systems does your SFA use to help with tracking?]

For all SFAs:

The next questions are about how your SFA tracks allowable program costs.

How does your SFA identify and track allowable program costs?

For SFAs that hold FSMC contracts and also do independent procurement:

Is there a difference in how your SFA tracks allowable program costs, between FSMC and independent vendor contracts?

For all SFAs:

How are accounting errors identified and rectified? [Probe based on answers to web survey questions IV.10 and IV.11. Probe: Does this differ for purchases that require a contract and those that do not, such as small or micro-purchases? If so, how?]

VII. SFA Training and Resources (5 min)

Next, I have a few questions for you about procurement-related training and technical assistance needs in your SFA and what resources are available to staff. First, here are the resources and training opportunities your SFA listed on the web-based survey as being provided by your State agency.

Interviewer: List procurement resources and training opportunities State agencies provided to SFAs from web survey responses.

What training needs are not being met?
What technical assistance is available?
Is there technical assistance your SFA needs that the State agency does not offer or has not offered?
Are there any resources your SFA does not have that would be helpful? [Probe: Such as documents describing best practices, access to staff with expertise on certain topics?]
What could be done to improve these resources and trainings from the State agency?

Now I am going to shift to asking you a few questions about resources and training made available locally.

What resources does your SFA offer in-house to help build staff capacity and expertise around procurement?
What training opportunities does your SFA offer in-house to help build staff capacity and expertise around procurement?

VIII. SFA Challenges and Best Practices (10 min)

The last part of this interview is about procurement challenges and best practices. To begin, I will ask you about challenges your SFA encounters with Federal, State, and local procurement standards.

Interviewer: For questions 1 and 2, solicit answers that address standards and supporting guidance at the Federal, State, and local levels. Probe as necessary to identify the particular Federal, State, and/or local standards that are creating challenges for the SFA.

What challenges does your SFA encounter in interpreting procurement standards?
Does your experience in interpreting procurement standards vary according to Federal, State, and local standards? [Probe: Are there standards or guidance that are hard to understand or you believe aren’t applicable to your SFA? Why? Is supporting guidance available to help you implement the standards? What is the source? Is it helpful?]
What challenges does your SFA experience when applying procurement standards to your procurement process?
Do the challenges you experience in applying procurement standards vary according to Federal, State, and local standards? [Probe: What standards or guidance are difficult for your SFA to put into practice or comply with? Why? Is supporting guidance available to help you implement the standards? What is the source? Is it helpful?]

Now I am going to ask about different aspects of the procurement process. By this, I am referring to the range of activities your SFA undertakes to procure goods and services. For example, think about assessing your procurement needs, drafting solicitations, evaluating bids/responses, managing the contract, keeping records, etc.

What aspects of the procurement process are easy for your SFA? Why are these processes easy?
What aspects of the procurement process are especially complex or burdensome for your SFA?
Why are these processes complex or burdensome?
How could the process be streamlined or simplified?
Do these complexities affect your SFA’s ability to procure local foods? How?
Do these complexities affect your SFA’s ability to try a different procurement arrangement? How? [Probe: For example, group purchasing or contracting with an FSMC.]
Does consolidating procurement across Child Nutrition programs affect your ability to comply with procurement regulations? How?

Interviewer: List the topics the SFA indicated best practices for, based on the web survey. Repeat question 12 for each topic separately.

What best practices does your SFA have around this topic? [Probe: What are your SFA’s best practices for addressing unavailable domestic product or prices that are perceived to be excessive? What are your SFA’s best practices for adhering to the Buy American provision? What are your SFA’s best practices around purchasing non-domestic products?]
Are there any templates or other resources your SFA uses that you think are particularly effective for helping to manage the procurement process?

Interviewer: List the topics from the web survey for which the SFA desired having best practices.

What type(s) of guidance would be most beneficial, and on which topic(s)?

Thank you very much for your time in answering these questions. On behalf of the study team, we appreciate your participation in the interview and helping us better understand procurement for Child Nutrition programs. As we review your responses we may reach out to you with short follow-up questions to make sure we have an accurate understanding of how your SFA conducts procurement.

Appendix C – SFA Procurement Practices Web Survey Debriefing Interview Guide

SFA Procurement Practices Web Survey Debriefing Interview Guide

Debrief of SFA PROCUREMENT PRACTICES WEB SURVEY

Date: ____________

SFA Site: ___________________

SFA Director Name: __________

Interviewee Name (if different from SFA Director): _____________________

Interviewer Name: ______________

Note-taker Name: _______________

Length of Debriefing Interview: _______ (minutes)

[note to Interviewer and note-taker: make sure to have the SFA PROCUREMENT PRACTICES WEB survey with you during the debriefing interview]

Hello, my name is [NAME]. I'm calling from 2M Research Services on behalf of the U.S. Department of Agriculture Food and Nutrition Service. May I speak with [respondent]?

[IF SPEAKING TO THE RESPONDENT, GO TO A]

[IF SPEAKING TO An administrative assistant, SAY:] Is there a direct line to reach him/her?

[IF YES, RECORD NUMBER IN SPACE BELOW. ENTER THIS NUMBER IN THE MESSAGE FIELD AT THE END OF THE CALL.]

RESPONDENT’S DIRECT PHONE NUMBER: _______________________________________

Is s/he available?

☐ YES [GO TO a]

☐ NO [GO TO b]

SCRIPT FOR WHEN RESPONDENT IS ON THE PHONE:

INTRODUCTION-1 [IF TRANSFERRED]

Hello, my name is [NAME] and I’m calling from 2M Research Services. [GO TO INTRODUCTION-2]

INTRODUCTION-2 [IF SPEAKING TO RESPONDENT]

Thank you again for helping us pretest our survey and interview materials for the “Study of School Food Authority Procurement Practices.”

We recently received your completed survey. Is now a good time to discuss your thoughts on it? This should take no more than 30 minutes.

☐ NO: Okay. That is not a problem. When would be a good time to call back?

Callback Date/Time: _____________________________________________________

[end interview, and inform respondent that you will try back at that date/time]

☐ YES: Great. Thank you.

[Note to timer/note-taker: Time the entire debriefing interview.

INTERVIEWER: BEGIN THE DEBRIEFING.]

RESPONDENT NOT AVAILABLE

Could you forward me to [REspondent’s NAME] voicemail so that I may leave a message?

☐ NO: Would you please leave a message for [REspondent’s NAME] mentioning that I called? Is there a good time to call back?

Callback Date/Time: _____________________________________________________

Voicemail

Hello, my name is [NAME] and I’m calling from 2M Research Services regarding the survey you completed for the “Study of School Food Authority Procurement Practices.” We sincerely appreciate your help in agreeing to pretest our survey and interview materials about your SFA’s procurement practices. We would like to speak with you to get your feedback on the survey. This should take no more than 30 minutes. Please call me at your earliest convenience at xxx-xxx-xxxx. Thank you.

Debrief of SFA Procurement Practices Survey

[Make sure to review the hard copy of the pretested survey prior to conducting the debriefing interview.]

For the pretest, I would like to get your feedback on the survey you completed. I’d like to ask you a few questions about any survey questions you thought were unclear or that were difficult to answer. Specifically, I am looking for your thoughts on the understandability and flow of the questions, and your perception of the time it takes to complete the survey. I would also like your opinion about the response choices provided, including whether we are missing response choices or have included some that are not appropriate or applicable.

Do you have the materials we sent you, the survey, the introductory letter and the Frequently Asked Questions, with you?

☐ NO: Would it be possible to get your completed survey so you can follow along during this debriefing?

☐ NO: that’s ok. we can continue without it.

Email Debriefing Questions

We sent you a letter describing the purpose of the survey with a list of Frequently Asked Questions. Did you find this letter helpful? Was anything in the letter unclear or confusing? What about the Frequently Asked Questions document? [probe for: reasons why and suggestions for improvement]

Survey Debriefing Questions

[Respondent’s name], can you tell us how long it took you to complete the survey (an estimate is fine)? Total time _________. Thank you.

Did you complete the survey all at one time, or did you come back to complete it at a later time? [Note to interviewer: If respondent indicates that s/he did not complete the survey all at once, ask respondent to elaborate why s/he did so. For example, did s/he run out of time? Need to look up information? Etc. probe for: reasons why and suggestions for improvement]

Did you ask others to help you complete the survey? If yes, who and why did you need that staff person’s help? [PROBE: on which sections of the survey did they seek help?]
Did you feel that the survey was too long, too short, or too complicated? [probe for: reasons why and suggestions for improvement]
What did you think of the flow of the questions? [probe for: specifics about the flow – order of question presentation, topic order, etc.] Do you have any suggestions to make the survey questions flow better?

What did you think about the how the questions were worded? Were any of the questions phrased in a way that was particularly difficult to follow? [probe for: reasons why and suggestions for improvement]

Were there any questions you had a hard time answering (either because you did not know the answer, or because you did not understand the question)? [probe for: reasons why and suggestions for improvement]

Were there any terms that were used that you are unfamiliar with? [probe for: specific terms] Do you have suggestions of other ways to phrase these terms?

Are there any questions you felt were repetitive? [probe for: reasons why and suggestions for improvement]

Section I. Solicitations and Contracts

The first section of the survey was about solicitations and contracts, including competitive solicitation methods to procure goods, services, and foods.

Regarding questions related to competitive solicitation methods, were any solicitation methods missing? [Q3] [probe for: reasons why]
Were there any factors that influence your competitive solicitation methods missing? [Q4] [probe foR: influencing factors]

Were there any types of contracts or contract provisions missing from the lists that we should consider adding? [Q5, 6, 10, 11] [probe for: which ones? How are the missing contracts defined?]

Were there any benefits or reasons for using certain contract types missing? [Q8 & 9] [probe foR: BENEFITS OR influencing factors]

Was the description of formal and informal procurement at the top of page 6 helpful? [probe foR: Other definitions of “formal” and “informal”]

The next questions in Section I asked about best methods for procuring food/goods/services.

Were any best practices for the procurement of foods, goods, and services not listed? [Q12-20, 23-34] [probe foR: Other procurement best practices]

Is there anything that you would like to mention/discuss about the questions related to 1) the best practices for the length of procurement contracts [Q27, 28],

2) issuing invitations for bids/requests for proposals for procurement contracts [Q29, 30],

3) product specifications [Q31, 32], or

4) local bonding requirements? [Q33, 34] [probe foR: methods/processes for each topic individually. Do not list all categories at once.]

Section II. Procurement Methods

[DEBRIEF ON THIS SECTION only if the respondent answered section II.]

Is there anything else we should have asked regarding the use of an FSMC? [Q1-10] [Probe: Please explain.]

Are there any other types of cooperative purchasing agreements we should ask about? [Q14] [Probe: what are they? Please explain.]

Is there anything else we should have asked regarding interstate or statewide cooperatives? [Q16-19] [Probe: Please explain.]

Was the question about locally sourced foods clear? [Q20]

[Probe: are there other procurement methods SFAs use to procure local foods?]

We do try to buy local, but it is not required to buy local. We want it to be price competitive and quality.

Was the question about procuring services clear? [Q23]

[Probe: should any other services be added? should any services be removed?]

Section III. Sourcing Foods

The third section asks about how your SFA sources foods, including USDA Foods and foods from local sources.

Are there any additional sources your SFA uses to procure services to convert raw or bulk USDA Foods into ready-to-use end products? [Q3] [Probe: please describe/explain.]

Are there any additional pass-through methods your SFA uses to obtain credit for the value of the USDA Foods in processed end-products? [Q4] [Probe: was the question clear and easy to understand.]

Are there any additional practices that your FSMC uses to support procurement from local sources? [Q7] [Probe: other promotion practices to support procurement from local sources.]

[review hard copy of survey and ask about any questions that appear to have been confusing or unclear.]

Section IV. SFA Planning and Monitoring

The fourth section of the survey asked about procurement planning and monitoring.

Were the steps we listed for the procurement process reflective of your process? Were any steps missing? [Q1-5] [probe for: Procurement methods.]

Were there any recordkeeping methods missing? [Q9] [probe for: recordkeeping methods.]

Were there any methods to identify accounting errors missing? [Q10] [probe for: methods TO identify accounTing errors.]

Is there anything we missed about how your SFA monitors contractor performance? [Q11] [probe for: methods SFA uses TO MONITOR CONTRACTOR PERFORMANCE.]

[review hard copy of survey and ask about any questions that appear to have been confusing or unclear.]

Section V. SFA Guidance and Training

The fifth and last section asked about guidance and templates when developing procurement solicitations.

Has any other guidance or templates been provided to your SFA for developing procurement solicitations? [Q3, 4, 6] [Probe: please describe/explain. From fns? from your state agency? from your lea? FROM THE INSTITUTE OF CHILD NUTRITION?]

Are there other procurement training topics that we should have included? [Q14, 15] [probe: Training for both pROCUREMENT and procurement solicitations. include training provided by fns, state agency, and lea]

[review hard copy of survey and ask about any questions that appear to have been confusing or unclear.]

General Feedback

We’re nearing the end of my questions. I just have a few more general questions. Please think about the overall experience of completing the survey.

Do you have any comments or suggestions for improving the overall understanding of the survey or adding topics that may have been missed? [probe for: SPECIFIC TOPICS OR QUESTIONS, PROVIDING INFORMATION IN ADVANCE TO PREPARE TO ANSWER QUESTIONS, SURVEY ORGANIZATION, ETC.]

Finally, we’d like to schedule a time for pretesting the interview component of the study. This is the final phase of the pretesting. This interview would be with our partner, Mathematica Policy Research. We would like to schedule this interview during the week of [insert appropriate week]. [open up Mathematica’s availability excel spreadsheet from the sharepoint.] The interview and debriefing should take no more than 2 and a half hours. What day and time works for you?

IF THE RESPONDENT INDICATES that THE SUGGESTED TIME frame CONFLICTS WITH THEIR SCHEDULE.

Ok, not a problem. There is also availability the week of [INSERT APPROPRIATE WEEK AND/OR DAY. AFTER A DAY IS GIVEN, PROVIDE THE AVAILABLE TIMES]. AFTER A TIME IS CONFIRMED, INSERT TIME AND DATE BELOW FOR MPR’S IDI.

Date: ____

Time: ___

Thank you for your feedback on the survey.

[end interview]

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Morgan Miller, MPIA
File Modified	0000-00-00
File Created	2021-01-21

Appendix I Procurement Practices - Final Memorandum on Pretest Results