Appendix A.10 Cognitive Pretest Memo

This memorandum describes the findings from pretesting the USDA Foods Distribution and Fees Survey, which was conducted by 2M Research (2M) in late May and early June 2020. The pretest was executed using a protocol that included a recruitment period, development of the cognitive testing instrument, and cognitive interviews with pretest participants (former State Distributing Agency [SDA] directors and staff) that assessed their comprehension of individual survey items. Information and feedback collected from the pretest will be applied to the study’s final web survey.

This memo first describes the methods used to execute the pretest, including (1) selecting pretest participants, (2) recruiting pretest participants, and (3) conducting the cognitive interviews. A discussion follows regarding the key findings of the pretest and study team recommendations for improving the web survey. The memorandum closes with the Subject Matter Expert (SME) review of the pretest memo, which captures the SME’s review and comments, along with any changes made to the pretest memo as a result of the SME’s review.

Pretest Methods

Respondent Selected for Pretest

Due to the study design of the web survey calling for a census of current SDA directors to participate in the study, this pool of potential respondents could not be used for the pretest. Instead, the pretest participants were identified internally by FNS. FNS Foods Distribution Division (FDD) selected five participants for the pretest from their internal contacts with previous SDA directors and director-level staff to invite them to participate in the pretest. The selected individuals divided into two subsets that included two current staff at FNS and three former SDA director-level or associate director-level staff. Both subsets of pretest invitees had recent program experience with USDA Foods.

Recruitment Procedures

In early May 2020, FNS FDD staff made initial contact with the individuals identified for the pretest to briefly describe the purpose of the study and the pretest while 2M concurrently developed a study invitation email for recruitment. This email described the study, the intent of the pretest, the pretest process, and associated time commitment for participating in the web survey and a cognitive interview. The FNS project Contracting Officer’s Representative (COR) for this study reviewed the draft invitation email, made suggested revisions and the invitation was finalized for use on May 21, 2020. 2M then sent the five pretest invitees the study invitation email to confirm their participation in the study and identify interview times for the pretest. Most pretest invitees responded to the pretest invitation email within 24 to 48 hours, affirming their participation in the study and identifying interview availability. Pretest participants then received a calendar invitation to the pretest interview and a follow-up confirmation and reminder email about the pretest a day prior to the scheduled interview time. The confirmation email further instructed participants to have their computer ready 30 minutes prior to the interview in an effort to avoid technical issues occurring during the interview time, and the email reminded participants to be prepared to answer the survey and interview questions based on their most recent State experience with USDA Foods. Exhibit 1 presents the characteristics of the participants recruited for the pretest.

Exhibit 1 - Pretest Respondent Characteristics

Participant Title	Most Recent State Affiliation
Former SDA Director	Missouri
Former Assistant SDA Director	Colorado
Former SDA Director	Connecticut
Program Analyst, Program Integrity and Monitoring Branch (former SDA Director)	Florida
Branch Chief, School Nutrition Programs Branch Special Nutrition Programs Division	Florida

Pretest interviews took place from May 29, to June 3, 2020. Exhibit 2 presents the communication documents and schedule used in the pretest recruitment process. Given the iterative nature of the recruitment process, there was some overlap in the timing of the various communications.

Exhibit 2 - Pretest Communications and Schedule

Pretest Recruitment Communication	Date Started	Date Sent
Email Invitation Letter to Selected Pretest Participants	May 21, 2020	May 24, 2020
2M Survey Appointment Confirmation Email	May 22, 2020	June 1, 2020
2M Email Reminder	May 28, 2020	June 2, 2020
2M Pretest Email: Survey Link and Survey PDF	May 29, 2020	June 3, 2020

Cognitive Interview Procedures

The goal of the cognitive interviews was to understand how respondents are interpreting the survey questions and what they think about when selecting a response. The cognitive testing focused on assessing the clarity and comprehension of the survey items and response options. About 45 minutes before each scheduled interview, 2M emailed a web survey link and a PDF version of the survey to the respondent with instructions not to view the content before the interview. At the beginning of the pretest, respondents were instructed to answer the web survey questions from the perspective of the State whose information and processes they were most familiar with. When respondents encountered questions for which they lacked specific factual information (e.g., access to financial data), respondents were asked to come up with a reasonable answer to the best of their ability. Respondents then completed the web survey during their interview appointments. After submitting the web survey, respondents were then cognitively debriefed on their responses to the survey instructions and questions using a combination of retrospective probing and think-aloud interview techniques.

With participants’ permission, 2M recorded the interviews to allow the interviewer to focus on the discussion and to support the analysis of the pretest interviews. A set of backup notes was developed for each interview by a notetaker over the course of each interview. Three of the interviews took 90 minutes to complete, while two of the interviews were completed within 75 minutes.

During each pretest interview, 2M used a series of scripted interview probes to obtain detailed insights into how respondents conceptualized and determined their responses to the web survey items. The pretest protocol (Appendix A) included a set of generic probes, as well as probes for specific questions to help discern cognitive clarity, question phrasing, format, and terminology. The interview probing was designed to identify specific cognitive issues respondents encountered, including issues of clarity, comprehension, recall, and sensitivity and to also obtain respondents’ insights into potential improvements. Over the course of each interview, 2M utilized these probes and instructed participants to think aloud and describe in detail their thoughts and how they arrived at an answer as they were presented with their responses to each of the survey questions. This process enabled the study team to understand participants’ response processes, including their interpretation of the questions, response selection strategies, and clarity of response options. In addition, 2M used these probes to follow up on any difficulties participants had with questions or response options, as well as to elicit suggestions for improvements.

Pretest Findings and Recommended Revisions

This section summarizes the pretest findings and revisions recommended in response to participants’ feedback. Findings from the cognitive pretest of the data collection instrument and recommendations for improvement are presented first, followed by a summary of the estimated response burden.

Findings from the Cognitive Interviews

Pretest participants generally thought the web survey was clear and covered most of the expected topics related to understanding and comparing fee structures for the distribution, warehousing, and administration of USDA Foods for school meals. However, participants did find a few questions difficult to answer, including a series of items for which the interview team observed participants struggle to select a response. The discussion below identifies the questions for which respondents specifically encountered difficulties. It presents summarized findings for these specific questions and a recommendation for revision based on the qualitative analysis of respondents’ challenges in answering these questions.

Introduction and “Which State do you represent?” Instructions. Most participants felt the introduction and instructions were clear and that SDA directors would have no difficulties understanding the topics the questionnaire covers. At the beginning of the web survey, respondents are asked to provide their name and contact information for the purposes of follow-up on answers they provided. There was an outstanding question regarding whether the survey requires input from more than one respondent and how SDA directors might complete this information request. However, pretest respondents indicated that even if SDA directors consulted others to complete the survey, they would only provide their name and contact information in response to this instruction. 2M recommends no changes be made to the Introduction and “Which State do you represent?” instructions.
Question 1. Overall, respondents felt that the response options for this question were not inclusive enough, as the response options did not provide an opportunity to indicate a choice related to delivery or a distributor. Three of the five participants indicated that the addition of such an option (State-owned or State-provided distributor) would allow respondents to more accurately reflect the operations within their States. 2M recommends addition of State-owned distributor and State-provided distributor as response options.
Question 3. The question is unintentionally misleading because the current wording does not direct respondents to think specifically about other programs that are associated with the National School Lunch Program (NSLP). All of the respondents initially selected no for this question, but then changed their answers after reviewing the response options associated with the following question, 3A. Most commented that because the question stem only referenced The Emergency Food Assistance Program (TEFAP), they did not think about programs under the umbrella of NSLP. Respondents indicated that it would be helpful to have additional examples of programs that could be administered with NSLP in the question stem. 2M recommends including another program example, specifically Summer Food Service Program (SFSP), in the question stem to give respondents the correct frame of reference, so they can provide an accurate response. It may be technologically feasible to display the list for 3A for the question as well to encourage the correct frame of reference for a response.
Question 6. Three respondents were familiar with providing both a case and dollar amount as a response to this question. However, some respondents thought that “dollar value” was not specific enough terminology, as they wondered whether what was being asked for was the commercial price, the value of donated foods, or the processing. “Dollar value” was also confusing terminology because there could be different minimum dollar values across items. Some respondents also noted that while most States have a minimum drop amount and/or value for delivery from State facilities, there are some States that do not. These respondents suggested adding “no minimum” as a response option to provide an answer option for these States. 2M recommends FNS clarify their intended information need for dollar value by including a brief definition to clarify this as the invoice amount or the sales minimum. 2M also recommends adding “no minimum required” as a response option, which will only be used when the respondent selected NA or provided any response for dollar value, pounds or cases.
Question 7. Respondents had a somewhat difficult time answering this question. Most believed the term “fee” to be ambiguous, as it could reference a distribution fee, a storage fee, or both types of fees. Some interpreted the question to be referencing only storage fees, while others included warehousing/ distribution and delivery fees in their rationale for response selection. As a result, respondents were uncertain of their answers. Because the terminology “fees” is not well understood in this question, respondents were not able to select the full range of response options to best represent their State operations. Some respondents also felt that “dwell time” is not an appropriate response option for a question on delivery fees. They suggested the addition of “distance from facility” as another response option. Depending on the original information need and intent of USDA for this question, 2M recommends that the question be revised to define the specific fees (storage fees or delivery fees or both) of interest. Additionally, if the question’s intended focus is both types of fees or delivery fees only, the response options should be revised to include “distance from facility.”
Question 9a. Respondents understood the question but believed the response options were not the best choices, as they did not reflect the frequency at which States reallocate direct delivery products across Recipient Agencies (RAs). All of the respondents selected more than 4 times a year for this question; however, their answers ranged from 8 times a year to close to 100 times per year. While the “more than 4 times a year” option could be selected, it was not the most accurate or precise response. Because the intervals in the response options are incorrect according to respondents, FNS will not be able to correctly capture the actual frequency at which States reallocate direct delivery products across RAs. 2M recommends revising the answer options to be “weekly,” “biweekly,” “once a month,” “quarterly”, “annually,” and “on demand” to better capture the frequency of this process. Additionally the question stem should be revised to read “How often do you reallocate direct delivery products across RAs?”

Questions 12-12a. Respondents described a range of circumstances that made answering question 12 challenging, including the organizational structure of state agencies, the requirements for a state match to use State Administrative Expense (SAE) funds, and some confusion about the subcategories listed in question 12a. Overall, respondents generally understood the definition provided prior to question 12, which explained SAE funds. Respondents also generally understood the question to be asking for information on how States distribute their SAE funds and how states are going to spend those funds, but respondents struggled to come up with a suitable answer. The issue with this question is that this question’s cognitive burden requires SDA directors to sort out various circumstances around matching dollars to arrive at an answer. Additional challenges included how to arrive at an approximate amount for question 12. Respondents confirmed their answers would vary greatly without a specific year for which to derive the average amount included in the question. Some respondents provided an average based on several years and their experience, while others provided either a response based on the last fiscal year or on the last approved SAE plan. Some respondents also commented that SDA directors would need to consult with their accounting departments to develop their responses to this question.
Many of the pretest respondents also struggled with determining how to allocate the approximate budget amount reported for question 12 across the options for question 12a. Some respondents seemed to think that staffing and salaries should not be part of the response options based on their experiences with SAE funds. Others thought that the answers were difficult to arrive at due to the differences in where child nutrition programs and food distribution programs might be housed within state agencies. For question 12, 2M recommends that FNS decide whether the most accurate information for its purposes should be based on fiscal year or what was reported in the most recent SAE plan/525 form. Respondents noted that asking for data from the last fiscal year that has been completely closed out might be the most straightforward way to frame this question, thus the language used could be “In school year (SY) 2018-2019, approximately how much SAE did the State agency allocate (not including SAE reallocation dollars) to support costs associated with the administration, storage, and distribution of USDA Foods in Schools?” Depending on FNS’s decision, the explanation of SAE funds preceding question 12 should include additional clarifying language, such as, “Use your State’s most recent SAE Plan/525 form as the basis for this response,” or, “Data should be for the most recent fiscal year (SY 2018-2019).” 2M recommends including instructions referencing the last approved SAE plan as a basis for answers to question 12a. Even with these changes, 2M notes that the question may still be challenging for respondents to answer due to the complexities of how states and their contractors might calculate the expense categories.
Question 13. Some respondents found this question difficult to answer. While most understood the question and the concept of “distribution charge” as it was explained, some struggled with how to develop a response based on factors such as the variability in their state’s fee payment structure, such as differences between public and private entities (schools). Another respondent commented they were not sure whether warehouse contracts or distribution contracts would come out of SAE specifically, and the respondent found the question confusing for this reason. Another respondent commented that the question was complex because a response would vary depending on what kinds of services the state had access to. 2M recommends clarifying how respondents should think about differences in their payment structure to select a yes response if fees are collected for at least some entities. An open-ended response should be added to allow states to explain what else they are charging RAs for. In further discussion, FNs determined that the question should read as” Answer this question based on whether the distribution charge is used for any type of RA” and chose not to add the additional open-ended response option.
Question 14,14A,14B. The question was universally ambiguous to respondents. Respondents did not know what services were being referenced in the question or whether to focus on fees associated with storage or delivery. While some tried to determine what the question was asking from the response options listed for Q14, no respondent could answer this question. The question is conceptually vague and, as currently presented, impossible for respondents to answer. 2M recommends additional language to clarify USDA’s intent in terms of the specific services being asked about, as well as the types of fees that are of interest. These modifications will clarify the question. The study team is not able to make a recommendation for the specific word modification for this question because we do not know the intent of the question from USDA’s perspective. Based on review of the pretest memo, FNS FDD determined that the question should be revised as “Consider the fees or “distribution charge” that you just described in the previous questions. Are storage and/or transportation services included in the fees charged directly from State-contracted facilities?”. Additionally, FNS indicated the question should be skipped by those answering no to question 13, and that question 14B should include the response option “RAs are invoiced for transportation only” for further clarification of this question.

Estimated Web Survey Burden

Based on the administration of this web survey in 2016, FNS indicated at the outset of the pretest an expected completion time of approximately 20 minutes. During the pretest, 2M observed the survey start and completion times for participants as they finished and submitted the web survey. On average, the survey required 19 minutes and 35 seconds. It should be noted that because the testers are not current SDA directors, their response times could only approximate the burden that actual respondents might experience. The estimated completion times do not account for possible information retrieval efforts on certain questions for which participants were not able to provide a response due to lack of access to financial records. It should be noted that, for these questions, some respondents stated that assistance from their SDA accounting department would be needed to provide a response.

SME Review

2M’s SME reviewed both the pretest memo and the revised web survey instrument and did not request any substantive differences. A minor update was made to the revised web survey as a result of the SME review. The SME did provide additional technical insight and comments about the challenges encountered by respondents for some of the question items discussed. These additional insights, presented below, support the recommendations made by 2M and provide additional technical insight into the cognitive issues described previously in this memo.

Question 9. Reallocations could take place several times daily and having a wide answer range between monthly and annually may make it difficult to obtain clear data. FNS may need to consider adding “quarterly” as an answer option: it may allow more accurate data to be obtained.
QuestionS 12-12A. The intent of the question is clear but obtaining accurate information from SDAs will be difficult based on each unique situation. The complexity of the expense breakdown may be hindered by obtaining accurate information from contractors or other agencies. Especially in state owned, state operated distribution facilities, determining the breakdown between salaries and operating expenses (truck leases, repairs and maintenance, etc.) may be more cumbersome. Additionally, for state contracted, privately operated facilities, the contractor may not be willing to share this information (or be able to accurately determine how these costs break down since most employees have several duties which may not all pertain to the USDA Foods distribution contract.) There are also states that distribute both USDA Foods and Community Feeding Programs (TEFAP/CSFP) simultaneously. Obtaining accurate data would be increasingly difficult to differentiate between these programs. Further complicating this issue are states where Child Nutrition Programs are housed at the Department of Education, but the Food Distribution Program is located at another agency, such as Department of Agriculture. Some Food Distribution Programs must have funds transferred and are not always fully funded. The answers may vary widely depending on how each respondent interprets and calculates the expense percentages.
Question 14. The intent of this question should be clarified. Overall, it seems that USDA is trying to determine the billing methodology in each state and what organization is billing for services. If this is the intended information requested, it may be clearer to separate these questions into two separate questions: what services are being billed, and who is billing for the services. 2M has not made these changes to the revised instrument as USDA would need to still confirm the intent of this question.

Based on the SME review, the 2M team reviewed the response options further for question 9A for a possible response option between monthly and annually. While 2M suggests that in cases where the reallocation process happens more frequently than monthly, respondents could select the newly proposed “on demand” response option, but the survey was modified further to add the quarterly option as suggested by the SME.

No additional SME recommendations were made to for the web survey instrument.

Appendix A: Cognitive Interview Pretest Protocol

USDA Foods Distribution and Fees Survey Cognitive Interview Guide

INTERVIEWER INTRODUCTION: Hello, my name is _________ and I am a researcher at 2M. I’m on the call with ______ another researcher at 2M who will provide technical assistance if needed and supports notetaking. Thanks again for agreeing to help us test the survey questions for the USDA Foods Distribution and Fees Survey on behalf of the U.S. Department of Agriculture, Food and Nutrition Service (FNS). The objective of the study is to describe in detail how State Distributing Agencies (SDAs) manage the operations, storage, and distribution of USDA Foods to school food authorities (SFAs) including associated fee structures, warehouse operations, and use of State Administrative Expense (SAE) funds. The study includes a web survey that will be administered to all SDA directors later this year.

First, I want to make sure you received our email with the survey link and pdf version of the survey. We have included some simple instructions to help you share your screen with us as well.

We’d like to ask that you share your screen with us first.

As we have shared previously, during the pretest, we are asking that you respond to questions from the perspective of the state whose information and processes you know best. We anticipate that there will be some questions for which you do not have information. For these items, we are asking you to come up with the best answer you can. After you complete the web survey, we will ask you to open the pdf as we ask you about how you arrived at your answers.

Please begin the web survey when you are ready.

[AFTER RESPONDENT COMPLETES WEB SURVEY] Thank you for completing the survey. Were you able to print out the pdf or do you have it on your computer screen? Now that you have completed the survey, we’d like to ask you some questions about how you arrived at your answers. We are interested in what you were thinking about to come up with your answers and be as detailed as possible in your response, including your thoughts and feelings on how easy or difficult the question was to understand. We want to hear when you think the questions are unclear, hard to follow, or when the response choices just don’t seem to fit for you. There are no wrong answers: please share as much as you can about what you were thinking about when you answered the questions.

With your permission, I’d like to record the interview so that we have a complete record of your responses. Even though we are taking notes, we’d like to make sure we’ve captured everything you shared in this interview. Everything that you tell me is confidential and will be kept private. Do I have your permission to record the interview?

Y/N

Alright, we will begin the interview.

Shape2

LIST OF GENERAL PROBES

INTERVIEWER INSTRUCTION: To be asked as needed for each question to understand respondent comprehension, clarity and recall:

Tell me what you were thinking about when you were coming up with your answer to the question?
Can you tell me more about this?
In your own words, what is the question asking?
How sure are you of your answer? What makes you sure of your answer?
How difficult was the question to answer? What made the question difficult to answer?
Please tell me what you think about the answer choices:
1. Are any of the choices hard to understand? What makes them hard to understand?
2. Are there any answer choices that you think don’t belong on this list? Why don’t they belong?
3. Are there any answer choices that are missing? What would you add?
What led you to select Don’t Know/No for that question?

INTERVIEWER INSTRUCTION: Tailored probes for specific questions are in light blue.

INTRODUCTION SCREEN

USDA Foods Distribution and Fees Survey

Thank you in advance for taking the time to complete this survey.

The United States Department of Agriculture (USDA) Food and Nutrition Service (FNS) has contracted with 2M Research (2M) to conduct a survey of State Distributing Agencies (SDAs). The study will allow FNS to better understand and compare fee structures for the distribution, warehousing, and administration of USDA Foods for school meals.

The survey should take approximately 20 minutes to complete.

If you have any questions about this study, please contact the study team by email at [email protected] or by phone at 1-8XX-XXX-XXXX (toll-free), Monday through Friday from 10a.m. to 6 p.m. Eastern Time. If you call outside of these hours, please leave a message, and we will return your call the following business day.

P: Are the instructions clear? Why or why not?

P: (If needed based on previous response): Is there enough detail here for SDA Directors to know what they are being asked for in the survey? Why or why not?

This information is being collected to assist the Food and Nutrition Service in understanding fee structures for the distribution, warehousing, and administration of USDA Foods for school meals. This is a voluntary collection and FNS will use the information to better understand and compare how State Distributing Agencies manage the operations and distribution of USDA Foods to School Food Authorities . This collection does not request any personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 0.33 hours per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, Room 555, Alexandria, VA 22314 ATTN: PRA (0584-0613). Do not return the completed form to this address.

INSTRUCTIONS

This survey has been optimized to run on a desktop computer and is best viewed in the latest versions of Chrome, Firefox, or Internest Exlporer.

Move forward through the questions by clicking on the “Next” button on the bottom right.

If you need to go back, click the “Back” button on the bottom left.

To exit the survey and come back to it later, click the “X” on the top right. All responses will be saved.The survey can be accessed again by clicking on the survey link that was provided to you in the email notifying you about the survey.

When you reach the end of the survey, click “SUBMIT.”

Please provide the following information:

Which State do you represent?

[Provide a drop down list that includes all states, the District of Columbia, Puerto Rico and U.S. Virgin Islands in alphabetical order]

Please provide your name, email address, and telephone number for any follow- up questions about the survey:

Name: __________________________________

Email Address:____________________________

Telephone Number:________________________

The web survey first asks you to provide some contact information as the respondent.

P: How easy or difficult are these instructions to understand?

P: Are there any reasons why providing a name and contact information as requested would be difficult?

P: Having just completed the survey, do you think SDA Directors will be able to answer all of these questions, or will they need to get information from others on their staff?

P: What would you do if multiple respondents are needed to answer these questions?

What type of distribution facility, facilities, or warehouse does [State] contract with? Select all that apply.

State-owned warehouse(s)
State-contracted commercial distributor
State-contracted commercial warehouse
State-contracted non-commercial warehouse (e.g., a food bank)
Other, please describe:__________________

None. All USDA Foods are delivered directly to Recipient Agencies (RAs), to processors, and/or by USDA DoD Fresh distributors.

P: What is the difference between state-owned and state-contracted?

How many warehouses/distribution facilities does the SDA manage for USDA Foods in Schools? Please include all facility types.

P: What different facility types were you thinking about as you answered this question?

P: How difficult was it to come up with a number of facility types for this question?

Do any of the SDA facilities store and distribute for other FNS programs, such as TEFAP, in the same facility/facilities with USDA Foods in Schools?
- Yes

3a. [if “yes”] Which program warehouses are co-located? Select all that apply.

Summer Food Service Program (SFSP)
Child and Adult Care Food Program (CACFP)
Commodity Supplemental Food Program (CSFP)
The Emergency Food Assistance Program (TEFAP)
Food Distribution Program on Indian Reservations (FDPIR)

P: Should this question provide more examples of other FNS programs that SDA facilities might store and distribute for to answer the question?

P: Are there enough examples of other FNS programs to help SDA directors consider them all in their response to this question?

Does the SDA separate the State into regions for warehousing and/or distribution?
- Yes

4a. [If “yes”] How many regions? [Drop down number list 1-30]

P: Did you think only about warehousing distribution or both of these when you answered the question?

P: How difficult was it to come up with a number of regions for this question?

Is transportation to Recipient Agencies (RAs) included in the contract for the state facility/facilities?
- Yes

5a. [If “yes”] How many times per month are items delivered?

As needed
Weekly

5b. [If “no”] How do RAs receive USDA Foods?

RA pick up

RA procures transportation services separately

Both, items are delivered to RAs and RAs arrange transportation.

5c. [If “both”] How many times per month are items delivered?

As needed
Weekly
Biweekly
Monthly
Bimonthly

IF R ANSWERS YES: PROBE ITEM OPTIONS FOR FIT

IF R SELECTED BOTH: PROBE FOR RATIONALE FOR WHY AND HOW THEY SELECTED A RESPONSE OPTION, BASED ON RA DELIVERY OR PICKUP

P: Please tell me what you think about the answer choices:

Are any of the choices hard to understand? What makes them hard to understand?
Are there any answer choices that you think don’t belong on this list? Why don’t they belong?
Are there any answer choices that are missing? What would you add?

What is the minimum drop amount and/or value for delivery from your State facilities? Select all that apply, and indicate the amount and/or value required.

____ Dollar value
____ Pounds
____ Cases

Not applicable: State facility/facilities does not deliver food.

P: What does minimum drop amount and/or value for delivery mean to you?

Do the fees charged vary based on any of the following? Select all that apply.

Region of the State
Type of storage (i.e., freezer, cooler, or dry)
“Dwell time” permitted to withdraw USDA Foods from the State warehouse or distributor
Other, please specify: _________

Fees charged do NOT vary

P: What does dwell-time mean to you?

P: What led you to select Fees charged do NOT vary for this question?

What types of warehouse/distribution does the SDA allow Recipient Agencies (RAs) to use for receiving USDA Foods? Select all that apply.

RA/Co-op contracted commercial distributor
RA/Co-op contracted commercial warehouse
RA/Co-op contracted non-commercial warehouse (e.g., a food bank)
Direct ship to RA-owned warehouse
Other, please describe:_________________________

None

P: How did you define warehouse/distribution in this question? What did that mean to you?

P: In your own words, what is the question asking?

Does your State have a process for reallocating USDA Foods from one RA to another for direct delivery products?

9.a. [If “yes”] How many times a year do you reallocate direct delivery products across RAs?

Annually
Twice per year
Three times per year
Four times per year
More than four times per year

9.b. [If “no”] When do you do move inventory from one RA to another? Select all that apply

January
February
March
April
May
June
July
August
September
October
November
December

If R ANSWERED YES:

P: Please tell me what you think about the answer choices

P: Are there any answer choices that are missing? What would you add?

10. As the SDA, do you have a policy where you reallocate bulk pounds at processors from RAs with high inventory balances to other RAs, often known as a sweep policy?

10.A [If “yes” to Q10] How many times a year do you sweep RA pounds?

Annually
Twice per year
Three times per year
Four times per year
More than four times per year

10.A.2 [If “yes” to Q10] What time of year do you do your sweep? Select all that apply.

January
February
March
April
May
June
July
August
September
October
November
December

P: How useful is the definition of sweep policy as presented in this question? Is this how SDA Directors understand this concept? Is there a better definition of sweep policy?

If R ANSWERED YES:

P: Please tell me what you think about the answer choices

P: Are there any answer choices that are missing? What would you add?

11. Do RAs enter orders directly in the web-based supply change management (WBSCM) system in your State (i.e. is WBSCM rolled down)?

Yes, all RAs enter order in WBSCM.
Yes, but only some RAs enter orders in WBSCM.
No, no RAs can access WBSCM.

State Administrative Expense (SAE) funds are allocated to each State based on a statutory formula to cover expenses for the administration of Child Nutrition Programs. However, States have some discretion in allocating SAE funds across individual program administrative costs within their State.

12. On average, approximately how much SAE does the State agency annually allocate (not including SAE reallocation dollars) to support costs associated with the administration, storage, and distribution of USDA Foods in Schools?

PLEASE FILL IN APPROXIMATE AMOUNT

12.a. As reported above, if SAE funds are used, approximately what percentage of the funds are used for the following:

Warehousing __________%

Distribution (for example, trucking) __________%

Staff Salaries _________________%

Training____________%

Other, please specify____________ __________%

P: An explanation of State Administrative Expense (SAE) funds is provided before answering this question. How helpful is that definition?

P: Given this explanation, how would an SDA Director approach answering this question?

P: How difficult is it to come up with an approximate amount for this question?

P: For what timeframe would you respond for, this year, the last fiscal year or something else?

P: What led you to select OTHER PLEASE SPECIFY for this question?

When SAE or other state funds are insufficient to cover the full costs of administering the Food Distribution Program (i.e., expenses for the storage, transportation, or associated administrative costs of USDA Foods in Schools), States are authorized to charge a fee to RAs participating in the program to close the funding gap. This fee is officially termed the “distribution charge,” but States often refer to it by different names (e.g., administrative fee, storage fee), and it may include fees charged directly to RAs by State-contracted warehouses.

13. Does your State charge such a fee to RAs?

13.A [If “yes” to Q13] What is the basis for the fee? Select all that apply.

Pounds of direct delivery, processed end product, and/or commercial product an RA sends through the State warehouse
Cases of direct delivery, processed end product, and/or commercial product an RA sends through the State warehouse
SFA meal counts
Other method

13.B [If “yes” to Q13] How do you collect the fees? Select all that apply.

Invoices sent directly to the RAs by the State
Invoices sent directly to the RAs by the State-contracted warehouse
Administrative offset to NSLP meal reimbursement

Other, please describe:_________________

13.C [If “yes” to Q13] In an average year, what is the approximate total amount of these fees collected by your State (and/or by the State-contracted warehouse) from all RAs for the storage and distribution of USDA Foods (not including any reimbursement due to SAE re-allocation)?

[$XX,XXX,XXX] PLEASE FILL IN APPROXIMATE AMOUNT

13.D [If “yes” to Q13] As reported above, if a “distribution charge” is used, as defined above, approximately what percentage of the funds is used for the following:

Warehousing__________%

Distribution (e.g., trucking, etc.)_____________%

Staff salaries________________%

Training______________________%

Other, please describe___________ _______________%

13.E [If “yes” to Q13] Since SY15-16, how many times were State distribution fees changed in consultation with the FNS Regional Office?

P: How helpful is the definition of distribution charge that is provided before answering this question?

P: How would you improve it? Is there a better way to describe the distribution charge?

P: Given this explanation, how would an SDA Director approach answering this question? Can you tell me more about how you arrived at your answer?

P: How readily available is the information needed to answer this question?

PROBE THE RESPONSE OPTIONS IF R SAID YES TO 13 A:

P: How did you understand the concept basis for the fee? Can you define that in your own words?

P: Please tell me what you think about the answer choices:

Are any of the choices hard to understand? What makes them hard to understand?
Are there any answer choices that you think don’t belong on this list? Why don’t they belong?
Are there any answer choices that are missing? What would you add?
IF R SELECTED OTHER: What is the other method you were thinking about?

PROBE THE RESPONSE OPTIONS IF R SAID YES TO 13 B:

P: Please tell me what you think about the answer choices:

Are any of the choices hard to understand? What makes them hard to understand?
Are there any answer choices that you think don’t belong on this list? Why don’t they belong?
IF R selected OTHER: What is the other option for collecting fees you were thinking about?

IF R ANSWERED Y TO 13C:

P: Please tell me what you think about the answer choices:

How did you come up with your answer to this question?
How difficult is it to come up with an approximate total amount of these fees collected by your State (and/or by the State-contracted warehouse) from all RAs for the storage and distribution of USDA Foods (not including any reimbursement due to SAE re-allocation)?

IF R ANSWERED Y TO 13D:

P: Please tell me what you think about the answer choices:

How did you come up with your answer to this question?
How difficult is it to come up with a percentage?

IF R ANSWERED Y TO 13E:

P: How did you come up with your answer to this question?

14. Are services included in the fees charged directly from State-contracted facilities?

Yes, for all facilities:

14.A. [If “yes for all facilities”] For what are the RAs invoiced?

Storage and transportation
Storage only
Transportation only

Yes, but not for all facilities:

14.B [If “yes but not for all facilities”] For what are the RAs invoiced? Select all that apply.

RAs are invoiced for storage only.
RAs are invoiced for storage and transportation.
The SDA pays the facility directly and later bills the RA.
The facility is State- owned/-leased.

14.C [If “no”] How does the RA receive the USDA Foods?

The SDA pays the facility directly and later bills the RA.
The facility is State- owned/-leased.

Other, please describe:________________

P: Tell me what you were thinking about when you were coming up with your answer to the question?

P: How difficult was the question to answer? What made the question difficult to answer?

If R ANSWERED YES to 14A or 14 B:

P: What was your answer to this question?

P: Please tell me what you think about the answer choices:

Are any of the choices hard to understand? What makes them hard to understand?
Are there any answer choices that you think don’t belong on this list? Why don’t they belong?
Are there any answer choices that are missing? What would you add?

P: How difficult was the question to answer? What made the question difficult to answer?

15. Does the “distribution charge,” as defined in Question 13, change based on storage time, sometimes known as dwell time?

15.A. [If “yes”] When do fees begin accruing?

Immediately
1-10 days
11-30 days
31-60 days
More than 60 days

P: What was your answer to this question?

P: Tell me what you were thinking about when you were coming up with your answer to the question?

P: What did “dwell time” mean to you?

16. Does your State, or do RAs in your State, send USDA Foods to processors for further processing?

P: What does “further processing” mean?

17. For the management of processing of USDA Foods at the State level: Which value pass-through methods does your State allow? Select all that apply.

Indirect discount (i.e., Net off invoice)
Direct discount
Rebate
Fee-for-service (including modified fee-for-service)

P: What is a value pass through method?

P: How helpful were the definitions provided for the response options? How would you improve the definitions provided?

P: How difficult was the question to answer? What made the question difficult to answer?

18. Does the State agency procure processing services?

18.A [If “yes”] Do RAs also procure processing services from additional processors?

18.B [If “no”] Do RAs procure processing services?

P: What does procure processing services mean to you?

P: Are there any answer choices that are missing? What would you add?

INTERVIEWER: We have completed the pretest interview. We’d like to thank you again for your time and responses. Your feedback will help us improve the survey so that it meets its purpose. Thank you again and have a great day.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	QRS USDA Foods Distribution and Fees Survey: Pretest Memo
Subject	AG-3198-C-15-0004
Author	Joshua Townley
File Modified	0000-00-00
File Created	2021-01-13

Appendix A.10 Cognitive Pretest Memo