Appendix F_Sample Pretest

APPENDIX F

SAMPLE PRETEST

OMB Control Number: 0584-XXXX
Expiration Date: XX/XX/20XX

PRETEST DEBRIEF PROTOCOL

RESPONDENT NAME:
RESPONDENT TITLE:
SFA NAME:
STATE:

Thank you for taking the time to speak with us today for the pre-test of the Operational Challenges in Child Nutrition Programs Survey. Do you have a copy of the survey in front of you? I also have a copy of your survey responses too, for reference.

If no: It will help our discussion if you have it in front of you. I can give you a minute to get it. (I can also email you a new copy of the survey now if necessary.)

This call should take no more than 45 minutes. Our goal today is to get your feedback on the survey, particularly if anything was confusing or difficult to answer, and your suggestions for revisions. We will first talk about your reactions to the survey overall, and then we will ask about specific survey questions.

Do you have any questions for us before we begin?

1. About how long did it take you to answer the entire survey, including time spent gathering information?

   1. Did any questions take particularly long to complete?

2. Now I want to know what you thought of the questions overall.

   1. What did you think of the organization and flow of questions?

   2. If any questions left blank: I see you left questions [numbers] blank. Can you tell me why?

   3. Did the survey use any terms you recommend changing or that you think we should define?

      

      The Food and Nutrition Service (FNS) is collecting this information to better understand the impacts of supply chain disruption on school districts nationwide. This is a mandatory collection. FNS will use the information to develop tailored resources, tools and flexibilities to support school districts in serving students healthy and nutritious meals. This collection does not request personally identifiable information (PII) under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to complete this information collection is estimated to average 50 minutes (0.8333 hours) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22314, ATTN: PRA (0584-XXXX). Do not return the completed form to this address.

      
      
      

   4. Is there any information or instruction you think we should add to the introduction to help SFAs understand the purpose of the survey or how to complete it?

3. In Question 2, do the response options encompass all possible SFA operations?

4. Do the response options provided in Question 3 make sense? (Western Region: Are they inclusive of SFAs in AS and CNMI?)

5. Looking at the response options for Question 4:

   1. Do we want to include a reference point? In other words, do we want to know if these challenges have emerged since pre-pandemic years, since last school year (SY 2021-2022), or simply whether they are present in the current school year (SY 2022-2023)?

   2. Are there additional challenges you would suggest we include, or existing challenges you suggest we reframe?

   3. Are there challenges particular to Farm to School not currently reflected in these options?

6. Looking at Question 6, will SFAs be able to answer questions about unpaid school meal debt as early as September, and is there anything else we should know about this issue?

7. Looking at Question 7:

   1. Does the current list of response options feel too long or burdensome?

      1. If yes, would it help to combine or eliminate some of the response options? If the latter, which could be eliminated?

   2. Are there additional strategies you would suggest we include, or existing strategies you suggest we reframe?

8. Looking at Question 8:

   1. Does the current list of response options feel too long or burdensome?

      1. If yes, would it help to combine or eliminate some of the response options? If the latter, which could be eliminated?

   2. Are there additional impacts you would suggest we include, or existing strategies you suggest we reframe?

   3. Is it clear what the response option “leveraging state contracts” means in this context, and would terms such as “piggybacking” help to clarify?

9. Looking at Questions 10, 12, and 14: For each of these questions, is it easier for SFA Directors to respond using one of the options provided, or to have a blank space to provide their own estimate (percentage increase/decrease)? Alternately, would it be easier to provide actual participation numbers so that we could calculate percentage increase/decrease?

10. Looking at Questions 11, 13, and 15: Are there other response options that should be included?

11. Looking at Question 19:

    1. Will SFAs be able to distinguish between federal procurement regulations (choice a) and state procurement regulations (choice b)?

    2. Is there any redundancy in the response options provided?

    3. Are there other response options you would suggest?

    4. Would you like to ask a similar question for any other challenges (e.g., insufficient bids)?

12. Are there other challenges (from response options in Questions 4, 5, 6 or elsewhere) that you’d like to see included in questions similar to Question 20 and Question 21, which assess changes in practices such as local food purchasing and scratch cooking?

13. Looking at Question 23, what other response options would you suggest we provide for why SFAs may not have accepted their Supply Chain Assistance fund?

14. One final question: We need to report whose feedback we solicited when we submit this survey to OMB for approval to distribute to all SFAs. Is it okay if we include your name, title and contact information in our list of pretest participants when we submit that document?

Those are all the questions I had for you. Is there anything else you’d like to add?

Thanks again for your feedback. It will be very helpful as we make our revisions.

2023 PULSE SURVEY: OPERATIONAL CHALLENGES

IN CHILD NUTRITION PROGRAMS

INTRODUCTION

In recent years, school nutrition professionals have met extraordinary challenges to ensure every child can get the food they need to learn, grow, and thrive. As State Agencies (SAs) and School Food Authorities (SFAs) continue to work together to administer and operate Child Nutrition Programs, the US Department of Agriculture (USDA) seeks to better understand the critical issues SAs and SFAs are facing.

This survey is mandatory and is intended to ensure the USDA has national, representative-level information to assess the scope, reach, and variation in challenges and mitigation strategies. The survey focuses on current operational challenges in school food service, including those related to school food procurement, program administration, and program finances. The results will be used to assist USDA’s Food and Nutrition Service (FNS) and its partners to enhance the toolbox for administrators and school nutrition professionals working hard to make sure students are served healthy and nutritious meals.

The survey has a maximum of 31 questions and is estimated to take approximately 20 minutes to complete. FNS is asking all SAs and SFAs that participate in the Child Nutrition Programs to complete this survey by January 1, 2023.

Please note that the survey includes questions about changes in overall food costs, labor costs, and student participation during the last several years. If this information is not readily available to you, we encourage you to provide your best estimates.

We sincerely appreciate your participation in this survey, particularly as many of you have little time available given the exceptional levels of effort you are putting into feeding children. As such, we have made this survey as short and simple to complete as possible.

If you have any questions about the survey, please email us at [email protected].

We thank you in advance for your contribution to this important effort.

Please click Next>> to begin the survey.

1. Since the start of the regular 2022-2023 school year, which Child Nutrition Programs has your SFA operated? Do not include programs operated during summer 2022. Select all that apply. 

   1. National School Lunch Program (NSLP)  

   2. School Breakfast Program (SBP)

   3. NSLP Afterschool Snack Service 

   4. Child and Adult Care Food Program (CACFP) At-Risk Afterschool Meals 

   5. Fresh Fruit and Vegetable Program (FFVP) 

   6. Special Milk Program (SMP) 

   7. Summer Food Service Program (SFSP) – select only if SFA operated SFSP during unanticipated school closures since the start of the regular 2022-2023 school year 

2. How are the schools in your SFA operating the school lunch and/or breakfast programs in school year 2022-2023? Select all that apply.

1. Household applications (standard operations)

2. Operating Provision 2/3

3. Operating the Community Eligibility Option

4. Universal free meals offered by the state

5. Operating other alternatives for NSLP and SBP

6. Operating an alternate provision(s) for only NSLP or only SBP

3. Which food service model(s) does your SFA use? Select all that apply.

1. Food Service Management Companies (FSMCs)

2. Vended meals

3. Purchase meals from other schools or central kitchens

4. Self-preparation (independent kitchen, base kitchen, or production kitchen)

4. What, if any, challenges has your SFA experienced related to supply chain challenges during school year 2022-2023? Select all that apply.

1. High food costs

2. High labor costs

3. High cost and/or limited availability of food service materials (e.g., plates, trays, utensils)

4. High cost and/or limited availability of food service equipment and/or parts

5. School food service staffing shortages

1. Challenges receiving deliveries on usual days, times, or locations

2. Receiving incomplete orders with missing or substituted items

3. Difficulty adhering to planned menus due to changing or limited food availability

4. Difficulty maintaining compliance with meal pattern requirements

5. Shortage of staff, time, equipment, or materials required for scratch cooking

6. Challenges finding or purchasing local food

7. Food vendors discontinuing participation in school food service operations

8. Insufficient bids for food service contracts

9. Lack of storage space to store additional food in the event of shortages

10. SFA has not experienced any challenges with school meal program operations in SY 2022-2023

11. I don’t know

5. What, if any, challenges has your SFA experienced related to USDA Foods during school year 2022-2023? Select all that apply.

   1. Challenges getting the products needed through USDA Foods processing program due to vendor capacity

   2. Challenges getting products needed through USDA Foods processing program due to distributor supply issues

   3. Challenges getting products needed through USDA Foods processing program due to distributor or processor product line reductions (SKU rationalization)

   4. Challenges getting products needed through USDA Foods processing program due to price increases

   5. SFA has not experienced any challenges related to USDA Foods during SY 2022-2023

   6. I don’t know

6. What, if any, challenges has your SFA experienced related to the return to standard operations during school year 2022-2023?

   1. Fewer parents/guardians submitting applications for free or reduced-price meals

   2. Challenges verifying or processing applications for free or reduced-price meals

   3. Additional questions or confusion from parents/guardians regarding shift from free meals for all students to standard program operations

   4. Challenges preparing and/or ordering the right amount of food due to uncertainty in participation

   5. Challenges meeting meal pattern requirements

   6. Unpaid school meal debt

   7. SFA has not experienced any challenges related to the return to standard operations in SY 2022-2023

   8. I don’t know

7. What, if any, impact(s) are the challenges your SFA is experiencing having on school meal operations in your SFA? Select all that apply. [Randomize display order.]

1. Reduced student participation 

2. Reduced sale of nonprogram (competitive) foods

3. Increased program costs 

4. Inability to offer enough reimbursable meals to participating children 

5. Inability to meet meal modification requirements for children with food and nutrition related disabilities 

6. Difficulty complying with meal pattern requirements

7. Inability to comply with regular procurement requirements 

8. Difficulty maintaining routine program documentation 

9. Inability to offer afterschool snacks/suppers 

10. Increased staff stress or workload  

11. Decreased staff morale 

12. Inability to fulfill other job requirements due to time spent on meal planning, procurement, or food service  

13. Increased negative feedback or complaints about school meals from students and/or parents/guardians

14. Changes in meal preparation methods

15. SFA has not experienced any of these impacts in SY 2022-2023

16. I don’t know

8. What, if any, strategies has your SFA used to address supply chain challenges during school year 2022-2023? Select all that apply. [Randomize display order.]

1. Aligning or streamlining product specifications with other SFAs

2. Aligning or streamlining product specifications with available vendor products

3. Using cooperative purchasing agreements

4. Communicating more with parents/guardians about menus and other changes

5. Increasing use of local vendors

6. Increasing use of USDA Foods direct delivery (brown box)

7. Increasing use of USDA DoD Fresh

8. Leveraging state contracts

9. Conducting emergency procurements

10. Increasing use of micro-purchases

11. Increasing use of scratch cooking

12. Purchasing foods directly from retail outlets

13. Partnering with restaurants, catering companies, or community partners (e.g., universities, hospitals)

14. Limiting or repeating weekly menu offerings

15. Requesting shorter bids

16. Making more frequent orders

17. Making more frequent menu substitutions

18. Increasing tracking of shipments

19. Increasing use of available products across multiple menu items

20. Seeking external funding to cover excess costs of serving school meals

21. Limiting service options (e.g., stopping breakfast in the classroom)

22. Receiving deliveries at nontraditional days, times, or locations

23. Picking up orders when usual delivery methods are not workable

24. SFA has not used any strategies to address challenges in school year 2022-2023

25. I don’t know

9. What, if any, actions has your SFA taken in anticipation of the return to standard operating procedures during school year 2022-2023?

   1. Increasing paid lunch prices

   2. Covering the cost of reduced price meals

   3. Communicating more with parents/guardians about program changes

   4. Offering universal school meals or receiving other financial assistance from the state

   5. No longer participating in NSLP

   6. No longer participating in SBP

   7. No longer participating in other USDA child nutrition programs (e.g., afterschool snack programs)

   8. SFA has not taken any actions in anticipation of the return to standard operating procedures

   9. I don’t know

10. You indicated that your SFA has faced challenges due to increased food costs. To the best of your knowledge, approximately how much have your food costs increased overall since the start of last school year (SY 2021-2022)? [ASK IF SELECTED Q4 = INCREASED FOOD COSTS]

1. Less than 10%

2. 10-24%

3. 25-49%

4. 50-74%

5. 75-99%

6. 100% or more

7. I don’t know

11. To the best of your knowledge, what factors have contributed to changes in your food costs? Select all that apply. [ASK IF SELECTED Q4 = INCREASED FOOD COSTS]

1. Changes in student participation in nutrition program(s)

2. Changes in food service vendors, including using new and/or multiple vendors to obtain food

3. Distributors increased costs

4. My SFA procures high quality food products, which have become more expensive

5. My SFA procures local food, which has become more expensive

6. Purchased more food directly from retail store(s) or warehouses

7. Purchased more food last-minute

8. Transported own food from vendor or other location more frequently

9. Vendors increased costs

10. I don’t know

11. Other (please specify)

12. You indicated your SFA has faced challenges due to increased labor costs. To the best of your knowledge, approximately how much have your labor costs increased overall since the start of last school year (SY 2021-2022)? [ASK IF SELECTED Q4 = INCREASED LABOR COSTS]

1. Less than 10%

2. 10-24%

3. 25-49%

4. 50-74%

5. 75-99%

6. 100% or more

7. I don’t know

13. To the best of your knowledge, what factors have contributed to changes in your labor costs? Select all that apply. [ASK IF SELECTED Q4 = INCREASED LABOR COSTS]

1. Wage increases

2. Need to hire additional staff for basic program operations

3. Need to hire additional staff to support scratch cooking

4. Increased overtime for existing staff

5. Staff turnover and training

6. I don’t know

7. Other (please specify)

14. You indicated your SFA has faced challenges leading to reduced student participation. To the best of your knowledge, approximately how much has student participation decreased across the child nutrition programs your SFA operates since the start of last school year (SY 2021-2022)? [ASK IF SELECTED Q5 = REDUCED STUDENT PARTICIPATION]

1. Less than 10%

2. 10-24%

3. 25-49%

4. 50-74%

5. 75-99%

6. I don’t know

15. To the best of your knowledge, what factors have contributed to decreases in student participation? Select all that apply. [ASK IF SELECTED Q5 = REDUCED STUDENT PARTICIPATION]

    1. Fewer parents/guardians submitting applications for free or reduced-price meals

    2. Challenges verifying or processing applications for free or reduced-price meals

    3. Questions or confusion from parents/guardians regarding shift from free meals for all

    4. Difficulty purchasing enough food to meet student demand

    5. Student dissatisfaction with menus and/or meals

    6. Shortage of staff, time, equipment, or materials required for scratch cooking

    7. Reduced sale of nonprogram (competitive) foods

    8. I don’t know

    9. Other (please specify)

16. In general, how did student participation in your SFA’s child nutrition programs last school year (SY 2021-2022) compare to participation in a typical year prior to the pandemic (e.g., SY 2018-2019 or before)?

1. Greater than pre-pandemic participation

2. Less than pre-pandemic participation

3. About the same as pre-pandemic participation

4. I don’t know

17. You indicated your SFA has received incomplete orders with missing or substituted items. To the best of your knowledge, how frequently do you receive incomplete orders with missing or substituted items? [ASK IF SELECTED Q4 = RECEIVING INCOMPLETE ORDERS WITH MISSING OR SUBSTITUTED ITEMS]

1. Rarely or never

2. Sometimes

3. About half of the time

4. Most of the time

5. Almost always or always

18. To the best of your knowledge, when you receive an incomplete order with missing or substituted items, approximately how many items in the order are affected? [ASK IF SELECTED Q4 = RECEIVING INCOMPLETE ORDERS WITH MISSING OR SUBSTITUTED ITEMS]

1. Few or none

2. About a quarter of items

3. About half of items

4. About three quarters of items

5. All or almost all items

19. You indicated your SFA has experienced food companies discontinuing participation in school food service operations. To the best of your knowledge, which of the following factors are driving this change? Select all that apply. [ASK IF SELECTED Q4=FOOD VENDORS DISCONTINUING PARTICIPATION IN SCHOOL FOOD SERVICE OPERATIONS]

1. Federal procurement regulations (please specify)

2. State procurement regulations (please specify)

3. Local procurement regulations (please specify)

4. Labor shortages

5. Food shortages

6. Labor costs

7. Food costs

8. Energy or fuel costs

9. Difficulty producing items that meet SFA specifications

10. Difficulty guaranteeing prices for bids due to changes in food costs

11. School food operations are no longer profitable

12. I don’t know

13. Other (please specify)

20. Compared to last school year (SY 2021-2022), is your SFA purchasing more, less, or about the same amount of local foods during school year 2022-2023?

    1. More

    2. Less

    3. About the same, and we do purchase some amount of local foods

    4. About the same, but we rarely or never purchase local foods

    5. I don’t know

21. Compared to last school year (SY 2021-2022), is your SFA using scratch cooking more often, less often, or about the same amount during school year 2022-2023?

    1. More often

    2. Less often

    3. About the same amount, and we do use scratch cooking

    4. About the same, but we rarely or never use scratch cooking

    5. I don’t know

22. Did your SFA accept all or some of their Supply Chain Assistance (SCA) fund?

1. Yes, SFA accepted all of it

2. Yes, SFA accepted some of it

3. No

4. I don’t know

23. If your SFA did not accept their Supply Chain Assistance (SCA) fund, do you know why? [ASK IF ANSWER Q18 = NO]

1. Didn’t know SCA funds were available

2. Too much time and/or effort required to apply to receive SCA funds

3. Too much time and/or effort required to accept and use SCA funds

4. Unsure how to code funds in local accounting system

5. My SFA did not need the SCA fund

6. I don’t know

24. At the end of SY 2021-2022, what was the status of your school food service account balance?

1. Operated at a surplus (i.e., revenues exceeded costs)

2. Broke even (i.e., revenues are about equal to costs)

3. Operated at a deficit (i.e., costs exceeded revenues)

25. You indicated your SFA was operating at a surplus at the end of SY 2021-2022. To the best of your knowledge, how has your SFA used surplus funds (or does your SFA plan to use surplus funds) for any of the following? Select all that apply. [ASK IF ANSWER Q20 = OPERATED AT SURPLUS]

1. Cover the cost of reduced-price meals

2. Hire new staff

3. Increase pay rate for staff

4. Improve meal quality

5. Purchase new equipment

6. I don’t know

26. What do you predict will be the status of your school food service account balance in the first quarter of SY 2022-2023?

1. Operate at a surplus (i.e., revenues exceeded costs)

2. Breakeven (i.e., revenues are about equal to costs)

3. Operate at a deficit (i.e., costs exceeded revenues)

27. What else would you like FNS to know about the challenges your SFA is experiencing, the impacts those challenges are having on school meal operations, or the strategies you are using to address those challenges? If there are particular strategies you would like to use to address your SFA’s challenges, but you are experiencing barriers to doing so, please explain.