Appendix A.18 Cognitive Pretest Memo

Appendix A.18 Cognitive Pretest Memo

Table of Contents

Pretest Findings Memo 2

Appendix A.1: Pretest – Recruitment Email to ADCC Sponsoring Organizations 8

Appendix A.2: Pretest – Recruitment Email to ADCC Program Operators 9

Appendix A.3: Pretest – Recruitment Phone Calls to ADCC Program Operators 10

Appendix A.4: Pretest – Survey of Food Safety Education Needs of ADCC 13

Pretest Findings Memo

In August 2018, 2M Research (2M) and Mathematica Policy Research (Mathematica) conducted a cognitive pretest of the Survey of CACFP Adult Day Care Centers Food Safety Education Needs. This memo describes the pretest activities, summarizes the findings from the pretests, and describes revisions made to the survey as a result of these findings. In addition to the memo, this revised submission includes the revised survey that was pretested in hard copy format and the final updated version of the hard copy survey and the web version of the survey that reflects findings from the pretest and FNS’s comments.

A. Pretest Activities

The pretest included a total of five respondents from California adult day care centers (ADCCs): two program administrators, two program directors, and one finance director. Two of the respondents were from independent centers, and three were from sponsored centers.

2M Research received an initial list of potential pretest participants from FNS, which included contact information on ADCC site program operators, both sponsored and independent, as well as contact information of sponsors. This list was obtained via an FNS regional liaison, who had requested it directly from the State CACFP director in California. The study team then provided FNS with a list of nine program operators from the sponsored and independent ADCCs selected for pretesting to have enough variation in the population of the cities where the sites were located. The recruitment period was from July 26, 2018, to August 3, 2018. The study team created a specific email address, CACFP Help Desk – [email protected] – to send and receive emails from ADCCs.

As the study team progressed through the list of potential pretest participants, the team found that email addresses of the site program operators of sponsored ADCCs were missing. To resolve this issue, the study team used a two-pronged approach: deducing the email address of the program operator using organization email information for the sponsor and authorized representative and conducting a web search. If neither strategy worked, the study team contacted the sponsor to obtain the email address of the site program operator. Initially, the study team sent the site recruitment email (Appendix A.1) to the sponsors and had to send a clarifying email to ADCC sponsors requesting contact information on the site program directors. The main contact from each ADCC sponsor was initially contacted to participate in the pretest. The main contact was then asked, and provided, the missing contact information on site program directors/administrators in the list from the State. For example, Daylight Adult Day Healthcare Center (ADHC) provided three contacts for three different sites (one in Los Angeles and two in Glendale) and Mad River ADHC provided another contact.¹ The study team had to reach out to more individuals than they initially anticipated. Since the study team had only allocated one week to recruit for the pretest, they also included the table of available time slots in the contact emails to the program operators (Appendix A.2). In addition, the study team adjusted recruitment efforts by calling the program operators (Appendix A.3) to reach the number of participants needed to pretest the survey. In total, the study team contacted these nine site operators to recruit at least five participants for the pretest. The maximum number of times for follow-up was five, either through email or phone call.

The majority of potential pretest participants (78 percent) did not respond to the initial and first follow-up emails. Only two contacts out of nine replied to confirm participation from the initial email. The study team decided, with approval from FNS, to call each remaining potential participant individually. Considering the time difference (PST and EST), the best time to reach out to the ADCCs was after 12:30 p.m. EST/9:30 a.m. PST. The study team received two agreements to participate through this method.² Overall, five different ADCCs agreed to participate: Mad River ADHC, Altamedix ADHC, Alzheimer’s Services – East Bay – Berkeley, Kings Rehabilitation Center, Inc. – Tulare Campus, and Shascade Skills Development Center.

Recruitment Strategy and Efforts	Number of Respondents
Number of Participants Contacted	9
Number of Participants Recruited by Email	2
Number of Participants Recruited by Phone	3
Number of Participants who Completed Pretest	5

2M scheduled 1-hour pretest interviews with each of the respondents and emailed the survey (revised draft version – “Adult Day Care Center Program Operator Survey” – Appendix A.4) to them prior to the scheduled interview. Respondents were instructed not to read the survey until their interview. When called at the scheduled time for their pretest interview, two of the five respondents reported that they were unavailable but rescheduled the interview for a later day or time.

Mathematica survey staff conducted 1-hour pretest interviews with each pretest respondent to solicit feedback. To minimize the burden of participating in this pretest, respondents were asked to complete the survey on hard copy³ during the interview, rather than on their own prior to the call. During the pretest interviews, respondents completed the survey one section at a time and read their responses aloud to the interviewer. The interviewer timed each section and asked respondents debriefing questions to assess whether respondents understood the meaning of the questions; whether they had difficulty answering the questions; and whether the response options were applicable, clear, and comprehensive. All pretest respondents were mailed a $30 Visa gift card to thank them for their time.

The time respondents took to complete the survey ranged from 8 to 20 minutes, although the respondent who completed the survey in 8 minutes reported that she had read the survey prior to the pretest interview. Excluding the respondent who completed the survey in 8 minutes, the average time to complete the survey was 17 minutes.

B. Pretest Findings and Survey Revisions

This section first summarizes general pretest findings and revisions made to the survey, and then presents specific findings and revisions for questions in each section. Although respondents initially reported that most of the questions were clear and the answer choices were comprehensive, further probing revealed that respondents had varying degrees of difficulty understanding some survey questions and question instructions. Several themes emerged that required broad revisions to the survey:

1. Respondents generally had difficulty following question instructions, and when probed, they reported that they did not read all of the question text (including the instructions). For example, respondents did not realize they could mark more than one answer for certain questions, or they reported assuming the question was asking about something they currently do instead of what they would like to do. As a result, the study team simplified question language and structure throughout the survey to reduce the cognitive burden on respondents and make the questions easier to understand and complete. The study team also added instructions at the beginning of the survey about how to answer the questions in the survey.

2. Respondents expressed varied understanding of terms used in survey questions. Respondents interpreted the terms “food safety training” and “staff directly involved with foodservice” differently. Respondents interpreted “food safety training” to include anything from a cursory mention of a food safety principle during an unrelated training to intensive, 16-hour food safety manager certification courses. When asked about “staff directly involved with foodservice,” some respondents reported thinking only about kitchen staff who prepare food, while others reported thinking about all staff who prepare, handle, and serve food. To address this issue, the study team added definitions for these terms and repeated these definitions in each section where the terms appear. In the hard copy version of the survey, these definitions have been retained at the beginning of each section. In the web survey, terms will be defined at first mention; later occurrences will show the definitions when a respondent hovers over the word or phrase.

3. Some respondents did not have the knowledge needed to respond to the survey questions. Several pretest respondents suggested that the most appropriate respondent would be the program director or CACFP point of contact, instead of the program operator. One respondent, a program director, said that the program operator would not be able to answer most of the survey questions. To accommodate a variety of job titles of survey respondents, the study team removed all references to the respondent as a “program operator” from the survey. In addition, the survey recruitment team will ask that the survey be filled out by a staff member at the center who knows the most about foodservice operations and food safety training and education needs of center staff.

Section A: Adult Day Care Center Background

In this section, two respondents did not realize they could select more than one response option for the first two questions, which focus on food preparation and meal/snack service. As mentioned above, the study team added instructions at the beginning of the survey to explain how to respond to the various types of questions in the survey, including “mark all that apply” questions. For Q1 and Q2, some respondents reported that they answered the question thinking only about one of the meals served at their centers. The study team modified the wording of Q1 to clarify that they should think about all meals and snacks served at their centers and changed “foods” to be “meals and snacks” in Q2.

Also, in Q2, the study team substituted the word “client” for “participant” to reflect the terminology used by pretest respondents. One pretest respondent suggested this change, and the other respondents simply used the term “client” while answering debriefing questions. In addition, the study team modified the response options for Q2 to reflect respondent feedback that it is rare for clients to get their own food through a serving line, but food can be served to clients in various ways (for example, from a serving cart that goes to tables). The study team reworded the response choice “foods are served to participants in a serving line” to be less specific, so that it covers the various ways that staff may portion and serve food to clients.

In Q3 and throughout the survey, respondents had different interpretations of the phrase “staff directly involved with foodservice,” depending on the question. When asked how they interpreted “staff directly involved with foodservice,” some respondents reported thinking only about kitchen staff who prepare food, while others reported thinking about all staff who prepare, handle, and serve food. One respondent reported thinking only about kitchen staff for Q3, which asks about how many staff are directly involved with foodservice. The same respondent then reported thinking about kitchen staff and staff who serve food for the questions in Section B, which ask about staff activities or training. To provide clarity and to ensure that respondents use a consistent definition throughout the survey, the study team added a definition of “foodservice staff” to Q3 and repeated the definition at the beginning of Sections B and C.

Section B: Current Food Safety Education

Throughout Section B, respondents interpreted food safety training to include anything from a cursory mention of a food safety principle during an unrelated training to intensive, 16-hour food safety manager certification courses. To minimize this variation in understanding the term, the study team included definitions of food safety training and food safety information (a simplified term for “educational resources”) at the beginning of Sections B and C. Throughout the survey, the study team used the simplified term of “food safety information” instead of “educational resources.”

Respondents had difficulty reporting the required number of hours of foods safety training both for themselves and other staff (Q7 and Q8). Some were unsure how to report food safety certification courses that they are only required to take once every 3–5 years. Others were only able to provide estimates. Most respondents reported that the requirements vary among their staff, so providing one answer for all staff was difficult. Given respondents’ difficulty in providing accurate responses to these questions and the variety of requirements for foodservice staff at a single center, the study team revised and consolidated these questions. Q5, which asked whether all foodservice staff are required to receive food safety training, was removed. Instead, respondents are now asked for the number of hours of food safety training that most foodservice staff are required to receive each year (now Q7 in the final updated survey). Asking about “most” staff eliminates the confusion respondents felt when trying to respond for all foodservice staff during the pretest. It also captures the most common response, instead of potentially reflecting the highest or lowest number of training hours when they vary by staff member. In addition, the study team added a question about whether all, some, or no foodservice staff receive food safety certifications through certification programs like ServSafe (now Q6 in the final updated survey). This new item is in response to all pretest respondents mentioning ServSafe as part of their food safety training. The study team felt it was important to parse food safety certification programs out of general food safety training in this question, since the training does not happen every year, and when staff participate, they could be receiving 8–16 hours of training, which may not reflect the norm for annual training requirements.

The study team reordered and reworded some of the response options to Q11 (now Q10, about where centers get food safety training and information) and Q13 (about where centers go for answers to their food safety questions). For example, several respondents expressed confusion about what “State agency” included. One reported that this response option was “broad,” and others wanted to confirm that we meant the California Department of Education (the name of the state agency that administers CACFP in California, where the pretest respondents were located). The study team clarified the wording of this response option and others and reordered them so that the response options that respondents are more likely to select appear first.

Some respondents did not notice the skip instructions in this section and answered questions they should have skipped, based on their responses. Although many respondents will answer the survey questions in a web instrument, the study team made small changes to accommodate any respondents completing the survey on hard copy. To make the survey easier to complete and reduce the number of instructions respondents are asked to follow, the study team removed the skip instructions from Q12 (now Q11), which asks about challenges related to receiving food safety training or information. All respondents are now asked about language needs for foods safety training or information (Q12a, now Q12). Since this question may not apply to everyone, the study team added a response option for “not applicable – We only use materials in English.” For Q12a (now Q12), Q11 (now Q10, about where centers get food safety training and information), Q12 (now Q11), and Q13 (about where centers go for answers to their food safety questions), the study team clarified the instructions so that respondents can select multiple response options or select a “not applicable” response. The study team moved the “not applicable” response options for these questions to the end to encourage respondents to read all of the response options before selecting an answer. Removing some skip patterns reduces the survey’s complexity for those completing it on hard copy and the need for data cleaning, while adding minimal time burden for the respondents to read additional response options and complete an additional question.

Section C: Food Safety Education Needs

The long instructions and complicated question structure in Section C (Food Safety Education Needs) was challenging for respondents. Several respondents thought that all of the questions asked about their current food safety practices, instead of their needs and preferences on these topics. The study team simplified the question wording for all three questions in this section and added an introduction to this section to emphasize that respondents should provide information on what they need and prefer, instead of what they currently do.

In column B of Q15, respondents were instructed to rank the three most important food safety topics for which they need training. In column B of Q16, they were instructed to indicate their preferred method of training for the top three priority food safety training topics they identified in the previous question. However, no respondents were able to correctly complete column B of Q16, and one was unable to correctly complete column B of Q15. Respondents said they were intimidated by the lengthy instructions of Q16 (about their preferred methods for receiving training) and assumed the instructions were the same as for the previous question. Based on this feedback, the study team simplified both questions so that respondents will identify the food safety topics for which they need new or additional training/information (Q15), and their preferred methods in general for receiving training/information (Q16). Although respondents will provide less detailed information in the revised versions of these questions, both questions still directly answer the research questions. To increase the data quality for these questions, the final questions require a yes/no or prefer/do not prefer response to each item listed, instead of simply instructing respondents to mark all that apply. Requiring a response to each item eliminates the ambiguity of a missing response, which could be interpreted as a “no” or “do not prefer” or as missing data because the respondent simply forgot to mark it.

Section D: Center and Contact Information

In Section D, the three respondents from sponsored centers did not understand Q18, which asks whether their center is sponsored or independent. Two incorrectly answered that their centers were independent, and one marked “don’t know.” To address this issue, the study team added definitions for sponsored and independent centers to this question. Two of these respondents also misinterpreted the definition for “affiliated centers” from Q19 (which asks whether their center is affiliated with the sponsoring organization) to mean that sponsored centers are owned by CACFP itself. For respondents that complete the web survey, the study team will remove Q18 because it will be available on the sample frame file, and Q19 will be programmed so that only respondents from sponsored centers will receive the question. However, in order to contextualize the question about center affiliation for respondents completing the survey in hard copy, Q18 (about whether their center is sponsored or independent) will still appear in the hard copy version.

Finally, in Q20, the study team separated out the fields for respondent contact information to reduce data cleaning efforts.

Appendix A.1: Pretest – Recruitment Email to ADCC Sponsoring Organizations

To: [EMAIL Address of ADCC Sponsoring Organizations Contact]

Subject: CACFP Adult Day Care Center Food Safety Education Needs – Pretest of Survey Instruments

Good afternoon:

The U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) has contracted with 2M Research (2M) and its subcontractor, Mathematica Policy Research (Mathematica), to identify the food safety education needs of adult day care center (ADCC) program operators. This study will help FNS to better understand knowledge gaps among ADCC program operators regarding understanding food safety education techniques and the best way for operators to obtain and receive needed information. To collect the above information, this study will use a brief (less than 20 minutes) survey of a nationally representative sample of ADCC program operators across the United States.

Soon, the study team will begin pretesting the survey. We are hoping to get in touch with the program operators for the following XXX sites in order to pretest survey instruments for the CACFP Food Safety Education Needs of Adult Day Care Centers Study. We understand that we contacted the sponsoring organization, but we are seeking contact information for the centers/sites. We are reaching out to you to request their email addresses.

[LIST of ADCC SITES]

Pretest activities include one survey and one interview; both will be completed while on the phone with a member of the study team. Each respondent will receive a $30 incentive for participating in the pretest. 2M representatives will schedule the interviews with each site’s program operator at a date and time convenient for them between Monday, August 6, and Tuesday, August 7.

For your records, attached are the correspondences that will be sent to the ADCC sites selected for pretesting. If you have any questions about the study, please do not hesitate to contact the study team at 1-833-728-7369 (toll-free) from 9:00 a.m. to 4:00 p.m. Monday – Friday EST or by email at [email protected].

Thank you in advance for your response.

Sincerely,

Dr. Hiren Nisar

Director of Analytics and Senior Economist | 2M Research

Project Director

Attached:

1. Pretest – Recruitment Email to ADCC Program Operators

2. 

   According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 8 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0613). Do not return the completed form to this address.

   Frequently Asked Questions (FAQs) for the study

Appendix A.2: Pretest – Recruitment Email to ADCC Program Operators

To: [EMAIL address of ADCC Program Operator]

Subject: CACFP Adult Day Care Center Food Safety Education Needs – Pretest of Survey Instruments

Good afternoon:

The U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) has contracted with 2M Research (2M) and its subcontractor, Mathematica Policy Research (Mathematica), to identify the food safety education needs of adult day care center (ADCC) program operators. This study will help FNS to better understand knowledge gaps among ADCC program operators regarding understanding food safety techniques and the best way for operators to receive needed information. This study will collect the information needed using a brief (less than 20 minutes) survey of a nationally representative sample of ADCC program operators across the United States.

Today, we encourage you to participate in a pretest of the survey for the study. For your participation, you will receive $30. Pretest activities include one survey and one interview; both will be completed within an hour while on the phone with a member of the study team.

To expedite the scheduling process, please fill in the table by placing an X in the cells that display times that you are available for the pretest and email the table back to me. A member of the study team will follow up with you shortly to confirm your interview time or provide other times convenient to you.

Time (PST)	Monday, August 6	Tuesday, August 7
9:00 a.m.
10:30 a.m.
12:00 p.m.
1:30 p.m.
3:00 p.m.

Your responses will not be disclosed or used in identifiable form for any purpose. Please also know that this study is not an audit—in this study we are asking you to be partners in assessing the range of food safety needs. For any assistance, you may contact the study team by phone at 1-833-728-7369 (toll-free) between 9:00 a.m. and 4:00 p.m. Monday to Friday EST or by email at [email protected].

We thank you for your important contribution to this research.

Sincerely,

Dr. Hiren Nisar, Project Director

Director of Analytics and Senior Economist | 2M Research

Enclosed:

1. 

   According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 3 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0613). Do not return the completed form to this address.

   Frequently Asked Questions (FAQs) for the study

Appendix A.3: Pretest – Recruitment Phone Calls to ADCC Program Operators

INITIAL CONTACT

Hello, my name is [YOUR NAME]. I'm calling from 2M Research on behalf of the USDA Food and Nutrition Service to follow up on the Adult Day Care Center Food Safety Education Needs Pretest Survey participation request for [RESPONDENT’S NAME]. Would that be you?

(IF SPEAKING TO THE RESPONDENT ON INITIAL CONTACT, GO TO B1.)

(IF SPEAKING TO SOMEONE ELSE, SAY:)

Is there a direct line to reach him/her, or could you please transfer me to (RESPONDENT’S NAME)?

(IF YES, RECORD NUMBER IN SPACE BELOW AND ENTER THIS NUMBER IN THE MESSAGE FIELD AT THE END OF THE CALL)

RESPONDENT’S DIRECT PHONE NUMBER:

Is he/she available?

☐ YES (GO TO B2. IF TRANSFERRED)

☐ NO (GO TO A. RESPONDENT NOT AVAILABLE)

A. RESPONDENT NOT AVAILABLE

(IF SPEAKING TO A PERSON WHO IS NOT THE RESPONDENT): We sent a request to [RESPONDENT’S NAME], to participate in the Adult Day Care Center Food Safety Needs Pretest Survey for USDA. This survey should be completed by the person at your center who knows the most about food service and food safety practices.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0613). Do not return the completed form to this address.

Do you know whether [RESPONDENT’S NAME] might have asked someone else at your center to participate in the survey pretest?

☐ Yes, knows name of new person

May I have the name, title, and contact information (email and phone) for that person? (ENTER THE CONTACT INFORMATION IN THE MESSAGE FIELD)

CONTACT:

Name: _______________

Job Title: _____________

Email: _______________

Phone: _______________

☐ No, do you know who at your center knows the most about food service or food safety? (CONTINUE BELOW)

May I have the name, title, and contact information (email and phone) for that person? (ENTER THE CONTACT INFORMATION IN THE MESSAGE FIELD)

CONTACT:

Name: ______________________________

Job Title: _____________

Email: _______________

Phone: _______________

Is he/she available?

☐ YES (GO TO B3a. IF TRANSFERRED)

☐ NO (GO TO A.1)

☐ No, does not know name or whether given to someone else (GO TO A.1)

A1. LEAVE A MESSAGE

Would you please leave a message for [RESPONDENT’S NAME] mentioning that (YOUR NAME) called from 2M Research to follow up on the Adult Day Care Food Safety Needs Pretest Survey? When is a good time to call back? If [RESPONDENT’S NAME] prefers, she/he can reach me toll-free at 1-833-728-7369 between the hours of 9:00 a.m. and 4:00 p.m. Monday through Friday EST. (END OF CALL)

Callback Date/Time:

If person on the phone transfers you to voicemail: (GO TO D. VOICEMAIL SCRIPT)

B. SCRIPT FOR WHEN RESPONDENT IS ON THE PHONE:

B1. If Speaking to Respondent on Initial Contact

We recently sent a request to your center inviting the person who knows the most about food safety to participate in the Adult Day Care Food Safety Needs Pretest Survey. We sent this invitation through email. (GO TO B3. STUDY INTRODUCTION)

B2. If Transferred

Hello, my name is [YOUR NAME], and I’m calling from 2M Research. We recently sent a request inviting your center to participate in the Adult Day Care Food Safety Needs Pretest Survey for the USDA Food and Nutrition Service to learn about your center’s food safety education needs. (GO TO B3. STUDY INTRODUCTION)

B3. Study Introduction

2M is conducting this ADCC food safety education needs survey for the USDA Food and Nutrition Service. We noticed that you haven’t replied to our emails, so we wanted to be sure that you received our request for you to participate in the pretest survey. Are you available to participate?

If yes, GO TO C1.

If no: Thank you for your time.

C. SCHEDULING PRETEST INTERVIEW

C1. If Participant Agrees to Participate

Thank you so much for your support. We have the following dates and times available (PROVIDE TIMES AND DATES). It will take about an hour to complete the pretest interview. We will send you a follow-up email with details on the interview. We want to be sure that we have the correct email to contact you. Is your email (SPELL OUT EMAIL ON FILE)? At the end of the interview, we will mail you a $30 gift card. Again, thank you for your support.

D. VOICEMAIL SCRIPT

Hello, may name is [NAME], and I’m calling from 2M Research on behalf of the USDA to follow up on the Adult Day Care Center Food Safety Education Needs Survey Pretest. This pretest survey should be completed by the person who knows the most about food service and food safety at your center. As a reminder, your cooperation in USDA nutrition studies is highly encouraged under the Child Nutrition Reauthorization Act of 2010. Please call us back at 1-833-728-7369 between the hours of 9:00 a.m. and 4:00 p.m. Monday through Friday EST so we can schedule the pretest. (END OF CALL)

E. FOLLOW-UP EMAIL TO CONFIRM PARTICIPATION AND PROVIDE INSTRUCTIONS

Hello XXX,

Thank you for agreeing to participate in pretesting the survey for the USDA Adult Day Care Center Food Safety Needs study. This email provides you with instructions for your interview scheduled on Tuesday August 7, 2018 at 10:30am PST. Below are some basic instructions for the interview:

1. Jeanette Holdbrook will call you at 707-822-4866 at your scheduled time of the interview. Please contact her via email at [email protected]  or phone at 609-275-2296 if you need to cancel or reschedule.

2. Attached is a PDF of the survey. If you can print it before the pretest interview, it would be helpful but please do NOT read it or complete it ahead of time. Having a pen ready for the interview is the only other thing you need to do to prepare for the interview.

3. At the end of the pretest interview, Jeanette will confirm your contact information to send you the $30 gift card.

Please let us know if you have any questions or if your phone number is incorrect.

Again, we appreciate your help for this important study.

Sincerely,

Dr. Hiren Nisar

Director of Analytics and Senior Economist | 2M Research

Project Director

Enclosed:

Survey of Food Safety Education Needs for ADCC Program Operator for the Pretest

Appendix A.4: Pretest – Survey of Food Safety Education Needs of ADCC (Hard Copy Survey)

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-0613). Do not return the completed form to this address.

1 The study team did not provide any assurances to the sponsoring organization point of contact about keeping their responses private, but the study team did provide assurances to the respondents that their answers will be kept private.

2 The respondent mentioned on the phone that the site had been closed for a full week due to wildfires in California while the study team was trying to recruit the site to participate in the pretest of the survey.

3 For the main study’s data collection, the study team has assumed that most respondents will complete a web version of the survey and hard copy surveys will be mailed to those who do not have email addresses, do not click on the web survey link, or request a hard copy during phone call reminders.