Appendix F Summary of Findings and Recommendations from Pretest Cognitive Interviews 031015

APPENDIX F

Summary of Findings and Recommendations from

Pretest Cognitive Interviews

Summary of Findings and Recommendations from

Pretest Cognitive Interviews

INTRODUCTION

Between April 1 - April 16, as part of the development of the sponsor and provider surveys for the main component of the CACFP Sponsor and Provider Characteristics Study, 2014 Kokopelli Associates conducted cognitive interviews by telephone to test the draft survey materials and questionnaires for the Study of the Characteristics of CACFP Sponsors and Providers. The testing was conducted to assess respondents’ understanding of the survey questions and of the recruitment materials that will be sent in advance of the actual survey to the sponsors and providers soliciting their participation in the survey.

This memorandum summarizes the methods and findings from the cognitive interviews. The cover letter and instructions sent to pretest respondents, cognitive interview debriefing guides, the pretested draft instruments (which included FNS comments on the first draft instruments submitted), and the complete set of eight revised sponsor and provider instruments are provided in separate documents as attachments to this memorandum.

PRETEST PARTICIPANTS

Pretest participants were recruited for the pretests from two States where staff on the Kokopelli team have informal contacts with CACFP sponsors and providers: California and New York. Following the protocol of the Federal Food and Nutrition Service (FNS), FNS first contacted the State CACFP Agency in each of these states to notify them that Kokopelli would be selecting CACFP sponsors, CACFP-participating centers, and CACFP-participating family day care homes from their states, whose identify would remain anonymous to FNS and the States. After receiving notice that the FNS notification was completed, Kokopelli staff reached out to 20 CACFP sponsors and providers to explain the study, the voluntary nature of the pretest, its purpose, and the honoraria that would be provided to participants. As part of the recruitment process, Kokopelli also provided sponsors and providers with information on the requirements of the pretest and the timeframes involved. The target respondent in CACFP sponsoring organizations was the individual responsible for coordinating CACFP functions within that organization. The target respondent in child care and Head Start centers was the center director or coordinator. And, for family day care homes, where there are usually only one or two providers, we recruited the lead provider. Fifteen respondents initially agreed to participate in the pretests.

One family day care home provider of the 15 recruited had to drop out due to her inability to complete the survey within the time limits of the pretest and we were able to quickly recruit an additional family day care home provider to take her place.

The survey questionnaires for each type of CACFP sponsor and participating CACFP site include both a common set of questions and some questions tailored for each type of respondent, resulting in a slightly different instrument for each type of respondent. As shown in Exhibit 1 below, we pretested seven different instruments with a total of 15 respondents. Six instruments―the child care center sponsor, Head Start Center sponsor, and family day care home sponsor, sponsored child care center, independent child care center, and Head Start center instruments―were each pretested with one respondent from each of the two states. We had anticipated that the family day care home providers would have the most difficulty completing the questionnaire because of the greater time constraints and perhaps a lower education level; hence, we pretested the family day care home questionnaire with three respondents and purposively chose and recruited homes that varied in size (the smallest serving six children and the largest serving 14 children).

Study of CACFP Provider and Sponsors Characteristics: Number of Pretest Participants by Type
Type of Sponsor	Number of Pretest Respondents
Child care center sponsor	2
Head Start center sponsor	2
Family day care home sponsor	2
Total All Sponsors	6
Type of Child Care Provider
Sponsored child care center	2
Independent child care center	2
Head Start center	2
Family day care home	3
Total All Child Care Providers	9
TOTAL ALL PRETEST RESPONDENTS	15

PROCEDURES

During the week of March 24, the recruited respondents received a brief telephone call to explain the pretest methods, timing, confirm their wiliness to participate, and stress that study staff would protect their confidentiality.

After this phone call, each pretest participant was sent the questionnaire corresponding to their sponsor or provider type, the study recruitment brochure, the study introductory letter (including a list of documents they would need to have with them to answer some of the questions), and pretest instructions. They were also sent a pre-paid Federal Express enveloped to return the completed pretest documents within one week of receipt. Kokopelli staff then called each participant two days after sending the pretest package to ensure its receipt and to schedule a telephone cognitive debriefing, and remind the participant to complete the survey, mark and “X” next to any question that they were not sure about or had any questions about, and to jot down the time they began and completed the questionnaire, noting separately the time they needed to compile necessary documents or consult with other staff to answer the survey questions.

All cognitive debriefing interviews were completed by telephone and lasted from 40 minutes to about an hour. All cognitive interviews were conducted by Ms. Vivian Gabor, an experienced survey methodologist. During the telephone interviews, the interviewer began by reading an informed consent to participants. The interviewer confirmed the amount of time the participant to complete (as written down by the participants on the returned survey instrument). Then, the interviewer used a debriefing guide to lead the participant in a discussion to review their written responses to the survey questions, understand whether they understood each survey question intent, and to identify questions or terms in the response choice that were confusing or difficult to understand, identify common “other” response choices, and obtain suggestions for rewording the question or response options. Each interview lasted approximately one hour, and participants’ organizations received a $25 honorarium after completing the cognitive pretest interviews.

FINDINGS

The cognitive interview participants found each of the survey instruments easy to complete. The six sponsors reported that it took them, on average, 45 minutes to complete the questionnaire, including time needed to obtain based data from their electronic or paper-based systems. The six center and three family day care home respondents each reported that it took them, on average, 52 minutes to complete the questionnaire and collect needed program data. The length of time it took center versus family day care home providers did not vary considerably and none of the family day care home providers reported spending more than one hour to complete the survey.

Specific revisions to the sponsor and center and home instruments based on the findings from the cognitive interviews are summarized in the sections that follow.

I. SPONSOR INSTRUMENTS: REVISIONS BASED ON PRETEST FINDINGS

Revisions Common Across Instruments

• Added underlining of key words for emphasis in introductory text and questions.

• Clarified instructions at beginning of many of the sections.

• Fixed some skip patterns where needed or deleted skips where no longer relevant.

• Changed the word “provider” to child care “site” or child care “center” in the child care center (CCC) sponsor, Mixed sponsor, and Head Start sponsor surveys because respondents consider the word “provider” to only refer to family day care homes.

• A “don’t know” response choice was added to several of the questions in all the surveys.

Revisions for Head Start Sponsor Instrument Only

Because Early Head Start centers were not mentioned specifically in the draft, Head Start sponsors assumed they should not be included in their responses. To address this issue we made changes both in the introductory text to the survey, to section titles and to the wording of individual questions. In the introductory text for the Head Start sponsor survey, we added language clarifying that respondents who sponsor a mix of Head Start and Early Head Start centers should include both in their responses. The words “Early Head Start centers” were also added to several questions to remind respondents that they should include these centers in their response.

Section by Section Findings and Revisions

Important: The question numbering referenced below reflects the numbering in the current revised draft instruments, except where otherwise noted.

General Characteristics of Sponsoring Organization

1. CCC sponsors indicated that some sites are mixed At-risk and regular CCC sites so question should be referred to as sites participating in At-risk rather than “At-Risk centers.”

Revision

• Q 5d of the CCC sponsor survey changed from “How many of these centers were At-Risk Afterschool centers?” to “How many of these centers were sites that participated in the At-Risk Afterschool component of CACFP?

2. Respondent from New York City explained that they use the term “city borough” rather than town or county to describe geographic areas within New York City.

Revision

• Added new response choice to Q 8 in the CCC sponsor survey, Q 7 in the Head Start sponsor survey, and Q 7 in the FDCH sponsor survey “Part of a city borough” ; and revised two response choices to add city borough: “An entire county or city borough” and “A group of counties or city boroughs”.

3. Several respondents misinterpreted the question about sponsors managing other USDA food and nutrition programs because of the use of the word “participate” in the draft instrument. When participate was changed to manage or administer they understood question intent.

Revision

• In Q 10 and 10a in the CCC sponsor survey, Q9 and 9a in the HS sponsor survey, and Q 8 and 8a in the FDCH sponsor survey “participate” was changed to “manage or administer”.

4. Several respondents noted that the list of response choices about non-USDA programs provided are “services” not “programs” and using the words “programs” limited their understanding of the question.

Revision

• For Q 11 in the CCC sponsor survey, Q 10 in the HS sponsor survey, and Q9 in the FDCH sponsor survey we changed “programs” changed to “services”.

5. To improve question flow and skip patterns in this section of the CCC and HS sponsor surveys, we also changed the question numbering so that what had been Q 8 in the CCC sponsor survey and Q7 in the HS sponsor survey became Q 7b and Q 6b respectively.

Training and Assistance Provided by State CACFP Agency

This section had been titled “Administration and operations of your CACFP program” but because of some deletions agreed to with FNS we pared down the section and created two separate section titles in the revised draft―one on training and assistance and a second section on electronic systems used for CACFP claims (discussed below.)

1. Many pretest respondents said that their answers about frequency, format and content of training would differ greatly if they were talking about the annual mandatory training versus additional trainings that State CACFP Agencies may offer and they felt that FNS should know their responses for each.

2. Many pretest respondents also noted that technical assistance should be clearly separated from training and that this is not “offered” by State CACFP agencies but rather may be available based on sponsor requests.

3. Respondents also did not understand the questions about “additional support services” needed and suggested we focus on topics they would like to more help with.

4. Respondents volunteered information about variations in training format (between mandatory and additional trainings) and information about satisfaction with trainings and suggested that these questions be added to the survey.

Revisions (based on the four findings highlighted above)

• The first question series in this section now asks specifically about annual mandatory CACFP training.

• The second question series in this section now asks about additional training, if any, provided by State CACFP agency.

• The third question series in this section now asks about technical assistance, if any, sponsors requested from their State CACFP Agency.

• The final two questions in this section were reworded to ask about “food, nutrition and/or CACFP-related topics” sponsors would like “additional training or assistance’ on.

• We inserted new questions about the format of annual mandatory training and the most common format of additional trainings (if any offered in past 12 months).

• We inserted new questions to determine sponsor satisfaction with State trainings and with the technical assistance provided by their State CACFP agency.

Electronic Systems

1. Respondents explained that as staff working on CACFP they are mostly involved with systems storing or sending claims data and were not knowledgeable about overall financial or accounting systems of their organization.

Revisions

• The introductory text and questions in this section were slightly revised to focus on sponsors’ use of systems specifically for CACFP claims.

• Wording in all questions were revised to focus on systems used for CACFP claiming.

2. Three of the six sponsor respondents did not understand the term “automated system”.

Revision

• Throughout this section the term “automated system” was revised to “electronic system”.

CACFP Staffing

1. Pretests revealed that respondents were not sure who to count as employees who “work on CACFP”. Several respondents thought FNS would want them to include anyone who does anything for CACFP no matter how little and thus chose to include personnel who carry out only one or two CACFP tasks per year. For example, one Head Start sponsor said that there were 42 staff working on CACFP because she had counted all of their organization’s family service workers who obtain information annually from families that is used in their application for CACFP. Other respondents counted each of the cooks and teachers at the participating sites who work on CACFP for the center site but not for the organization as a CACFP sponsor.

Revision

• Qs 17a and 18 in the CCC sponsor, 16a and 17 in the HS sponsor survey, and 15a and 16 in the FDCH sponsor survey were revised to describe the population of interest as those who “work on the CACFP on a regular basis.

2. Respondents pointed out that time spent on various functions may vary in a given month because of the time of the year or other things going on in the organization. Thus, they suggested that the question about percentage of time spent on specific functions be rephrased to ask about percentage of time usually or typically spent on each CACFP functions.

Revisions

• Copy edits made to the introductory text introducing questions about staff time spent on CACFP to clarify how respondents should answer the questions in this section.

• The introductory clause, “in a typical month” was added to the beginning of Qs 19-21 in the CCC sponsor survey, Qs 18-19 in the HS sponsor survey, and Qs 17-19 in the FDCH sponsor survey.

3. Respondents suggested that the question about having any nutrition staff was too vague. Several indicated that anyone can call themselves a nutritionist. So we edited this question to ask whether any employees have formal degree or training and/or credentials in nutrition.

Revision

• What had been one question in the survey was expanded to two. Qs 22 and 22a in the CCC sponsor survey, 20 and 20a in the HS sponsor survey, and 20 and 20a in the FDCH sponsor survey now read as follows:

Does your organization’s CACFP employ anyone who has a degree or formal training in nutrition?

Yes
No

Do any of these individuals have a registered dietitian (R.D.) or an RDN credential?

Yes

No

Don’t know

4. During pretest discussions we recognized that the questions about staff turnover flowed better right after asking about number of employees who work on the CACFP on a regular basis. Thus we moved the questions on staff turnover (Qs 18 18a, and 18b in the CCC sponsor survey, Qs 17, 17a, 17b in the HS sponsor survey, and Qs 16, 16a and 16b in the FDCH sponsor survey) up into the “CACFP staffing” section right after the questions about the number of employees working on CACFP on a regular basis and added an introductory sentence. The wording of these turnover questions was also simplified based on pretester feedback.

Training Provided by Sponsor to Sites (CCCs, HS centers or FDCHs)

1. Respondents had several questions and comments about this short section. First, they almost uniformly did not understand the term “in-service.” Second, they strongly recommended that training be defined to focus on more structured training and exclude any informal training they provide during monitoring visits in their responses or that may be provided ad hoc in response to sites’ telephone or email requests.

Revisions

• To define what kind of training should be included and how to differentiate training from other help sponsors provide their CACFP sites, the introductory text for this section was revised, as follows:

In this section, we are interested in the CACFP-related training your organization provided to family day care home providers during the past 12 months. In your responses, do not include any informal training you or your staff may provide during monitoring visits or in response to individual requests for assistance.

• The term “in-service” was removed from all text and questions in this section.

2. 2. Center and Head Start sponsors noted that the response choice “caregivers” in the question about types of child care staff who received training is not a term they use to describe staff in center classrooms.

Revision

• The second response choice in Q 23a in the CCC sponsor survey and Q 21a in the HS sponsor survey was revised from “caregivers” to “classroom staff.”

3. In answering the question about frequency of training, large sponsors counted the total number of trainings they provide to all centers rather than trainings provided per center.

Revision

• Wording of question and response choice for Q 23c in the CCC sponsor survey, 21c in the HS sponsor survey, and 22b in the FDCH sponsor survey was revised to help the respondent focus on an individual typical site they sponsor:

“Thinking about a typical (center or family day care home) that you sponsor, how many times in the past 12 months did your organization provide CACFP training for that (center or home)?”

Monitoring Visits

1. Respondents were consistently confused about the overlap or difference between the response choices related to menu- and meal-related areas reviewed in their monitoring efforts

Revision

• We reworded and reorganized the response choices for Q 27 and 28 in the CCC sponsor survey, Q 25 and 26 in the HS sponsor survey, and Q 26 and 27 in the FDCH sponsor survey so that the first of these two questions asks about two most important “claiming and menu-related areas reviewed” and the second asks about two most important “meal-related areas observed and reviewed.”

2. Two pretest respondent filled in “menu production records completed (with quantities)” as an “other, please specify” response choice for menu-related areas they review during monitoring. During the pretest one respondent said that this is one of the most common deficiencies she finds during monitoring visits.

Revision

• “Menu production records are completed with quantities” was added to the list of response choices for the question regarding “claiming and menu-related areas reviewed” (Q 27 in the CCC sponsor survey, Q 25 in the HS sponsor survey, and Q 26 in the FDCH sponsor survey) because two pretest respondent filled this choice in their other response choice and said this was the most important menu item they monitor for and find a need for more training on.

• “Failure to fill out menu production records correctly” was added as a response choice to the survey question about most common deficiencies found during monitoring visits (Q 30 in the CCC sponsor survey, Q28 in the HS sponsor survey, and Q 29 in the FDCH sponsor survey).

3. Within the response choices for the question about the most common deficiencies found, several respondents did not understand the choice “conduct or conditions that threaten the public health or safety” or its relevance to CACFP monitoring.

Revision

• This response choice was deleted from Q 30 in the CCC sponsor survey, Q 28 in the HS sponsor survey, and Q 29 in the FDCH sponsor.

4. Center sponsor respondents explained that “provider not present” is only a relevant monitoring issue with FDCHs.

Revision

• “Provider not present” was deleted as a response choice for Q30 in the CCC sponsor survey and Q 28 in the HS sponsor survey.

5. Pretest respondents pointed out that the first monitoring visit to a new CACFP site should always be announced. We assumed then that FNS did not want this first visit included in responses to this question and made the revision below.

Revision

• Q 32 in the CCC sponsor survey, Q 29 in the HS sponsor survey, and Q 30 in the FDCH sponsor survey were revised to begin the questions with the phrase “Not including the first visit made to new sites,…”

Satisfaction with State CACFP Agency

Some pretest respondents told us that they were not able to answer some of these satisfaction questions. For example, some respondents were not at the organization when they first applied to participate as a sponsor for CACFP.

Revision

• We added don’t know and not applicable response options to the satisfaction question series in Q 33 of the CCC sponsor survey, Q 31 of the HS sponsor survey and Q 32 of the FDCH sponsor survey.

Sponsors Perceptions of the CACFP

1. When asked to list the most important benefits of the CACFP, many of the respondents (sponsors and providers) emphasized that their responses to this question would be more meaningful to FNS if they were able to rank them as the first, second and third most important benefits of CACFP. We agree and in fact considered this option in our earlier internal drafts of the survey instruments.

Revision

• Q 34 in the CCC sponsor survey, Q 32 in the HS sponsor survey and Q 33 in the FDCH sponsor survey are revised to read as follows:

“The following is a list of possible benefits of the CACFP. Please rank the three benefits you consider to be the most important, with “1” being the most important, “2” being the second most important, and “3” being the third most important. (Rank 3)”

2. Pretest respondents pointed out that they may consider the program not burdensome at all or if they consider it burdensome overall this does not mean that every area of the program is burdensome. Their responses illustrated that that the draft survey as structured to focus on each area of burden did not allow responses to choose the option of “no burden at all” for the overall program or to differentiate their perceived level of burden for different aspects of the program.

Revisions

• Q 35 in the CCC sponsor survey, Q 33 in the HS sponsor survey, and Q 34 in the FDCH sponsor survey had “no burden at all” added as the first response option.

• A new skip pattern was inserted so that any follow-up questions in this section are only answered only if respondent answers a high or very high burden level to Q35 in the CCC sponsor survey, Q 33 in the HS sponsor survey, or Q 34 in the FDCH sponsor survey.

• Should respondents report a high or very high level of burden with CACFP, the follow-up questions in this section are also now organized with a lead question about the level of burden for that aspect of the program. For each area of potential burden, respondents now skip out of the follow-up question about what activities cause burden in any area unless they report a high or very high burden level in that area.

As an example one of these screener burden questions now reads:

How would you rate the level of burden for your organization for performing CACFP enrollment activities?

No burden at all 

GO TO QUESTION 36

Very low burden 

Low burden 

Neither high nor low 

High burden 

Very high burden 

3. Regarding the questions about why child care centers or family day care homes leave the CACFP, no respondents reported collecting information from centers or homes on why they left. If FNS wants to maintain this question, we propose revising the question wording as described below and have added two response options recommended by pretest respondents and look forward to FNS input on this option.

Revisions

• Q 43 in the CCC sponsor survey and Q 42 in the FDCH sponsor survey were edited to begin with “What do you think are the two most common reasons…” instead of “What does this information show as the two most common reasons?”

• Unannounced site monitoring visits and serious deficiency process were added as response choices to this question based on what pretest respondents provided as “other, please specify” responses.

II. CENTER AND HOME INSTRUMENTS: REVISIONS BASED ON PRETEST FINDINGS

Revisions Common Across Instruments

• Added underlining of key words for emphasis in introductory text and questions.

• Clarified instructions at beginning of many of the sections.

• Fixed some skip patterns where needed or deleted skips where no longer relevant.

• Changed the word “provider” to child care “site” or “center” in the CCC, independent child care center (ICCC), and Head Start center surveys because respondents consider the word “provider” to only refer family day care homes.

• A “don’t know” or “not applicable” response choice was added to a few of the questions in all the surveys.

• Clarified that centers and homes should be answering the survey only for the site where they work and that providing direct child care services at the address on the survey cover letter. In the majority of the pretests with the lead CACFP staff at CCC and HS centers we learned that the individual child care site where the respondent works is managed or administered by an umbrella organization that has more than one child care center. We were told that the meal claiming process, meal claims, and other data are kept at the central office level. Hence these respondents were not sure whether they should be providing information on their site alone or for all the CACFP-participating sites their organization manages or administers. This was most evident in questions related to general characteristics of the site, CACFP staffing, and meal claims and service. All pretest respondents confirmed that if the term “site” was clearly emphasized throughout the survey they would have known to respond to the questions based only on their experience with their individual site.

• In the introduction at the beginning of the center surveys we added the following text:

• When completing this questionnaire, please think ONLY of the child care site at the address listed in the cover letter that came with the questionnaire packet.

• Base your answers on your experiences with this site only.”

• Where appropriate, in the questions and response choices in each of the center instruments we also replaced the word “center” with the word “site”.

Revisions Specific to the Head Start Center Instrument

In the introductory text for this instrument, we added language clarifying that respondents who run a mix of Head Start and Early Head Start classrooms should include both in their responses. The words “Early Head Start” were also added to several questions to remind respondents that they should include these classrooms in their response if they serve meals and/or snacks and participate in CACFP for those classrooms.

Section by Section Findings and Question Revisions

Important: The question numbering referenced below reflects the numbering in the current revised draft instruments, except where otherwise noted.

Child Care Site Schedule

Centers we pretested reminded us that many sites have split schedules for child care, with half-day a.m. and p.m. sessions. Thus they found it difficult to fill out the schedule table as written since there are gaps between the morning and afternoon sessions (e.g., 8:30-11:30 am and 12:30- 3:30 pm). Additionally, we realized it would be important for the survey analysis to know whether each responding site has split child care sessions so that we can understand why reported child enrollment and attendance numbers may be greater than the number of children they are licensed to serve. Several pretest respondents explained that their offices are open for more days or hours than their child care center serves children and suggested that if FNS wants the child care center hours only this should be clarified in the questions.

Revisions

• A new question was added to the center surveys (Q9 in the CCC and ICCC surveys and Q 8 in the HS center survey) asking if the site has split (a.m./p.m.) child care sessions. If the respondent says “yes” then she or he is directed to fill out the hours separately for their morning and afternoon sessions (Qs 9a and 9b in the CCC and ICCC surveys and Q 8a and 8b in the HS center survey). If respondent says “no” to having split sessions, he or she is directed to skip these split session tables and provide their hours in a table in Q 10 in the CCC and ICCC surveys and Q 9 in the HS center survey.

• The wording of the questions about the center schedule was also revised to reflect the interest in child care hours only. Specifically, the words “What hours does your (center or home) operate each day of the week” was revised to “What hours does your site usually provide care for children each day of the week?

Enrollment

Some pretest respondents were not sure whether they could double count children in the various questions about part-time and weekend enrollment and center respondents were not clear whether they should include enrollment from all sessions if they have split sessions of child care.

Revisions

• Instructions for Q 12 in the CCC and ICCC surveys and Q 11 in the HS center survey clarified that respondents should combine the enrollment from both sessions if they have split sessions.

• A clause was added to Qs 12b and 12c in the CCC and ICC surveys and Q 11b and 11c in the HS center survey to clarify that respondents can include the same children in their counts for these questions even if they fit in more than one response category.

Meal Service and Menus

1. Most center and provider respondents were not clear about the difference between the question about total number of meals and snacks, by type, claimed for reimbursement versus meals they served, so early on in the pretests we made a revision that was pretested with the second half of respondents , was easily understood by respondents and we think will allow Kokopelli to better analyze and provide information for FNS’ research question in this area: “What percentage of meals served by CACFP providers are reimbursable vs. non-reimbursable by CACFP, by meal type (breakfast, lunch, supper, AM snack, PM snack)?”

Revision

• The order of the two survey questions about reimbursable versus non-reimbursable meals served was switched (Qs 19 and 20 in the CCC and ICCC surveys and Q 18 and 19 in the HS center survey) and the wording revised as follows:

Please provide the total number of each type of meal and snack that were claimed for your child care site for CACFP in October 2014.

Breakfast |___|___|___|___|___|

Morning snack |___|___|___|___|___|

Lunch |___|___|___|___|___|

Afternoon snack |___|___|___|___|___|

Supper |___|___|___|___|___|

Evening snack |___|___|___|___|___|

Please provide the total number of each type of meal and snack that your child care site served to the children in October 2014, but were not claimed for CACFP.

(SAME RESPONSE OPTIONS AS ABOVE)

2. Pretest respondents were confused by the intent of the questions about sources of their menus. They asked if FNS wants to know about the sources of their menus only if they did not produce all of their own menus from scratch or if FNS would be interested in knowing about the sources they use to develop their menus, even if they write them or tailor them for their child care site. We revised the two questions on this subject to simplify the survey questions and allow us to more directly answer FNS’ research question in this area: “What is the source of menus used by CACFP providers?”

Revision

• In all center and home surveys (Qs 22 and 22a in the CCC and ICCC surveys and Qs 21 and 21a in the HS center survey) the respondents are first asked “What are the sources of the menus used at your (child care site or family day care home)?” Then, for those who check more than one source they are asked, “What is the primary source of the menus used….”

3. Center respondents uniformly did not understand the question intent and response choices regarding where there meals are “cooked.” For example, a few respondents said they heat up and cut up food that is brought to them from their sponsor or a central kitchen in their umbrella organization and considered that “cooking”. So these two questions were significantly revised and ready to serve was substitute for “cooked” with clear definitions. Additional response options were also added for the places that could prepare site’s meals for them, based on pretester respondent input. (Note that the wording for these questions is the same as the comparable questions in the draft At-risk Center (ARC) survey including edits suggested by FNS in their 5/7/14 comments on the draft ARC instrument.

Revisions

• Qs 23 and 23a in the CCC and ICCC surveys and Qs 22 and 22a in the HS center survey (which were moved up and were Q29 and Q29a or Q28 and Q28a, respectively, in the previous drafts) were revised with the wording shown below:

Q 23 or 22:

Are all, some, or none of the meals you serve prepared by another organization (e.g., a food bank, commercial food service vendor, or CACFP sponsor) and provided to your site as “ready to serve”? (By “ready to serve” we mean you can serve the meal as it was prepared for you with only minimal work such as heating it up or cutting it into portion sizes.)

All meals are provided to us by another

organization ready to serve 

Some meals are provided to us ready to serve

and some meals are prepared on site 

No meals are provided to us “ready to serve;”

all meals are prepared at our site   GO TO QUESTION 24

Q 23a or 22a:

Where are most of the meals you serve prepared? (Check one box)

At a central kitchen of my organization

or my CACFP sponsor 

A local school that is not my sponsor 

A commercial food service vendor 

A local restaurant or delicatessen with

a catering permit 

At a food bank or emergency kitchen 

At a homeless shelter 

At another community site 

Other 

(Please specify)

Languages Spoken

The question about primary versus other languages used by child care staff was uniformly misunderstood by pretest respondents and respondents wanted to know what the survey means by “using a language”. Several explained that many different languages are spoken to the children at their sites, but most said English is the “main” language spoken.

Revision

• Qs 24b and 24c in the CCC and ICCC surveys and Qs 23b and 23c in the HS center survey (which were 22b and 22c or 21b and 21c, respectively, in the previous drafts) were revised as shown below (Here we include both the previous and the revised draft to illustrate how the questions were modified and improved and will result in similar and likely more useful information for the analysis):

Draft question wording in pretest instrument

Q 22b or 21b: What is the primary language you and your staff use with the children at your child care center? (Check all that apply)

Q22c or 21c: What other languages do you and your staff use with the children at your child care center? (Check all that apply)

Revised draft question wording

Q 24b or 23b: What languages do you and your staff speak when talking with the children at your child care site? (Check all that apply)

Q 24c or 23c: What is the main language you and your staff speak when talking with the children at your child care site? (Check one box)

Staffing

1. Several center pretest respondents did not understand the term child to adult “ratio” so we asked respondents what term would be clearer and the revision below was made. They also said it was hard to define “usual” in that it may vary over the course of the day, for example during after school hours.

Revision

• Qs 27 and 28 in the CCC and ICC surveys and Qs 26 and 27 in the HS center survey were revised to replace the term “child to staff ratio” with “number of children per adult” and a specific time (10:00 a.m.) was included as part of the question, as shown below

Q 27 or 26:

What is the usual number of children per adult at this site at 10:00 a.m. on weekdays, for groups of 3 to 5 year olds?

Number of children per adult |___|___|

Q 28 or 27:

Is this different during weekends or evenings that your child care site is in operation?

(IF RESPONDENT ANSWERS YES, HE/SHE IS SKIPPED TO THE FOLLOWING QUESTION)

Q 28a or 27a:

What is the usual number of children per adults for groups of 3 to 5 year olds served during weekends or evenings at this site?

Number of children per adult |___|___|

2. The question about training for food service staff was interpreted very inconsistently by pretest respondents. Specifically, in answering the lead question defining the number of staff involved in food service task, many respondents said that no staff have food service tasks as their primary function, though they did like the definition of “food service” tasks. Furthermore, when answering the question about training on food service topics, pretest respondents explained that there would be a difference in their answer depending on whether FNS is interested in food service training provided as part of the annual training and/or interested in training beyond this annual State CACFP training.

Revisions

• In Q 29 in the CCC and ICCC surveys and Q 28 in the HS center survey the definition of food service tasks was maintained but instead of asking about the number of employees who “have one or more of the following food service tasks as their primary job function..”, the question was revised to ask how many employees “work on any of the following food service tasks..”.

• Next, instead of asking how many of these employees received any food service training, revised Q 29a in the CCC and ICCC surveys and Q28a in the HS center survey ask about the number of employees who received food service training as part of the mandatory annual CACFP training; and revised Q 29b in the CCC and ICCC surveys and Q 28b in the HS center survey ask about the number of employees who received food service training that was not part of the mandatory annual CACFP training.

How Child Care is Funded

This section title was revised to clarify its content because there was some confusion among providers when asked for their overall feedback on this section of the survey.

1. CCC and ICCC respondents seemed at first to be confused about this section and what the discounts were for. When we reread the questions changing the order of the questions to first ask about fee amounts, then discounts, and then whether additional, separate fees are charged for meals, they understood and were able to respond.

Revision

• The question order was changed consistent with pretest respondent feedback to improve the flow of the section so that respondents are first asked about fee amounts, then about any discounts to those fees, and then whether a separate fee is charged for meals.

2. A few pretest respondents were also concerned that the wording of the first question about discounts, as worded in the pretest draft, erroneously assumed that child care centers and providers could have different rates for children receiving subsidies versus those where parents pay the fee.

Revision

• Q 33c in the CCC and ICCC surveys and Q 27c in the FDCH survey were edited from “Does your child care center offer any discounts to families that do not receive any type of subsidy for child care?” to “Does your child care site (or “Do you” in the case of FDCHs) offer any child care discounts to families that pay for their care?”

3. One reviewer suggested adding another response choice to the list of potential discount policies and we believe this is likely to be a common response selected.

Revision

• We added “Children of people that work at the child care site (or family day care home) or sponsoring agency.” as a response choice for Q 33d in the CCC and ICCC surveys and 27d in the FDCH survey―a multiple choice question asking about the basis for sites’ child care discounts.

Training and Assistance by Sponsoring Organization (or State CACFP Agency for ICCCs)

• The title of this section was changed to more accurately reflect the focus of the questions.

• Two questions were added asking respondents about their level of satisfaction with the training and the technical assistance from their CACFP sponsor.

• This section of the ICCC survey was revised to make all changes that were made in the comparable section of the CCC sponsor survey.

CACFP Monitoring Visits

Questions and response choices in this section of the center and home surveys were revised to make them consistent with the questions and response choices to sponsoring agencies in the comparable section of the sponsor surveys.

Satisfaction with CACFP

Some pretest respondents gave us “don’t know” responses or said certain questions in this section were not applicable to them. For example, one center site director said she has not direct experience with the processing of CACFP meal claims because all of her meals are reimbursed to her umbrella organization which prepares the meals and delivers them to her site).

Revision

• “Don’t know” and “not applicable” response choices were added to all of the questions in this section.

Perceptions of the CACFP

The majority of the sponsor and one provider pretest respondent suggested that the question asking them to choose the three most important perceived benefits of the CACFP should be revised to have them rank among those they think are the three most important benefits. When asked to list the most important benefits of the CACFP, many of the respondents (sponsors and providers) emphasized that their responses to this question would be more meaningful to FNS if they were able to rank them as the first, second and third most important benefits of CACFP. We agree and in fact considered this option in our earlier internal drafts of the survey instruments.

Revision

• In Q48 of the CCC and ICCC surveys, Q 43 of the HS center survey, and Q 41 of the FDCH survey response choices the question wording was revised as follows:

“The following is a list of possible benefits of the CACFP. Please rank the three benefits you consider to be the most important, with “1” being the most important, “2” being the second most important, and “3” being the third most important. (Rank 3)”

Submitted by:

Kokopelli Associates, LLC

Policy Research and Evaluation

Submitted by:

Kokopelli Associates, LLC

Policy Research and Evaluation

May 12, 2014 1