OMB Control Number 0584-NEW: Child and Adult Care Food Program (CACFP) Family Day Care Homes Meal Claims Feasibility Study
Appendix A-4. Summary Report of a Pilot Study
For the CACFP Family Day Care Home (FDCH) Meal Claim Feasibility Study, Manhattan Strategy Group (MSG) designed and developed two digital data collection platforms, the Meal Service Reporting System (MSRS) and the Child Attendance Reporting System (CARS). MSRS enables FDCH providers to report meal serving times through a smartphone mobile application or a website. It does not replace the current procedures that providers use to file their monthly meal claims. CARS will collect the child’s daily attendance from parents through a text messaging system or a website. We use these data, in combination with secondary data collected from sponsors, to estimate improper payment.
In addition to the technologies, we developed detailed step-by-step user guides for MSRS and CARS. We organized the draft user guides to provide a brief rationale for the feasibility study, describe the data to be reported and how it would be used and protected, and teach parents and providers how to use the technology tools for their daily reporting responsibilities. The user guides present step-by-step instructions, accompanied by screenshots, for accessing the technology platforms from mobile phone and web, reporting information correctly for each screen or situation (i.e., a drop-off), and making corrections to information once reported.
We
conducted a pilot test as part of the technology development effort
to test the functionalities of the data collection platforms with a
small number of providers and parents. The pilot test included
three components: (a) cognitive testing of CARS and MSRS
(1
week), (b) pilot testing instruments with real data collection (4
weeks), and (c) exit interviews with parents and providers (1 week).
During the cognitive test in late August 2015, we collected feedback
from parents and providers on the interface design of the platforms.
We then implemented MSRS and CARS for the entire month of September
2015 to test their functionality with a small number of providers and
parents. To better understand the experiences of parents and
providers during the pilot test, we conducted exit interviews in
early October, the week immediately after the pilot test month.
In this memorandum, we document the implementation of each of the three components in the pilot test. We start with a description of the recruitment process for the cognitive testing and the pilot test, then we summarize both the procedures for implementing each component and the findings and lessons learned.
We used a two-stage process recruitment strategy for the pilot study, conducting recruitment activities first for the cognitive testing and then for the pilot test. Before beginning the recruitment process, the MSG team identified Maryland for cognitive testing and Texas for the pilot study. The two states are different in both geographic location and size, and also provide diverse sponsors, providers, and parents to supply critical feedback on the technology developed for this feasibility study.
We conducted the recruitment process for the pilot study from May to August 2015. Specifically, recruitment of sponsors, providers, and parents for the cognitive testing took place in May and June 2015 and recruitment for the September pilot occurred in late July and August 2015.
MSG began the recruitment process for the pilot study by obtaining permission for the study and seeking State CACFP agency contact information from Food and Nutrition Service Regional Offices (FNSROs) for the two States identified for the pilot study. The MSG team developed a request letter, which was reviewed and approved by FNS. The request letter provided a detailed overview of the study and the technology platforms to be piloted, and included a request for support for the study. FNS sent the request letter to the FNSROs on behalf of the study. Within 2 weeks, MSG received formal permission to proceed, and obtained the State CACFP agency contact information.
With permissions from FNSROs, the MSG team prepared and sent a study invitation letter to the respective CACFP State agencies in Texas and Maryland. In the letter, we provided an overview of the study and the technology platforms to be tested. We also requested the State agency’s assistance in identifying up to three sponsoring organizations for participation in the cognitive testing or the pilot test in September. (We made this request because there is no national database of CACFP participants that we can use to identify potential sponsors for the feasibility study.) We asked State agencies to provide the sponsor name and contact information, and the number of currently active FDCHs associated with each of these sponsors, by tiering status, to help the MSG team select diverse sponsoring organizations for the pilot study. We asked State agencies to respond within a 2-week time period. MSG conducted follow-up contacts with both Maryland and Texas representatives to confirm receipt of the letter and to obtain the needed information.
Both Maryland and Texas provided the contact information for three sponsoring organizations for the pilot study. We developed a set of study invitation materials for sponsoring organizations that provided a more detailed description of the rationale for the pilot study, the technology platforms, and pilot test activities. In the invitation letter, we also requested information about 10 FDCHs that are currently active and claimed reimbursement in May 2015 (or the most recent month available) with enrollment of at least five children. Specifically, we asked for these FDCHs’ contact information, tiering status, and enrollment information. We asked sponsoring organizations to provide the necessary information electronically or in paper format within a 3-week period.
The MSG team conducted follow-up calls with sponsors in Maryland and Texas to confirm receipt of the letter, review the data request, answer questions about the study, and plan for receipt of the needed information. We sent letters to all six sponsoring organizations via email, inviting the sponsors to participate in the pilot study on a voluntary basis. At the time, FNS recommended that we not make participation in the study mandatory unless and until we ran into difficulty recruiting sufficient voluntary participants. MSG followed up with the sponsors via phone. One sponsor in Maryland required approval from the Institutional Review Board; MSG, in consultation with FNS, determined that the timeline for the pilot study would not permit the time required for a full, human subject review process, and decided not to pursue that sponsoring organization any further. Two sponsors in Texas turned down the invitation, citing concerns over potential burdens. In the end, two sponsors in Maryland and one in Texas agreed to participate and provided the requested information for FDCH providers.
Provider Recruitment. The MSG team requested and received a list of 20 total FDCHs from the two Maryland sponsoring organizations. Among these FDCHs, we contacted 10 providers and successfully recruited two to participate in the cognitive testing. After reviewing the FDCHs, the MSG team prioritized those FDCHs with more than seven children enrolled,1 and with diverse tiering status, length of program participation, and methods providers used for claiming meals. These steps were taken to ensure that those selected for the cognitive testing would be a diverse set of providers with robust enrollments to support the selection of parents for the cognitive testing.
MSG initially selected five providers from this list to receive the study invitation materials to participate in the cognitive interviewing. MSG experienced some significant challenges in initially getting providers to respond to the study invitation letter. Among this first set of providers, MSG was able to recruit one provider within the initial 2-week period to the cognitive testing. After 10 days of calling the first set of providers, the MSG team sent the invitation letter to a second set of five providers to mitigate the unforeseen challenge in contacting providers. We successfully identified another provider for the cognitive interviewing.
Parent Recruitment. The MSG team invited parents to participate in the cognitive testing if their children were enrolled in the FDCHs we contacted for cognitive testing. We sent them study materials including an invitation letter describing the study and the cognitive interviewing, as well as a list of FAQS for parents to review. The MSG team followed up with parents to confirm their receipt of the letter and to answer their questions about the interviewing. MSG sent 29 parent invitations; after several days of call attempts, parents began responding to the study invitation requests. MSG successfully recruited nine parents for the cognitive testing, although these efforts took longer than initially planned.
In summary, the recruitment experience for providers and parents required more effort than the MSG team initially anticipated. The timing of the recruitment over summer vacation months when providers and parents have irregular schedules was certainly a major challenge. Based on our conversations with the sponsors, we also determined that inclusion of the mandatory participation statement is an important tool to gain the attention of the providers. It also became clear that the recruitment strategy should be revised to increase the amount of contact with the sponsor in order to involve them more directly in supporting the recruitment of providers to the study.
The recruitment for the September pilot test in Texas began at the end of July. Based on the experience with the cognitive test recruitment, MSG revised the recruitment strategy and actively engaged the sponsor in Texas to notify the selected providers about the pilot test and communicate with them about the expectation to participate in the study. The sponsor identified concerns that providers might have in response to the invitation later, which we shared during this contact. We then revised the letter to clarify details related to the pilot test, and emphasize the discretion we would take in conducting it. As a result, our recruitment conversation with the providers was productive in establishing buy-in for the pilot test.
After the sponsor in Texas made additional contact with providers for the study, the MSG team sent the study materials to four providers. These materials included an invitation letter and a list of FAQS to help providers understand the pilot test. Instead of waiting for the providers to gather child enrollment information, which could have been another deterrent, MSG asked the sponsor to provide child enrollment information for the four providers selected for the pilot test. This strategy was successful as well.
Four providers having agreed to participate in the pilot test, MSG began to recruit parents for the September pilot test. The parent recruitment for the pilot test took place between August 7 and August 24, 2015. The MSG team started with two of the four providers; all of the parents for these two providers received the study invitation materials, which included a letter describing the pilot test and the provider’s role in the pilot, as well as a set of FAQS tailored for parents. MSG found that this recruitment of parents was both slow and challenging. Many parents refused to participate or were unreachable during the first week of outreach. We then started recruiting additional parents from the remaining two providers. A total of nine parents eventually agreed to participate.
The intent of the cognitive testing was to understand how providers and parents interpret the functions designed for MSRS and CARS, as shown on the website and/or their mobile phones, so we could improve the design of the technologies and the clarity of the user guides. Of particular concern was how intuitive the systems were and whether respondents would provide the type of data needed and anticipated by the technology platforms. The cognitive protocols assessed whether the respondent could correctly interpret the instruction/question; whether the respondent could retrieve the information needed to answer the question or perform the function; whether the respondent was willing to provide the desired response or whether they simply did not know or could not tell what to provide; and whether the respondent could provide the answer in the format requested.
The final protocols included the interview guide for the cognitive interview, a set of CARS/MSRS screenshots depicting the screens in both systems for smartphone and website applications alike, and a series of vignettes to review during the cognitive testing. Each question in the protocol included a set of possible probes to be used to obtain further detailed information from the respondents, for both the provider and the parent interviews. The cognitive test materials were accompanied by an informed consent document. Prior to each interview, MSG obtained an informed consent from each study participant. Once participants consented, we audio recorded the interviews. Interviewers also took detailed notes using the interview protocols. Cognitive testing was conducted with nine parents and two FDCHs.
Providers did not demonstrate any cognitive issues related to comprehension, clarity, recall, or sensitivity for a majority of the statements presented in the MSRS system and its user guide. Providers shared some suggestions for improved language for two statements, but did not feel that the statements were difficult to understand. Providers considered the reporting requirements for the MSRS system reasonable and less burdensome than what they normally would report to a sponsor. In response to probes about the likelihood of a provider using MSRS daily for a month, providers shared that they believed compliance with the system might depend on the recordkeeping practice they already used for meal claims. One of the providers in the interview was an official trainer of providers in partnership with her sponsor who observed that some providers, especially those who have been in the CACFP for a long time, may be more negligent about the daily reporting requirements.
The cognitive interviews also indicated that respondents displayed general understanding of the system’s functions, but highlighted several areas where improvements would be needed to support the participation of providers. While they clearly understood the steps for logging into the system and reporting a closure, providers noted that the system’s menu was not in an order that coincided with the workflow that providers complete on a daily basis. Both providers indicated that the system needed to mimic the daily order of operations at FDCHs and be as simple as possible to follow, especially because using MSRS is an additional reporting burden to providers. Relatedly, providers commented that some of the language used in the MSRS system menu did not make sense to providers or was not intuitive. The interviews also revealed a set of potential user issues associated with the meal reporting function. Providers also identified some additional user issues with the display of information on the Summary page, and recommended altering the display so that providers can obtain a monthly summary.
Based on the feedback from providers in the cognitive interview, MSG revised the system’s menu page, reordered the workflow of the system, and altered the Summary page as described by providers. We also changed the wording as suggested by providers for the meal reporting and My Info screens. We did not add instructions to the MSRS meal reporting page, but instead provided clearer instructions in the MSRS user guide.
Overall, respondents view texting CARS as feasible for most parents. However, about half of the parents interviewed also suggested there might be some challenges in getting parents to add another task to their busy schedules, even if texting is fairly easy. All of the respondents demonstrated a consistent understanding of what to report. They clearly grasped the concepts of drop-off, pick-up, and absences. For example, all nine parents uniformly described an absence as a time when the child was not attending day care, excluding times when the day care is closed.
All of the respondents understood that CARS is an electronic child attendance reporting system that could be accessed via mobile phone or computer. All parents appreciated the language about the study not altering the provider’s operations and noted this was critically important to them. All of the respondents found the statement about data protections to be necessary to support their participation in the study.
An emerging finding from the interviews suggests that parents would be able to provide a fairly accurate reporting of drop-off and pick-up time based on their daily routines. About half of the parents suggested that if they were using CARS, they would text immediately after dropping off or picking up their children, while the remaining parents indicated they would likely text a drop-off time around lunchtime and a pick-up time after children had been settled for the evening. A majority of parents indicated they would not be able to text their children’s school bus drop-off or pick-up times, confirming our expectation that parents know only the suggested schedule and would be unable report the exact time. All nine parents indicated they would be willing to text additional drop-off or pick-up times for a child’s doctor’s appointment, because they would readily know this information and the frequency of this type of event would be low.
All respondents reported that CARS text reminders would be extremely helpful in reminding them to report daily. They would likely tolerate receiving two text reminders per day, but not additional texts. When presented with the user scenarios, all of the parents provided the correct text for their child in response to the specific scenario. Parents found the CARS website very intuitive to navigate and had a sound understanding of how to enter correct data for their child’s attendance.
Based on the outcomes of the cognitive testing, MSG recommended very few changes for the CARS system. The two improvements that MSG made were to alter the timing of the reminder to be sent at 8 p.m., and to replace SET with SEND when selecting reporting time. We addressed additional suggestions, revising and clarifying one of the statements during the revision of the user manual.
The findings supported the conclusion that, overall, parents and providers successfully navigated the CARS and MSRS systems, with minor changes. The findings also supported the conclusion that users provided the expected types of responses, and had no comprehension issues with the data requests. MSG developed recommendations based on the findings, and updated the user guides and systems to accommodate providers and parents better.
We now proceed to summarize the activities conducted for the September pilot test, starting with a description of the recruitment efforts.
The
pilot test officially started on September 1 and concluded on
September 30, 2015. A key component of the pilot test was the
onboarding process MSG undertook to provide pilot test participants
with access to the technology platforms, familiarize them with how to
use the technology platforms correctly, and confirm understanding of
the daily reporting responsibilities of parents and providers. To
meet this need, the MSG team developed study onboarding procedures
that we implemented a week and a half before the official start of
the September
pilot test.
All providers and parents successfully began to report data in MSRS and CARS on September 1. Over the course of the pilot test, MSG maintained a data dashboard that displayed selected system information/data variables, which allowed the field operations staff to monitor reporting from respondents on a daily basis. The study team staffed the study hotline and monitored email daily during the pilot test between 8:30 am and 5:30 pm.
MSRS. We followed up with providers if we did not receive meal service data for a day and a closure was not reported. One provider reported inconsistently. Between September 11 and 28, the study team made nine attempts to contact the provider via phone on separate days at different times and regularly left voicemail messages during these calls. However, the provider did not respond.
CARS. No non-response follow-up outreach was required for parents during the CARS pilot test. Although a few parents missed reporting data, no parent reached the 3-day threshold established by the MSG team for non-response, follow-up contact. No parent missed two consecutive days of reporting in CARS.
We used two sources of information to assess the performance of MSRS and CARS: (1) user responses during the pilot test month and (2) user feedback from exit interviews. Furthermore, we determined the functionality of MSRS and CARS by assessing the response rate, user experience, and potential burden on the users. In this section, we first describe response rates and response patterns based on 135 responses from providers2 and 189 responses from parents in the pilot test month. We then discuss issues with accessing and using the technologies and synthesize user experience from the exit interviews. Last, we look at the reporting burden as perceived by providers and parents.
Response Rate. The daily response rate for providers is calculated as follows: the number of weekdays a provider reported any data for any child divided by the number of weekdays at least one child was present. The response rate at the provider level is 84.8 percent. One provider, who was responsible for 48.4 percent of reporting, had a particularly low response rate of 59.5 percent. If in fact she had responded at rates similar to those of the other providers, the overall response rate would have been a substantially higher 92.2 percent.
During the study month, we expected to receive 153 entries for all children who were present on weekdays in September. MSRS received a total of 135 entries from providers, of which 18 entries were reported for children who were absent. We took out these 18 entries and calculated the response rate with the remaining 117 entries (135–18), which yielded a response rate of 76.5 percent at the child daily record level.
The
response rate did not vary widely over the course of the month; the
highest response rate, 87.5 percent, was observed in the first week;
the lowest, 82.4 percent, in the third week. Although errors of other
types are possible, an MSRS entry is considered valid if an FDCH is
open and the provider
reported meal serving time in a chronological order within the
opening hours. No provider reported partial closure or modified
opening or closing times during the
pilot test.
The overwhelming majority of providers report serving breakfast, lunch, and afternoon snacks as opposed to any other meal option. Breakfast, lunch, and afternoon snacks were reported 133, 114, and 121 times, respectively, while the only other meals reported, morning snacks, were offered on only nine occasions.
The parents’ response rate was 100 percent, with all parents reporting at least some data on every weekday except for Labor Day, on which eight parents failed to report their children absent. Most parents responded regularly and accurately, although some performed better than others. In 94.3 percent of cases, parents reported both drop-off and pick-up times within 5 hours of the actual drop-off or pick-up. This reduces the likelihood that the data suffer from problems with respondent recall.
Data Quality. Providers reported valid open/closure information and meal service data 95 percent of the time. The overwhelming majority of providers reported serving breakfast, lunch, and afternoon snacks as opposed to any other meal option. Breakfast, lunch, and afternoon snacks were reported 167, 186, and 180 times, respectively, while the only other meals reported, morning snacks, were offered on only nine occasions.
Parents reported valid data in 95.3 percent of cases. While parents generally provided reliable, plausible responses throughout the month, their highest error rates occurred in week 2, when 26 percent of responses contained at least one error. However, by week 5, only 3 percent of responses contained errors, and in general error rates were very low after week 2. Only five out of 196 text messages (2.6 percent) were resubmissions or corrections. In four of these cases, pick-up times were revised, while in one case an absence notification was resubmitted.
To understand more deeply the experiences of parents and providers during the pilot test, we conducted exit interviews in early October, the week immediately after the pilot test month. We formulated the exit interview protocol to assess the user’s experience with either CARS or MSRS mobile and web applications, and also to ascertain how respondents arrived at the data provided during the pilot month. Furthermore, we developed a semi-structured interview protocol for both CARS and MSRS for use with parents and providers. We conducted the exit interviews by telephone, and designed these to last 30 to 60 minutes for each audience.
MSRS Exit Interview Findings. The exit interview findings suggest that MSRS is fairly easy for providers to use on a regular basis. The MSRS smartphone application worked well and was the primary mode for providers to communicate with the database. Three providers used the smartphone application almost exclusively; the fourth provider reported that she used the website exclusively.
Providers find the MSRS system easy to use and less burdensome than the reporting required by sponsors for providers to receive reimbursement. Three of the four providers found that the daily reporting generally took 15 minutes or less.
Providers were motivated to report daily in MSRS over the pilot test month because they believed the MSRS reporting was simply another requirement. They were thus able to assimilate it into their schedules with little extra burden. These providers described themselves as “rule-followers” and stated that “if I’m asked to do something, I want to follow through and do it.” It should be noted that one of the four providers reported feeling challenged to balance the reporting for the sponsor and MSRS. She would need to access a laptop and report very detailed information for the sponsor-reimbursement report. She was typically reporting for the sponsor first and accessing MSRS afterwards, which she sometimes forgot to do during the pilot month. When she was busy with day care children and other family needs, she easily forgot MSRS reporting. This provider specifically made the recommendation for MSRS to include some reminders to help providers remember to report daily. For some of the providers, the $100 stipend for participation in the pilot test was also sufficient motivation for a consistent response.
All providers indicated that MSRS was easy to navigate and that the menu and logical flow of the system made the reporting conceptually easy to complete. Providers did not find the information that MSRS asked for to be burdensome. All of the providers used the batch feature when reporting meals, which reduced their reporting time even further.
Providers found the training they received was sufficient for them to install, access, and use MSRS successfully. While all the providers received a user manual, only half of them read the manual again after the training; the training was enough to get them to report the correct data consistently. None of the providers reported any technical issues that required them to refer to the manual for guidance.
Providers also identified some potential benefit to using MSRS. They explained that the monthly summary in MSRS can help them submit claims for reimbursement. For this reason, they stressed their desire to download and print the monthly summary from MSRS.
The exit interview findings also suggest that providers were fairly motivated to provide accurate data reflecting the status of children in their homes at the times of the meal service, the types of meals served, and the time the meals were served. Providers were asked which methods they used to report the most accurate meal times in the system. All the providers had some method for noting the times meals were served and then recording this information in the MSRS system. For three of the providers, reporting often happened right after the meals were served. These providers reported entering data into the MSRS soon after the end of the meal service. One provider described completing her reporting in MSRS in the evening, after the day care had closed.
CARS Exit Interview Findings. Exit interview findings suggest that CARS is highly intuitive and easy for parents to use. Parents seemed to think daily reporting in CARS was a minimal to moderate burden and quickly built the reporting into their daily routine. The exit interviews support this suggestion. Half the parents indicated they typically reported data as they completed the drop-off or pick-up of their child. Other parents described efforts to coordinate information from another parent who texted the pick-up or drop-off time so that it could be reported. Parents generally believed that it merely took approximately 5 minutes per day to provide text responses for CARS during the study month. Parents also indicated they had little difficulty coming up with a time to report that was supported by another source, either their watch, cellphone, or another readily available clock, like one at the provider.
All the parents found the system’s daily AM and PM reminders extremely valuable in reminding them to report. While 44 percent of the parents described reporting prior to receiving the reminders, the majority of parents needed the reminders to prompt them to report at that time. Parents decided to participate in and comply with the study, largely because they wanted to do something positive for the food program (CACFP).
MSG asked parents participating in the pilot study to try both the mobile and the website versions of CARS for the pilot test. Findings from the exit interviews indicate that despite this request, parents demonstrated a very strong preference for the text-based version of CARS. While about half of the parents did log in to the website at least one time, only three parents actually reported using the CARS website to provide their child’s attendance data. Parents ascribed their preference for the text-based version to their easy access to their cell phones over the course of the day.
The outcomes of the pilot study provide strong evidence that MSRS and CARS are effective tools in capturing meal serving time and child attendance. Such information is critical for the feasibility study to estimate meal claim errors. The pilot test also validates our assumptions that it is reasonable for providers and parents to implement and use MSRS and CARS as expected. We are able to confirm that providers and parents in general had easy access to mobile phones or the Internet and were willing to use them for the purpose of this study. A majority of providers exclusively used smartphones to access MSRS. The pilot test also confirms the importance of having a web version of MSRS; indeed, one provider exclusively used the reporting website via a laptop computer. Almost all parents preferred using CARS via mobile phone, primarily due to easy access to text messaging. This suggests that the primary purpose of a web application for parents would be as a backup in the unanticipated event of a missing or nonfunctional phone.
Providers and parents find MSRS and CARS easy to learn and intuitive to use. They are able to quickly integrate the technologies into their daily routines and report meal serving time and child attendance on a daily basis. Parents and providers can access and use the technology platforms through different mobile phone systems and web browsers. No users experienced technical issues due to system failures in the pilot test month.
1
We prioritized our sampling list to first contact those providers
with more than seven children enrolled to ensure
a robust
parent sample for the next phase of recruitment for the cognitive
testing. We believed that the increased enrollment would offset
cases when the provider’s enrollment reflected parents of
multiple children, thus reducing the number of actual parents in the
home. This approach was also established to minimize the burden for
the FDCHs, as our recruitment strategy was developed to minimize the
chance we would need to go to more homes
for the cognitive
testing due to a limited number of parents.
2 The number of MSRS responses providers submitted for all children during the weekdays in September 2015.
A-4.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Erika Gordon |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |