Pilot Study Memo to OMB

,

MEMORANDUM

P.O. Box 2393 Princeton, NJ 08543-2393 Telephone (609) 799-3535 Fax (609) 799-0005 www.mathematica-mpr.com

TO: Bridget Dooling and Jennifer Park

the

FROM: Susan Jenkins, OPE/ACL/HHS DATE: 6/15/2015

SUBJECT: Results of the OMB Requested pilot study for the outcomes evaluation of the Elderly Nutrition Services Program

As part of the approval for the information collection provisionally approved under OMB Control number # 0985-0037, OMB requested that ACL conduct a pilot study to study the likely success of the proposed approach for collecting outcome data from Local Service Providers and members of the intervention and comparison groups. Therefore, Mathematica, ACL’s contractor for this information collection, conducted a small-scale pilot of the client outcomes study to test the operational aspects of data collection for this evaluation. The pilot included (1) the client interview with the 24-hour dietary recall and client outcomes questionnaire, and (2) the local service provider (LSP) menu survey. The purpose of pilot-testing these instruments was to provide information on respondent burden, ASA24 (the web-based automated self-administered 24-hour recall tool) administration and features, and the usefulness of supplemental forms and scripts in collecting the 24-hour recall and menu data. Below describes the data collection aspects addressed in this pretest:

1. Respondent burden and fatigue for the client outcomes interview. Given the age and health of the study population, Mathematica assessed whether clients could complete the 24-hour dietary recall and client outcomes questionnaire in the 77 minutes estimated for these instruments. They tested whether placing the dietary recall at the beginning, middle, or end of the client interview might affect the overall length. They also assessed levels of respondent fatigue and engagement to determine the best placement of the dietary recall.

2. Use of ASA24 features for the 24-hour dietary recall. Mathematica previously adapted procedures for the ASA24 to conduct is as an interviewer-administered instrument instead of a self-administered instrument, as it was originally designed. They anticipated that interviewers would need to conduct interviews in-person to engage respondents and to collect dietary recall data. The ASA24 includes visual aids of containers and portion sizes, which can help respondents estimate the amount consumed. The pilot provided the opportunity to test the usefulness of these visual aids with a frail, elderly population.

3. Use of ASA24 features for the menu survey. While ASA24 is designed to be a tool for collecting information on individual dietary intake, the pretest offered the opportunity to test the feasibility of using it to collect menu data, and thereby assess the food and nutrient content of offered/delivered meals. This involved having interviewers modify the questions in ASA24 to ask respondents about everything offered or served for the midday meal.

4. Identifying respondents for the menu survey. The pretest allowed Mathematica to better understand how well the planned approach to contacting sites and identifying the menu survey respondent allowed them to interview the person who is most knowledgeable about the meals served at each site.

Pilot recruitment and sample. Mathematica contacted two Area Agencies on Aging (AAAs) in central New Jersey and the one AAA in the state of Delaware (a State Unit on Aging that is also an AAA) to recruit local service providers to participate in the pilot. These three agencies participated in the process evaluation of the Elderly Nutrition Services Program but were not among the 115 AAAs selected to provide LSPs to participate in the outcomes evaluation. From these three agencies, Mathematica recruited four LSPs to participate in the pilot. Two were providers of home-delivered meals only, and two were providers of congregate and home-delivered meals. These four LSPs offered a total of five sites from which Mathematica recruited meal program clients and menu survey respondents. They also recruited an AAA in Massachusetts that is a direct provider of meal services to complete only the menu survey portion of the pilot.

Each site handled the recruitment of clients to participate in the interviews. Some sites provided Mathematica with the names and telephone numbers of interested clients, and an interviewer called the clients directly to schedule the interviews. Other sites scheduled times for interviewers to be on-site and lined up clients for each interview slot themselves. Interviewers met in person with all client participants, at the congregate meal site or at their homes (if the client received home-delivered meals), to administer the client outcomes questionnaire and dietary recall.

In general, interviewers scheduled a time to meet in person with the menu survey respondent at their sites to collect information on one midday meal. During that first meeting, Mathematica’s interviewers provided respondents with a menu form and asked them to record everything served during the congregate and home-delivered meals for two additional days. Interviewers then scheduled a follow-up telephone call to collect the meal information (including food details and portion sizes) for those two additional days.

Training interviewers. Mathematica trained three interviewers to conduct client and menu survey interviews for the pilot. The contractor selected these interviewers because they are experienced Mathematica field interviewers who had been trained on ASA24 for another project. This reduced the length of training because they needed to be trained on study-specific procedures only and not on the use of ASA24 itself.

Mathematica provided the training during a 1.5 hour webinar. The training included a brief background on the study and the purpose of the pilot, then focused on data collection procedures for the client interviews and menu survey. Trainers instructed interviewers to ask each pilot participant debriefing questions after each interview to identify procedures that worked well for the survey administration and those that did not. The trainers also instructed interviewers to observe and document their assessment of challenges during the interview. After the training, each interviewer completed a practice client dietary recall and menu survey interview with one of the trainers, to ensure they could correctly administer the dietary recall and menu survey. They also were given time to practice administering the hard-copy client interview instrument.

A. Results

Client interviews (including dietary recall). Mathematica completed 12 interviews with congregate meal clients and 20 interviews with home-delivered meal clients from five meal program sites (Table 1). Clients ranged in age from 41 to 97, and 56 percent of the clients were female.

Table 1. Characteristics of pilot test client respondents

	Congregate meal clients (N = 12)			Home-delivered meal clients (N = 20)			Both congregate and home-delivered meal clients (N = 32)
	Number	Percentage		Number	Percentage		Number	Percentage
Age
40–59	0	0%		4	20%		4	13%
60–69	1	8%		2	10%		3	9%
70–79	4	33%		3	15%		7	22%
80–89	5	42%		5	25%		10	31%
90 and over	2	17%		6	30%		8	25%
Gender
Female	8	67%		10	50%		18	56%
Male	4	33%		10	50%		14	44%

On average, the dietary recall took 35 minutes to complete (median = 32 minutes; range = 10 to 80 minutes); the client outcomes questionnaire took 72 minutes (median = 66 minutes; range = 38 to 183 minutes). Table 2 shows the average length of the dietary recall and client outcomes questionnaire, by type of client and whether the recall was administered at the beginning, middle, or end of the interview. Based both on interviewer perception and the recorded time to complete the interviews, placement of the recall at the start or end of the interview did not have a meaningful effect on the length of time needed to complete the interviews. However, interviewers preferred starting the interview with the recall, as the repetitive structuring of questions in the recall helped set the pace for the rest of the interview. Many clients (13 of 32) felt that the interview was long. After completing the first few interviews, interviewers were instructed to begin asking clients whether they would prefer completing the interview in two sessions: one in person and a follow-up by telephone. Clients overwhelmingly preferred completing the interview in person in one session (17 of 19 clients).

Table 2. Average length (in minutes) of dietary recall and client outcomes questionnaire, based on placement of dietary recall and client type

	Congregate meal clients					Home-delivered meal clients
	Placement in dietary recall					Placement in dietary recall
	All (N = 12)	Beginning (N = 4)	Middle (N = 4)	End (N = 4)		All (N = 20)	Beginning (N = 7)	Middle (N = 7)	End (N = 6)
Dietary recall	32	30	38	27		37	39	44	27
Client outcomes questionnaire	58	56	67	51		82	73	84	87
Total (recall and client outcomes questionnaire)	90	86	105	78		119	111	129	114

Interviewers noted several respondent characteristics that posed a challenge to administering the interview. These included difficulty hearing (two clients), blindness or visual impairment (two clients), forgetfulness (five clients), confusion (three clients), and fatigue or low levels of engagement (six clients). In general, these challenges were noted among the home-delivered meal clients. In three instances, a friend, family member, or caregiver helped the client complete the interview. Having a copy of their menu from the LSP¹ helped several clients who had difficulty remembering their intake from the previous day. About two-thirds of the clients also indicated that seeing images of foods and portion sizes on the laptop screen helped them estimate their intake.

Menu survey interviews. Mathematica completed six menu survey interviews for this pilot with respondents from each site where meals were served or prepared for delivery. One used a central kitchen to prepare meals, one used a caterer, three used on-site production, and one prepared and delivered frozen meals. In general, the site manager was the most knowledgeable respondent for the menu survey and was able to complete the first in-person menu survey interview. In one instance, the manager was familiar with the foods served but needed to consult with the cook to obtain details and portion sizes for the foods. One exception was the respondent whose organization distributes frozen meals to clients who are not in the same geographical location. Mathematica interviewed the corporate dietitian via telephone about the three most popular lunch-type meals, since there is no single midday meal to ask about in their meal delivery model.

Interviewers found the activities they conducted during the first in-person meeting greatly aided in collecting the same data by telephone for the second and third days. The first interview typically was the longest (38 minutes, on average)² but also trained the respondent on the information needed, such as food details and portion sizes. The menu form that interviewers asked respondents to complete for the second and third days was also a helpful memory aid. Thus, the interview to complete the menu survey for the second and third days by telephone was more efficient. In some cases, it took less time than the initial interview, even though the interviewer asked for two meals’ worth of menu data instead of one.

One benefit of meeting the menu survey respondent in person is to show the respondent the portion size visuals in ASA24. Some respondents indicated these images were very helpful in estimating portion sizes; others repored they were unnecessary because they already had portion size information for their menu items. One issue the interviewers identified with ASA24 is that the options for selecting portion sizes in ASA24 are not always in the same units reported by the respondents. For example, if a site manager reports “ounces” of a deli meat, the interviewer must first select “slices” to access the portion size visuals and can then enter the amount in ounces in ASA24.

B. Recommendations

1. Client interviews

Respondent burden and fatigue for the client outcomes interview

Reduce respondent burden. Responses to the Office of Management and Budget estimated that burden for this data collection would be approximately 1.28 hours (77 minutes) for both the outcomes survey (0.78 hours) and the dietary recall (0.5 hours). However, most interviews took longer. Because pilot sites identified participants who were willing and able to complete the survey, the sample for the full study presumably will include even more people who have significant barriers to completing the interview (such as cognitive impairments or insufficient stamina). Therefore, we recommend:

• Reducing the overall number of items in the client outcomes survey by approximately 42 percent (30 minutes). The length of the dietary recalls (35 minutes, on average) cannot be shortened while still meeting the objectives of the study. Therefore, the client outcomes questionnaire is the only instrument where burden can be reduced.

  • A revised survey is included in this package which includes a reduced number of questions to achieve the response time reduction.

• Incorporating a “frail skip” in the client outcomes survey. The contractor can train interviewers on the appropriate situations to use this skip (for example, if the respondent has difficulty focusing). The skip would allow the interviewer to bypass noncritical sections of the survey to significantly reduce its length.

  • The revised survey included in this package has the “Frail Skip” questions highlighted

• Interviewing a proxy respondent, when appropriate. In a handful of cases, the client completed the interview with the assistance of a friend, caregiver, or family member. Completing an interview with two people providing responses adds time to the interview process. Mathematica can train interviewers to identify when a proxy is needed and limit questions to only those that a proxy could answer.

Conduct the full interview (including dietary recall) in person. Despite the length of the interview, nearly all the clients asked preferred completing the client outcomes survey and dietary recall in person, in one sitting. They were not in favor of completing part in person, with a telephone follow-up to complete the rest. Interviewing in person, as originally planned, will also mitigate several challenges in interviewing this population (such as hearing impairments, sight impairments, and the respondent staying engaged throughout the interview).

Use of ASA24 features for the 24-hour dietary recall

Begin the interview with the dietary recall. Interviews that began or ended with the dietary recall were somewhat shorter than those where interviewers conducted the recall in the middle. Although interviewers did not perceive a meaningful difference in the length of the overall interview with recall, they preferred starting the interview with the recall. The pattern of questions in the recall helped to set the tone and pace for the rest of the interview.

2. Local Service Provider Menu survey

Use of ASA24 features for the menu survey

Provide respondents with a menu form to record menu details before the second menu survey interview. Some respondents found the portion size visuals in ASA24 very helpful for estimation, while others found them unnecessary. However, most respondents found that completing the menu form was helpful in preparing for the telephone interview on the second and third days. Therefore, interviewers will provide a copy at the end of the in-person interview. The contractor will incorporate a field for portion size into the menu form so respondents who might not readily have this information remember to obtain portion size information for the subsequent telephone interview.

Conduct the first menu survey interview in person, and the second by telephone. At the same time, the pilot confirmed that in most cases it is both feasible and efficient to collect data from menu survey respondents in person for the first day’s menu, followed by collecting the data for the second and third days by telephone. Reviewing the probes and screens in ASA24, as well as receiving instructions on using the menu forms in the initial meeting with the interviewer prepared respondents for the next request for menu survey data.

Instruct interviewers on how to handle differences between reported portion sizes and portion size options in ASA24. Mathematica will adapt training materials to incorporate examples of foods where there is a discrepancy in portion size units and how best to enter these foods in ASA24. They will provide instructions for documenting situations where the reported portion size unit is not available in ASA24.

Identifying respondents for the menu survey

Add instructions to training materials on how to identify the appropriate menu survey respondent. The pilot confirmed that the questions developed allowed Mathematica to identify the person best suited to provide data for the menu survey (typically, a site manager). However, it also uncovered situations in which the best respondent may not be a site manager or even a person geographically close to the clients being served. Mathematica can adapt procedures for interviewing the best menu survey respondent based on the type of meal delivery system. This would include collecting all three days of menu data by telephone from respondents located outside the immediate geographic area of the meal site.

Attachments:

1. Revised client survey with track changes

2. Revised client survey-clean version

1 Some congregate and home-delivered meal clients had a copy of the menu from the site distributing or serving the meal.

2 The interview with the corporate dietitian (not reflected in this average) took fewer than 30 minutes to collect information on all three menu options.

An Affirmative Action/Equal Opportunity Employer