ENSP OMB Part B (formatted) REVISED FINAL PROCESS AND COST ONLY11-18-13....

PART B

Part B of the Justification for this information collection activity, the Evaluation of Older Americans Act Title III-C Elderly Nutrition Services Program, addresses the five points outlined in Part B of the OMB guidelines.

B.1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

In this section, we describe the procedures we will use to select the sample of Area Agencies on Aging (AAAs)¹ and Local Service Providers (LSPs)² for the evaluation of the Title III-C Elderly Nutrition Services Program.

The rest of Section B.1 covers:

• Universe of Potential Respondents to the Process Evaluation and Cost Study

• Multi-stage Sampling

• Sample Development and Selection of AAAs and LSPs

• Sampling Frame and Identification of Program Sites, Routes, and Participants

• Sample Size

• Response Rates

• Analysis of Nonresponse Bias

B.1.1. Universe of Potential Respondents To the Process Evaluation and Cost Study

The Universe of Potential Respondents includes all State and Territorial Units on Aging (SUAs), AAAs in the contiguous United States, LSPs in the contiguous United States. There are 56 SUAs, 629 AAAs, and more than 30,000 LSP organizations.³ In fiscal year 2010, 96.4 million congregate meals were served in Congregate Nutrition Services to 1.7 million congregate participants in the Older Americans Act (OAA) Nutrition Program.⁴

B.1.2. Multi-stage Sampling

The census of SUAs will be independent of the remaining samples. The sample of AAAs and LSPs will be selected in stages, as described below. Because these samples are not nested within SUAs, the SUA census is not considered part of the sampling design for AAAs, and LSPs.

B.1.3. Sample Development and Selection of AAAs and LSPs

The sampling frame for selecting AAAs will be an electronic file provided by the Administration on Aging (AoA). The sampling frame for LSPs will be developed through contact with those AAAs subsampled (as discussed below) for LSP and individual data collection. The sample of AAAs will serve two purposes. We will use a sample of 300 AAAs to gather data at the AAA level.

The initial sample of 300 AAAs will be selected as a stratified random sample. There will be two explicit strata: (1) the certainty stratum, and (2) the noncertainty stratum. The certainty stratum will ensure that very large AAAs are not excluded from the sample. Within the noncertainty stratum, implicit stratification will be used to ensure the representativeness of the sample. The certainty stratum will include those AAAs large enough to be sampled with certainty in a probability-proportional-to-size (PPS) sample of 100 AAAs⁵ with a measure of size (MOS) defined as the estimated number of participants; all of these will be included in the initial sample of 300. The rest of the sampled 300 AAAs will be selected from the noncertainty stratum. Within this stratum, implicit strata will be defined by region of the country, size (estimated number of participants), and whether the AAA serves urban or rural areas. The AAAs selected in the noncertainty stratum will be chosen with equal probability (within that stratum).

We will use the full sample of 300 AAAs for collecting data at that level. We will select a subsample of 100 from the 300 to serve as PSUs for the samples of LSPs for the process and cost studies, and, for the client outcomes study, samples of participants and the comparison group. This subsample will include all AAAs in the certainty strata, plus others selected using PPS methods with the number of participants as the MOS.⁶ In selecting the PPS subsample, we will implicitly stratify on the same characteristics used in selecting the initial sample of AAAs.

The next step will be to create LSP sampling frames for each of the 100 AAAs in the subsample using information provided by those AAAs. After the selection of the AAAs and recruitment of these agencies for the study, each AAA will be asked to provide information on its LSPs. We will determine from each AAA whether each of its LSPs has only one program (congregate or home-delivered) or both programs and the number of people served (by those programs). After AAAs have been recruited and it has been determined what information is available about LSPs, the project will evaluate what the best size measure would be for selecting LSPs. A MOS based on numbers of participants will serve well both for the cost study and for the later client outcomes study. The MOS may be based on the total number of participants or may be a composite that incorporates estimates for the different target groups. A PPS sample of LSPs will then be selected using the chosen MOS.

We will select the LSPs after the sampling frame for the LSPs is complete and verified. An initial sample of LSPs will be selected separately (with PPS) within each participating AAA. Before selecting the samples of LSPs, we will examine their distribution to determine if stratification is necessary to ensure adequate numbers of congregate meal sites and home-delivery distribution sites. To achieve a target of 200 participating LSPs, we will choose an initial sample of approximately 222 LSPs so that approximately 200 cooperative LSPs will remain after nonresponse. The approach to sampling LSPs is:

• In AAAs with one or two LSPs, select and attempt to recruit all.

• In AAAs with three or four LSPs, select two with PPS for recruitment and keep the other(s) in reserve in case of nonresponse or ineligibility.

• In AAAs with five or more LSPs, select four with PPS and randomly select two for recruitment, keeping the other two in reserve in case of nonresponse.

B.1.4. Sampling Frame and Identification of Program Sites, Routes, and Participants

We will ask each sampled LSP to provide information on the program(s) (congregate nutrition, home-delivered nutrition, or both) it runs. For the congregate nutrition program, we will request information on the day, time, and location of each site where meals are provided. For participants in the home-delivered nutrition program, we will request information on the delivery routes, schedule for the deliveries, the type of meals provided (hot or frozen), and the quantity of meals provided at a delivery (single day or multiple days). Although this information may change, it is needed to prepare for selection of sites and delivery routes for the cost study. Assuming that the same LSPs are used to select clients for the outcomes study, this information may need to be verified and updated when the client outcomes study is conducted.

For the cost study, we will select one congregate meal site in each LSP that has one or more sites. In LSPs with home delivery, we will sample one distribution site and one or two routes within each site.

B.1.7. Sample Size

Table B.1.6.1 summarizes the planned sample sizes.

Table B.1.6.1. Summary of Sample Sizes

Respondent Group	Sample Selected	Number of Responses
State and Territorial Units on Aging	56	56
Area Agency on Aging	300	300
Local Service Provider (process survey and cost study)	222a	200a

^aApproximate.

B.1.8. Response Rates

The SUA survey is a census of a small universe (N=56); all SUAs will be requested to complete the survey. While ACL anticipates 100 percent participation because other evaluation activities conducted by ACL have reached those levels, it is possible that some SUAs will decide not to or will be unable to participate. In that case the number of responses will be smaller. However, even if there is non-response, ACL believes that it will be low and that it is reasonable to anticipate a response rate of at least 90 percent. Since the SUA survey is a census of a small universe, statistical analyses are not planned and we do not propose to weight survey responses to account for any non-response. Given these considerations and the anticipated response rate of at least 90 percent, no non-response bias analysis is planned for this group.

For the AAAs, we expect 95 percent cooperation. At the LSP level, a 90 percent response rate is expected.

B.1.9. Analysis of Nonresponse Bias

The levels of non-response for the Process Evaluation and Cost Study are extremely low. As a result there is no plan to analyze the data received for non-response bias.

B.2. Procedures for the Collection of Information

Procedures for the collection of information addressed below include:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed for the purpose described in the justification

• Unusual problems requiring specialized sampling procedures

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden

B.2.1. Statistical Methodology for Stratification and Sample Selection

This is described in subsections B.1.1, B.1.2, B.1.3, B.1.4, and B.1.6.

B.2.2. Estimation Procedures

Very little non-response is expected from the organizations involved in the Process Evaluation and Cost Study.

B.2.3. Degree of Accuracy Needed: Statistical Power and Minimum Detectable Differences

Next, we present expected precision for estimates at the AAA, and LSP. Because the SUA survey is a census, there is no sampling error. The projections for individual-level estimates pertain to the outcome study to be conducted under another contract. The analysis at the individual level will include making estimates for subgroups of individuals and comparing those groups. The tables in this section present the groups and their sample sizes.

The precision of any estimate (standard error of a point estimate) or the minimum detectable difference (MDD) for comparing two groups depends not only on the variability of the measurement but also on the sample sizes and increases in variance due to design effects.⁷ These design effects are:

• The design effect due to weighting (Deffw)

• The design effect due to clustering (Deffc)

• The overall design effect is the product of the two (Deff = Deffc * Deffw)

Because the design calls for nearly equal probabilities and a high response rate is expected, it is reasonable to assume values for Deffw of 1.05 for AAAs, and 1.1 for LSPs.

The design effect of clustering is a function of the number of cases per PSU (b = n/a where n is the sample size (number of interviews) and a is the number of PSUs) and the intracluster correlation (ICC). Thus:

Deffc = 1 + ICC (b-1)

Different measures have different values of ICC. A range of ICC = .01 to ICC = .05 is reasonable. The calculations below assume an average ICC of .03. Furthermore, the effect of clustering is reduced with comparing two groups from the same PSUs. Kish 1965 found that Deffc for comparisons was about 80 to 90 percent of those for point estimates. The MDDs presented below for the outcome study assume that Deffc is 85 percent of Deffc for point estimates. Tables B.2.3.1, B.2.3.2, and B.2.3.3 present standard errors and half width 95 percent confidence intervals for point estimates and MDDs for comparisons. The MDDs are calculated for 80 percent power and a two-tail test. The examples for the outcome study are based on the proportion of elderly who are food insecure, approximately eight percent (Coleman-Jensen et al. 2011).

Table B.2.3.1. Standard Errors and Half-Width Confidence Intervals for AAAs and LSPs^a

Group	Sample Size	Deff C	Deff	Standard Error	I/2 Confidence Interval
All AAAs	300	1.0	1.05	2.96	5.81
AAAs in subsample	100	1.0	1.05	5.15	10.09
LSPs	200	1.1	1.13	3.77	7.38

^a An LSP characteristic or percentage reporting cost above or below a certain amount. The ½ confidence intervals are based on an estimate of 50 percent, and are thus the maximum.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

A multistage sample is required because no national sampling frame of participants exists and because in-person data collection that wil occur under Phase 2 (not covered by this Information Collection) requires clustering to be cost-efficient.

B.2.5. Data Collection Methods

1. SUA Process Survey

The contractor will elicit the support of the AoA Regional Offices. Before telephone contact, we will send an outreach package by Federal Express to the AoA regional contacts and make a courtesy telephone call to seek their support. After we have established contact with the 10 AoA regions, we will send the 56 SUA outreach packages by FedEx and begin recruitment calling. We will enlist the support of the SUA director and request name and contact information of the designated respondent if it is someone other than the director. If an alternate proxy respondent is identified, we will request that the SUA director give the survey materials to that person, and a survey specialist will attempt to contact that person.

2. The Outreach Packages Will Include:

1. A cover letter

2. A brochure

1. A survey worksheet

These materials are described in detail in subsection B.3, Methods to Maximize Response Rates and Deal with Nonresponse.

After the respondent is identified, we will contact them and urge them to complete the survey. In some instances, we will complete the survey with the respondent by telephone. Other respondents will complete the paper survey and return it to the contractor.

3. AAA and LSP Process Surveys

The AAA and LSP process surveys will be web-based. Web surveys offer maximum flexibility to respondents and minimize errors associated with data entry of hard-copy surveys. High response rates are achievable when support is available to help respondents during the field period. For this purpose, each AAA and LSP will be contacted to (1) identify appropriate respondent(s), (2) provide technical assistance to complete the survey, and (3) monitor completion. Reminder emails will be sent to encourage timely submission of completed surveys.

Recruitment for this data collection will be similar to that for the SUA process survey. Recruitment materials will be sent by Fed Ex that includes (1) a brochure; (2) a list of frequently asked questions (FAQ) about the study’s purpose, the role of the AAA’s and LSP’s and information on whom to contact with questions; and (3) a letter inviting their participation in this study of elderly nutrition services. LSPs will participate in the LSP survey, the cost study, and the menu survey. They will facilitate sampling participants for the studies of congregate and home-delivered participants for the client outcomes study.

4. Meal Cost Data Collection

We will collect the costs of selected LSPs’ congregate and home-delivered meals to calculate an average cost of congregate meals and an average cost of home-delivered meals. Using initial information collected from the LSPs, we will tailor the cost data collection worksheets to each LSP’s particular circumstances, such as whether meals are prepared at the site or in a central kitchen and transported to the site. The structured worksheets, along with detailed instructions and support provided by trained Mathematica analysts, will ensure that the study collects consistent information across the LSPs.

From each of the LSPs selected, we will randomly choose a congregate meal site and/or a meal distribution site from which to collect costs. The four tailored worksheets—facility/site labor costs, meal delivery labor costs, nonlabor costs, and central administrative labor costs—will collect information on the real resources involved in meal production and distribution and obtain unit price data to value those resources.

We anticipate much variation among LSP program staff in how they conceptualize average costs. In addition, the variation among LSPs in the accounting systems and reports may make it difficult to collect the requisite data using a standardized question-and-answer approach or a self-administered protocol. These data need to be collected by people who understand the analytic objectives and can tailor their questioning and overall approach based on the level of understanding of the LSP respondent, the accounting system, and available accounting reports, to collect accurate and consistent cost data for all LSPs.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

To maximize response rate for this study, we will develop multimode data collection systems that ensure high quality data collection while minimizing burden on respondents. Table B.3.1 summarizes the data collection mode and number of responses for each survey. (See Appendix C for the data collection protocols.) A discussion of nonresponse analysis is in subsection B.1.9.

Table B.3.1. Survey and Collection Mode

Survey	Estimated Number of Responses	Mode
Process Survey
SUA (mail and fax-back survey)	56	Paper SAQ
AAA (web and fax-back survey)	300	WEB/SAQ
LSP (web and fax-back survey)	200	WEB/SAQ
Cost Survey
LSP (paper self-administered questionnaire)	200	Paper SAQ

We will encourage greater participation through contact and recruitment materials that are relevant to each sample group (Dillman 2000). (See Appendix B for contact and recruitment materials.) Here, we present our strategies for maximizing response rates by survey.

B.3.1. Process Surveys

The process survey will examine the strategies, activities and resources of the Title III-C organizations at each of the three levels of the Aging Network: (1) SUAs, (2) AAAs, and (3) LSPs. We will initiate the contacts at the AoA region level and proceed to the SUA level and from there to the AAAs and LSPs. At each level in the Aging Network, we will not only request endorsement for the next level but also ask the respondent to directly communicate that support to the next level in the Aging Network.

A key element in a high recruitment success rate will be the recruitment materials. Dillman (2000) showed that clear, well-written, and persuasive survey materials assist in higher response rates. Recruitment materials include a cover letter, project brochure, and survey worksheet.

Cover letter. The cover letter will explain the purpose of the Title III-C evaluation and will contain endorsements from other agencies or individuals that support the evaluation.

Brochure. The trifold brochure will contain information on the purpose and importance of the study, key components of the study, contact information for the sponsoring and contracting agencies, and responses to frequently asked questions, with a toll-free number and email address in case the recipient has additional questions.

Survey worksheet. The survey worksheet will contain summary information regarding the interview. By knowing in advance the types of information we seek, the SUA director will be able to identify the best respondent for the survey (him- or herself or another staff member). This will provide time for the selected respondent to prepare for the survey and help reduce the burden on the respondents.

B.3.1.a. SUA Process Surveys

All telephone contact with AoA regional officers and SUA directors will be made by senior-level staff. To ensure high response rates once the SUA process survey is in the field, telephone reminder calls will be made to SUA directors to complete the survey. Follow-up letters will be sent and additional reminders will also be made.

B.3.1.b. AAA and LSP Process Surveys

Planned communication with AAAs and LSPs during the data collection is needed to maximize response rates.

B.3.2. Cost Survey

We will recruit LSPs for the cost data collection when we recruit them for the LSP process survey. After we recruit the LSPs, experienced Mathematica analysts will be assigned responsibility for a set of LSPs. An important part of the analysts’ responsibilities will be to establish rapport with the LSP’s main contact to encourage the LSP’s participation throughout the study. For the cost study, the analysts will also provide the LSPs with the technical assistance necessary to complete the cost worksheets. After the LSP submits the worksheets, the analysts will follow up, as necessary, to complete any missing data. This will ensure an accurate calculation of the LSP’s meal costs.

B.4. Tests of Procedures or Methods to be Undertaken

The procedures, materials, and instruments developed for the evaluation are similar to those that have been developed, tested, and administered for other elderly nutrition studies. The process, menu, and client outcome surveys were pretested with fewer than 10 respondents. Table B.4.1 shows the survey instrument, the number of agencies, and the number of respondents who participated in the pretest.

Table B.4.1. Data Collection Pretest Activities

Survey/Instrument	Number of Agencies	Number of Respondents
Process Surveys
SUA (fax-back survey)	3	3 SUA directors
AAA (web and fax-back survey)	4	4 AAA directors
LSP (web and fax-back survey)	3	3 LSP directors

1. Process Surveys

The process surveys are designed to examine the strategies, activities, and resources of the Title III-C organizations at three levels: (1) SUAs, (2) AAAs, and (3) LSPs. Pretests for all instruments took the form of cognitive interviews, and respondents were also asked to review the study recruitment procedures, contact materials, and technical assistance procedures. Each respondent completed the survey independently, and then senior project staff members had a 30-minute follow-up conversation with each respondent to debrief.

In November 2010, the SUA process survey was pretested with three respondents from California, Massachusetts, and Texas. In January 2012, the AAA process survey was pretested with four respondents from Iowa, Kansas, Massachusetts, and Michigan. The LSP process survey was pretested in April 2012 with three respondents from Ohio, Kansas, and Wyoming.

During the debriefing, senior staff members noted questions that needed clarification, questions that required adjustments, and those that needed to be reworded. The time required for each respondent to complete the interview was also recorded. The results of the pretest were used to revise the surveys.

The results of the three pretests are summarized below:

• SUA. No additional modifications were identified.

• AAA. The respondents’ opinion of the survey was positive overall. Most respondents found both the length and complexity of the survey appropriate, but several noted that new AAA directors or administrators may not be able to complete the survey as quickly or as easily as more experienced directors. The respondents recommended that future respondents be provided a list of necessary sources of data beforehand to facilitate prompt completion of the survey and fax-back form.

• LSP. Two respondents found the survey long, but one found that it took less time than anticipated. The respondents found the questions pertaining to finances, number of volunteers, mileage, and unduplicated counts of participants difficult to answer. However, respondents also reported that most data needed to complete the survey were readily available. One respondent suggested making the results of the survey available to participants in summary form, to encourage completion.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The design for the study was developed in conjunction with Mathematica Policy Research, under the direction of: Rhoda Cohen, project director; John Hall, senior statistician; Mary Kay Crepinsek, senior researcher, and James Mabli, senior researcher. Ms. Cohen may be reached at (609) 275-2324 or [email protected]; Mr. Hall may be reached at (609) 275-2357 or [email protected]; Ms. Crepinsek may be reached at (617) 301-8998 or [email protected]; Dr. Mabli may be reached at (617) 301-8997 or [email protected].

In addition, Jennifer Klocinski and Susan Jenkins of the Office of Performance and Evaluation, Administration on Aging, Department of Health and Human Services reviewed the study design and instruments. Ms. Klocinski may be reached at (202) 357-0146 or [email protected]. Dr Jenkins may be reached at (202)357-3591 or [email protected].

References

Biggs, D., B. deVille, and E. Suen. A Method of Choosing Multiway Partitions for Classification and Decision Trees. Journal of Applied Statistics. 1991, 18, 49-62.

Coleman-Jensen, Alisha, Mark Nord, Margaret Andrews, and Steven Carlson. Household Food Security in the United States in 2010. ERR-125. Washington, D.C.: U.S. Department of Agriculture, Economic Research Service, September 2011. Available at http://www.ers.usda.gov/Publications/ERR125/ERR125.pdf.

Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: Wiley, 2000.

Kass, G. V. An Exploratory Technique for Investigating Large Quantities of Categorical Data. Applied Statistics. 1980, 29, 119-127.

Kish, L. Survey Sampling. New York: John Wiley and Sons, 1965.

Magidson, J. SPSS for Windows CHAID Release 6.0. Belmont MA: Statistical Innovations, Inc., 1993.

Ponza, Michael, James C. Ohls, and Barbara E. Millen. Elderly Nutrition Program Evaluation Final Report, Volume I: Title III Evaluation Findings. Report submitted to the U.S. Department of Health and Human Services, Office of the Secretary, Administration on Aging, and Office of Assistant Secretary for Planning and Evaluation. Princeton, NJ: Mathematica Policy Research, July 1996.

Ponza, Michael, James C. Ohls, and Barbara E. Millen. Elderly Nutrition Program Evaluation Final Report, Volume II: Title VI Evaluation Findings. Report submitted to the U.S. Department of Health and Human Services, Office of the Secretary, Administration on Aging, and Office of Assistant Secretary for Planning and Evaluation. Princeton, NJ: Mathematica Policy Research, July 1996.

1 AAAs plan, coordinate, and advocate for the development of a comprehensive service delivery system to meet the needs of older adults in a specific geographic area. They administer state and federal funds for community-based services. http://www.tjaaa.org/glossary-of-terms.aspx.

2 Area agencies normally contract with local for-profit or nonprofit or public providers (LSPs) to deliver benefits. The contract service providers nationwide, providing care under the act, are the largest single network of long-term care providers in the country. An agency may be allowed to directly provide supportive services, nutrition services, or in-home services if it can prove that it can provide these services more effectively. http://www.longtermcarelink.net/eldercare/area_agencies_on_aging.htm.

3 http://www.n4a.org/files/advocacy/campaigns/oaa/OAA_Backgrounder_Final.pdf.

4 Services are available to people age 60 or older and the spouse of an older person regardless of age.

5 These are sampled with certainty at this point so they will be included in the subsample of 100.

6 We will obtain the MOS from the SUA.

7 A design effect is defined as the increase in sampling variance, relative to a simple random sample with the same number of observations. Thus for a sample size of n Deff = (Var actual|n)/(Var SRS|n).

8