PART B
Part B of the Justification for this information collection activity, the Evaluation of Older Americans Act Title III-C Elderly Nutrition Services Program, addresses the five points outlined in Part B of the OMB guidelines.
B.1. Respondent Universe and Sampling Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
In this section, we describe the procedures we will use to select the sample of Area Agencies on Aging (AAAs)1 and Local Service Providers (LSPs)2 for the evaluation of the Title III-C Elderly Nutrition Services Program. We also describe the selection of samples of program participants and a comparison sample of nonparticipants. These individual-level samples will not be selected as part of this survey, but will be sampled as part of a subsequent survey (for a study of client outcomes) from within the same AAAs and LSPs sampled for this survey.
The rest of Section B.1 covers:
Universe of Potential Respondents
Multi-stage Sampling
Sample Development and Selection of AAAs and LSPs
Sampling Frame and Identification of Program Sites, Routes, and Participants
Identification of a Matched Comparison Group and Selection of the Nonparticipant Sample
Consent
B.1.1. Universe of Potential Respondents
The Universe of Potential Respondents includes all State and Territorial Units on Aging (SUAs), AAAs in the contiguous United States, LSPs in the contiguous United States, and, for the client outcomes study, participants in the congregate and home-delivered meals programs and a comparison group of nonparticipants. There are 56 SUAs, 629 AAAs, and more than 30,000 LSP organizations.3 In fiscal year 2010, 96.4 million congregate meals were served in Congregate Nutrition Services to 1.7 million congregate participants in the Older Americans Act (OAA) Nutrition Program.4 In fiscal year 2010, Home-Delivered Services served meals to approximately 870,000 participants who receive home-delivered meals in the OAA Nutrition Program.5
The census of SUAs will be independent of the remaining samples. The sample of AAAs, LSPs and individuals (participants and comparison group members) will be selected in stages, as described below. Because these samples are not nested within SUAs, the SUA census is not considered part of the sampling design for AAAs, LSPs, and individuals.
B.1.3. Sample Development and Selection of AAAs and LSPs
The sampling frame for selecting AAAs will be an electronic file provided by the Administration on Aging (AoA). The sampling frame for LSPs will be developed through contact with those AAAs subsampled (as discussed below) for LSP and individual data collection. The sample of AAAs will serve two purposes. We will use a sample of 300 AAAs to gather data at the AAA level. A subsample of 100 (of the 300) AAAs will serve as Primary Sampling Units (PSUs) for selecting samples to collect LSP and individual data.
The initial sample of 300 AAAs will be selected as a stratified random sample. There will be two explicit strata: (1) the certainty stratum, and (2) the noncertainty stratum. The certainty stratum will ensure that very large AAAs are not excluded from the sample. Within the noncertainty stratum, implicit stratification will be used to ensure the representativeness of the sample. The certainty stratum will include those AAAs large enough to be sampled with certainty in a probability-proportional-to-size (PPS) sample of 100 AAAs6 with a measure of size (MOS) defined as the estimated number of participants; all of these will be included in the initial sample of 300. The rest of the sampled 300 AAAs will be selected from the noncertainty stratum. Within this stratum, implicit strata will be defined by region of the country, size (estimated number of participants), and whether the AAA serves urban or rural areas. The AAAs selected in the noncertainty stratum will be chosen with equal probability (within that stratum).
We will use the full sample of 300 AAAs for collecting data at that level. We will select a subsample of 100 from the 300 to serve as PSUs for the samples of LSPs for the process and cost studies, and, for the client outcomes study, samples of participants and the comparison group. This subsample will include all AAAs in the certainty strata, plus others selected using PPS methods with the number of participants as the MOS.7 In selecting the PPS subsample, we will implicitly stratify on the same characteristics used in selecting the initial sample of AAAs.
The next step will be to create LSP sampling frames for each of the 100 AAAs in the subsample using information provided by those AAAs. After the selection of the AAAs and recruitment of these agencies for the study, each AAA will be asked to provide information on its LSPs. We will determine from each AAA whether each of its LSPs has only one program (congregate or home-delivered) or both programs and the number of people served (by those programs). After AAAs have been recruited and it has been determined what information is available about LSPs, the project will evaluate what the best size measure would be for selecting LSPs. A MOS based on numbers of participants will serve well both for the cost study and for the later client outcomes study. The MOS may be based on the total number of participants or may be a composite that incorporates estimates for the different target groups. A PPS sample of LSPs will then be selected using the chosen MOS.
We will select the LSPs after the sampling frame for the LSPs is complete and verified. An initial sample of LSPs will be selected separately (with PPS) within each participating AAA. Before selecting the samples of LSPs, we will examine their distribution to determine if stratification is necessary to ensure adequate numbers of congregate meal sites and home-delivery distribution sites. To achieve a target of 200 participating LSPs, we will choose an initial sample of approximately 222 LSPs so that approximately 200 cooperative LSPs will remain after nonresponse. The approach to sampling LSPs is:
In AAAs with one or two LSPs, select and attempt to recruit all.
In AAAs with three or four LSPs, select two with PPS for recruitment and keep the other(s) in reserve in case of nonresponse or ineligibility.
In AAAs with five or more LSPs, select four with PPS and randomly select two for recruitment, keeping the other two in reserve in case of nonresponse.
B.1.4. Sampling Frame and Identification of Program Sites, Routes, and Participants
We will ask each sampled LSP to provide information on the program(s) (congregate nutrition, home-delivered nutrition, or both) it runs. For the congregate nutrition program, we will request information on the day, time, and location of each site where meals are provided. For participants in the home-delivered nutrition program, we will request information on the delivery routes, schedule for the deliveries, the type of meals provided (hot or frozen), and the quantity of meals provided at a delivery (single day or multiple days). Although this information may change, it is needed to prepare for selection of sites and delivery routes for the cost study. Assuming that the same LSPs are used to select clients for the outcomes study, this information may need to be verified and updated when the client outcomes study is conducted.
For the cost study, we will select one congregate meal site in each LSP that has one or more sites. In LSPs with home delivery, we will sample one distribution site and one or two routes within each site.
For the client outcomes study, the target sample size will be 1,200 completed interviews with program participants selected using the procedures described below. To the extent possible, the numbers of interviews will be spread evenly across AAAs and LSPs. However, AAAs selected with certainty may be allocated a slightly larger sample size of participants. In addition, if there are fewer than 200 participating LSPs, this may introduce some imbalance (for example, all the participant interviews in a AAA may be assigned to its only LSP).
The initial sample size of congregate nutrition participants will be 800, with a target of 600 completed interviews. To select the participants in the congregate nutrition program, a specific location and date and time within the data collection period will be selected. Field staff will contact the LSP before any data collection and arrive at the location before the meal to get information for the sample selection. Sampling worksheets (either hard-copy or electronic) will enable the field staff to conduct the congregate nutrition participant sample selection with minimal disruption to the meal. If all participants are present at the same time, the field staff can use an electronic worksheet to identify which participants are selected. The field staff can select a fixed number of participants from the participants present at the meal.
If participants arrive on a flow basis, the field staff will use a systematic sampling procedure to select every nth participant arriving using a hard-copy sampling worksheet. This procedure will result in some variability in the sample sizes selected but is necessary to make on-site sampling feasible.
All 600 respondents will complete a 24-hour dietary recall. A stratified (with AAAs being the implicit strata) random sample of 166 of the 600 respondents will be selected for a second 24-hour recall. It is expected that 150 will complete the second recall. At 6 and 12 months after the initial interview, we will attempt brief telephone follow-up interviews with those who completed the initial interview. It is expected that 540 6-month and 459 12-month interviews will be completed.
The home-delivered nutrition sample will be selected by the field staff and will include 800 participants in the home-delivered nutrition program, to achieve 600 completed interviews. For participants in the home-delivered nutrition program, the LSP will provide a description of the delivery routes, a schedule for delivery, and a count of participants before data collection; when the field staff arrive at the LSP, they will receive a current list of participants. Based on the information from the LSP, sampling procedures for the LSP will be developed and sent to the field staff before the data collection visit. To expedite the sample selection, an electronic worksheet will be available to the field staff to select the client sample based on the most current list of participants receiving meals during the data collection visit. Statisticians will be available on-call to the field staff to answer any questions and to advise of methods if unusual circumstances occur or if significant changes have occurred in the delivery schedule or the count of participants.
All 600 respondents will complete a 24-hour dietary recall. A stratified (with AAAs being the implicit strata) random sample of 166 of the 600 respondents will be selected for a second 24-hour recall. It is expected that 150 will complete the second recall. At 6 and 12 months after the initial interview, brief telephone follow-up interviews will be attempted with those who completed the initial interview. It is expected that 540 6-month and 459 12-month interviews will be completed.
B.1.5. Identification of a Matched Comparison Group and Selection of the Nonparticipant Sample for the Outcomes Study
Assessing program impacts requires that survey responses and other measured outcomes of program participants be compared with those of a group of nonparticipants with similar characteristics living in the same geographic areas. This comparison group will be selected from the population of Medicare beneficiaries living in the same zip codes as the interviewed Title III-C program recipients. To select the comparison group, the project will request the most recent Medicare enrollment and claims data for beneficiaries living in these zip codes from the Centers for Medicare & Medicaid Services Research Data Assistance Center.
Sampling the comparison group for nonparticipants in the congregate nutrition and home-delivered nutrition programs has three critical steps:
Requesting Medicare data
Surveying program participants and identifying them in the Medicare data using their names, addresses, and social security numbers (SSNs)
Matching participants to nonparticipants
Sampling the comparison group requires that the participants be interviewed first. At the end of each cycle (week) of the participant survey, the project team will use the Medicare name and address file and SSNs, along with the names, addresses, and SSNs of interviewed program participants, to identify meal recipients in Medicare data. As soon as interviewed program participants from that week’s data collection are identified in the Medicare data, the evaluation team will create a flag identifying program participants. Next, the evaluator will use Medicare claims and enrollment information for both participants and nonparticipants living in the zip codes served by the LSPs for which participants were interviewed to estimate a propensity score matching model. Specifically, the evaluator will estimate a logistic regression predicting program participation (separate models for home-delivered nutrition and congregate nutrition programs) as a function of age, race, sex, service utilization (for example, number of inpatient admissions, physician visits, any use of home health care), diagnoses (indicators for specific chronic conditions), original reason for Medicare entitlement (age versus disability), and Part B state buy-in indicator (proxy for dual eligibility).
Estimating the propensity score model across areas served by several sampled LSPs in multiple AAAs will increase model stability, while still allowing the matching of participants (in either the home-delivered nutrition or the congregate nutrition program) to nonparticipants using the predicted probability of participation from the model as well as the zip code of residence. For each beneficiary who received home-delivered meals, the evaluator will identify, among those who did not receive home-delivered nutrition, the several (up to 25) beneficiaries with closest propensity scores (that is, the “nearest neighbors”) who live in the same zip code as the program participant. The evaluation team will then repeat this exercise for recipients of congregate meals using the predictions from the propensity score model for congregate nutrition program participation.8 In addition, to ensure obtaining the best matches for each program participant, the evaluator will implement nearest neighbor matching with replacement—that is, the same nonparticipant could serve as a potential comparison group member for more than one program participant.
The study team will screen the remaining comparison group members (about eight per program participant) via telephone by asking questions to verify that they are eligible nonparticipants. They must be at least 60 years old and not participating in any AoA nutrition program. Screening potential matches is a cost-effective strategy to ensure visits are made only to eligible nonparticipants. After being screened and recruited into the study, the resulting names and addresses will be used to construct the interview samples for both the home-delivered nutrition and congregate nutrition comparison groups. Assuming that up to 25 percent of screened comparison group members will be ineligible or will refuse to participate in an interview, the evaluator can expect to have about six comparison group members per program participant for scheduling interviews. The evaluator will mail out a confirmation letter and study brochure to these recruited members.
The sample sizes for each of the comparison groups will be the same as for the participant groups. This includes the second 24-hour recall and the 6- and 12-month follow-up telephone interviews.
Consent will be given by the participant. If a respondent is too ill to participate in an interview or has cognitive, hearing, speech, or vision impairments, a family member or caregiver may serve as a proxy for consent and the interview.
The sample sizes for each respondent group are discussed in subsections B.1.2 through B.1.4. Table B.1.6.1 summarizes the planned sample sizes.
Table B.1.6.1. Summary of Sample Sizes
Respondent Group |
Sample Selected |
Number of Responses |
State and Territorial Units on Aging |
56 |
56 |
Area Agency on Aging |
300 |
300 |
Local Service Provider (process survey and cost study) |
222a |
200a |
Congregate Participant Survey + 24-hr Dietary Recall |
800 |
600 |
Congregate Participant Additional 24-hr Dietary Recall |
166 |
150 |
Congregate Participant Follow-Up At 6 Months |
600 |
540 |
Congregate Participant Follow-Up At 1 Year |
540 |
459 |
Congregate Nonparticipant Survey + 24-hr Dietary Recall |
800 |
600 |
Congregate Nonparticipant Additional 24-hr Dietary Recall |
166 |
150 |
Congregate Nonparticipant Follow-Up At 6 Months |
600 |
540 |
Congregate Nonparticipant Follow-Up At 1 Year |
540 |
459 |
Home-Delivered Participant Survey + 24-hr Dietary Recall |
800 |
600 |
Home-Delivered Participant Additional 24-hr Dietary Recall |
166 |
150 |
Home-Delivered Participant Follow-Up At 6 Months |
600 |
540 |
Home-Delivered Participant Follow-Up At 1 Year |
540 |
459 |
Home-Delivered Nonparticipant Survey + Dietary Recall |
800 |
600 |
Home-Delivered Nonparticipant Additional 24-hr Dietary Recall |
166 |
150 |
Home-Delivered Nonparticipant Follow-Up At 6 Months |
600 |
540 |
Home-Delivered Nonparticipant Follow-Up At 1 Year |
540 |
aApproximate.
No nonresponse is anticipated at the SUA level. For the AAAs, we expect 95 percent cooperation. At the LSP level, a 90 percent response rate is expected. For the client outcomes study, we expect a 75 percent gross completion rate for all groups for the initial interview and dietary assessment. The expected client outcomes response rates and their components are:
Home-delivered participants’ initial interviews: eligibility determined for 95 percent,9 90 percent eligible, 85 percent completion among eligibles for a marginal response rate of 84 percent10
Congregate participants’ initial interviews: eligibility determined for 100 percent, 90 percent eligible and 83 percent completion among eligible for a marginal response rate of 83 percent
For the additional dietary recalls, the completion rate (and marginal response rate) will be 90 percent. For each of the follow-ups (six months and one year), we expect eligibility to be determined for 100 percent, 95 percent to be eligible and a 95 percent marginal response rate. Cumulative response rates can be determined by multiplying the marginal response rate by the response rates at earlier stages.
B.1.9. Analysis of Nonresponse Bias
The methods described in subsection B.2.2 for weighting will be used for an analysis of nonresponse bias. Because values of dependent variables are not known for the population, the analysis will extend the identification of variables correlated with response propensity to see if any of these are also correlated with study outcomes.
B.2. Procedures for the Collection of Information
Procedures for the collection of information addressed below include:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
B.2.1. Statistical Methodology for Stratification and Sample Selection
This is described in subsections B.1.1, B.1.2, B.1.3, B.1.4, and B.1.6.
Analysis weights will be prepared for each file and then merged onto the data files. Most of this work will occur after the data collection period has ended, so that the weights can reflect adjustments for nonresponse. Weights are needed for each of the planned analysis levels. The levels for which weights are needed include:
AAA level
LSP level
Individual level (participants and comparison group)11
At the individual level, weights are needed for participants in the congregate nutrition program and the home-delivered nutrition program and for those in the comparison groups. The sampling design incorporates multiple stages of selection and subsampling, and the weights need to account for each stage and subsampling implemented for this study. The first step in weighting is to calculate sampling weights for each case (the inverse of its probability of selection). After the sampling weights are computed, the sampling weights will be adjusted to compensate for nonresponse.
The sampling weights for LSPs and clients will need to account for the multiple stages of selection. Weight adjustments will occur at each stage of selection to ensure accurate representation from each sampling stage.
We will select the AAAs and the LSPs using software that can directly compute the sampling weights. The client selection will often be conducted on-site during the data collection period. Sampling worksheets (both electronic and hard-copy) that the field staff use will help ensure complete information from this stage of selection. The sampling worksheets will be monitored on a continual basis to allow for speedy resolution of inconsistencies or omissions. Data from these worksheets will be used to compute the final stage of sampling weights.
After data collection is complete and the initial weights are computed, we will conduct an analysis to assess the response patterns. The nonresponse analysis will consist of univariate and bivariate cross-tabulations, and will also include multivariate analysis to detect patterns that interactive effects may mask. For example, a natural cross-tabulation would be by age and gender and may show no distinct pattern. However, a multivariate analysis using age, gender, race or ethnicity and geographic area (AoA region or an urban versus nonurban dichotomy) may show substantial variation. To determine these multivariate interactions, we will conduct a chisquared automatic interaction detector (CHAID) analysis in SPSS to find possible significant predictors. CHAID is normally attributed to Kass (1980) and Biggs et al. (1991), and its application in SPSS is described in Magidson (1993).
The segments identified by CHAID as being associated with nonresponse can then be used in the classical weighting class method to compute weight adjustments and to use the inverse of the class response rate as the adjustment value in that class. The adjusted weight is the product of the sampling weight and the adjustment factor. The weighting classes developed by CHAID ensure sufficient counts of respondents in each class to make the adjustment more stable (that is, have a smaller variance).
After the nonresponse-adjusted weights are computed, a poststratification adjustment will be made in which the weighted sums of the response-adjusted weights are aligned to the known totals of congregate and home-delivered nutrition recipients. We expect that such counts are available from the annual State Program Report (SPR). If SPR totals are not available, data collected from the first phase sample of 300 AAAs can be used to create estimated totals for the poststratification.
B.2.3. Degree of Accuracy Needed: Statistical Power and Minimum Detectable Differences
Next, we present expected precision for estimates at the AAA, LSP, and individual levels. Because the SUA survey is a census, there is no sampling error. The projections for individual-level estimates pertain to the outcome study to be conducted under another contract. The analysis at the individual level will include making estimates for subgroups of individuals and comparing those groups. The tables in this section present the groups and their sample sizes.
The precision of any estimate (standard error of a point estimate) or the minimum detectable difference (MDD) for comparing two groups depends not only on the variability of the measurement but also on the sample sizes and increases in variance due to design effects.12 These design effects are:
The design effect due to weighting (Deffw)
The design effect due to clustering (Deffc)
The overall design effect is the product of the two (Deff = Deffc * Deffw)
Because the design calls for nearly equal probabilities and a high response rate is expected, it is reasonable to assume values for Deffw of 1.05 for AAAs, 1.1 for LSPs, and 1.25 for individuals.
The design effect of clustering is a function of the number of cases per PSU (b = n/a where n is the sample size (number of interviews) and a is the number of PSUs) and the intracluster correlation (ICC). Thus:
Deffc = 1 + ICC (b-1)
Different measures have different values of ICC. A range of ICC = .01 to ICC = .05 is reasonable. The calculations below assume an average ICC of .03. Furthermore, the effect of clustering is reduced with comparing two groups from the same PSUs. Kish 1965 found that Deffc for comparisons was about 80 to 90 percent of those for point estimates. The MDDs presented below for the outcome study assume that Deffc is 85 percent of Deffc for point estimates. Tables B.2.3.1, B.2.3.2, and B.2.3.3 present standard errors and half width 95 percent confidence intervals for point estimates and MDDs for comparisons. The MDDs are calculated for 80 percent power and a two-tail test. The examples for the outcome study are based on the proportion of elderly who are food insecure, approximately eight percent (Coleman-Jensen et al. 2011).
Table B.2.3.1. Standard Errors and Half-Width Confidence Intervals for AAAs and LSPsa
Group |
Sample |
Deff C |
Deff |
Standard |
I/2
Confidence |
All AAAs |
300 |
1.0 |
1.05 |
2.96 |
5.81 |
AAAs in subsample |
100 |
1.0 |
1.05 |
5.15 |
10.09 |
LSPs |
200 |
1.1 |
1.13 |
3.77 |
7.38 |
a An LSP characteristic or percentage reporting cost above or below a certain amount. The ½ confidence intervals are based on an estimate of 50 percent, and are thus the maximum.
Table B.2.3.2. Standard Errors and Half-Width Confidence Intervals for Food Insecurity
Group |
Sample |
Deff C |
Deff |
Standard |
I/2
Confidence |
All Participants |
1,200 |
1.33 |
1.66 |
1.01 |
1.98 |
All Nonparticipants |
1,200 |
1.33 |
1.66 |
1.01 |
1.98 |
Congregate Participant |
600 |
1.15 |
1.44 |
1.33 |
2.61 |
Congregate Participant Follow-Up at 6 Months |
540 |
1.13 |
1.41 |
1.39 |
2.72 |
Congregate Participant Follow-Up at 1 Year |
459 |
1.11 |
1.39 |
1.49 |
2.92 |
Congregate Nonparticipant |
600 |
1.15 |
1.44 |
1.33 |
2.61 |
Congregate Nonparticipant Follow-Up at 6 Months |
540 |
1.13 |
1.41 |
1.39 |
2.72 |
Congregate Nonparticipant Follow-Up at 1 Year |
459 |
1.11 |
1.39 |
1.49 |
2.92 |
Home-Delivered Participant Survey |
600 |
1.15 |
1.44 |
1.33 |
2.61 |
Home-Delivered Participant Follow-Up at 6 Months |
540 |
1.13 |
1.41 |
1.39 |
2.72 |
Home-Delivered Participant Follow-Up at 1 Year |
459 |
1.11 |
1.39 |
1.49 |
2.92 |
Home-Delivered Nonparticipant Survey |
600 |
1.15 |
1.44 |
1.33 |
2.61 |
Home-Delivered Nonparticipant Follow-Up at 6 Months |
540 |
1.13 |
1.41 |
1.39 |
2.72 |
Home-Delivered Nonparticipant Follow-Up at 1 Year |
459 |
1.11 |
1.39 |
1.49 |
2.92 |
Table B.2.3.3. Minimum Detectable Differences for Food Insecurity
Group 1 |
Group 2 |
Sample
Size |
Deff C |
Deff |
MDD |
All Participants |
All Nonparticipants |
1,200 |
1.13 |
1.41 |
3.68 |
Congregate Participant |
Congregate Nonparticipant |
600 |
0.98 |
1.23 |
4.87 |
Congregate Participant Follow-Up at 6 Months |
Congregate Nonparticipant Follow-Up at 6 Months |
540 |
0.96 |
1.2 |
5.07 |
Congregate Participant Follow-Up at 1 Year |
Congregate Nonparticipant Follow-Up at 1 Year |
459 |
0.94 |
1.18 |
5.45 |
Home-Delivered Participant Survey |
Home-Delivered Nonparticipant Survey |
600 |
0.98 |
1.23 |
4.87 |
Home-Delivered Participant Follow-Up at 6 Months |
Home-Delivered Nonparticipant Follow-Up at 6 Months |
540 |
0.96 |
1.2 |
5.07 |
Home-Delivered Participant Follow-Up at 1 Year |
Home-Delivered Nonparticipant Follow-Up at 1 Year |
459 |
0.94 |
1.18 |
5.45 |
B.2.4. Unusual Problems Requiring Specialized Sampling Procedures
A multistage sample is required because no national sampling frame of participants exists and because in-person data collection requires clustering to be cost-efficient. Statistical matching is required to identify a comparison group. These methods are described in subsection B.1.
B.2.5. Data Collection Methods
The contractor will elicit the support of the AoA Regional Offices. Before telephone contact, we will send an outreach package by Federal Express to the AoA regional contacts and make a courtesy telephone call to seek their support. After we have established contact with the 10 AoA regions, we will send the 56 SUA outreach packages by FedEx and begin recruitment calling. We will enlist the support of the SUA director and request name and contact information of the designated respondent if it is someone other than the director. If an alternate proxy respondent is identified, we will request that the SUA director give the survey materials to that person, and a survey specialist will attempt to contact that person.
A cover letter
A brochure
A survey worksheet
These materials are described in detail in subsection B.3, Methods to Maximize Response Rates and Deal with Nonresponse.
After the respondent is identified, we will contact them and urge them to complete the survey. In some instances, we will complete the survey with the respondent by telephone. Other respondents will complete the paper survey and return it to the contractor.
The AAA and LSP process surveys will be web-based. Web surveys offer maximum flexibility to respondents and minimize errors associated with data entry of hard-copy surveys. High response rates are achievable when support is available to help respondents during the field period. For this purpose, each AAA and LSP will be contacted to (1) identify appropriate respondent(s), (2) provide technical assistance to complete the survey, and (3) monitor completion. Reminder emails will be sent to encourage timely submission of completed surveys.
Recruitment for this data collection will be similar to that for the SUA process survey. Recruitment materials will be sent by Fed Ex that includes (1) a brochure; (2) a list of frequently asked questions (FAQ) about the study’s purpose, the role of the AAA’s and LSP’s and information on whom to contact with questions; and (3) a letter inviting their participation in this study of elderly nutrition services. LSPs will participate in the LSP survey, the cost study, and the menu survey. They will facilitate sampling participants for the studies of congregate and home-delivered participants for the client outcomes study.
We will collect the costs of selected LSPs’ congregate and home-delivered meals to calculate an average cost of congregate meals and an average cost of home-delivered meals. Using initial information collected from the LSPs, we will tailor the cost data collection worksheets to each LSP’s particular circumstances, such as whether meals are prepared at the site or in a central kitchen and transported to the site. The structured worksheets, along with detailed instructions and support provided by trained Mathematica analysts, will ensure that the study collects consistent information across the LSPs.
From each of the LSPs selected, we will randomly choose a congregate meal site and/or a meal distribution site from which to collect costs. The four tailored worksheets—facility/site labor costs, meal delivery labor costs, nonlabor costs, and central administrative labor costs—will collect information on the real resources involved in meal production and distribution and obtain unit price data to value those resources.
We anticipate much variation among LSP program staff in how they conceptualize average costs. In addition, the variation among LSPs in the accounting systems and reports may make it difficult to collect the requisite data using a standardized question-and-answer approach or a self-administered protocol. These data need to be collected by people who understand the analytic objectives and can tailor their questioning and overall approach based on the level of understanding of the LSP respondent, the accounting system, and available accounting reports, to collect accurate and consistent cost data for all LSPs.
The menu survey responds to the AoA’s interest in a nutritional analysis of meals offered to participants in congregate and home-delivered nutrition programs. The core program meal is typically a hot lunch, although some programs also offer meals at other eating occasions and meals may be cold or pre-prepared. Congregate and home-delivered meals may be prepared at a congregate nutrition site, a central kitchen or other off-site location, or catered by local food service vendors or restaurants. In addition, congregate nutrition sites may offer cafeteria or restaurant-style meals (pre-portioned servings) or family-style or buffet meals (self-service).
Menu survey data will be collected in the same 200 program sites sampled for the client outcomes study. Within this sample, the LSP administrator will identify a respondent knowledgeable about the meals offered and/or delivered to clients. Data will be collected for the core or “mid-day” meals using the U.S. Department of Agriculture (USDA) Automated Multiple Pass Method (AMPM). The AMPM menu interviews will elicit detailed food descriptions, food preparation, and serving size information for three days of both congregate and home-delivered meals (when they differ) at each site.
Field interviewers will inform respondents of the three days selected for the survey and instruct them on using the measuring guides (food model booklet and household measuring tools) and the Menu Survey Guide and Forms (see Appendix A). The respondent will be asked to list the menu items available to clients in the regular mid-day meal on the form provided, after the meals are served/delivered. Specially trained nutrition staff will then conduct two AMPM menu interviews with each respondent by telephone—one interview for menu day 1 and one for menu days 2 and 3. The respondent can use the menu form during the interview to help remember or report on only the items included in the mid-day meals.
Obtaining reliable input from a sample of program participants and eligible nonparticipants is arguably the most important and most challenging component of the study. The goal of the client outcomes study is to collect information on nutrient intake, socialization, health status, service referrals, program experiences, and related variables from three target groups of seniors: (1) congregate nutrition participants, (2) home-delivered nutrition recipients, and (3) a matched sample of nonparticipants who live in the same zip code.
A significant challenge is recruiting participants and, especially, a matched sample of nonparticipants. To address this challenge, we will tailor recruiting materials and procedures to each target group. The materials will include a study brochure to provide information on the study’s purpose, the role of seniors who agree to participate, and whom to contact with questions. In addition, we will use an IRB-approved consent form; forms to confirm date, time, and location of data collection; and flyers to post at meal sites.
Recruiting congregate nutrition participants will occur in conjunction with LSP meal provision. Trained field interviewers will visit identified meal sites approximately one hour before a meal is served. Those seniors selected to participate will be given a brochure, a brief introduction to the study and what they are being asked to do, and an opportunity to ask questions. If the person agrees to participate, the interviewer will request contact information and schedule a time to meet on the following day. Interviews may take place at any location, although we anticipate most will take place in the respondent’s home. All selected seniors will be told that they will be mailed a check for $40 following completion of the interview.
We will recruit home-delivered nutrition recipients by telephone. We will request a list of home-delivered nutrition recipients from the relevant LSP/AAA and select a sample of home-delivered nutrition recipients from this list. Each sampled home-delivered nutrition participant will be given a brochure with his or her meal delivery shortly before being contacted by a field interviewer. Information about who is likely to need a proxy or an interviewer who speaks a language other than English will also be collected at the time of the telephone contact.
We will recruit nonparticipants by telephone. For each study participant, we will use Medicare files to identify up to 10 potential matches in the same zip code. We will contact potential matches and ask them questions to verify that they are eligible nonparticipants. They must be 60 years old or older and not participating in any AoA nutrition program. Screening potential matches is a cost-effective strategy to ensure visits are made only to eligible nonparticipants. For those who are eligible, an interview will be scheduled and a confirmation letter and study brochure will be mailed to them.
Following recruitment, trained field interviewers will visit the homes (or other designated sites) at a time that is convenient to the respondent. Each 60-minute interview consists of two instruments: (1) a client outcomes survey, and (2) a 24-hour dietary recall (AMPM). To thank seniors for completing the study requirements, each will receive $40. A subsample of 25 percent will be asked to do a second dietary recall one week later (see below) and will receive an additional $20 for doing so.
The single 24-hour recall to be collected from all sample members will be collected in person in respondents’ homes in conjunction with the client outcomes survey. Interviewers will use USDA’s AMPM software (Agricultural Research Service, Food Surveys Research Group, Beltsville, MD) on laptop computers. The software uses a standardized five-step process to maximize respondents’ ability to recall and report foods and beverages consumed. These steps are:
Quick list. Collects an uninterrupted listing of all foods and beverages consumed in a 24-hour period, the day before the interview.
Forgotten foods. Probes for any foods that may have been forgotten during the quick list, including beverages, sweets, snacks, fruit, vegetables, cheese, and breads.
Time and occasion. Collects the time the respondent began to consume each reported food item and what he or she would call the eating occasion for the food (such as breakfast, lunch, dinner, or snack).
Detail cycle. Through standardized questions, probes for descriptions of each item, including the quantity consumed, where the food was obtained, whether it was eaten at home, and any additions to the food. Also probes for other items that may have been eaten between the reported eating occasions.
Final probe. Collects information on any additional foods not previously mentioned.
The recall will cover all the foods and beverages eaten in the preceding 24-hour period (midnight to midnight). Interviews will be scheduled so that respondents (or appropriate proxies) only report intakes for days the Title III-C program meal is typically available, which is Monday through Friday. For most home-delivered nutrition participants who receive meals daily, this will capture their intake of the program meal. We recognize that not all individuals who consume a meal at a congregate nutrition site on the day of sampling will consume a program meal all days of the week. Data from the 1993–1995 evaluation conducted by Mathematica for AoA revealed that close to two-thirds of congregate nutrition participants received a program meal five days a week, and another one-fourth received meals three or four times per week (Ponza et al. 1996). Thus, the likelihood of capturing congregate nutrition participants’ intake of the Title III-C program meal with the 24-hour recall is still quite high (about 85 percent).
About a week after the first 24-hour recall, a randomly selected 25 percent subsample of respondents will complete a second recall over the telephone.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
To maximize response rate for this study, we will develop multimode data collection systems that ensure high quality data collection while minimizing burden on respondents. Table B.3.1 summarizes the data collection mode and number of responses for each survey. (See Appendix C for the data collection protocols.) A discussion of nonresponse analysis is in subsection B.1.9.
Table B.3.1. Survey and Collection Mode
Survey |
Estimated
Number |
Mode |
Process Survey |
||
SUA (mail and fax-back survey) |
56 |
Paper SAQ |
AAA (web and fax-back survey) |
300 |
WEB/SAQ |
LSP (web and fax-back survey) |
200 |
WEB/SAQ |
Menu Survey |
||
LSP Menu Survey (AMPM) (day 1 menu) (CAPI) |
200 |
CAPI/CATI |
LSP Menu Survey (AMPM) (days 2,3 menus) (CATI) |
200 |
CAPI/CATI |
Cost Survey |
||
LSP (paper self-administered questionnaire) |
200 |
Paper SAQ |
Client Outcomes Survey |
||
Home-Delivered Nutrition Program Participant Survey (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Participant Survey (6-month follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Participant Survey (1-year follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Nonparticipant Survey (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Nonparticipant Survey (6-month follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Nonparticipant Survey (1-year follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Participant (CATI/CAPI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Participant (6-month follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Participant (1-year follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Nonparticipant (CATI/CAPI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Nonparticipant (6-month follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Nonparticipant (1-year follow-up) (CAPI/CATI) |
600 |
CAPI/CATI |
24-Hour Dietary Recall |
|
CAPI/CATI |
Home-Delivered Nutrition Program Participant Dietary Recall (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Participant Dietary Recall (6-month follow-up) (CAPI/CATI) |
150 |
CAPI/CATI |
Home-Delivered Nutrition Program Nonparticipant Dietary Recall (CAPI/CATI) |
600 |
CAPI/CATI |
Home-Delivered Nutrition Program Nonparticipant Dietary Recall (6-month follow-up) (CAPI/CATI) |
150 |
CAPI/CATI |
Congregate Nutrition Program Participant Dietary Recall (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Participant Dietary Recall (6-month follow-up) (CAPI/CATI) |
150 |
CAPI/CATI |
Congregate Nutrition Program Nonparticipant Dietary Recall (CAPI/CATI) |
600 |
CAPI/CATI |
Congregate Nutrition Program Nonparticipant Dietary Recall (6-month follow-up) (CAPI/CATI) |
150 |
CAPI/CATI |
We will encourage greater participation through contact and recruitment materials that are relevant to each sample group (Dillman 2000). (See Appendix B for contact and recruitment materials.) Here, we present our strategies for maximizing response rates by survey.
The process survey will examine the strategies, activities and resources of the Title III-C organizations at each of the three levels of the Aging Network: (1) SUAs, (2) AAAs, and (3) LSPs. We will initiate the contacts at the AoA region level and proceed to the SUA level and from there to the AAAs and LSPs. At each level in the Aging Network, we will not only request endorsement for the next level but also ask the respondent to directly communicate that support to the next level in the Aging Network.
A key element in a high recruitment success rate will be the recruitment materials. Dillman (2000) showed that clear, well-written, and persuasive survey materials assist in higher response rates. Recruitment materials include a cover letter, project brochure, and survey worksheet.
Cover letter. The cover letter will explain the purpose of the Title III-C evaluation and will contain endorsements from other agencies or individuals that support the evaluation.
Brochure. The trifold brochure will contain information on the purpose and importance of the study, key components of the study, contact information for the sponsoring and contracting agencies, and responses to frequently asked questions, with a toll-free number and email address in case the recipient has additional questions.
Survey worksheet. The survey worksheet will contain summary information regarding the interview. By knowing in advance the types of information we seek, the SUA director will be able to identify the best respondent for the survey (him- or herself or another staff member). This will provide time for the selected respondent to prepare for the survey and help reduce the burden on the respondents.
All telephone contact with AoA regional officers and SUA directors will be made by senior-level staff. To ensure high response rates once the SUA process survey is in the field, telephone reminder calls will be made to SUA directors to complete the survey. Follow-up letters will be sent and additional reminders will also be made.
B.3.1.b. AAA and LSP Process Surveys
Planned communication with AAAs and LSPs during the data collection is needed to maximize response rates.
We will recruit LSPs for the cost data collection when we recruit them for the LSP process survey. After we recruit the LSPs, experienced Mathematica analysts will be assigned responsibility for a set of LSPs. An important part of the analysts’ responsibilities will be to establish rapport with the LSP’s main contact to encourage the LSP’s participation throughout the study. For the cost study, the analysts will also provide the LSPs with the technical assistance necessary to complete the cost worksheets. After the LSP submits the worksheets, the analysts will follow up, as necessary, to complete any missing data. This will ensure an accurate calculation of the LSP’s meal costs.
The menu survey will be conducted by CATI, as opposed to paper forms. Because CAI modes reduce respondent burden, we expect that this will have a positive effect on response rates.
B.3.4. Client Outcomes Survey—Dietary Recall
The client outcomes survey and dietary recall will take about 60 minutes to administer. We will offer respondents $40 to complete the interview. A subsample will participate in a second dietary recall and will receive an additional $20.
We will use recruiting materials and procedures that are tailored to each target group (see Table B.3.4). The following recruitment materials will be used: a flyer/poster, a study brochure, a letter, a consent form, a confirmation letter, and a screener. The study brochure will provide information regarding the study’s purpose, the role of seniors who agree to participate, and information on whom to contact with questions. The consent form will be IRB approved. The flyer/poster will be posted at meal sites and handed out during congregate meals. All materials will be prepared in both English and Spanish.
Table B.3.4. Tailored Recruitment Materials and Procedures to Ensure High Response Rates
Target Group |
Advance Study Information |
How/Where Recruited |
Recruitment Materials |
Screener |
Congregate Nutrition Participants |
Posters and meal sites, flyers available to recipients |
At meal site |
Letter Brochure Consent Form |
No |
Home-Delivered Nutrition Participant |
Brochure delivered with meal |
By telephone |
Letter Brochure Consent Form |
No |
Nonparticipants |
None |
By telephone |
If eligible, and have verbal consent, send brochure and confirmation letter |
Yes |
B.4. Tests of Procedures or Methods to be Undertaken
The procedures, materials, and instruments developed for the evaluation are similar to those that have been developed, tested, and administered for other elderly nutrition studies. The process, menu, and client outcome surveys were pretested with fewer than 10 respondents. Table B.4.1 shows the survey instrument, the number of agencies, and the number of respondents who participated in the pretest.
Table B.4.1. Data Collection Pretest Activities
Survey/Instrument |
Number of Agencies |
Number of Respondents |
Process Surveys |
||
SUA (fax-back survey) |
3 |
3 SUA directors |
AAA (web and fax-back survey) |
4 |
4 AAA directors |
LSP (web and fax-back survey) |
3 |
3 LSP directors |
Menu Survey (LSPs) |
2 |
2 LSP administrators (4 interviews) |
Client Outcome Survey |
|
4 Nonparticipants |
|
|
3 Home-delivered nutrition participants |
|
|
2 Congregate nutrition participants |
The process surveys are designed to examine the strategies, activities, and resources of the Title III-C organizations at three levels: (1) SUAs, (2) AAAs, and (3) LSPs. Pretests for all instruments took the form of cognitive interviews, and respondents were also asked to review the study recruitment procedures, contact materials, and technical assistance procedures. Each respondent completed the survey independently, and then senior project staff members had a 30-minute follow-up conversation with each respondent to debrief.
In November 2010, the SUA process survey was pretested with three respondents from California, Massachusetts, and Texas. In January 2012, the AAA process survey was pretested with four respondents from Iowa, Kansas, Massachusetts, and Michigan. The LSP process survey was pretested in April 2012 with three respondents from Ohio, Kansas, and Wyoming.
During the debriefing, senior staff members noted questions that needed clarification, questions that required adjustments, and those that needed to be reworded. The time required for each respondent to complete the interview was also recorded. The results of the pretest were used to revise the surveys.
The results of the three pretests are summarized below:
SUA. No additional modifications were identified.
AAA. The respondents’ opinion of the survey was positive overall. Most respondents found both the length and complexity of the survey appropriate, but several noted that new AAA directors or administrators may not be able to complete the survey as quickly or as easily as more experienced directors. The respondents recommended that future respondents be provided a list of necessary sources of data beforehand to facilitate prompt completion of the survey and fax-back form.
LSP. Two respondents found the survey long, but one found that it took less time than anticipated. The respondents found the questions pertaining to finances, number of volunteers, mileage, and unduplicated counts of participants difficult to answer. However, respondents also reported that most data needed to complete the survey were readily available. One respondent suggested making the results of the survey available to participants in summary form, to encourage completion.
In April 2012, we pretested the client outcome survey with nine respondents in California and New Jersey: four nonparticipants, three home-delivered nutrition participants, and two congregate nutrition participants. Senior staff members conducted telephone interviews using the client outcomes survey instrument with the nine respondents, but did not administer the 24-hour dietary recall. As with the process surveys, senior staff members noted questions that needed clarification, adjustment, or rewording. The time required for each respondent to complete the interview was also recorded. The results of this pretest were used to revise the survey.
The results of the pretest are summarized below:
Respondents reported little to no trouble answering the questions and, overall, thought the interview questions were easy to answer. Respondents’ responses were mixed regarding the length of the survey and familiarity of terms. Respondents found the thank you payment an appropriate incentive.
The menu survey pretest examined (1) how the USDA AMPM for 24-hour dietary recall data collection would perform when used to collect menu data; (2) the clarity, utility, and time burden of a “memory prompt” tool developed for this survey; and (3) the time required for respondents to complete the AMPM menu interviews. The pretest was conducted with two LSP administrators, including a nutrition services assistant vice president of programs and a program executive director. Both LSPs operate central commissary kitchens that prepare meals for congregate nutrition sites and homebound clients.
We contacted each respondent by telephone to explain the purpose of the pretest, collect information on LSP characteristics, and schedule the menu survey interviews and debriefing. We sent the draft “memory prompt” form and instructions by email. A Mathematica nutritionist conducted four menu survey interviews using the modified AMPM protocol: three interviews collected menu data for a single meal at congregate sites, and one interview collected both a home-delivered meal and a congregate site meal. As part of the debriefing, the nutritionist solicited respondent feedback on the memory prompt and suggestions for identifying survey respondents who would be knowledgeable about the relevant details of the meals at their program sites. We summarize the results here:
Feasibility of AMPM menu interview. The interviews using AMPM went as expected, and no additional modifications to the menu survey protocol were identified.
Memory prompt. Neither LSP administrator thought that the memory prompt was essential for menu survey respondents to recall the mid-day meal items that were offered or delivered, because menus are readily available. However, the administrators indicated that explicit instructions regarding which meal and the types of food and beverage items to be reported would be very useful. Mathematica revised the form and renamed the tool the “Menu Survey Guide” (see Appendix A). The field interviewers will provide training on the use of the guide when they provide the materials to the menu survey respondents. Completing the associated menu forms took less than five minutes per menu day.
Menu interview time burden. The interviews for a single mid-day congregate site meal took 17 to 21 minutes to administer. The interview to collect both a home-delivered meal and congregate meal lasted 35 minutes (13 and 22 minutes, respectively). Assuming that half of LSPs offer both meal types and that the menu items will differ, we weighted the longer interview accordingly. As a result, the burden estimate for the three-day menu survey was increased by about 40 minutes overall.
Menu survey respondents. In the two programs with central kitchens, local site staff would not know important details about preparation methods and major ingredients of the foods in meals offered and delivered to clients. The LSP administrators recommended that central kitchen managers be the primary respondent in these sites. Although the pretest did not involve programs with on-site meal preparation, the LSP administrators believed that local staff in these sites would be the most appropriate respondents. We will ask LSP administrators to identify the single most knowledgeable respondent for the menu survey in their site.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The design for the study was developed in conjunction with Mathematica Policy Research, under the direction of: Rhoda Cohen, project director; John Hall, senior statistician; Mary Kay Crepinsek, senior researcher, and James Mabli, senior researcher. Ms. Cohen may be reached at (609) 275-2324 or [email protected]; Mr. Hall may be reached at (609) 275-2357 or [email protected]; Ms. Crepinsek may be reached at (617) 301-8998 or [email protected]; Dr. Mabli may be reached at (617) 301-8997 or [email protected].
In addition, Jennifer Klocinski and Susan Jenkins of the Office of Performance and Evaluation, Administration on Aging, Department of Health and Human Services reviewed the study design and instruments. Ms. Klocinski may be reached at (202) 357-0146 or [email protected]. Dr Jenkins may be reached at (202)357-3591 or [email protected].
Biggs, D., B. deVille, and E. Suen. A Method of Choosing Multiway Partitions for Classification and Decision Trees. Journal of Applied Statistics. 1991, 18, 49-62.
Coleman-Jensen, Alisha, Mark Nord, Margaret Andrews, and Steven Carlson. Household Food Security in the United States in 2010. ERR-125. Washington, D.C.: U.S. Department of Agriculture, Economic Research Service, September 2011. Available at http://www.ers.usda.gov/Publications/ERR125/ERR125.pdf.
Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: Wiley, 2000.
Kass, G. V. An Exploratory Technique for Investigating Large Quantities of Categorical Data. Applied Statistics. 1980, 29, 119-127.
Kish, L. Survey Sampling. New York: John Wiley and Sons, 1965.
Magidson, J. SPSS for Windows CHAID Release 6.0. Belmont MA: Statistical Innovations, Inc., 1993.
Ponza, Michael, James C. Ohls, and Barbara E. Millen. Elderly Nutrition Program Evaluation Final Report, Volume I: Title III Evaluation Findings. Report submitted to the U.S. Department of Health and Human Services, Office of the Secretary, Administration on Aging, and Office of Assistant Secretary for Planning and Evaluation. Princeton, NJ: Mathematica Policy Research, July 1996.
Ponza, Michael, James C. Ohls, and Barbara E. Millen. Elderly Nutrition Program Evaluation Final Report, Volume II: Title VI Evaluation Findings. Report submitted to the U.S. Department of Health and Human Services, Office of the Secretary, Administration on Aging, and Office of Assistant Secretary for Planning and Evaluation. Princeton, NJ: Mathematica Policy Research, July 1996.
1 AAAs plan, coordinate, and advocate for the development of a comprehensive service delivery system to meet the needs of older adults in a specific geographic area. They administer state and federal funds for community-based services. http://www.tjaaa.org/glossary-of-terms.aspx.
2 Area agencies normally contract with local for-profit or nonprofit or public providers (LSPs) to deliver benefits. The contract service providers nationwide, providing care under the act, are the largest single network of long-term care providers in the country. An agency may be allowed to directly provide supportive services, nutrition services, or in-home services if it can prove that it can provide these services more effectively. http://www.longtermcarelink.net/eldercare/area_agencies_on_aging.htm.
4 Services are available to people age 60 or older and the spouse of an older person regardless of age.
5 Services are available to people age 60 or older and homebound and the spouse of an older person regardless of age.
6 These are sampled with certainty at this point so they will be included in the subsample of 100.
7 We will obtain the MOS from the SUA.
8 Because non-congregate nutrition recipients will be selected for the interview sample, some of the nearest neighbors so identified may be congregate nutrition recipients themselves. These beneficiaries will need to be screened at the time of interview, as described in Section D.
9 Not all home-delivered nutrition participants listed on the sampling frame will meet eligibility criteria. We will not be able to determine eligibility for those we are unable to contact.
10 The marginal response rate is the percent for whom eligibility is determined multiplied by the completion rate among eligibles.
11 These weights will be constructed as part of the separate client outcomes study.
12 A design effect is defined as the increase in sampling variance, relative to a simple random sample with the same number of observations. Thus for a sample size of n Deff = (Var actual|n)/(Var SRS|n).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | BCollette |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |