Elder nutrition ICR - ACL response to OMB 3-21-14

ENSP OMB Comments

Note to Reviewers: The following pages include three rounds of comments and responses. The second response from the Administration for Community Living (ACL) is in this color and the third response is starts immediately below.

Questions and responses from the conversation between OMB and ACL on 3-19-14

OMB Question: 1-Where does the linking of data files occur? In other words, please walk through the handling of SSN from collection to disposition.

ACL Response: The actual linking will occur with The Research Data Assistance Center (ResDAC) the Center for Medicare & Medicaid’s (CMS) Data Request Center. ResDAC is a CMS contractor (Contract Number HHSM-500-2013-00166C) that provides free assistance to academic, government and non-profit researchers interested in using Medicare and/or Medicaid data for their research. ResDAC is staffed by a consortium of epidemiologists, public health specialists, health services researchers, biostatisticians, and health informatics specialists from the University of Minnesota.

Specifically, the evaluation contractor, Mathematica, will ask service recipients who are participating in the study for their SSN during an in-person interview. The request will occur at the end of the interview and after the interviewer reads text informing the respondent that their SSN will only be used to link their survey data with their Medicare records for the purposes of examining possible effects of nutrition on health and that the SSN will not be released to anyone else or used for any other purpose. Respondents will also be informed that providing their SSN is completely voluntary: respondent can provide their full SSN, the last four-digits of their SSN, or decline to provide their SSNs. Their decision will have no effect on their current or future benefits. If provided, the respondent’s SSN will be immediately entered into the interviewer’s computer and immediately encrypted. The SSN will not be written on any paper forms or otherwise recorded. Interviewers will transmit the encrypted data to Mathematica’s secure server. Then, using a secure transfer site, the SSN and other relevant information such as gender, year of birth, and zipcode will be transmitted to ResDAC in accordance with a Data Use Agreement that specifies that “the User agrees to use the data only for purposes that support the User’s study, research or project referenced in this Agreement, which has been determined by CMS to provide assistance to CMS in monitoring, managing and improving the Medicare and Medicaid programs or the services provided to beneficiaries; and the User agrees to ensure the integrity, security, and confidentiality of the data by complying with the terms of this Agreement and applicable law, including the Privacy Act and the Health Insurance Portability and Accountability Act.” (The complete text of the Data Use Agreement is a separate file that has been sent with this document and is available at: http://www.resdac.org/sites/resdac.org/files/RIF_DataUseAgreement.pdf)

The Medicare data that ResDAC transmits to Mathemetica will not contain SSN. Rather individual records will include a Beneficiary Identification number (BebeID) that will allow Mathematica to link initial Medicare data with Medicare data after the 12-month follow up period. The BeneID, be used to identify an individual. At this point, SSN will be removed from the encrypted Mathematica database.

In addition to the Data Use Agreement, ACL is entering into an Inter-Agency Agreement with CMS that specifies the uses for the data, the authority under which this study is being conducted, and the roles and responsibilities of each agency.

In terms of data security and protection as an Operating Division of HHS, Administration for Community Living (ACL), follows established HHS policies for IT Security and Privacy, including:

• HHS Policy for Privacy Impact Assessments (PIA) (2009-0002.001 2/9/2009)

• HHS Policy for Responding to Breaches of Personally Identifiable Information (PII), (2008-0001.003 2/09/2009)

• HHS Policy for IT Security and Privacy Incident Reporting and Response (2010-0004 4/05/2010)

 A comprehensive list is found at http://www.hhs.gov/ocio/policy/#Security.

OMB Question: 2-What is the informed consent text:

ACL Response: At the start of the survey process individuals will be read the following consent statement:

This survey has two parts. The first part of the survey is about your participation in the nutrition program at [NAME OF PROGRAM SITE] and your satisfaction with aspects of the nutrition program there. The second part of the survey is about what you ate and drank over the past 24 hours. Your participation is voluntary but we would really like your help. This survey is for research purposes only and will help to improve services for older adults in the future. All of your answers will be kept strictly confidential. Your eligibility for services from this and other programs will not be affected by your decision to participate. The entire survey takes about 75 minutes to complete. We’ll mail you a $xx gift card for completing the survey.

Immediately prior to being asked for their Social Security Number, respondents will be read the following statement:

Mathematica Policy Research will conduct statistical research by combining your survey data with health and other related records. To obtain these records, we need your social security number. We will not release it to anyone, including any government agency, for any other reason. Providing this information is voluntary. There will be no effect on your benefits if you do not provide it.

If respondent is reluctant to give number or if respondents ask if they must give number they will be read the following statement:

It is extremely useful to have this information to be able to link to health records such as Medicare records. About a year from now, the information you gave me can be used to see how health habits and diet at one point in your life influence how healthy you are in the future. If you prefer, you can give us only the last four digits of your social security number, and we can use this number to access your records.

If respondent cites privacy concerns as the reason they are reluctant to give their SSN they will be read the following statement:

I understand your concern. Mathematica has never had a breach of confidentiality in the more than 40 years we have been conducting research studies. I do not have access to this information after I type it. Once I complete the interview all the information is sent to a secure facility. Only one or two people have access to the file to use it for our health research. If you prefer, you can give us only the last four digits of your social security number, and we can use this number to access your records.

Respondents who decline to provide their SSN will continue to be included in the study and the data that they did provide will remain in the system for use in analysis and reporting.

OMB Question: 3-ACL’s response to OMB from December 2013, mentioned that without SSN the linkage rates of meal recipients to their Medicare records would range from 33%-78%. What are the expected linkage rates if either full SSN or last 4-digits of SSN are used?

ACL Response: Linkage rates with full SSN are much higher than rates not using SSN. Data reported from CMS¹ shows that linking without SSN resulted in linkages with 48% and 70% of individuals. When full SSN was used those rates increased to 89%. The research team for this data collection has experience using full SSN and achieving linkage rates of 95% and higher. Scholarly articles support this finding. Specifically, an article by Weiner, et al, (2003) reports linkage rates of 97% using only SSN². Grannis et al (2002) report linkage rates between 90% and 95% when using full SSN³ alone and linkage rates of up to 99% when using full SSN combined with other identifiers such as first name, last name, year of birth, and gender. According to Grannis et al, “Linkage criteria that include SSN combined with variables from both name and birth date maximize the match rate while keeping the false positive rate near zero.” ACL has been unable to find definitive information about linkage rates using the last 4-digits of SSN. But, there are articles that discuss the fact that because the last 4-digits of SSN are not unique to a single person, and may be shared with up to 10,000 other people, it is not sufficient to uniquely identify an individual⁴. In addition, in a communication with CMS on 3-20-14, staff of CMMI estimate that linkage rates using the last 4-digits of SSN can be expected to be between 70%-86%.

OMB Question: 4-Is there a need for a Systems of Records Notice (SORN) for this collection?

ACL Response: ACL completed a Privacy Impact Assessment (PIA), and determined that because the data from the system will not be retrieved or retrievable using PII there is no need for a SORN. Once linkage with ResDAC is complete, the system will not contain any PII other than zipcode.

OMB Question: 5-How long will SSN be kept in the database?

ACL Response: SSN will only be collected from meal recipients. SSN will be used solely in the request to CMS/ResDAC for the Medicare records for those individuals. The resulting file created by CMS will not contain SSN. Once received and checked against the Mathematica data request, SSN will be removed from Mathematica’s encrypted database.

EARLIER QUESTIONS/ANSWERS

Part A

A.9. Explanations of Any Payment or Gift to Respondents (page 8)

OMB comment: ‘As mentioned on our call, please provide scientific evidence that the incentive will reduce nonresponse bias, justifications for both the incentive amount ($40 vs. $20), and form of payment (cash vs. gift-card).’

ACL Response: The role of incentives in increasing survey response rates has been widely documented (Holbrook et al. 2008, Singer et al. 1999, Singer & Ye 2013). Some research has found that the level of incentives matters; in other words, higher incentives engender higher response rates (Rodgers 2011; Datta et al. 2001; Colicchia et al. 2012). There is also evidence that higher incentives help in reducing non-response bias without compromising the quality of responses (Singer & Kulka 2001; Castiglioni & Pforr 2007).

A report issued by the Council of Professional Associations on Federal Statistics (1993) identifies circumstances under which incentives should be considered. One circumstance is very relevant to this proposed data collection: “when there are unusual demands or intrusions on the respondent (e.g. lengthy interview).” The client outcomes survey and dietary recall is estimated to take 75 minutes, and those who are selected to complete a second dietary recall (at a later date) will spend an additional 30 minutes completing it. Each follow-up survey is expected to take respondents about 10 minutes to complete over the phone. Therefore, respondents will be offered a $40 gift card to complete the client outcomes survey, a $20 gift card to complete the second dietary recall for those selected, and a $10 gift card to complete each of the 6- and 12-month follow-up surveys by phone. Based on research described above that indicates (1) incentives increase response rates, (2) higher incentives lead to higher response rates, and (3) higher incentives help in reducing non-response bias, these amounts seem reasonable for the amount of burden that will be placed on respondents.

Incentives will be offered in the form of a Visa gift card. The reason for this is that the respondent population includes many adults who are frail may not have the ability to leave their home to cash a check, but could use the gift card online or over the phone. Therefore, the Visa gift card is the likely the best form of providing the incentive to the whole population of respondents.

OMB Response 9/10/13: Please include this justification in your supporting statement. Please integrate an incentive experiment into your planned pilot (offering a reduced amount) and evaluate the corresponding response rates observed between treatment groups.

ACL Response 11/13: While ACL was happy to include a pilot to test the benefits of differing levels of incentives, the budget for the contract under which the outcome evaluation will be conducted was reduced based on competing funding priorities within ACL. As a result, ACL has reduced the incentive to consumers from the $40 mentioned in the original response to OMB to $25. With this reduction, ACL is not sure that there is still a need for a pilot study, but will include one if OMB requests one

.

Another change made based on budget limitations was using widely accepted gift cards rather than limiting the incentives to Visa gift cards as specified in the original response to OMB. This change was made based on fees associated with VISA gift cards

A.10. Assurance of Confidentiality (page 8)

OMB comment: ‘Please include a statement of FISMA-compliance and/or a statement of adherence to the Privacy Act Systems of Records Notices (SORN) requirements.’

ACL Response: Protecting the confidentiality and integrity of sensitive data is of the utmost importance to ACL and its contractor, Mathematica. As frequent collectors and users of ACL/AoA and other federal agencies’ data, Mathematica has adopted federal standards for the use, protection, processing, and storage of data. Their security policies, procedures, and technical safeguards are consistent with the Privacy Act of 1974, the Federal Information Security Management Act of 2002, OMB memoranda regarding data security and privacy, and National Institute of Standards and Technology security standards and guidance. Mathematica secures individually identifiable and other sensitive project information and strictly controls access to sensitive information on a need-to-know and least privilege basis. In addition, data is encrypted in transit and at rest using Federal Information Processing Standard 140-2 compliant cryptographic modules and is securely destroyed at the earliest opportunity. Further, all Mathematica staff are required to complete security awareness training which reviews potential workplace security threats along with policies and procedures for avoiding them. Staff members working on this particular project have also signed non-disclosure forms as required under the contract with ACL.

OMB Response 9/10/2013: To our knowledge, ACL does not have a confidentiality statute. We recommend striking reference to the word “confidentiality” in this response and otherwise incorporating it into the supporting statement. The agency should also confirm that it has a) a system of records notice for this study and b) completed a privacy impact assessment before information collection begins.

ACL Response 11/13: ACL is reviewing the requirements associated with System of Records Notices and is completing a Privacy Impact Assessment related to this study. Appropriate process will be in place before PII is collected as part of this evaluation.

A.11. Justification for Sensitive Questions (page 9)

1. Social Security Number

OMB comment: ‘It is OMB policy guidance that agencies do not collect respondents’ full social security number, or the last four digits. Are there any other ways for the agency to get the information it needs without collecting SSNs?’

ACL Response: Using Social Security Numbers (SSNs) together with names and addresses of Medicare beneficiaries will ensure accurate identification of treatment group respondents in Medicare claims and enrollment data, necessary for estimating the propensity score matching model. In the absence of SSNs, identification of treatment group beneficiaries in Medicare data will have to rely on using names and addresses alone, which will increase the time, level of effort, and cost to the government in identifying these beneficiaries while also reducing the quality of the matches (or data). For instance, differences in the spelling of names between the survey response and Medicare enrollment data and possible changes in address will require closer scrutiny and a more elaborate system for ensuring that respondents are correctly mapped to their Medicare records. Also, the possibility of errors in identifying beneficiaries cannot be completely ruled out, which would lead to higher variance in our estimates and reduce ACL’s ability to draw conclusion from, and inform policy with, these data.

To further support our request to obtain and use SSNs, OMB’s memo M-13-17 dated July 26, 2013, to the heads of federal departments and agencies entitled “Next Steps in the Evidence and Innovation Agenda, specifically encourages agencies to link “data across programs and levels of government while fully protecting privacy” (pg. 6). The purpose is to lower costs of evaluations and improve the quality of data collected. The use of Medicare records and the SSN of interviewed congregate and home delivered meal participants will allow the research team to efficiently identify an optimal comparison sample for the evaluation. The quality of the outcomes data using Medicare records will be improved because the research team will be able to access respondent-specific data about chronic diseases and hospitalizations both retrospectively and prospectively from the date of the survey.

ACL and the contractor for this effort, Mathematica, are committed to protecting the security of all study data and, in particular, the confidentiality of Personally Identifiable Information (PII) that respondents provide. Mathematica will be able to assure respondents that their identities and the data they provide—or data that others provide with their consent—will be kept in the strictest confidence, will be used only for the purposes explained to them, and will not be linked to other data, except for research purposes. The assurances and the language that will be used with respondents will be approved by Mathematica’s external institutional review board.

OMB Response 9/10/2013: We believe the reference to M-13-17 is incorrect in the present context. That memo pertains to facilitating linkages to extant data; it does not promote the collection of social security numbers. As ACL is aware, M-06-16, M-07-16 and M-07-19 require agencies to eliminate the unnecessary collection and use of social security numbers. The collection of SSN for this information collection is not necessary for the proposed information collection; participants can be appropriately matched by name, address, and other direct and indirect identifiers collected as part of this study. Please remove reference to SSN collection in your request materials.

ACL Response 11/13: ACL understands the seriousness of the collection of SSN. Wherever possible, ACL tries to reduce or eliminate the collection of SSN. However, in the case of this evaluation, the collection of SSN is necessary. Without SSNs, the research team will not be able to link participants to their own health records in the Medicare file with a reasonable level of accuracy or efficiency. Without confidence in the linkages of meal recipients with their Medicare records, the research team will have less information to use when matching program participants with non-participants which would increase the number of unobserved variables on which the groups may differ and substantially increase bias in the estimates of the effects of meal program participation on nutrition, food security, and socialization outcomes. Our ability to distinguish program effects from differences in characteristics associated with program outcomes will be diminished as a result. Not having SSNs will also introduce substantial error in the measurement of long-term health outcomes, preventing identification of differences in outcomes associated with program participation versus differences in outcomes as the result of design error. Because of the sensitive nature of some of the data ACL is requiring the contractor to prepare and submit a request for approval to a recognized Institutional Review Board (IRB) for Research Involving Human Subjects. All study materials and instruments for the program participants and nonparticipants will be submitted to and approved by the IRB.

Examples included below show that similar efforts of matching individuals’ to their Medicare records without SSN, resulted in linkage rates between 33% and 78%. These rates are unacceptable. For example, calculations conducted by the research team to determine the number of respondents needed for this study show that with the full sample the minimum detectable difference (MDD), at an 80% confidence level, in food security would be 4.16 points. While some attrition is expected, and accounted for in their calculations, if there is an additional 20% attrition rate because of our inability to link meal recipients with their health data the design effect (the amount of the detectable difference attributable to the design rather than actual differences between groups) increases by 15% and the MDD increases by 4%. With 66% attrition the design effect increases by 15% and the MDD increases by 15%. For selected sub-groups the effect is more pronounced. For example, with congregate meal participants at the 6 month follow up, with 20% attrition the design effect increases by 31% and the MDD by 8%; at 60% attrition the design effect increases by 53% and the MDD by 30%. Such increases in the design effect and the minimal detectable difference mean that the team may not be able to detect actual differences between meal recipients and members of the comparison group.

1. The central outcomes for the study are dependent on the use of SSN. Health outcomes are explicitly listed in the authorizing legislation as an objective for the ENSP. Title III-C of the Older Americans Act states that an objective of the ENSP is “to promote the health and well-being of older individuals by assisting such individuals to gain access to nutrition and other disease prevention and health promotion services to delay the onset of adverse health conditions resulting from poor nutritional health or sedentary behavior.” Further, the subsection in the legislation on Study of Nutrition Projects states that “…study shall, to the extent data are available, include—an evaluation of the effect of the nutrition projects authorized by such Act on—(i) improvement of the health status, including nutritional status, of participants in the projects;…”

The Client Outcomes Survey will be administered to consumers and the matched sample of non-consumers who participate in the evaluation. The survey will be one source used to estimate the effect of program participation on individuals’ nutrition, food security, socialization activities, and health. However, because the information in the survey is self-reported, these data have limitations based on individual’s memories and other factors that affect the quality of self-reported data. To augment the self-reported health data collected in the survey, and to reduce the number of questions respondents are asked about their health (i.e., response burden), Medicare data have been identified as the best source of information regarding the health status and health care utilization patterns for this study population.

We are requesting to use SSN because linking two or more data files is easiest when there is a common, unique identifier such as SSN. In the absence of such an identifier, the linking task becomes much more difficult and less accurate. Furthermore, the complexity of the linking task is amplified if the files are large, the quality of the matching variables is poor, or the matching variables are not comparable due to differences in measurement or timing (Herzog et. al., 2007). 

The research team has experience linking with and without SSN. In their experience, it is far easier and faster to find high quality matches when SSN is available. Although it is possible to link data files without SSN, those linkages are often lower quality and require more time for manual review. On the surface, first and last name, date of birth, gender, and address appear to provide sufficient information to accurately link records across multiple files. However, in practice, these measures often fall short of correctly and uniquely linking records. The quality of these variables varies across files making data linking more challenging and reducing accuracy (Herzog et. al., 2007).  In other words, we may be able to match study participants with Medicare records, but will not be able to guarantee that they are matched with their own Medicare records. Problems encountered using common matching variables are presented below:

Name – Even after data cleaning and standardization, first, last and middle names present many challenges when used for linking records.

• Common first and last names limit the ability to uniquely link an individual’s records

• First and last names can be spelled differently across multiple files (Catherine vs. Katherine) and can also contain typos

• The occurrence of nicknames or shortened first names can vary across files

• Last names often change over time due to marriage, divorce and adoption

• Suffixes such as Jr. and Sr. can be recorded differently across files or not at all

• First and last names can be transposed within a data file

• First and last names can become truncated in data extracts

Address – A complete and accurate address can be very helpful in linking an individual across two files. However, these variables also contain many issues that limit their ability to link records.

• Address can change over time and so may not match across files. Therefore, while an exact match on address is a good indication of the same person, a different address does not necessarily mean it is a different person.

• Different data files may record different types of addresses (mailing vs. residential)

• Often only a partial addresses is available due to missing information

• Misspellings of street names and cities

Date of Birth – Given the relatively few unique values, especially when working within a certain timeframe or age group, date of birth has limited ability to uniquely identify an individual. In addition, date of birth can include typos and transposed dates (month and day).

Gender – With only two possible values, gender is better suited as a confirmatory measure rather than a linking variable. Furthermore, while a basic measure, gender is not always 100 percent accurate.

Below are two recent examples of problems encountered by Centers for Medicare & Medicaid Services (CMS) contractors when attempting to match consumers to their Medicare records without SSN.

Example 1: Lessons Learned from CMS’ Pilot Evaluation of the Chronic Disease Self-Management Program. For this pilot, CMS’ contractor attempted to match consumers with their Medicare records. Linking using direct and indirect identifiers was associated with a 78% match rate. Specific issues that reduced the match rate, and are applicable to the current proposed data collection, were:

• Variation between the data received and the data needed: Difficulties arose due to variations in the order of address elements and variations in the address elements themselves such as the variety of abbreviations available for the same word (i.e., ST, Str., or Street). Additionally, the CMS beneficiary mailing address field might have included “in care of” information that was inconsistent with a Medicare beneficiary’s residence information. While a majority of street addresses were correctly parsed, substantial effort was required and sufficient error remained. Thus, the address field was used as a validation field during manual review rather than a linking field.

• Time and resources needed to develop an appropriate scoring method for matching: Establishing an appropriate scoring method required several reviews of the linking sequence. After assigning field scores using SAS© 9.3 Generalized Edit Distance commands, a manual review of field scores and cumulative scores compared to appropriateness of linkages was completed. Ultimately, a combination of cumulative score and individual field scores resulted in an iterative process. To assure appropriateness of linkages, a final manual review was included for linkages that did not exceed the maximum cumulative score, but did exceed one or more of the individual field scores. This allowed the maximum opportunity to find appropriate linkages. However, the time required for manual review of questionable linkages may become an important limitation for the feasibility of a study as sample size increases. For example, of the 71 Chronic Disease Self-Management Program (CDSMP) participants that did not have scores below both cumulative and field score thresholds, 64 found Medicare beneficiary data similar enough to justify manual review. After manual review, only 6 of the 64 CDSMP participants (9%) were linked to Medicare beneficiary identification numbers. The manual review of 64 participants including validation and incorporation of the 6 into the final sample took approximately 2 hours.

• Underlinking based on patterns of similarities in variable responses: The potential for under-linking may exist when another Medicare beneficiary with a name and date of birth similar to a study participant is found prior to finding the actual beneficiary data for the study participant. For example, the automated process may identify “Jane Coe” as the appropriate beneficiary for linking to “Jane Doe,” the study participant, prior to finding “Jane Doe” in the beneficiary file. The similarities in names would produce a matching score low enough to remove “Jane Doe” from further linking consideration.

Example 2: Lessons Learned from the Retrospective Study of Community-Based Wellness and Prevention Programs Final Report. CMS contracted with Acumen, LLC, to conduct an evaluation of health service utilization and medical costs in Medicare beneficiaries participating in evidence-based wellness programs to assess whether these programs have the potential to improve beneficiary health outcomes and reduce health resource utilization. The final report, which has not been released to the public, was completed in spring 2013. The researchers used ACL participant data containing date of birth, zip code, and gender to match ACL program participants with their Medicare records to determine whether participants, as compared to non-participants, had different patterns of health care utilization and lower health care costs. The researchers ended up with a 33% linkage rate- a level that is unacceptable for the ENSP evaluation.

2. Introduction of bias into the treatment and comparison groups. While considered the most rigorous approach, random assignment is not an option for the evaluation of the ENSP. Specifically, random assignments would require denying meals to people who need them and the record keeping needed for random assignments is largely outside of the capacity of local meal providers. As a result, the group of nonparticipants will likely differ from the ENSP participants in terms of some set of characteristics. The difference in outcome measures that reflect these differences in individuals’ characteristics, rather than true program effects, is often referred to as selection bias. The more comprehensive the information used in the process to match participants and nonparticipants, the more we will be able to minimize selection bias.

The research team proposes to link each meal recipient who participates in the evaluation to the Medicare data file using his or her SSN in order to obtain information about his or her health status and health care utilization patterns. The research team will then use this health status and utilization data, in addition to variables collected in the survey (i.e., age, gender, race and ethnicity, and zip code), to match meal recipients with other individuals in the Medicare database to construct the comparison group. This will greatly reduce selection bias by increasing the similarity between meal recipients and matched nonparticipant groups (i.e., the team will have more variables upon which to match). The inability to use SSNs to match meal recipients to potential nonparticipants is likely to substantially increase bias in our estimates of the effects of meal program participation on nutrition, food security, and socialization outcomes.

If the research team uses SSNs for matching participant Medicare records to find the comparison group, it will take about 11 weeks between participant and nonparticipant interviews in the same geographic area. Without SSNs, the research team will have to rely on probabilistic and deterministic sampling procedures and the gap between participant and nonparticipant interviews will increase to at least 29 weeks. (Please see Exhibit 1 and 2 for timelines associated with data collection using or not using SSN respectively.) Research suggests that the delay in the identification of, and collection of data from, members of the comparison group will likely result in bias in estimations of food security and nutritional status between the two groups⁵ as a result of price movements and/or food availability resulting from changing weather conditions, production cycles, changeovers of models, and holidays.

3. Increased burden to respondents and data collection. If SSNs are not used there could be a significant number of study participants that cannot be reliably matched to their own Medicare records. If this occurs then, in addition to reducing the quality of the comparison group, the research team will not have health data for those respondents. As health outcomes are central to the goals of the ENSP, health status and utilization data would need to be collected directly from study participants. This will require additional burden for respondents and additional costs to the evaluation for the development and testing of new questions for the client outcomes survey and 6- and 12-month follow-up surveys to capture these data in self-report form.

,

Part B

B.1.6. Consent (page 6)

Text from OMB package: ‘Consent will be given by the participant. If a respondent is too ill to participate in an interview or has cognitive, hearing, speech, or vision impairments, a family member or caregiver may serve as a proxy for consent and the interview.’

OMB Comment: ‘What biases must be considered when using this strategy vs. selecting another individual from the respondent pool?’

ACL Response: In making the decision on whether or not to allow proxy respondents, the research team weighed several factors. Allowing for proxy respondents means that the study will be more likely to have outcome measure estimates that are representative of the entire population served by Title-III C meal programs because it will include frail individuals who need a proxy in order to participate in the interview. Not allowing for proxy respondents will mean excluding frail individuals who cannot respond to the survey on their own, and the estimates produced would, thus, be more representative of the non-frail population served by Title III-C meal programs. This is problematic for several reasons including that ACL is required to collect information about services to frail individuals. In particular, the Older Americans Act (OAA), which authorizes the Elderly Nutrition Services Program, specifically requires that ACL must provide reports on its services that include “statistical data and an analysis of information regarding the effectiveness of the State agency and area agencies on aging in targeting services to older individuals with greatest economic need and older individuals with greatest social need, with particular attention to low-income minority individuals, older individuals residing in rural areas, low-income individuals, and frail individuals (including individuals with any physical or mental functional impairment)” (Section 207).

The survey instrument contains questions that collect objective, not subjective, information that a family member or caregiver, who is knowledgeable about the respondent’s dietary intake and who lives in the household or provides care to the individual during waking hours, can accurately answer.

Each situation will be independently evaluated to determine the need for a proxy respondent, recognizing that self-reporting is preferred. If a suitable proxy cannot be identified, that individual would be considered ineligible and another individual would be selected from the respondent pool.

OMB Response 9/10/13: Thank you. Please incorporate into the Supporting Statement.

ACL Response 11/13: ACL will incorporate this into the supporting statement.

B.1.8. Response Rates (page 7)

Text from OMB package: ‘No nonresponse is anticipated at the SUA level.’

OMB comment: ‘Please update to account for the potential of nonresponse.’

ACL Response: The SUA survey is a census of a small universe (N=56); all SUAs will be requested to complete the survey. While ACL anticipates 100 percent participation because other evaluation activities conducted by ACL have reached those levels, it is possible that some SUAs will decide not to or will be unable to participate. In that case the number of responses will be smaller. However, even if there is non-response, ACL believes that it will be low and that it is reasonable to anticipate a response rate of at least 90 percent. Since the SUA survey is a census of a small universe, statistical analyses are not planned and we do not propose to weight survey responses to account for any non-response. Given these considerations and the anticipated response rate of at least 90 percent, no non-response bias analysis is planned for this group.

OMB Response 9/10/13: Please incorporate into the supporting statement that, although you hope to achieve a 100% response rate, you will conduct nonresponse bias analysis if the observed response rate is lower than 80%.

ACL Response 11/13: ACL will incorporate this into the supporting statement.

2. Client Outcome Survey (page 20)

OMB comment: ‘What was the justification for senior staff not administering the 24-hour dietary recall? And, what has staff done to address the length of the survey and improve respondents’ familiarity of terms?’

ACL Response: Senior staff administered the pretest to be able to evaluate responses and identify problems with the survey administration. The decision to only pretest the Client Outcomes Survey and not include the 24-hour dietary recall was based on Mathematica’s understanding that the 24-hour dietary recall has been validated and was successfully used with older adults in the Center for Disease Control’s National Health and Nutrition Examination Survey (NHANES). Based on the discussions with OMB staff, the research team now proposes to pilot the dietary recall data collection procedures along with the Client Outcomes Survey. During the proposed pilot test (see below), Mathematica will conduct an in-depth examination of respondent burden and will provide recommendations for revising procedures and study materials as necessary. ACL and Mathematica are also working to identify questions or sections of the questionnaire that could be cut to decrease respondent burden. The results of the pilot will help inform decisions on changes to the client outcomes survey. Well-trained professional field interviewers will administer the Client Outcomes Survey and 24-hour dietary recall to participants and non-participants during the full-scale data collection in Phase 2 of the study.

Regarding familiarity with terms, during the pretest, several participants who were receiving home-delivered meals were unfamiliar with the name of the agency that provided them with the meals. They indicated that they were more accustomed to the name “meals on wheels.” As a result, the research team changed the instrument to reference meals on wheels as well as the name of the agency that actually provides the meals.

OMB Response 9/10/13: Please incorporate the proposed pilot into the supporting statement, including research questions, method of evaluation (particularly with the reliability of the recall instrument), sample size determination, and recruitment method. Please also include an incentive experiment (offering a lower incentive amount as well as that currently proposed and evaluating the resulting response rates). OMB review of the pilot results prior to launch of the main study will be a condition of clearance.

ACL Response 11/13: ACL will incorporate the proposed pilot (described below) of the outcome data collection instruments into the supporting statement.

The pilot testing of the Client outcomes survey and 24-hour dietary recall will focus on respondent burden and fatigue. The research team will test three different question ordering patterns, placing the dietary recall at the beginning, middle, and end of the interview. They will observe and compare the total length of the interview as well as levels of respondent fatigue and engagement between the three different ordering patterns to determine the optimal order. They will test the three ordering patterns with equal numbers of Congregate Meal (CM) and Home Delivered Meal (HDM) participants: 5 CM and 5 HDM participants will be administered each ordering pattern, for a total of 30 completed interviews. The research team will also observe aspects of the interview and refine their training approach to address issues such as respondent hearing, sight, and motor impairments; low levels of engagement; and situations best suited for identifying proxies. The pilot test will also provide an opportunity to implement several features of the ASA24 dietary recall interview. Little is known about the ability of older adults to estimate portion sizes using the ASA24 food portion visual aids on a computer screen. The pilot will help determine whether these aids are sufficient or if further portion size estimation guides (such as measuring cups) would be useful. The research team will also determine how well the new source of food module, to be released by NCI in September 2013, distinguishes where respondents obtained the foods and beverages they ate (from a program meal versus other sources). The pilot will allow observation of the general feasibility of the interviewer assisted ASA24 dietary recall. For example, the web-based format requires a high-speed Internet connection and we can detect and correct any logistical problems that arise. Additionally, the team will test how well the protocols and scripts, specially developed for this study and intended to facilitate interviewer-assisted administration, meet the study needs.

With regard to the pilot of the Menu survey, to our knowledge, the ASA24 has never been used to collect menu data to assess the food and nutrient content of offered/delivered meals. A previous pilot to test the feasibility of using AMPM to collect menu data suggested that this innovative approach would work in congregate settings and for home-delivered meals. However, the ASA24 uses a different method of estimating portion sizes. Whereas the AMPM provides respondents with a food model booklet and measuring cups and spoons and prompts them to refer to these throughout the interview, the ASA24 requires respondents to review food images on the computer screen. The pilot test will examine Local Service Provider (LSP) respondents’ reporting of portion sizes during the menu survey interviews. The research team will also explore the process of identifying a local respondent who is most knowledgeable about the meal. This respondent’s position and title will vary by site, as meals may be prepared at a congregate nutrition site, a central kitchen or other off-site location, or catered by local food service vendors or restaurants. Once the team identifies the respondent, they will determine his or her knowledge of food details such as preparation methods, brands, and recipes. The team will also compare the utility of respondents using a memory prompt developed for this study to recall foods in midday meals versus using a printed menu, in terms of whether the menu includes an accurate listing of foods actually served and whether it is available to respondents when they are interviewed. Finally, we will test the performance of scripts developed for the menu survey that facilitate interviewer-assisted administration.

As discussed in the response to a previous question, as a result of lower funding levels available for this evaluation, the incentive level has been reduced from $40 to $25. Therefore, ACl no longer proposes to conduct a pilot study to assess the value of different levels of incentives.

Email from OMB

OMB comment: ‘We recommend the agency engage in a pilot study of the outcome evaluation before launch the main study version.’

ACL Response: ACL proposes conducting a small scale pilot of the client outcomes survey with the 24-hour dietary recall and the menu survey using senior staff to test the operational aspects of the data collection. Pilot-testing the menu survey with agency respondents and the client outcomes survey and 24-hour dietary recall with program participants will provide valuable information related to respondent knowledge and burden.

Results of a previous pretest of the client outcomes survey were positive. This proposed pilot test will build on these findings, focusing on how the introduction of the 24-hour dietary recall affects respondent burden and fatigue. The pretest determined how long the client outcomes survey takes without the recall. The pilot will be used to compare to the time it takes to administer the client outcomes survey and dietary recall together. Further, during the pilot, we will place the recall at different points in the interview, such as in the middle or at the end, and observe and compare the length of administration as well as levels of respondent fatigue and engagement.

Mathematica will contact local service providers of congregate and home-delivered meals in the Princeton, New Jersey area in order to identify a convenience sample of 6 providers who agree to participate in the pilot. Mathematica will make arrangements to interview 2-3 congregate and 2-3 home-delivered meal participants at each site, as well as arrange for a time to complete the menu survey with the appropriate individual at the site. In total we will administer the client outcomes survey and 24-hour dietary recall to 30 congregate or home-delivered meal participants from these 6 sites, and complete the menu survey with an individual at each site. We will make recommendations for revising procedures and study materials as necessary, and will provide a memorandum describing the results of the pilot to OMB.

OMB Response 9/10/13: Please see above comment.

ACL Response 11/13: ACL will incorporate the proposed pilot into the supporting statement.

References

Castiglioni, L., & Pforr, K. “The effect of incentives in reducing non-response bias in multi-actor survey.” Paper presented at the 2nd ESRA Conference (Prague). June 2007.

Colicchia, M., Czaplewski, M., &, Jaszczak, A. “Refusal Conversion Incentives and Participation in a Longitudinal Study of Older Adults.” Survey Practice, vol. 5, no. 3, 2012. Available: http://surveypractice.org/index.php/SurveyPractice/article/view/22/pdf_1

Council of Profession Associations on Federal Statistics. “Providing Incentives to Survey Respondents, Final Report.” 1993. Available: http://www.copafs.org/reports/providing_incentives_to_survey_respondents.aspx#incentives

Datta, A.R., M.W., & Walker, J.R.“Evaluation of a Monetary Incentive Payment Experiment in the National Longitudinal Survey of Youth.” Paper presented at the 2001 Federal Committee on Statistical Methodology Conference. Available: http://www.fcsm.gov/01papers/Horrigan.pdf

Holbrook, A. L., Krosnick, J. A. and Pfent, A. “The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms,” in Advances in Telephone Survey Methodology (eds J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. d. Leeuw, L. Japec, P. J. Lavrakas, M. W. Link and R. L. Sangster), John Wiley & Sons, Inc., Hoboken, NJ, USA, 2007.

Herzog, Thomas N., Fritz J. Scheuren, William E. Winkler. Data Quality and Record Linkage Techniques.  New York: Springer, 2007. Print.

Rodgers W.L. “ Effects of increasing the incentive size in a longitudinal study.” Journal of Official Statistics. vol. 27, no. 2, 2011, pp. 279-299

Singer, E., Gebler, N., Raghunathan, T., Van Hoewyk, J., & McGonagle, K. “The effect of incentives on interviewer-mediated surveys.” Journal of Official Statistics vol. 15, no. 2, 1999, pp. 217-230.

Singer, E., & Kulka, R.A. “Paying respondents for survey participation.” 2011. Available: http://aspe.hhs.gov/hsp/welf-res-data-issues02/04/04.htm

Singer, E., & Ye, C. “The use and effects of incentives in surveys.” The ANNALS of the American Academy of Political and Social Science, vol. 645, no. 1, 2013, pp. 112-141.

Exhibit 1: 20 Week Comparison Group Identification Period with SSN

Exhibit 2: 38 Week Comparison Group Identification Period without SSN

1 Acumen. (2013) Retrospective study of community-based wellness and prevention programs: Final report. Page 173

2 Weiner, M. Stump, T.E., Callahan, C.M., Lewis, J.N, & McDonald, C.J. (2003). A practical method of lining data from Medicare claims and a comprehensive electronic medical records system. International Journal of Medical Informatics, Vol. 71, issue 1 pages 57-69.

3 Grannis, S.J, Overhage, J.M, & McDonald, C.J. (2002) Analysis of Identifier Performance using a Deterministic Linkage Algorithm, Proc. AMIA Symp. 305–309.

4 Larsen, M.D, and Yuan Zhao (2012). “A Study of Factors Affecting Record Linkage  in Federal Statistical Databases.” FCSM Research Conference 2012. http://www.fcsm.gov/12papers/Larsen_2012FCSM_X-B.pdf

5 Bureau of Labor Statistics. August 2013. Consumer Price Index Frequently Asked Questions. Available at [http://www.bls.gov/cpi/cpifaq.htm]. Last accessed October 11, 2013.

Gregory CA, Coleman-Jensen A. Do food prices affect food security: Evidence from the CPS 2002-2006. Selected Paper prepared for presentation at the 2011 Agricultural & Applied Economics Association and Northeastern Agricultural and Resource Economics Association Joint Annual Meeting, Pittsburgh, Pennsylvania, July 24- 26, 2011. Available at [http://ageconsearch.umn.edu/bitstream/103265/2/foodprice_foodsecurity.pdf ]. Last accessed October 11, 2013.

Lee JS, Frongillo EA, Jr. Nutrition and health consequences are associated with food insecurity among U.S. elderly persons. J Nutr. 2001; 131:1503–09.

Mullens, Richard. Rising food costs take bite out of budgets. The Tampa Tribune. September 2013. Available at [http://tbo.com/news/business/rising-food-costs-take-bite-out-of-budgets-20130922/]. Last accessed October 10, 2013.

Nord, Mark, and Linda S. Kantor. "Seasonal variation in food insecurity is associated with heating and cooling costs among low-income elderly Americans." The Journal of nutrition 136.11 (2006): 2939-2944.

Rose, D., Gundersen, C. & Oliveira, V. (1998) Socio-Economic Determinants of Food Insecurity in the United States: Evidence from the SIPP and CSFII Datasets. Technical Bulletin 1869. Economic Research Service, U.S. Department of Agriculture, Washington, DC.

Todd, Jessica E., Lisa Mancino, Ephraim Leibtag, and Christina Tripodo. Methodology Behind the Quarterly Food-at-Home Price Database. Technical Bulletin No. 1926, U.S. Dept. of Agriculture, Economic Research Service, April 2010.

Wenzlau, Sophie. Global Food Prices Continue to Rise. Worldwatch Institute. April 2013. Available at [http://www.worldwatch.org/global-food-prices-continue-rise-0 ]. Last accessed October 10, 2013.

Page 12 of 12