Response to Desk Officer questions

Subject: OMB Questions for Senior Corps Longitudinal Study

1)      Can CNCS provide more clarity on why and how the volunteer and caregiver studies (combined) make sense for one combined longitudinal study? 

a.     There is much justification provided on the basis for the volunteer study but not as much for the caregiver study

There are three reasons for combining the volunteer and caregiver studies.

First, this study is designed to measure progress towards two goals of the 2011-2015 strategic plan: Increasing the impact of national service on community needs and increasing the number of adult national service volunteers. Progress toward the first goal is measured by the number of caregivers receiving respite services (output) and the change in caregivers’ satisfaction with respite services (outcome). The Volunteer Study addresses Goal 2 of the Strategic Plan: Increasing the number of adult national service volunteers. This outcome is measured by volunteer satisfaction and retention, self-rated health, and symptoms of depression and loneliness.

Second the combined study seeks to leverage an existing data collection process specific to Senior Corps for maximum efficiency and minimum burden. The caregiver and volunteer surveys will be distributed concurrently with approximately 2,300 instruments administered by state office and grantee staff. Combined as such this study will require only one work plan, one clearance process, one TWG and FWQ, one training and technical assistance plan, and one data collection process. It will utilize existing access to new caregivers as they come on board for the first time, making survey distribution more economical and efficient.

Third, the Caregiver survey will provide the necessary baseline data for a future longitudinal evaluation.

b.     How does the prior national study for Senior Corp compliment and differ from this proposal? 

The prior study was a cross-sectional design. Although the longitudinal study is still correlational, this research tracks changes in volunteers and caregivers over time, allowing for some insight into the processes that contribute to volunteer satisfaction and caregiver stress.

2)      There is some mention about the Michigan Health Study. Can CNCS elaborate on what exactly was done with that data (in terms of comparisons?)

Health and Retirement Study data from the Michigan Health Study (http://hrsonline.isr.umich.edu/) was used in the Senior Corps Evaluation to generate a matched comparison group. This element of the study allowed comparisons between Senior Companion Program (SCP) clients and a matched comparison group of HRS respondents who did not receive SCP services. The SCP survey included several variables that were also contained in the HRS datasets including background and demographic characteristics, and questions on life satisfaction, self-efficacy, social loneliness, and emotional loneliness.

3)      Under confidentiality section of SS A, there is mention of confidential information being collected. Can CNCS cite the appropriate SORN used to protect the respondents PII?

A SORN is not required because the information maintained by the researchers is not retrieved by personal identifier and, as a result, does not require a SORN under the statute. Data is categorized by grantee organization. PII is protected through segregating, de-identifying, and restricting access to the data.

4)      Incentives- We do not believe that the 10 or 20 incentives to the org and participants are necessary. These do not believe to be “hard to reach” populations and the incentive amounts are and have not proven to change behavior or improve response rates.

The Foster Grandparent and SCP volunteers and caregivers are in fact, hard to reach populations. These volunteers and caregivers are mostly low-income, minority adults with a mean age greater than 72. They have limited English proficiency and almost 10 percent of these populations primarily speak a language other than English.

CNCS does not maintain a primary relationship with these volunteers or caregivers. It neither serves as the primary source of information nor is the initial point of contact. Consequently, CNCS’s ability to contact these populations is encumbered by the fact that they are more difficult to reach by phone or through the mail. Commonly these populations fear being victims of phone scams, and are suspicious of mail from unknown senders; especially at the initial point of enrollment. These populations are also far less likely to be connected to the Internet, as they are often not computer literate. Given this population profile, the best method to administer the survey to these volunteers and caregivers would be in person.

Please note that the timing of the baseline data collection is critical. If CNCS misses the window to collect the baseline due to multiple rounds of contacts and persuasion to participate, participants will have already started the intervention; thus confounding our ability to test the impact of the intervention, which is the main goal of the study.

The limited option through which these populations can be reached means that CNCS will need to engage in multiple rounds of in-person contact to encourage participation, which is an unlikely scenario without any incentive. Recognizing that the most effective way to administer the survey – in person – is costly, these multiple rounds of likely unsuccessful contacts to encourage participation are not cost effective, particularly as there are multiple waves of data collection. The cost of multiple rounds of contacts to persuade outweighs the proposed incentive to increase participation. Given the costs of the study and the potential uses of the findings it appears prudent to take steps to ensure a viable sample.

In addition, in longitudinal data collection the use of incentive has been shown to be cost effective due to the savings incurred by reducing the costs of follow-ups with non-respondents across waves of data collection (Rogers, 2011). There are mixed findings in the research literature. Certain types of incentives, timing of incentives, and amounts of incentives may not be effective, but the overall recommendation is that an incentive is an effective means to increase response rate and reduce nonresponse bias. A recent study on the use of incentives in the Annals of the American Academy of Political and Social Science, Singer and Ye (2012) conclude that:

• “Incentives increase response rates to surveys in all modes, including the Web, and in cross-sectional and panel studies;…

• Monetary incentives increase response rates more than gifts, and prepaid incentives increase them more than promised incentives or lotteries, though they are difficult to implement in Web surveys;…

• Incentives, thus, have clear potential for both increasing and reducing nonresponse bias. If they can be targeted to sample members who would otherwise fail to respond”

This article represents a balanced perspective in that it presents findings that suggest that some types of incentives are not effective as well as findings that suggest incentives are effective. The general conclusion based on research published in the most recent decade is that the incentives are effective means to increase the response rate.

Additionally, a substantial body of research including experimental and meta-analyses does support the use of incentives to increase response rates (Brick et al. 2005; Church 1993; Edwards et al. 2002, 2005; James and Bolstein 1992; Shettle and Mooney 1999; Singer et al. 1999; Singer, Van Hoewyk, and Maher 2000; Yammarino, Skinner, and Childers 1991). See attached references.

Senior Corps and the contractor asked a working group of Senior Corps grantees about incentives. The group strongly endorses an incentive to encourage participation given the critical window of opportunity to engage participants in the study. The incentive will target individuals who would otherwise not respond. Engaging in follow-up efforts to retain non-respondents would be more costly than the proposed incentive.

CNCS’s experience in the previous evaluation is that these populations do not accept calls from strangers nor are they open to communicating with strangers. Again this goes to the issue of the propensity of these populations to be the target of scam. Thus it is important to use the grantee/organization as the agency that conducts the enrollment. At baseline CNCS is unknown to these populations; and they have begun to build a trusting relationship with the Senior Corps grantees in their community. The $10 to the organization is reimbursement for estimated expenses related to the enrollment of the participants.

5)      What particularly about the piloted study inform and change the proposed evaluation?  Additionally, why was the generic not used to reach a larger population vs. the 9 and under?

The cross-sectional study of FGP and SCP volunteers and SCP caregivers provided very valuable experience with strategies to enhance data collection adherence from grantees, volunteers and beneficiaries.

• The volunteer surveys has improved our understanding of questions that volunteers may need support to answer , as well as languages other than English are most commonly used among volunteers.

• The SCP client survey, which was administered to a convenience sample of SCP grantees provided valuable experience for working directly with respondents. The proposed evaluation of both volunteers and caregivers includes multiple protocols, gleaned from prior experience, to assist with the introduction of the research by grantees to potential longitudinal study participants.

• The FGP and SCP volunteer cross-sectional study identified differences between the FGP and SCP volunteers and control volunteers and non-volunteers from the Health and Retirement Study. The study supported the feasibility of simulating questions from the Health and Retirement Study as well as the potential value in examining health status, mobility and well-being as outcomes in the longitudinal study.

• The cross-sectional study also provided grantees with experience participating in survey research. This increase in grantee research capacity will be important in supporting a high response rate.

• Although using the generic clearance process was considered, it was determined that the scope and size and purpose of the instrument exceeded the guidelines for the generic process. The proposed instrument is not a customer satisfaction survey, is intended for evaluation purposes and is expected to generate publishable findings. It was also felt that the instrument placed a burden on respondents that exceeded the intent of the process.

6)      Race and Ethnicity questions in the questionnaire must follow OMB guidelines:

We will modify the question to conform to the OMB guideline.