Senior Companion Longitudinal Evaluation OMB Supporting Statement B 3_25_15

Supporting Statement

Senior Corps Longitudinal Evaluation

Part B: Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The respondent universe consists of 178 SCP grantees and 313 FGP grantees who are receiving funds from CNCS during fiscal year 2015. All FGP and SCP grantees will enroll volunteers and caregivers in the study. CNCS proposes to enroll all new caregivers and all new volunteers that enter the program during a six month enrollment window from May through October 2015. SCP grantees are projected to enroll 926 new caregivers receiving respite care; and FGP and SCP grantees are expected to enroll 1,224 new volunteers at baseline.

• Sample Size for the Caregiver Study

The focus of the Caregiver Study is the caregivers who are nested within grantees/projects and/or stations. The unit of analysis is the caregiver. SCP grantees will enroll caregivers receiving respite services. The study design will involve two stages: a plan for selecting SCP grantees and a plan to enroll new caregivers receiving respite services.

Selecting SCP grantees: The universe of grantees consists of 142 SCP grantees/projects that provide respite services will participate.¹ Assuming a grantee/project cooperation rate of 92 percent as in the first evaluation, the projected number of grantees that will enroll caregivers is 142.

Enrolling caregivers: CNCS will enroll all new caregivers that apply for respite services during the enrollment window between May through October 2015.

Projected number of new caregivers: CNCS anticipates enrolling 926 caregivers at baseline. The projected number of new caregivers is based on data collection from the first evaluation. Although the first evaluation collected performance measurement data on caregivers receiving respite services for more than one year, CNCS asked grantees/projects to report the number of caregivers receiving respite services in their program for less than one year. The SCP grantees reported, on average, 13 caregivers that had been receiving respite services for less than one year. Using this average, CNCS projects that during a period of 12 months there will be an average of 1,852 new caregivers across the 142 SCP grantees. The enrollment window is expected to be six months (May through October). Assuming the average number of new caregivers per grantee is 13, during the enrollment window CNCS estimates an enrollment of 926 caregivers. Assuming an 80 percent retention rate at the conclusion of the post-survey, CNCS anticipates the final sample size will be 740 caregivers.

Minimum detectable effect size: The effective sample size will allow analysis for two major subgroups such as, for example, intensity of services received defined by number of hours of respite care received as low to moderate, and high intensity with a 95 percent confidence interval of less than +/-5 percentage points. Table 3 shows the minimum detectable effect size for analysis of responses at baseline and at the conclusion of the post-survey with a power of 0.80 and alpha value of 0.05. A sensitivity analysis shows that a baseline sample size of 926 respondents will detect an effect size of about 0.05 for a target proportion of 0.50. Two subgroup analyses with a total of equal size of 463 respondents per subgroup and a baseline group proportion of 0.50 will detect an effect of about 0.09. A sensitivity analysis shows that an effective sample size of 740 responses at the conclusion of the post-survey will detect an effect size of about 0.05. Two subgroup analyses with a total of 370 respondents per subgroup detect an effect size of about 0.10.

Table 3: Caregiver Study Minimum Detectable Effect Sizes for One- and Two-Sample Proportion Tests^a

	Single Group Analysisb		Two-group Analysesc
Baseline Proportion	Baseline Survey (N=926)	Post-Survey (N=740)		Baseline Survey (N=926)	Post-Survey (N=740)
0.1	0.031	0.035		0.062	0.070
0.3	0.044	0.049		0.087	0.098
0.5	0.046	0.051		0.092	0.102
0.7	0.040	0.045		0.081	0.090
0.9	0.024	0.027		0.049	0.054
a Power=0.80. α=0.05
b One-sample proportion test Wald z test (Ho: p = p0 vs Ha: p ≠ p0).
c Two-sample proportions test Pearson's chi-squared test (Ho: p2 = p1 vs Ha: p2 ≠ p1).

• Sample Size for the Volunteer Study

The focus of the Volunteer Study is the volunteers who are nested within FGP and SCP grantees and stations. The unit of analysis is the volunteer. The study design will involve two stages: a plan for selecting FGP and SCP grantees and a plan to enroll new volunteers.

Selecting FGP and SCP grantees: The universe of grantees consists of 170 SCP grantees and 309 FGP grantees. CNCS desires to understand volunteer satisfaction and retention, self-rated health, symptoms of depression and loneliness among new SCP and FGP volunteers separately. All FGP and SCP grantees will participate. To achieve this goal, CNCS proposes to divide the sample into two strata, one for SCP and one for FGP. Given that FGP enrolls more volunteers than SCP, stratifying the two programs increases the probability of enrolling a sufficient minimum number of participants to permit separate analyses for each group of volunteers as well as having a large enough sample for the matched comparison analysis with the HRS for the SCP volunteers. In the first evaluation CNCS conducted a census of all active SCP and FGP volunteers. The cooperation rate in the previous evaluation was 96 percent for SCP grantees and 99 percent for FGP grantees. CNCS anticipates a similar level of grantee cooperation with the proposed data collection. Thus CNCS anticipates 170 SCP grantees and 309 FGP grantees will enroll volunteers in the study.

Projected number of new volunteers: The projected number of new volunteers during the enrollment window (May through October) is based on data collected from the first evaluation of Senior Corps programs (ESC). In the first ESC, SCP grantees had an average of six volunteers in the program for less than one year at the time of the survey. The FGP grantees had an average of seven volunteers in the program for less than one year at the time of the survey. Using this average, CNCS projects that during a period of 12 months there will be an average of 1,020 new volunteers across SCP grantees, and an average of 2,163 new volunteers across FGP grantees. Based on the first ESC, August through October are peak months individuals will become volunteers. CNCS estimates that 60 percent of new volunteers will begin the program during the six months enrollment window. Thus during the six months enrollment window CNCS projects 612 new SCP volunteers and 1,297 new FGP volunteers.

Enrolling volunteers: CNCS proposes to assign half the sample to FGP programs and half to SCP programs. CNCS proposes to enroll 612 new SCP volunteers and 612 new FGP volunteers during the enrollment window which is expected to be between May through October 2015. This yields a sample size of 1,224 new volunteers at baseline. Assuming an 80 percent retention rate for each follow-up, CNCS anticipates the effective sample size will be 979 volunteers at the first follow-up and 783 volunteers at the second follow-up.

Minimum detectable effect size: The effective sample size will allow analysis for FGP and SCP separately, with a 95 percent confidence interval of less than +/-5 percentage points. Table 4 shows the minimum detectable effect size for analysis of responses at baseline and at the conclusion of the first follow-up and the second follow-up with a power of 0.80 and alpha value of 0.05. A sensitivity analysis shows that a baseline sample size of 1,224 respondents will detect an effect size of about 0.04 for a target proportion of 0.50. Two subgroup analyses with a total of 612 respondents per subgroup and a baseline group proportion of 0.50 will detect an effect of about 0.08. At follow-up a sample size of 979 respondents will detect an effect size of about 0.045 for a target proportion of 0.50. Two subgroup analyses with a total of 979 respondents will detect an effect size of about 0.089 for a baseline group proportion of 0.50.

A sensitivity analysis shows that an effective sample size of 783 responses at the conclusion of the second follow-up will detect an effect size of about 0.05 for a target proportion of 0.50. Two subgroup analyses with a total of 392 respondents per subgroup detect an effect size of about 0.10 for a baseline group proportion of 0.50.

Table 4: Volunteer Study Minimum Detectable Effect Sizes for One- and Two-Sample Proportion Tests^a

	Single Group Analysisb			Two-group Analysesc
Baseline Proportion	Baseline Survey (N=1,224)	1st Follow-up Survey (N=979)	2nd Follow-up Survey (N=783)	Baseline Survey (N=1,224)	1st Follow-up Survey (N=979)	2nd Follow-up Survey (N=783)
0.1	0.027	0.030	0.034	0.053	0.060	0.068
0.3	0.038	0.043	0.048	0.076	0.085	0.095
0.5	0.040	0.045	0.050	0.080	0.089	0.100
0.7	0.035	0.039	0.044	0.071	0.079	0.087
0.9	0.022	0.024	0.026	0.043	0.047	0052
a Power=0.80. α=0.05
b One-sample proportion test Wald z test (Ho: p = p0 vs Ha: p ≠ p0).
c Two-sample proportions test Pearson's chi-squared test (Ho: p2 = p1 vs Ha: p2 ≠ p1).

2. Describe the procedures for the collection of information including: Statistical methodology for stratification and sample selection, Estimation procedure, Degree of accuracy needed for the purpose described in the justification, Unusual problems requiring specialized sampling procedures, and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

CNCS will provide JBS a list of the FGP and SCP participating grantees. The list of grantees will include the name of the grantee or sponsor organization, name and contact information for the grantee’s project director, the state and city where the project is operating. JBS will contact each grantee to provide the following information about their stations including: (a) Each station’s unique identifier  (e.g., name); (b) name and contact information of the staff person who coordinates enrollment of caregivers or volunteers; (c) City, state, zip code and 4-digit zip for each station.

Once a caregiver or volunteer consents to participate, the FGP/SCP project staff will review the informed consent form with each participant. Project staff will remit the signed consent form to JBS. For each enrolled volunteer the project will indicate the anticipated date of orientation. Project staff will provide the survey to the caregiver during the intake for respite services, and for volunteers during orientation prior to the start of service. Attendance at orientation is required of all new volunteers. Therefore, it is reasonable to expect for those volunteers who agreed to participate in the study that this would be an opportune time to complete the survey. Participants can return their completed surveys in a sealed envelope to the project staff that will then mail the surveys to JBS, or participants have the option to mail their completed surveys directly to JBS using a pre-paid postage envelope. JBS anticipates encountering situations involving volunteers or caregivers that may have a sensory or physical impairment. In cases where the respondent has an impairment, such as a hearing or speech impairment, the respondent may personally choose someone to assist them in completing the survey. A trained JBS interviewer will contact volunteers to complete the survey by telephone if a completed survey is not received within a week following their orientation. For each enrolled caregiver, the project will indicate the anticipated date for the in-take process to be completed. A trained JBS interviewer will contact the caregiver to complete the survey by telephone if a completed survey is not received within a week following the completion of intake for respite services.

• Calculating Weights

The proposed design is a census of all eligible new caregivers receiving respite care and new volunteers within a specified enrollment window. This design eliminates the need to calculate sampling weights. However, regardless of design, JBS will calculate a non-response weight adjustment. JBS will compute survey weights that account for differential non-response at each round of data collection. The possible reasons for nonresponse are an eligible respondent who declines to participate in the follow-up; a respondent that becomes ineligible (ie, the person is deceased); and unknown eligibility in that the respondent could not be located as such it is unknown whether the respondent is still eligible (ie., not know if the respondent is deceased). Follow-up survey weights will account for both the propensity to participate in the baseline survey and the follow-up. The follow-up survey weights will be used in analyzing the combined baseline and follow-up data. Analyses using the weights will be representative of the original study population. Since the data collection will be integrated with data from the HRS data to form a matched comparison group, JBS will evaluate whether it is necessary to make weight adjustments for additional missing data arising from the matching process.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

JBS will calculate response rate for each round of data collection, and will conduct non-response bias analysis is the response rate is below 80 percent. In this section CNCS describes the retention efforts to maximize response rates and minimize noon-response bias. CNCS assumes an 80 percent response rate for each follow-up for both studies. The proposed strategy for maximizing response rates includes building points of access for participants to stay in contact with the research, as well as instituting regular, ongoing contact between follow-ups. As previously discussed, the data collection for both volunteers and caregivers will ask for contact information including landline telephone, cell phone, current address, any other seasonal address and e-mail, and similar information on additional contacts, for up to two other relatives and friends. The data collection will also ask respondents for the ways that are best to use when contacting them for the follow-up survey.

An equally important component of retention is to collaborate with the grantees/projects. Findings from the first evaluation of Senior Corps programs showed that response rates were higher when participants had been contacted in advance by someone they knew and trusted (e.g., the FGP or SCP project director or the Senior Companion). Participants who remain associated with Senior Corps will come to trust the project staff and will be more likely to agree to remain in the study if the invitation comes from the project or station staff. For participants who are no longer associated with the program, JBS will engage in greater efforts to contact these participants through the additional contact information obtained in the preceding round of the data collection. The frequency with which this occurs will depend on the attrition rate.

Non-response bias analysis: Even with the most aggressive and comprehensive retention efforts, there is a possibility of attrition in that some respondents that participated in the baseline survey may not be found for the follow-up (i.e., the respondent is deceased), or the respondent, even if found, may decline to participate in the follow-up. All completion rates at follow-up are conditioned on the respondent having been a baseline respondent, since follow-up data collection will be attempted only for the volunteers and caregivers that enrolled and provided informed consent at baseline. If the completion rate is less than 80 percent at the follow-up data collection JBS will conduct a non-response bias analysis to determine whether the follow-up respondents differ significantly from the baseline volunteers and caregivers that enrolled in the study.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

As discussed earlier, JBS conducted a pilot test of the surveys with less than nine respondents per instrument variant in English and Spanish. This pilot test included the following: content review focusing on completeness of instructions and questions for respondents; clarity of instructions and questions for respondents; sequence of questions; relevance of questions for target audience; use of language that target audience understands; use of appropriate and relevant terms; and appropriateness of format. JBS also conducted cognitive interviews during pre-test to determine the extent to which respondents’ understandings of the questions match their intended meaning. Appendix H summarizes the survey pre-test procedures and findings.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

1. The following individuals were consulted on statistical aspects of the design:

JBS International, Inc.: Annie Georges, PhD (650.373.4938); Susan Gabbard, PhD (650.373.4949)

State University of New York at Buffalo: Robert L. Wagmiller, Jr., PhD ([email protected])

Technical Working Group Members: Robert A. Applebaum, PhD ([email protected]); Michele Carlson, PhD ([email protected]); Dawn C. Carr, PhD ([email protected]); Tara Gruenewald, PhD ([email protected]); Joyce Miller, PhD ([email protected]); Suzanne Kunkel, PhD ([email protected]); Jacqui Smith, PhD ([email protected]);

2. Name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency

JBS International, Inc.

555 Airport Boulevard, Suite 400

Burlingame, CA 94010

References

Allison, P. D. (2002). Missing data. Thousand Oaks, CA: Sage Publications, .

Beckler, D., & Ott, K. (2006). Indirect monetary incentives with a complex agricultural establishment survey. ASA Section on Survey on Survey Research Methods. 2741-2748. Retrieved from https://www.amstat.org/sections/srms/proceedings/y2006/Files/JSM2006-000059.pdf

Blossfeld, H., Hamerle, A., & Mayer, K. U. (1989). Event history analyses: Statistical theory and application in the social sciences. Hillsdale, NJ and Hove and London: Lawrence Erlbaum Associates.

Jäckle, A., & Lynn, P.. (2008). Respondent incentives in a multi-mode panel survey: Cumulative effects on nonresponse and bias. Survey Methodology 34 (1): 105–17.

Kolwnikov, A., & Kennedy, C. (2014). Evaluating three approaches to statistically adjust for mode effects. Journal of Survey Statistics and Methodology, 2, 126-158.

Schaffer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.

Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS of American Academy of Political and Social Science. 645, 112-141. DOI: 10.1177/0002716212458082. Retrieved from http://ann.sagepub.com/content/645/1/112.

Singer, J., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.

Zhang, F. (2010). Incentive experiments: NSF experiences. (Working Paper SRS 11-200). Arlington, VA: National Science Foundation, Division of Science Resources Statistics.

1 ^{The projected number of SCP grantees that provide respite care is based on CNCS’s performance measure data for fiscal year 2014 and projections for fiscal year 2015, and the first Evaluation of Senior Corps (ESC) programs. In the ESC, CNCS conducted a census of SCP grantees that were renewing their grants in fiscal year 2013. Of the 39 SCP grantees that participated in the ESC, 34 grantees reported the number of caregivers receiving respite care. Based on this data, CNCS assumes that approximately 87 percent of SCP grantees provide respite care. Of these 34 grantees, 31 grantees administered the caregiver respite survey, a cooperation rate of 92 percent. The number of participating grantees may be lower. At the last evaluation 10 out 61 grantees, 16% received some sort of exemptions – ie., institutional settings such as nursing homes, and grantees that relinquished their grants.}

Page 5 of 5