Evaluation of Senior CorpsRetired Senior Volunteer Program
Part A: Justification
Necessity of the Data Collection
The Corporation for National and Community Service (CNCS) has been shifting toward a system where the Retired Senior Volunteer Program (RSVP) grantees organize their volunteer services by work plans that are linked to performance measurement categories. To better guide RSVP, CNCS needs information on volunteers, their activities related to work plans and their volunteer outcomes. In order to provide to provide this information CNCS seeks approval to conduct an evaluation of RSVP volunteers.
A work plan is a logically related combination of an activity that leads to an output and a desired outcome. An example of a work plan would be to provide companionship (activity) to individuals needing independent living services (output) to increase the individuals’ level of social support (outcome). CNCS has set up three categories of work plans: Primary Focus Areas, Other Focus Areas/Capacity Building, and Community Priorities. CNCS performance measurement requirements set targets for volunteer effort that call for at least 25 percent of volunteers to be working in a Primary Focus Area, and no more than 30 percent working in Community Priorities, and the remainder involved in other (non-primary) Other Focus Areas/Capacity Building. CNCS instructs grantees that ten percent of volunteers should be working on a work plan that results in outcomes.
Through its strategic plan, CNCS has identified and developed a set of performance measures for each focus area as well as for capacity building. In each service area, there are objectives (e.g., school readiness) and a mapping of activity-output combinations to one or more outcome measures. CNCS has agency-wide priority output and outcome measures for the Primary Focus Areas, though other measures are permitted for several objectives. Grantees selecting work plans in community priorities (not in the six Primary Focus Areas or Other Focus Areas/Capacity Building) are asked to report on their success or failure to achieve self-determined targets.
The proposed studywill include information on how volunteer characteristics and outcomes differ by types of work plans as well as information on distribution of volunteers across types of work plans. This information would include the hours, types of services undertaken and how volunteers combine their service activities across the performance measure categories. It would be useful to have information on the characteristics of volunteers engaged in different work plans, and to be able to determine the number of work plans and performance measure categories in which individual volunteers engaged in. Further, as the type and amount of service a given volunteer does is likely to impact the benefits to the volunteer, it would be helpful to have a better understanding of how service activities, hours and engagement relate to volunteer psychosocial health.
Purpose of the Data Collection
The purpose of the data collection is to examine the distribution of volunteers across types of work plans, including hours engaged in specific service activities, and types of service activities. CNCS proposes to administer a 30-minute telephone survey to 1,200 RSVP volunteers via computer-assisted telephone interviewing (CATI).The exploratory and descriptive evaluation will examine whether there are differences in psychosocial health for volunteer of different characteristics and for volunteers whoseparticipation in service activities that fall under different performance measurement categories (Primary Focus Area, Community Priorities, and Other Focus Areas/Capacity Building).Further the study is designed to allow CNCS to conduct additional studies. For example, CNCS could use this datato compare RSVP volunteers with a similar group of adults drawn from the Health and Retirement Survey (HRS) using propensity score matching; another possible future study is to link this data to performance measurement information on work plans supplied by the RSVP grantees at the end of their grant performance.
Use of Electronic Media
The survey instrument for RSVPwill be administered by telephone. Informing respondents about the surveys will also be done by telephone. Trained interviewers will administer the survey.Interviewers will enter the data directly into a computer assisted telephone interviewing (CATI) system. Using a CATI system to administer the survey minimizes both respondent burden and cost to the government. JBS, the independent contractor collecting and analyzing the data, has extensive expertise collaborating with Senior Corps grantees and conducting evaluation studies with CNCS volunteers. Their previous experiences with data collection efforts with the affected population will be valuable in terms of achieving high retention and response rates while minimizing burden and costs to the government.
Identification of Duplication
The instrument to be administered to volunteers includes performance measures developed by CNCS to gauge grantees’ performance in the six focus areas.The RSVP data collection is not a duplication of previous CNCS data collection efforts. The objective of the proposed data collection is to understand how RSVP volunteers are distributed across work plans in the six focus areas, and also to have descriptive information on volunteers psychosocial health. This is CNCS’s first attempt to measure the characteristics of RSVP volunteers, and their individual contributions to RSVP work plans. Therefore, the information to be collected does not exist with another organization or with CNCS.
Impacts on Small Businesses
The proposed data collection will require input from RSVP granteesthat are often small, non-profit entities, but will not impact small businesses. The grantees will be asked to share the list of volunteers linked to CNCS funds with JBS, the external evaluator. JBS will use the list of volunteers to draw the sample.
Consequence to Federal Program or Policy
The Domestic Volunteer Service Act of 1973, as amended (Public Law 93-113), directs CNCS to assess the impact and effectiveness of Senior Corps Programs at least once every three years. In addition, CNCS’s 2011-2015 Strategic Plan outlines the agency’s commitment to building an evidence base that will allow for informed decisions and the allocation of resources. The proposed data collection is one of the activities the agency is undertaking to build its capacity to contribute to this evidence base with data collected through uniform, outcome-based performance measures.The proposed data collection will provide information to CNCS on how to strengthen its national service programs so that adult volunteers participating in the RSVP program achieve stronger social ties and social support, life satisfaction, and self-efficacy. The RSVP program is open to all people age 55 and over. The volunteers decide how, where and how often they want to serve. Volunteers do not receive monetary incentives. The data collection will form the basis for improving and strengthening services the sponsoring organizations provide to their volunteers to meet community challenges.
Special Circumstances
There are no special circumstances that would require the collection of information in any other ways specified.
Publication in the Federal Register and Consultation withOther Agency
An agency’s notice in the Federal Register was completed.The 60-day Federal Notice was published on Tuesday, November 12, 2013 in the Federal Register, volume 78, number 218, pages67342-67343.
CNCS consulted with a Field Working Group (FWG) whose four members have experience with the RSVP program. At a meeting held on December 18, 2013 FWG members were asked to review and comment on the RSVP survey developed for this evaluation. FWG members provided comments on each of the survey questions, the consent process and burden on the sponsoring organization. Their comments were incorporated prior to pre-testing the survey instrument with nine RSVP volunteers.
Gifts or Payments
Respondents will not receive any incentive for participating in the study.
Assurance of Anonymity
Upon contacting respondents, trained interviewers will obtain consentand will provide information on who is conducting the study and why the study is being conducted. Interviewers will describe the background, research activities, risks and benefits, and provide assurance ofanonymity. Respondents will also receive information on whom to contact if they have questions. Respondents will be informed of the voluntary nature of the research and their right to end participation at any time. Respondents will be informed that their responses will be anonymous.
Questions of Sensitive Nature
The survey instruments do not include any questions of a sensitive nature. However, some respondents may feel that questions on income might be personal in nature. During the consent process, respondents will be informed of their right not to respond to any of the survey questions.
Burden Hours
The burden hours for each category of respondents subject to this clearance are described in the table below.
Category of Respondent |
No. of Respondents |
Participation Time (Minutes) per Respondent |
Burden Hours per Respondent |
Burden Hours All Respondents |
Survey of RSVP volunteers |
1,200 |
30 |
0.50 |
600 |
|
|
|
|
|
RSVP Project directors (list of volunteers) |
33 |
60 |
1.00 |
33 |
JBS research staff conducted pre-test of the survey instrument with eight current RSVP volunteers recommended by RSVP Project Directors. The pre-test was followed by a cognitive interview to review survey organization and content, and what problems were encountered. The cognitive interviews were essential to ensuring data quality, allow researchers to determine the extent to which respondents’ understandings of the questions match their intended meaning.
Testers reported in general the instructions were easy to follow, and for the most part the questions were clear. Testers did identify instances of unclear wording and problems with response options for the question asking about when they began volunteering with the RSVP program, and the questions asking for the number of days they volunteered and the number of hours they spent volunteering. The instruments were revised to address the problems testers identified. The revisions include close-ended questions for the interviewer to probe when they encounter a respondent who is unable to recall the specific dates, or number of days or hours.
The time to complete each instrument varied and depended on the characteristics of those completing the instrument. Based on the pretest results, it should take the average respondent 30 minutes to complete the survey.
The estimated annualized cost to respondents for the burden hours for collection of the information is:
RSVP volunteers:The estimated annualized cost for respondents is $5,268 which is based on an estimated hourly wage rate of $8.78 and total burden hours of 600.Hourly wage estimates for elderly adults were based on average Social Security benefit for retired workers. The average monthly Social Security benefit for a retired worker was $1,230 at the beginning of 2012. Assuming a 35-hour work week, the average monthly benefit yields $8.78 per hour. The estimates were obtained from“Average monthly Social Security benefit for a retired worker”, accessed at http://ssa-custhelp.ssa.gov/app/answers/detail/a_id/13/~/average-monthly-social-security-benefit-for-a-retired-worker.
RSVP Project Directorsto compile their volunteer list: The estimated annualized cost for respondents for the grantee survey is $1,130which is based on an estimated hourly wage rate of $34.24 and total burden hours of 33.Wage estimates for nonprofit fields were taken from Wages in the Nonprofit Sector: Management, Professional, and Administrative Support Occupations, accessed at (http://www.bls.gov/opub/mlr/cwc/wages-in-the-nonprofit-sector-management-professional-and-administrative-support-occupations.pdf) and are from 2007.
Estimated Total Annual Cost Burden to Respondents
The collection of this information does not have any capital and start-up cost; and it does not have any operation and maintenance cost.
Estimated Annual Costs to Federal Government
For the survey design, sampling, implementation and analysis of the data collected the estimated annual cost to the Federal government is $140,000.This number is based on the portion of the contract with JBS International that is devoted to the design, sampling, implementation, and analysis of the data collection for the RSVP studywhich will provide descriptive information on the distribution of volunteers across work plans, and the characteristics of volunteers under the performance measures categories.
Explain any Program Changes
This is an application for new collection. There are no program changes.
Dissemination of Information
The data will be analyzed and a final report will be submitted to CNCS. CNCS will include results in their Performance and Accountability Reporting to Congress and in Congressional Budget Justification. In addition, research data files will be prepared and submitted to CNCS for future research.
The proposed RSVP survey instrument is included as Appendix A. The RSVP volunteer survey will collect information on the volunteer’s Length of service and hours served , Type of volunteer activities, volunteer hours per activity, volunteer experience, self efficacy and social support, demographic background, and employment, retirement and income.Demographics and background measures include age, gender, race/ethnicity, education, marriage and veteran statuses. The survey also includes questions on whether the volunteers feel that their capabilities and skills align with their role as a volunteer, and the support they have received from the volunteer station.
The survey instrument also includes information on the volunteers gathered from the sponsoring organization. The data from the sponsoring organization will be collected via telephone conversations with project directors or managers. As previously stated, the project directors will be asked to provide a list of volunteers to JBS which will be used for sampling. In addition, the project directors will be asked to identify the primary work plan for each of their sampled volunteers, the station where the sampled volunteers carry out their volunteer activities. Additional information about the sponsoring organizations such as geographic location of the stations (rural or urban geographic location), and the demographic characteristics of the community where the station is located will be obtained from publicly available sources from the Census Bureau.
JBS will conduct exploratory and descriptive analyses to assess data quality and reliability of the survey items. This is an important initial first step to detect errors, determine if there are any violations of statistical assumptions, determine the relationships among variables, assess the direction and size of relationships between explanatory and outcome variables, and generate hypotheses. At the end of this initial analysis, we expect to have a solid knowledge base about the data characteristics of the variables of interest and the relationships among them. In this initial phase of the analysis, JBS will determine the response rate, screen the data for incomplete or missing data, detect outliers in responses, and combine items into scales.
Since the study will be based on a sample of volunteers, if the response rate is less than 80 percent JBS will conduct a non-response bias analysis to determine if the final sample of respondents reflects the population from which it was obtained, and whether the final sample is significantly different from the population from which it was drawn.Non-response bias for the RSVP evaluation at the grantee and station level will also be investigated using data on grantee size (i.e., number of volunteers) and location obtained as part of the sampling frame. A possible data source for the non-response bias is CNCS’scensus ofthe population of FGP and SCP volunteers in a previous study. These variables that are in common in both surveys can be used for the non-response bias analysis.Table 1 lists the overlapping questions between FGP and SCP surveys and the proposed RSVP volunteer survey which will inform the non-response bias analysis. The FGP and SCP volunteers are low-income seniors and receive a stipend for their volunteer service; the RSVP population does not have an income restriction. To account for these differences analysis that compares the RSVP to the FGP and SCP population would be limited to the subgroup of RVSP volunteers whose grantee and/or volunteer station is in the same state and who are low-income.
Table 1. Overlapping questions between FGP, SCP and RSVP Volunteers Surveys |
|
FGP/SCP Survey |
RSVP Survey |
When did you begin volunteering with the Foster Grandparent Program/The Senior Companion Program? (Month and Year) |
When did you begin volunteering with the RSVP program? (Month and Year) |
Age |
In what month and year were you born? |
Gender |
Are you male or female? |
Do you consider yourself Hispanic or Latino?Do you consider yourself primarily: White/ Caucasian; Black/ African American; American Indian or Alaskan Native; Asian Native Hawaiian, or Pacific Islander; Other. |
Are you of Hispanic or Latino origin? What is your race? White; Black or African American; Asian; Native Hawaiian or other Pacific Islander; American Indian or Alaska Native; Other |
Veteran Status: Active duty or Reserve Component Military family Veteran Family of veteran |
Which of the following apply to you? Active duty or Reserve Component; Military family; Veteran; Family of veteran; None, not a veteran |
What is the highest grade of school or year of college you completed? No formal education; Grades 1- 11; Grade 12 (High School Diploma or GED); Some College; Associate’s Degree; Bachelor’s Degree/ College Graduate; Some graduate school; Completed a graduate/professional degree; Other |
What is the highest grade or year of school or year of college you completed?No formal education; Grades 1-11; Grade 12 (High School Diploma or GED); Some College Associate’s Degree; Bachelor’s Degree/College Graduate; Some graduate school; Completed a graduate/professional degree; Other |
Are you currently married, with a partner as if married, separated, divorced, widowed, ornever married? |
Are you currently married, with a partner as if married, separated, divorced, widowed, or never married? |
How many hours did you spend per week in the past four weeks doing volunteer work forreligious, educational, health- related or other charitable organizations, including yourvolunteer time with FGP, SCP, and/or RSVP? 0 hours per week 1- 10 hours per week11- 20 hours per week greater than 20 hours per week |
Last month, on approximately how many days did you volunteer with the Retired and Senior Volunteer Program? Last month, approximately how many hours would you say you volunteered with the Retired and Senior Volunteer Program? Do you volunteer with other organizations other than RSVP?In the past month, how many hours did you volunteer with these other organizations? |
How much of the time do you feel that there are people you feel close to?Often Some of the time Hardly ever or never |
How much of the time do you feel that there are people you feel close to? Would you say often, some of the time, or hardly ever or never? |
How much of the time do you feel that you lack companionship?Often Some of the time Hardly ever or never |
How much of the time do you feel that you lack companionship? Would you say often, some of the time, or hardly ever or never? |
I can do the things that I want to do: Strongly disagree; Somewhat disagree; Slightly disagree; Slightly agree; Somewhat agree; Strongly agree |
Do you agree or disagree with this statement: I can do the things that I want to do? Strongly disagree; Somewhat disagree; Slightly disagree; Slightly agree; Somewhat agree; Strongly agree |
Internal consistency will be examined using Cronbach’s Alpha. Cronbach’s alpha is a summary measure, and provides a sense of the overall consistency across a series of items.
JBS will identify the extent of missing values for the items during the exploratory analysis phase. The exploratory analysis will provide information about the source of missing values, and this will allow JBS to select, if needed, the most appropriate imputation method. It is important to understand whether missing values are related to observed characteristics of respondents.Once we understand the sources and extent of missing data, JBSwill review, select, and apply the most efficient method, based on careful consideration of the dataset and the type of missing data.
JBS will conduct descriptive analysis and multivariate analysis. Table 2 outlines the research questions, the descriptive analysis and statistical tests.
Table 2. Research Questions, Proposed Analysis and Statistical Tests |
||
Research Questions |
Type of Analysis/Statistical Test |
Survey Questions (see Appendix A) |
|
Frequencies, cross tabulations Chi-square test |
Q5, Q7, Q8, Q9, Q10, Q11 |
|
Frequencies, Mean, Median, Range One-way ANOVA, T-test, chi-square test |
Q12 through Q29; Q33 through Q34 |
|
Frequencies, cross tabulations, Mean, Median, Range One-way ANOVA, T-test |
Q12 through Q29; Q10, Q11, Q30 through Q32 |
|
Frequencies Wilcoxon-Mann-Whitney test, chi-square test |
Q12 through Q29; q10, Q11, Q30 |
|
Mean, Median, Range One-way ANOVA, T-test |
Q8, Q9, Q10, Q11, Q12 through Q29; Q32 |
|
Frequencies, Mean, Median, Range Exploratory factory analysis, confirmatory factory analysis, latent class analysis Wilcoxon-Mann-Whitney test, chi-square test, One-way ANOVA, T-test RMSEA, CFI, TLI, BIC, and Log-likelihood |
Q35through Q40; Q41 through Q47; Q48 through Q56 |
|
Cross tabulations, correlation, covariance Logistic regression, ordered logistic regression, multiple regression, ANOVA Chi-square test, Pearson correlation coefficients, Spearman correlation coefficients |
Q7, Q8, Q9, Q35through Q40, Q41 through Q47; Q12 through Q29; Q48 through Q56 |
|
Cross tabulations, correlation, covariance Logistic regression, ordered logistic regression, multiple regression, ANOVA Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test |
Q10, Q11, Q12 through Q29; Q35 through Q40 |
|
Cross tabulations, correlation, covariance Logistic regression, ordered logistic regression, multiple regression, ANOVA Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test |
Q12 through Q29, Q35 through Q40 |
|
Cross tabulations, correlation, covariance Logistic regression, ordered logistic regression, multiple regression, ANOVA Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test |
Q12 through Q29, Q35 through Q40 |
|
Cross tabulations, correlation, covariance Logistic regression, ordered logistic regression, multiple regression, ANOVA Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test |
Q35 through Q40, Q41 through Q56 |
The outcomes of interest for regression models are measured on a scale of 1 to 4, ranging from strongly disagree (1) to strongly agree (4).We expect the RSVP data will have a nested structure with volunteers nested within stations or sites, and stations nested within project. JBS will calculate robust standard errors to account for the complex sampling and the nested structure of the data. JBS will use the appropriate statistical models such as logistic regression where the outcome measures are either categorical or dichotomous, or an ordered logistic model which is an appropriate model for discrete event data being collected on the outcome measures.
CNCS desires that the design of this study allow for two possible additional analytic studies in the future. The first potential future study would be to compare RSVP volunteers with a similar group of adults drawn from the Health and Retirement Survey (HRS) using propensity score matching. The second potential future study would involve linking data from the volunteer survey to performance measurement information on workplans supplied by the grantees at the end of their grant performance. As currently designed, the study design would permit CNCS to conduct these additional future studies.
The RSVP survey includes overlapping questions with the Health and Retirement Study (HRS) that would permit, in a future study, the use of the data collected during this study to form a matched comparison group for other evaluation activities. For example, CNCS could form a matched comparison group using the HRS data where RSVP volunteers are the treatment group matched to a comparison group of HRS respondents with similar characteristics who were not volunteers. Table 3belowshows the list of common questions in the RSVP and HRS datasets.
Table 3. Common Questions in the RSVP and HRS Surveys
RSVP Survey Instrument |
HRS Data |
Items for propensity score model |
Items that are outcomes to compare between the two groups |
Questions –35-44, 46-55 |
Common questions |
Demographic and background questions 41-44, 46-47; employment, retirement and income questions: 48-55 |
Self-efficacy questions 35-36; social loneliness question 37-38; emotional loneliness 39-40 |
Questions 7-34, 43, 56-58 |
Not in the HRS data |
|
|
Table 3 below illustrates the analytic steps that an analysis might use to complete the propensity score matching (PSM) with the data.
Table 4.Potential Analytic Steps for a Propensity Score Matching Study with the HRS Data
Recommended Steps in Future Analysis with the RSVP DAta |
How and what could be done |
Data preparation and harmonization |
|
Weight adjustments |
|
Stratification and segmentation |
|
Estimate the propensity score |
|
Matching algorithm |
|
Test properties of the statistical match |
|
JBS will produce a public access and agency specific data file for this data collection. The data files will be clean, well-organized, and adhere to privacy and security standards. JBS will create a de-identified and anonymous Public Use Data File. JBS will follow guidance on government-wide data sharing in all Federal statutes pertaining to data collection, privacy, and human subject research and take adequate steps to ensure that no individual is identifiable. For the Public Use Data File, JBS will produce a sampling and methodology report that will include a copy of the instrument, description of the methodology, and guidance on using post-sampling weights.
Activity/Task |
End Date |
Sample selection |
6/1/14 |
Data collection |
8/1/14 |
Conduct data analysis |
9/1/14 |
Final Report |
9/15/14 |
Submit data files to CNCS for future research |
9/26/14 |
Display of Expiration Date for OMB Approval
The expiration date for OMB approval of the information collection will be displayed on the first page of the instruments.
Explanation of Exceptions in Item 19, "Certification for Paperwork Reduction Act Submissions"
There are no exceptions claimed in Item 19, "Certification for Paperwork Reduction Act Submissions"of OMB Form83-I.
Page
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | ageorges |
File Modified | 0000-00-00 |
File Created | 2021-01-28 |