RSVP SS Part A 5.29.14

RSVP SS Part A 5.29.14.docx

RSVP Volunteer Program Survey

OMB: 3045-0157

Document [docx]
Download: docx | pdf

Evaluation of Senior CorpsRetired Senior Volunteer Program


Part A: Justification


  1. Necessity of the Data Collection


The Corporation for National and Community Service (CNCS) has been shifting toward a system where the Retired Senior Volunteer Program (RSVP) grantees organize their volunteer services by work plans that are linked to performance measurement categories. To better guide RSVP, CNCS needs information on volunteers, their activities related to work plans and their volunteer outcomes. In order to provide to provide this information CNCS seeks approval to conduct an evaluation of RSVP volunteers.


A work plan is a logically related combination of an activity that leads to an output and a desired outcome. An example of a work plan would be to provide companionship (activity) to individuals needing independent living services (output) to increase the individuals’ level of social support (outcome). CNCS has set up three categories of work plans: Primary Focus Areas, Other Focus Areas/Capacity Building, and Community Priorities. CNCS performance measurement requirements set targets for volunteer effort that call for at least 25 percent of volunteers to be working in a Primary Focus Area, and no more than 30 percent working in Community Priorities, and the remainder involved in other (non-primary) Other Focus Areas/Capacity Building. CNCS instructs grantees that ten percent of volunteers should be working on a work plan that results in outcomes.


Through its strategic plan, CNCS has identified and developed a set of performance measures for each focus area as well as for capacity building. In each service area, there are objectives (e.g., school readiness) and a mapping of activity-output combinations to one or more outcome measures. CNCS has agency-wide priority output and outcome measures for the Primary Focus Areas, though other measures are permitted for several objectives. Grantees selecting work plans in community priorities (not in the six Primary Focus Areas or Other Focus Areas/Capacity Building) are asked to report on their success or failure to achieve self-determined targets.


The proposed studywill include information on how volunteer characteristics and outcomes differ by types of work plans as well as information on distribution of volunteers across types of work plans. This information would include the hours, types of services undertaken and how volunteers combine their service activities across the performance measure categories. It would be useful to have information on the characteristics of volunteers engaged in different work plans, and to be able to determine the number of work plans and performance measure categories in which individual volunteers engaged in. Further, as the type and amount of service a given volunteer does is likely to impact the benefits to the volunteer, it would be helpful to have a better understanding of how service activities, hours and engagement relate to volunteer psychosocial health.




  1. Purpose of the Data Collection


The purpose of the data collection is to examine the distribution of volunteers across types of work plans, including hours engaged in specific service activities, and types of service activities. CNCS proposes to administer a 30-minute telephone survey to 1,200 RSVP volunteers via computer-assisted telephone interviewing (CATI).The exploratory and descriptive evaluation will examine whether there are differences in psychosocial health for volunteer of different characteristics and for volunteers whoseparticipation in service activities that fall under different performance measurement categories (Primary Focus Area, Community Priorities, and Other Focus Areas/Capacity Building).Further the study is designed to allow CNCS to conduct additional studies. For example, CNCS could use this datato compare RSVP volunteers with a similar group of adults drawn from the Health and Retirement Survey (HRS) using propensity score matching; another possible future study is to link this data to performance measurement information on work plans supplied by the RSVP grantees at the end of their grant performance.


  1. Use of Electronic Media


The survey instrument for RSVPwill be administered by telephone. Informing respondents about the surveys will also be done by telephone. Trained interviewers will administer the survey.Interviewers will enter the data directly into a computer assisted telephone interviewing (CATI) system. Using a CATI system to administer the survey minimizes both respondent burden and cost to the government. JBS, the independent contractor collecting and analyzing the data, has extensive expertise collaborating with Senior Corps grantees and conducting evaluation studies with CNCS volunteers. Their previous experiences with data collection efforts with the affected population will be valuable in terms of achieving high retention and response rates while minimizing burden and costs to the government.


  1. Identification of Duplication


The instrument to be administered to volunteers includes performance measures developed by CNCS to gauge grantees’ performance in the six focus areas.The RSVP data collection is not a duplication of previous CNCS data collection efforts. The objective of the proposed data collection is to understand how RSVP volunteers are distributed across work plans in the six focus areas, and also to have descriptive information on volunteers psychosocial health. This is CNCS’s first attempt to measure the characteristics of RSVP volunteers, and their individual contributions to RSVP work plans. Therefore, the information to be collected does not exist with another organization or with CNCS.


  1. Impacts on Small Businesses


The proposed data collection will require input from RSVP granteesthat are often small, non-profit entities, but will not impact small businesses. The grantees will be asked to share the list of volunteers linked to CNCS funds with JBS, the external evaluator. JBS will use the list of volunteers to draw the sample.


  1. Consequence to Federal Program or Policy


The Domestic Volunteer Service Act of 1973, as amended (Public Law 93-113), directs CNCS to assess the impact and effectiveness of Senior Corps Programs at least once every three years. In addition, CNCS’s 2011-2015 Strategic Plan outlines the agency’s commitment to building an evidence base that will allow for informed decisions and the allocation of resources. The proposed data collection is one of the activities the agency is undertaking to build its capacity to contribute to this evidence base with data collected through uniform, outcome-based performance measures.The proposed data collection will provide information to CNCS on how to strengthen its national service programs so that adult volunteers participating in the RSVP program achieve stronger social ties and social support, life satisfaction, and self-efficacy. The RSVP program is open to all people age 55 and over. The volunteers decide how, where and how often they want to serve. Volunteers do not receive monetary incentives. The data collection will form the basis for improving and strengthening services the sponsoring organizations provide to their volunteers to meet community challenges.


  1. Special Circumstances


There are no special circumstances that would require the collection of information in any other ways specified.


  1. Publication in the Federal Register and Consultation withOther Agency


  • An agency’s notice in the Federal Register was completed.The 60-day Federal Notice was published on Tuesday, November 12, 2013 in the Federal Register, volume 78, number 218, pages67342-67343.

  • CNCS consulted with a Field Working Group (FWG) whose four members have experience with the RSVP program. At a meeting held on December 18, 2013 FWG members were asked to review and comment on the RSVP survey developed for this evaluation. FWG members provided comments on each of the survey questions, the consent process and burden on the sponsoring organization. Their comments were incorporated prior to pre-testing the survey instrument with nine RSVP volunteers.


  1. Gifts or Payments


Respondents will not receive any incentive for participating in the study.


  1. Assurance of Anonymity


Upon contacting respondents, trained interviewers will obtain consentand will provide information on who is conducting the study and why the study is being conducted. Interviewers will describe the background, research activities, risks and benefits, and provide assurance ofanonymity. Respondents will also receive information on whom to contact if they have questions. Respondents will be informed of the voluntary nature of the research and their right to end participation at any time. Respondents will be informed that their responses will be anonymous.


  1. Questions of Sensitive Nature


The survey instruments do not include any questions of a sensitive nature. However, some respondents may feel that questions on income might be personal in nature. During the consent process, respondents will be informed of their right not to respond to any of the survey questions.


  1. Burden Hours


The burden hours for each category of respondents subject to this clearance are described in the table below.


Category of Respondent

No. of Respondents

Participation Time (Minutes) per Respondent

Burden Hours per Respondent

Burden Hours All Respondents

Survey of RSVP volunteers

1,200

30

0.50

600






RSVP Project directors (list of volunteers)

33

60

1.00

33


JBS research staff conducted pre-test of the survey instrument with eight current RSVP volunteers recommended by RSVP Project Directors. The pre-test was followed by a cognitive interview to review survey organization and content, and what problems were encountered. The cognitive interviews were essential to ensuring data quality, allow researchers to determine the extent to which respondents’ understandings of the questions match their intended meaning.


Testers reported in general the instructions were easy to follow, and for the most part the questions were clear. Testers did identify instances of unclear wording and problems with response options for the question asking about when they began volunteering with the RSVP program, and the questions asking for the number of days they volunteered and the number of hours they spent volunteering. The instruments were revised to address the problems testers identified. The revisions include close-ended questions for the interviewer to probe when they encounter a respondent who is unable to recall the specific dates, or number of days or hours.


The time to complete each instrument varied and depended on the characteristics of those completing the instrument. Based on the pretest results, it should take the average respondent 30 minutes to complete the survey.

The estimated annualized cost to respondents for the burden hours for collection of the information is:


  1. Estimated Total Annual Cost Burden to Respondents


The collection of this information does not have any capital and start-up cost; and it does not have any operation and maintenance cost.


  1. Estimated Annual Costs to Federal Government


For the survey design, sampling, implementation and analysis of the data collected the estimated annual cost to the Federal government is $140,000.This number is based on the portion of the contract with JBS International that is devoted to the design, sampling, implementation, and analysis of the data collection for the RSVP studywhich will provide descriptive information on the distribution of volunteers across work plans, and the characteristics of volunteers under the performance measures categories.


  1. Explain any Program Changes


This is an application for new collection. There are no program changes.


  1. Dissemination of Information


The data will be analyzed and a final report will be submitted to CNCS. CNCS will include results in their Performance and Accountability Reporting to Congress and in Congressional Budget Justification. In addition, research data files will be prepared and submitted to CNCS for future research.


  • Survey Instrument

The proposed RSVP survey instrument is included as Appendix A. The RSVP volunteer survey will collect information on the volunteer’s Length of service and hours served , Type of volunteer activities, volunteer hours per activity, volunteer experience, self efficacy and social support, demographic background, and employment, retirement and income.Demographics and background measures include age, gender, race/ethnicity, education, marriage and veteran statuses. The survey also includes questions on whether the volunteers feel that their capabilities and skills align with their role as a volunteer, and the support they have received from the volunteer station.


The survey instrument also includes information on the volunteers gathered from the sponsoring organization. The data from the sponsoring organization will be collected via telephone conversations with project directors or managers. As previously stated, the project directors will be asked to provide a list of volunteers to JBS which will be used for sampling. In addition, the project directors will be asked to identify the primary work plan for each of their sampled volunteers, the station where the sampled volunteers carry out their volunteer activities. Additional information about the sponsoring organizations such as geographic location of the stations (rural or urban geographic location), and the demographic characteristics of the community where the station is located will be obtained from publicly available sources from the Census Bureau.


  • Non Response Bias and Reliability

JBS will conduct exploratory and descriptive analyses to assess data quality and reliability of the survey items. This is an important initial first step to detect errors, determine if there are any violations of statistical assumptions, determine the relationships among variables, assess the direction and size of relationships between explanatory and outcome variables, and generate hypotheses. At the end of this initial analysis, we expect to have a solid knowledge base about the data characteristics of the variables of interest and the relationships among them. In this initial phase of the analysis, JBS will determine the response rate, screen the data for incomplete or missing data, detect outliers in responses, and combine items into scales.


Since the study will be based on a sample of volunteers, if the response rate is less than 80 percent JBS will conduct a non-response bias analysis to determine if the final sample of respondents reflects the population from which it was obtained, and whether the final sample is significantly different from the population from which it was drawn.Non-response bias for the RSVP evaluation at the grantee and station level will also be investigated using data on grantee size (i.e., number of volunteers) and location obtained as part of the sampling frame. A possible data source for the non-response bias is CNCS’scensus ofthe population of FGP and SCP volunteers in a previous study. These variables that are in common in both surveys can be used for the non-response bias analysis.Table 1 lists the overlapping questions between FGP and SCP surveys and the proposed RSVP volunteer survey which will inform the non-response bias analysis. The FGP and SCP volunteers are low-income seniors and receive a stipend for their volunteer service; the RSVP population does not have an income restriction. To account for these differences analysis that compares the RSVP to the FGP and SCP population would be limited to the subgroup of RVSP volunteers whose grantee and/or volunteer station is in the same state and who are low-income.


Table 1. Overlapping questions between FGP, SCP and RSVP Volunteers Surveys

FGP/SCP Survey

RSVP Survey

When did you begin volunteering with the Foster Grandparent Program/The Senior

Companion Program? (Month and Year)

When did you begin volunteering with the RSVP program? (Month and Year)

Age

In what month and year were you born?

Gender

Are you male or female?

Do you consider yourself Hispanic or Latino?Do you consider yourself primarily: White/ Caucasian; Black/ African American; American Indian or Alaskan Native; Asian Native Hawaiian, or Pacific Islander; Other.

Are you of Hispanic or Latino origin?

What is your race? White; Black or African American; Asian; Native Hawaiian or other Pacific Islander; American Indian or Alaska Native; Other

Veteran Status:

Active duty or Reserve Component Military family Veteran Family of veteran

Which of the following apply to you? Active duty or Reserve Component; Military family; Veteran; Family of veteran; None, not a veteran

What is the highest grade of school or year of college you completed? No formal education; Grades 1- 11; Grade 12 (High School Diploma or GED); Some College; Associate’s Degree; Bachelor’s Degree/ College Graduate; Some graduate school; Completed a graduate/professional degree; Other

What is the highest grade or year of school or year of college you completed?No formal education; Grades 1-11; Grade 12 (High School Diploma or GED); Some College

Associate’s Degree; Bachelor’s Degree/College Graduate; Some graduate school; Completed a graduate/professional degree; Other

Are you currently married, with a partner as if married, separated, divorced, widowed, ornever married?

Are you currently married, with a partner as if married, separated, divorced, widowed, or never married?

How many hours did you spend per week in the past four weeks doing volunteer work forreligious, educational, health- related or other charitable organizations, including yourvolunteer time with FGP, SCP, and/or RSVP?

0 hours per week 1- 10 hours per week11- 20 hours per week greater than 20 hours per week

Last month, on approximately how many days did you volunteer with the Retired and Senior Volunteer Program?

Last month, approximately how many hours would you say you volunteered with the Retired and Senior Volunteer Program?

Do you volunteer with other organizations other than RSVP?In the past month, how many hours did you volunteer with these other organizations?

How much of the time do you feel that there are people you feel close to?Often Some of the time Hardly ever or never

How much of the time do you feel that there are people you feel close to? Would you say often, some of the time, or hardly ever or never?

How much of the time do you feel that you lack companionship?Often Some of the time Hardly ever or never

How much of the time do you feel that you lack companionship? Would you say often, some of the time, or hardly ever or never?

I can do the things that I want to do:

Strongly disagree; Somewhat disagree; Slightly disagree; Slightly agree; Somewhat agree; Strongly agree

Do you agree or disagree with this statement: I can do the things that I want to do?

Strongly disagree; Somewhat disagree; Slightly disagree; Slightly agree; Somewhat agree; Strongly agree


Internal consistency will be examined using Cronbach’s Alpha. Cronbach’s alpha is a summary measure, and provides a sense of the overall consistency across a series of items.


  • Data Imputation

JBS will identify the extent of missing values for the items during the exploratory analysis phase. The exploratory analysis will provide information about the source of missing values, and this will allow JBS to select, if needed, the most appropriate imputation method. It is important to understand whether missing values are related to observed characteristics of respondents.Once we understand the sources and extent of missing data, JBSwill review, select, and apply the most efficient method, based on careful consideration of the dataset and the type of missing data.


  • Research Questions,Proposed Analysis and Statistical Tests

JBS will conduct descriptive analysis and multivariate analysis. Table 2 outlines the research questions, the descriptive analysis and statistical tests.


Table 2. Research Questions, Proposed Analysis and Statistical Tests

Research Questions

Type of Analysis/Statistical Test

Survey Questions (see Appendix A)

  1. What is the distribution of volunteers across the unique work plans(activities-outputs-outcomes) within the RSVP performance measure categories?

Frequencies, cross tabulations

Chi-square test

Q5, Q7, Q8, Q9, Q10, Q11

  1. What proportion of volunteers’ time is dedicated to various service activities within the performance measure categories?

Frequencies, Mean, Median, Range

One-way ANOVA, T-test, chi-square test

Q12 through Q29; Q33 through Q34

  1. How are volunteer efforts concentrated within each performance measure category?

Frequencies, cross tabulations, Mean, Median, Range

One-way ANOVA, T-test

Q12 through Q29; Q10, Q11, Q30 through Q32

  1. What is the average number of work plans per volunteer?

Frequencies

Wilcoxon-Mann-Whitney test, chi-square test

Q12 through Q29; q10, Q11, Q30

  1. On average, how many hours does an individual volunteer contribute to service activities in a particular work plan?

Mean, Median, Range

One-way ANOVA, T-test

Q8, Q9, Q10, Q11, Q12 through Q29; Q32

  1. What are the demographic characteristics and psychosocial health status of RSVP volunteers?

Frequencies, Mean, Median, Range

Exploratory factory analysis, confirmatory factory analysis, latent class analysis

Wilcoxon-Mann-Whitney test, chi-square test, One-way ANOVA, T-test

RMSEA, CFI, TLI, BIC, and Log-likelihood

Q35through Q40; Q41 through Q47; Q48 through Q56

  1. Is there a relationship between volunteer characteristics (e.g., years of service, hours volunteering, age) and psychosocial health status?

Cross tabulations, correlation, covariance

Logistic regression, ordered logistic regression, multiple regression, ANOVA

Chi-square test, Pearson correlation coefficients, Spearman correlation coefficients

Q7, Q8, Q9, Q35through Q40, Q41 through Q47; Q12 through Q29; Q48 through Q56

  1. Is there an association between volunteer service activities (e.g., type of work plan, service activity) and psychosocial health status?

Cross tabulations, correlation, covariance

Logistic regression, ordered logistic regression, multiple regression, ANOVA

Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test

Q10, Q11, Q12 through Q29; Q35 through Q40

  1. Is there is an association between volunteer service activities in certain performance measure categories and the psychosocial health of volunteers?

Cross tabulations, correlation, covariance

Logistic regression, ordered logistic regression, multiple regression, ANOVA

Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test

Q12 through Q29, Q35 through Q40

  1. Is the number of hours engaging in volunteer service activities on a particular performance measure category associated with the psychosocial health of volunteers?

Cross tabulations, correlation, covariance

Logistic regression, ordered logistic regression, multiple regression, ANOVA

Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test

Q12 through Q29, Q35 through Q40

  1. Are volunteer characteristics and service activities associated with the psychosocial health of volunteers?

Cross tabulations, correlation, covariance

Logistic regression, ordered logistic regression, multiple regression, ANOVA

Pearson correlation coefficients, Spearman correlation coefficients, Chi-square test, Wald chi-square, Likelihood ratio test, R-square, adjusted R-square, T-test

Q35 through Q40, Q41 through Q56


The outcomes of interest for regression models are measured on a scale of 1 to 4, ranging from strongly disagree (1) to strongly agree (4).We expect the RSVP data will have a nested structure with volunteers nested within stations or sites, and stations nested within project. JBS will calculate robust standard errors to account for the complex sampling and the nested structure of the data. JBS will use the appropriate statistical models such as logistic regression where the outcome measures are either categorical or dichotomous, or an ordered logistic model which is an appropriate model for discrete event data being collected on the outcome measures.


  • Feasibility of Future Studies

CNCS desires that the design of this study allow for two possible additional analytic studies in the future. The first potential future study would be to compare RSVP volunteers with a similar group of adults drawn from the Health and Retirement Survey (HRS) using propensity score matching. The second potential future study would involve linking data from the volunteer survey to performance measurement information on workplans supplied by the grantees at the end of their grant performance. As currently designed, the study design would permit CNCS to conduct these additional future studies.


The RSVP survey includes overlapping questions with the Health and Retirement Study (HRS) that would permit, in a future study, the use of the data collected during this study to form a matched comparison group for other evaluation activities. For example, CNCS could form a matched comparison group using the HRS data where RSVP volunteers are the treatment group matched to a comparison group of HRS respondents with similar characteristics who were not volunteers. Table 3belowshows the list of common questions in the RSVP and HRS datasets.


Table 3. Common Questions in the RSVP and HRS Surveys


RSVP Survey Instrument

HRS Data

Items for propensity score model

Items that are outcomes to compare between the two groups

Questions –35-44, 46-55

Common questions

Demographic and background questions 41-44, 46-47; employment, retirement and income questions: 48-55

Self-efficacy questions 35-36; social loneliness question 37-38; emotional loneliness 39-40

Questions 7-34, 43, 56-58

Not in the HRS data





Table 3 below illustrates the analytic steps that an analysis might use to complete the propensity score matching (PSM) with the data.



Table 4.Potential Analytic Steps for a Propensity Score Matching Study with the HRS Data


Recommended Steps in Future Analysis with the RSVP DAta

How and what could be done

Data preparation and harmonization

  • Common variables in the RSVP and HRS datasets are aligned in terms of their definition and measurements, and their distributions are aligned as closely as possible to each other so as to minimize differences among common variables across datasets.

  • The analyst wouldhandle missing data imputation similarly for both datasets.

  • The HRS data will be appended to the RSVP data to create a combined data file. A variable to indicate whether a case belongs to the RSVP (i.e., the treatment group) or a control case from the HRS data.

Weight adjustments

  • Although both datasets have common variables on which to match them, the data are produced for different purposes, are based on different designs, and the data may not have been collected at the same time. The RSVP data will be from a self-weighting PPS sample. In this case, the analyst would use unweighted HRS data when estimating the impact of the treatment. The analyst would not be able to make inference to the whole population of RSVP volunteers.

Stratification and segmentation

  • The analyst would assess whether stratifying the combined RSVP-HRS data into discrete cells would be increase efficiency. In such a case, the propensity scores would be estimated separately within cells, and matching of treatment to control cases would be conducted within each cell. The analyst would have to consider whether the segmentation approach could achieve a tighter fit in matching treatment cases to control cases.

  • A reason for exploring this option is that the HRS data oversampled African American, Hispanic, and Florida respondents. Thus, it is important to explore whether matches of certain types of cases should be required and/or avoided.

Estimate the propensity score

  • Once the matched comparison is formed the analyst would estimate a logistic regression model on the combined RSVP-HRS dataset where the dependent variable is whether a respondent is a treatment case (e.g.,RSVP volunteer) and the common variables in the combined data are independent variables (excluding the variable for which we want to detect impact of volunteering).

  • For example, if the analyst is trying to assess the impact of volunteering on self-efficacy, then when estimating the propensity score, the self-efficacy measures will not be included as independent variables to estimate the propensity score. If data are segmented, as discussed in the prior step, the analystwould estimate a cell-specific propensity score in order to achieve a tighter fit in matching.

Matching algorithm

  • Several possible matching algorithms could be employed to match the treatment cases to the control cases. For example, the nearest neighbor matching procedure, caliper matching, Mahalanobis metric matching. Giving the expected large sample for the RSVP data, it would be possible to use sampling without replacement meaning each comparison group respondent will be included as a matched case only once

Test properties of the statistical match

  • Once the above steps are complete, the analyst would check the key assumptions and verify that the model specification to create the comparison group is appropriate and that the results do not exhibit any bias.

  • For example, the analyst would check that the PSM balances the characteristics between the treatment and comparison group. If differences between treatment and control group persist after the match, the model specification to improve the balance between the groups should be refined, or, potentially, consider a different matching approach.


  • Data File for Future Research

JBS will produce a public access and agency specific data file for this data collection. The data files will be clean, well-organized, and adhere to privacy and security standards. JBS will create a de-identified and anonymous Public Use Data File. JBS will follow guidance on government-wide data sharing in all Federal statutes pertaining to data collection, privacy, and human subject research and take adequate steps to ensure that no individual is identifiable. For the Public Use Data File, JBS will produce a sampling and methodology report that will include a copy of the instrument, description of the methodology, and guidance on using post-sampling weights.


  • Timeline (End dates)

Activity/Task

End Date

Sample selection

6/1/14

Data collection

8/1/14

Conduct data analysis

9/1/14

Final Report

9/15/14

Submit data files to CNCS for future research

9/26/14


  1. Display of Expiration Date for OMB Approval


The expiration date for OMB approval of the information collection will be displayed on the first page of the instruments.


  1. Explanation of Exceptions in Item 19, "Certification for Paperwork Reduction Act Submissions"


There are no exceptions claimed in Item 19, "Certification for Paperwork Reduction Act Submissions"of OMB Form83-I.



Page 11 of 11


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
Authorageorges
File Modified0000-00-00
File Created2021-01-28

© 2024 OMB.report | Privacy Policy