Senior Companion SSA9.24.13

Senior Companion SSA9.24.13.docx

Senior Corps Independent Living Evaluation Impact Study

OMB: 3045-0154

Document [docx]
Download: docx | pdf

Senior Corps Independent Living Impact Evaluation Study


Part A: Justification


  1. Necessity of the Data Collection


The Corporation for National and Community Service (CNCS) seeks approval to conduct a study to assess the impact of Senior Companion Program (SCP) on clients’ self-efficacy, life satisfaction, and perceived social ties and social supports. This study will include a comparison group from the Health and Retirement Study (HRS) dataset developed and collected by the University of Michigan. The study will contribute to CNCS’s ability to report on progress towards two of the goals of its Strategic Plan. The first goal is to increase the impact of national service on community needs in communities served by CNCS-supported programs. The second goal is to strengthen national service so that participants engaged in CNCS-supported programs consistently find satisfaction, meaning, and opportunity.


  1. Purpose of the Data Collection


The data collection will be used for three primary purposes. First, the data will be used to assess the impact of Senior Companion services on established clients’ self-efficacy, life satisfaction, and perceived social ties and social supports. Established clients are individuals who have been receiving services for a year or more. Second, the data will be used to determine the feasibility of conducting future studies with this population. Third, the data will be used to inform the feasibility of using the same instrument to conduct an evaluation of CNCS’s RSVP Program.


The data collection is divided into two parts. The first part is the main study and entails the administration of a 20-minute telephone survey to 1,980 established SCP clients via computer-assisted telephone interviewing (CATI) to measure self-efficacy, social ties, social support, well being and social demographic characteristics. The main study is designed to provide preliminary estimates of the impact of Senior Companion services, and to test the feasibility of future studies with this population. The data from the main study will be matched to similar respondents from the HRS dataset to form a comparison group using propensity score matching. The treatment group will be SCP clients matched with a comparison group of respondents from the HRS data with similar characteristics who did not receive Senior Companion services. Senior Companion grantees will also be asked to provide a list of the clients they serve that will be used as the sampling universe. This main study is large enough to provide Senior Corps with a first look at outcomes from its SCP services involving supporting independent living seniors. CNCS recognizes that the quasi-experimental for the main study is subject to limitations and threats to internal validity that will be noted in all reports of the results. At the same time, this study provides a test of the survey instrument that is being considered for a future longitudinal study.


The second part of the data collection is exploratory. It is a pilot and feasibility study designed as a pre/post study with new recipients of Senior Companion services. For the pre/post study, CNCS anticipates conducting two follow-up data collections to look at the feasibility of conducting multiple rounds of data collection among this population, and to gather preliminary exploratory data about whether and when changes in health benefits following entry into the Senior Companion program might occur. New recipients are adults who at baseline are not receiving any Senior Companion services. The anticipated sample size for the pilot pre/post study is 500 new Senior Companion clients. The data analysis for the pilot pre/post survey will be exploratory. The exploratory data from the longitudinal pilot study of new clients combined with the results from the main study of established clients will be used to provide information allowing CNCS to assess the feasibility of conducting future longitudinal studies to examine changes in outcomes among new clients in the Senior Companion Program. The results from this study will also inform the feasibility of using the same instrument to conduct an evaluation of CNCS’s RSVP Program and services.


  1. Use of Electronic Media


The survey instrument for SCP clients (both established and new clients) will be administered by telephone. Informing respondents about the surveys will also be done by telephone. Trained interviewers will administer the survey. Interviewers will also be trained on how to handle situations involving respondents with special needs, such as cognitive or physical impairments. In the event that a SCP client is unable to provide survey responses, a proxy respondent will respond on the client’s behalf. A modified version of the instrument will be administered to respondents who require a proxy reporter due to cognitive impairment. Our approach to administer the instrument is further described below. Interviewers will enter the data directly into the CATI system. Using the CATI system to administer the survey minimizes both respondent burden and cost to the government. JBS, the independent contractor collecting and analyzing the data, has extensive expertise conducting evaluation studies with the population affected by this study. Their previous experiences with data collection efforts with the population of elderly clients with special needs indicate that a telephone survey is the best method to achieve high retention and response rates while minimizing burden and costs to the government.


  1. Identification of Duplication


The instrument to be administered to clients includes performance measures developed by CNCS to gauge grantees’ performance in serving SCP clients. The instrument also includes items taken from the HRS questionnaire to be used in forming a matched comparison group. However, the data collection is not a duplication of the HRS data because it involves a sample of elderly clients with special needs who receive Senior Companion services. By contrast the HRS data is a national representative sample of non-institutionalized adults age 50 and older, and collects a diverse set of information on health care, housing, assets, pensions, employment, and disability, all which are intended to inform a broad discussion about retirement. One of the objectives of the proposed data collection is to understand the impact of CNCS-funded services on the clients served by that program. The HRS does not collect information about Senior Companion services from their respondents. Therefore, the HRS data are not appropriate for the purpose of this data collection. The HRS data will be used to form a comparison group using propensity score matching methods to compare to SCP clients. Also, this is CNCS’s first attempt to assess the impact of its services on elderly clients using quasi-experimental methods. Therefore, the information to be collected does not exist with another organization or with CNCS. Lists of SCP clients exist only at the grantee level.


  1. Impacts on Small Businesses


The proposed data collection will require input from SCP grantees that are often small, non-profit entities, but will not impact small businesses. The grantees will be asked to share the list of clients served using CNCS funds with JBS, the external evaluator. JBS will use the list of clients to draw the sample.


  1. Consequence to Federal Program or Policy


The Domestic Volunteer Service Act of 1973, as amended (Public Law 93-113), directs CNCS to assess the impact and effectiveness of Senior Corps Programs at least once every three years. In addition, CNCS’s 2011-2015 Strategic Plan outlines the agency’s commitment to building an evidence base that will allow for informed decisions and the allocation of resources. The proposed data collection is one of the activities the agency is undertaking to build its capacity to contribute to this evidence base with data collected through uniform, outcome-based performance measures and studies with a comparison group. The proposed data collection will provide information to CNCS on how to strengthen its national service programs so that clients receiving CNCS-supported services achieve stronger social ties and social support, life satisfaction, and self-efficacy. The Senior Companion Program’s dual purpose is to engage individuals age 55 and older, especially those with limited income, in volunteer service to meet community needs, and to provide a high quality experience. The program’s funds are used to support Senior Companion volunteers in providing supportive, individualized services to help adults with special needs maintain independence. The data collection will form the basis for improving and strengthening these services to SCP clients.


  1. Special Circumstances


There are no special circumstances that would require the collection of information in any other ways specified.


  1. Publication in the Federal Register and Consultation with Other Agency


  • An agency’s notice in the Federal Register was completed. The 60-day Federal Notice was published on Tuesday, November 27, 2012 in the Federal Register, volume 77, number 228, pages 70739-70740.

  • CNCS consulted with a select group of four Technical Working Group (TWG) members with expertise in psychometrics, experimental design, survey design, and analysis of clustered data. CNCS and its contractor JBS had extensive discussions with the TWG members to review and revise the instrument previously submitted with the 60-day Federal Notice. The purpose of the discussions was to ascertain whether the survey items would be able to identify a matched comparison group from the Health and Retirement Study (HRS) that is similar in characteristics to the SCP clients, and to determine the set of items required to reliably measure the key outcomes. Based on this discussion and the TWG’s feedback on the instrument was revised to include additional questions which will permit JBS to reliably and efficiently form a matched comparison group with the HRS. Additional survey items were added to improve the reliability of the key outcome constructs. JBS consulted with TWG members on the most effective procedures to determine when respondents with cognitive impairment would be unable to complete a telephone survey on their own.


  1. Gifts or Payments


Respondents will not receive any incentive for participating in the study.


  1. Assurance of Confidentiality


Upon contacting respondents, trained interviewers will explain the study and will provide information on who is conducting the study and why the study is being conducted. Interviewers will describe the background, research activities, risks and benefits, and provide assurance of confidentiality. Respondents will also receive information on whom to contact if they have questions. Respondents will be informed of the voluntary nature of the research and their right to end participation at any time. Respondents will be informed that their responses will be anonymous. A proxy reporter will be used with the client’s consent in cases where it is determined that the client will not be able to do the interview. In situations where a proxy reporter is used, the proxy’s agreement will be obtained to conduct the interview on behalf of the client. Proxies will be informed of the voluntary nature of the survey and their right to terminate participation at any time. Proxies will be informed that responses will be anonymous.


JBS anticipates encountering situations involving respondents who may need a proxy due to cognitive or physical impairment. In cases where the respondent has a physical impairment, such as for example hearing or speech impairment, JBS will obtain the respondent’s permission to select an individual who will assist the respondent in providing responses to the questions. In cases where the respondent has a cognitive impairment, a proxy reporter will be asked about the respondent’s demographics and health characteristics to obtain data on non-response. The organization of the survey instrument is discussed in greater detail to accommodate each of these situations under the Dissemination of Information section starting on page 6.


  1. Questions of Sensitive Nature


The survey instruments do not include any questions of a sensitive nature. However, some respondents may feel that questions on income might be personal in nature. During the consent process, respondents will be informed of their right not to respond to any of the survey questions.


  1. Burden Hours


The burden hours for each category of respondents subject to this clearance are described in the table below.


Category of Respondent

No. of Respondents

Participation Time (Minutes) per Respondent

Burden Hours per Respondent

Burden Hours All Respondents

Main study of established SCP clients

1,980

20

0.33

653

Pilot pre/post/post study of new SCP clients*

500

60

1

500

SCP Project directors (list of clients)

50

60

1.00

50

* Participation time for new SCP clients reflects 20 minutes for pre-test and 20 minutes for each of two post-tests.


JBS research staff conducted an internal pre-test of survey instrument with eight current SCP clients recommended by SCP Project Directors. The pre-test procedure included how long it took to complete the survey. The pre-test was then followed by a cognitive interview to review survey organization and content, and what problems were encountered. The cognitive interviews were essential to ensuring data quality, allow researchers to determine the extent to which respondents’ understandings of the questions match their intended meaning.


Testers reported in general the instructions were easy to follow, and for the most part the questions were clear. Testers did identify instances of unclear wording and problems with response options for some questions. The instruments were revised to address the problems testers identified. Based on the internal testing we revised some of the survey items to improve clarity.


The time to complete each instrument varied and depended on the characteristics of those completing the instrument. Based on the pretest results, it should take the average respondent 20 minutes to complete the survey.


Appendix A summarizes the process JBS used to conduct the pre-tests as well as the results of the pre-test pilot.

The estimated annualized cost to respondents for the hour burdens for collection of the information is:

  • Established and new clients who are elderly seniors: The estimated annualized cost for respondents is $10,123 which is based on an estimated hourly wage rate of $8.78 and total burden hours of 1,153. Hourly wage estimates for elderly adults were based on average Social Security benefit for retired workers. The average monthly Social Security benefit for a retired worker was $1,230 at the beginning of 2012. The estimates were obtained from “Average monthly Social Security benefit for a retired worker”, accessed at http://ssa-custhelp.ssa.gov/app/answers/detail/a_id/13/~/average-monthly-social-security-benefit-for-a-retired-worker.

  • SCP Project Directors to compile their client list: The estimated annualized cost for respondents for the grantee survey is $1,742 which is based on an estimated hourly wage rate of $34.24 and total burden hours of 50. Wage estimates for nonprofit fields were taken from Wages in the Nonprofit Sector: Management, Professional, and Administrative Support Occupations, accessed at (http://www.bls.gov/opub/cwc/cm20081022ar01p1.htm ) and are from 2007.


  1. Estimated Total Annual Cost Burden to Respondents


The collection of this information does not have any capital and start-up cost; and it does not have any operation and maintenance cost.


  1. Estimated Annual Costs to Federal Government


For the survey design, sampling, implementation and analysis of the data collected the estimated annual cost to the Federal government is $306,360. This number is based on the portion of the contract with JBS International that is devoted to the design, sampling, implementation, and analysis of the data collection for the main study which will provide preliminary estimates of the impacts of Senior Companion services and the exploratory feasibility pilot pre/post study.


  1. Explain any Program Changes


This is an application for new collection. There are no program changes.


  1. Dissemination of Information


The data will be analyzed and a final report will be submitted to CNCS. CNCS will include results in their Performance and Accountability Reporting to Congress and in Congressional Budget Justification. In addition, research data files will be prepared and submitted to CNCS for future research.


  • Survey Instrument

The survey questions begin with Part 1, Page 2. Our approach in determining if a respondent needs an assistant is to ask basic information the respondent is expected to know. The first three questions in the survey are birth date, ethnicity and marital status. Interviewers will be trained to recognize if a respondent is unable to answer or there is indication that the respondent is confused or unsure about how to answer these basic questions. If it is determined that a respondent cannot respond to the survey independently the interviewer will attempt to identify an assistant or proxy, or terminate the interviewer if necessary. JBS’ protocol makes a distinction between an assisted interview, where an individual assists the respondent with the survey, and a proxy reporter where the individual would have to provide the answers for the respondent due to severe physical or mental impairment. For assisted interview JBS anticipates that the respondent can often suggest an individual who can serve as an assistant. The respondent’s Senior Companion volunteer will not be an acceptable assistant. In the latter situation where a proxy reporter is used because the respondent has cognitive or physical impairment that prohibits the respondent to independently respond to the questions, JBS will administer a modified survey instrument. The modified version of the survey includes questions that a proxy reporter can be expected to answer such as demographic characteristics. Questions that use the first person “I” or use “you” will be modified to use the third person “the client”. The survey instrument collects the relationship of the assistant or proxy to the respondent so that those responses can be analyzed separately, if necessary. Specifically, Table 1 shows the specific questions in the instrument JBS will administer to each type of respondent:


Table 1. Survey Questions to be Administered to Different Types of Respondents


Respondents answer for themselves

Respondents answer for themselves with assistance

Proxy Reporters Answer for the Respondents

All Questions

All Questions

Questions 6, 7, 8, 9, 12, 13, 14, 15, 16, 25, and 27-37


  • Exploratory Analysis

The exploratory analysis will be used to assess data quality and reliability of the survey items. This is an important initial first step to detect errors, determine if there are any violations of statistical assumptions, determine the relationships among explanatory variables, assess the direction and size of relationships between explanatory and outcome variables, and generate hypotheses. At the end of this exploratory analysis, we expect to have a solid knowledge base about the data characteristics of the variables of interest and the relationships among them. During the exploratory phase, we will assess the validity of the instrument and estimate the reliability of the measures. In the exploratory analysis phase, JBS will determine the response rate, screen the data for incomplete or missing data, detect outliers in responses, and combine items into scales. Since the study will be based on a sample of clients, JBS will conduct a non-response bias analysis to determine if the final sample of respondents reflects the population from which it was obtained, and whether the final sample is significantly different from the population from which it was drawn. . CNCS has censused the population of SCP clients in a previous study, however no demographic information was collected. The specific variables that can be used for the non-response bias analysis are number of hours the client is receiving independent living services, and the set of questions listed under ‘Part 2: Performance Measure’ of the survey instrument. Some information on clients will also be obtained from grantees as part of the sampling process, including gender, primary language spoken, whether the grantee/station has prior information that the client may need an assistant or proxy to complete the survey, and how long the client has been receiving independent living services. JBS will use this information in its non-response bias. Non-response bias at the grantee level will also be investigated using data on grantee size (i.e., number of SCP clients served) and location obtained as part of the sampling frame.


We describe procedures for imputing missing data, propensity score matching, and statistical modeling for the main study with 1,980 respondents. The analysis for the pilot pre/post survey that consists of 500 respondents will be exploratory only.


  • Data Imputation

For the main study, JBS will identify the extent of missing values for the items during the exploratory analysis phase. The exploratory analysis will provide information about the source of missing values, and this will allow JBS to select, if needed, the most appropriate imputation method. It is important to understand whether missing values are related to observed characteristics of respondents, or if in the case of the longitudinal data, there is a loss of individuals during follow-up. Once we understand the sources and extent of missing data, JBS will review, select, and apply the most efficient method, based on careful consideration of the dataset and the type of missing data.


  • Propensity Score Matching with HRS Data

For the main study, JBS will create a matched comparison group using the HRS data. The treatment group will include SCP clients matched with a comparison group of HRS respondents with similar characteristics who would not have received Senior Companion services. Table 2 below shows the list of common variables in the SCP and HRS datasets.


Table 2. Common Variables in the SCP and HRS Datasets


Senior Corps Survey Instrument

HRS Data

Items for propensity score model

Items that are outcomes to compare between the two groups

Questions 6 - 25; 26 - 38

Common questions

Demographic and background questions 6-10, and 28-38; health questions 12-19

Life satisfaction question 11; self-efficacy questions 20-21; social loneliness question 22-23; emotional loneliness 24-25

Questions 26 and 27

Not in the HRS data





Table 3 below describes the analytic steps to complete the propensity score matching (PSM).



Table 3. Propensity Score Matching


Activity

How and what will be done

Data preparation and harmonization

  • Common variables in the SCP and HRS datasets will be aligned in terms of their definition and measurements, and their distributions will be aligned as closely as possible to each other so as to minimize differences among common variables across datasets. For example, JBS will examine the ‘age’ variable in both data sets to assess whether the range of values for this variable is similar across both datasets. If they are not, then the ‘age’ variable will be harmonized across both datasets.

  • JBS will handle missing data imputation similarly for both datasets.

  • The HRS data will be appended to the SCP data to create a combined data file. JBS will add a variable to indicate whether a case belongs to the treatment group from the SCP data or a control case from the HRS data.

Weight adjustments

  • Although the SCP data and the HRS data have common variables on which to match them, the data are produced for different purposes, are based on different designs, and the HRS data will likely be from different years than the SCP data. The SCP data will be from a self-weighting PPS sample. We will use unweighted HRS data when estimating the impact of the treatment. We will not make inference to the whole population of SCP clients. The HRS oversamples minority population, given that the SCP population is more heavily minority and low-income, the unweighted HRS data will more likely resemble the SCP population.

Stratification and segmentation

  • JBS will assess whether it would be beneficial and more efficient to segment the combined SCP-HRS data into discrete cells. The propensity scores would be estimated separately within cells, and matching of treatment to control cases would be conducted within each cell. The segmentation approach might achieve a tighter fit in matching treatment cases to control cases.

  • A reason for exploring this option is that the HRS data oversampled African American, Hispanic, and Florida respondents. Thus, it is important to explore whether matches of certain types of cases should be required and/or avoided.

Estimate the propensity score

  • JBS will estimate a logistic regression model on the combined SCP-HRS dataset where the dependent variable is whether a respondent is a treatment case (e.g., SCP client) and the common variables in the combined data are independent variables (excluding the variable for which we want to detect impact of Senior Companion services).

  • For example, if we are trying to assess the impact of Senior Companion services on psychosocial health outcomes, then when estimating the propensity score, the psychosocial health measures will not be included as independent variables to estimate the propensity score. If data are segmented, as discussed in the prior step, JBS will estimate a cell-specific propensity score in order to achieve a tighter fit in matching.

Matching algorithm

  • JBS will use the nearest neighbor matching procedure to match the treatment cases to the control cases. JBS will use sampling without replacement meaning each comparison group respondent will be included as a matched case only once. .

Test properties of the statistical match

  • JBS will check the key assumptions and verify that the model specification to create the comparison group is appropriate and that the results do not exhibit any bias.

  • JBS will check that the PSM balances the characteristics between the treatment and comparison group. If differences between treatment and control group persist after the match, JBS will refine the model specification to improve the balance between the groups, or, potentially, consider a different matching approach.


  • Statistical modeling

The impact of Senior Companion services will be estimated as the average of the difference in outcomes between treatment and the comparison group. We expect the SCP data will have a nested structure; that is, SCP clients will be nested within sites or stations, and stations will be nested within project. The HRS sample is selected under a multi-stage area probability sample design, and oversamples minority respondents. JBS will use multilevel/hierarchical models to estimate the impact of the Senior Companion treatment using Generalized Linear Mixed Models (GLMM), which are the appropriate multilevel models for complex and clustered samples like the SCP and HRS data where the outcome measures are either categorical or dichotomous.


  • Feasibility of Future Studies

JBS will collaborate and consult with CNCS to identify and understand the issues to be addressed in future studies of SCP. These issues might include research objective, design, expected data analysis method, detectable power, sample size, type I error, variability, and effect size. JBS will use the results, including the pilot pre/post survey of new clients, as well as prior results from other studies (such as reports and papers that used the HRS data and peer-reviewed articles), to arrive at an informed set of values for the type of information needed to carry out a power analysis for future studies. JBS will conduct several analyses using reasonable ranges in each input component used in the power analysis to obtain a range of estimated sample size options and effect sizes. Using the results from the power analysis, JBS will provide cost estimates needed to implement future studies.


  • Limitations

CNCS recognizes that the quasi-experimental for the main study is subject to limitations and threats to internal validity that will be noted in all reports of the results. Specifically, the main study seeks to match SCP clients using propensity score matching procedures to a completely independent sample survey using common measures from both collections that will be collected at different points in time by different organizations and under different conditions. Any observed differences may be due to these and other factors rather than effects of the SCP program per se. However, CNCS believes that this research design will provide potentially useful insights about potential effects of the SCP program.


  • Data File for Future Research

JBS will produce a public access and agency specific data file for this data collection. The data files will be clean, well-organized, and adhere to privacy and security standards. JBS will create a de-identified and anonymous Public Use Data File. JBS will follow guidance on government-wide data sharing in all Federal statutes pertaining to data collection, privacy, and human subject research and take adequate steps to ensure that no individual is identifiable. For the Public Use Data File, JBS will produce a sampling and methodology report that will include a copy of the instrument, description of the methodology, and guidance on using post-sampling weights.


  • Timeline (End dates)

Activity/Task

Main Study

Pre- and First Post Survey

Second Post Survey

Sample selection

6/28/13

6/28/13

6/28/13

Data collection

10/31/13

10/31/14

9/1/2017

Conduct data analysis

2/10/14

2/10/14

2/10/18

Final Report including feasibility of future studies

7/17/2014

5/15/2014

6/17/2018

Submit data files to CNCS for future research

9/15/2014

5/15/2014

6/17/2018


  1. Display of Expiration Date for OMB Approval


The expiration date for OMB approval of the information collection will be displayed on the first page of the instruments.


  1. Explanation of Exceptions in Item 19, "Certification for Paperwork Reduction Act Submissions"


There are no exceptions claimed in Item 19, "Certification for Paperwork Reduction Act Submissions" of OMB Form 83-I.


Appendix A


Draft - Senior Corps Pre-Test Memorandum
Independent Living Survey (Study 3)

Independent Living Pre-Test: Respondent Interviews

The pre-test survey process with individuals was initiated once the draft of Study 3 Impact Computer Assisted Telephone Interview (CATI) survey was completed in April. The pre-test consisted of six English language survey respondents, who were recruited from Senior Corps programs that are working with JBS International as part of the Field Working Group. Two Spanish-speaking pre-testers took part in the pre-test. Spanish language pre-testers were JBS family members, chosen because they are 55 and older. They were not or have ever been a Senior Corps volunteer. The Spanish language survey was administered to the respondents in Spanish. Seven of the eight respondents were women. They ranged in age from 55 to 92 years old.

Summary of Pre-test Process

The Study 3 English language survey had two pretesting phases. The first phase pre-tested the survey in the original form. Three individuals, including one individual who is hearing impaired, were administered the survey in English at one point from April 17 through May 6, 2013. The second phase of the Study 3 English language pre-testing process included three respondents that completed a variant of the survey between May 9 - May 14, 2013. There were the following differences between the original survey and the variant survey:

  • The variant survey began with the demographic questions.

  • The answer options differed for the question, “Please think about your life-as-a-whole. How satisfied are you with it? Are you satisfied or not satisfied? [Check one box]”. The variant survey asked the question in a yes or no format that then further broke down their yes or no response on an agree/disagree scale, whereas the original survey only had a five point agree/disagree scale.

  • The last question’s wording was, “Have a memory-related disease, such as Alzheimer’s disease?” in the original survey, but was substituted with “Had any emotional, nervous, or psychiatric problems?” in the variant survey.

The Spanish language survey was pre-tested with two individuals during May 13-30, 2013. These respondents were selected from the networks of JBS’ Spanish speaking staff.

The process of pre-testing the CATI survey and conducting the interviews began with the interviewer contacting the respondents to schedule a time to administer the CATI survey. The interview could be completed on the first contact call or scheduled at a later date. Respondents’ time to complete the survey varied in both languages administered, from 18 minutes to nearly one hour. Most respondents took about 22 minutes to complete the survey, with only one respondent taking more than the average time to complete the survey. The feedback interviews were conducted soon after the survey was completed with pre-test respondents, and they were administered in eight to 16 minutes.

Below, a summary of the survey pre-test findings for both English and Spanish language respondents is provided.

Pre-tester Comments

Instructions

English and Spanish language pre-test respondents stated that the instructions were clear and they had no problem with following the instructions as read by the CATI interviewer. The exception was one Spanish language respondent who commented on the question “Because I have a Senior Companion”. Since this Spanish language pre-tester was not a Senior Corps client, it was understandable that it was a challenge to take on a Senior Corps Companion client perspective.

Questions

The questions, for the most part, posed no problems for respondents. There were several questions in the Social Loneliness and Emotional Loneliness sections that the interviewer needed to repeat because respondents were not clear what was being asked on the first pass. The two questions most often repeated for both English and Spanish language respondents were, “How much of the time do you feel that you lack companionship? Would you say often, some of the time, or hardly ever or never?” and “How much of the time do you feel that there are people you can turn to? Would you say often, some of the time, or hardly ever or never?” Respondents took time when answering this question, as they needed to reflect on what was being asked. The interviewer noted that both questions had multiple sentences that must be read to the respondent. This might have accounted for the lag in response and question repetition. One of the English language respondents stated that she was hearing impaired and, at times, she needed several questions repeated more than twice.

The interviewer noted that none of the pre-testers had any difficulty with the length of the survey or with the different wordings in the questions that varied between survey versions. The interviewer felt that breaking the scale down for the question, “Please think about your life-as-a-whole. How satisfied are you with it? Are you satisfied or not satisfied?” into multiple questions, made the item easier to administer, as respondents seemed to have less trouble understanding the scale, and the options flowed better.

One Spanish language respondent suggested breaking the “Because I have a Sr. Companion Volunteer…” table into two tables, as he found the table too cumbersome and long to answer.

Some of the respondents’ answers regarding the survey questions included the following:

  • “The questions were understandable and clear, I understood everything.”

  • “There were no questions that were confusing, everything was pretty understandable.”

Respondents did not exhibit reluctance in answering any of the survey questions.

Survey Format

Nearly all of the respondents felt that the length of the survey was appropriate. While the demographic questions may serve as a way to distinguish if a respondent is cognitively impaired, the interviewer noted that respondents were more engaged at the beginning and provided thoughtful answers to the scaled items. The interviewer also suggested that the demographic questions be kept at the end as it also provided some relief for both the interviewer and respondent. Additionally, the interviewer noted that when the scaled questions were at the front of the interview, the respondents spent more time discussing them and reflecting about their own lives and experiences, especially for questions about Social Loneliness and Emotional Loneliness. Furthermore, the interviewer observed that the survey response time was longer in some cases because some respondents had a tendency to diverge from the survey questions and discuss concerns or issues that would be tangentially related to the question. Numerous respondents would take time to think out loud before providing a response.

Recommendations to address those respondent issues and future surveys

Overall, the respondents for the variant survey and original survey did not appear to have any problems understanding instructions, questions, or the survey format. Below are some recommendations that respondents and interviewers provided that would improve future surveys:

  • Include “none” as answer option for the question “Veteran Status.”

  • Provide a write-in option for the question, “Do you consider yourself primarily…”

  • Consider deleting the income question. Two respondents felt that future respondents may find this question too personal.

  • Ask how comfortable a Senior Companion client is communicating with their volunteer, and if the volunteer is able to clearly communicate with the client. Good language communication and same language spoken, can lead to better cooperation between client and volunteer.

  • Break the table with the question, “Because I have a Sr. Companion Volunteer…” into two tables, as the current table seemed too long to some respondents.

  • Retain the survey as a CATI format, as respondents preferred this format over a mail or self-completed survey.

The following are JBS’ three recommendations for Study 3 – Impact CATI Survey which take into consideration the pre-test respondents’ feedback:

  • Administer the original survey question sequence with the demographic questions at the end of the survey.

  • Retain original survey questions with the exception of the answer options for question “Please think about your life-as-a-whole. How satisfied are you with it? Are you satisfied or not satisfied? [Check one box]”. Use the answer options from the variant survey, which provided the answers in a in a yes or no format and broke down the agree/disagree scale.

  • Keep question “Had any emotional, nervous, or psychiatric problems?” used in the variant survey, as there were no problems with the question as observed by respondents.


Page 10 of 10


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
Authorageorges
File Modified0000-00-00
File Created2021-01-29

© 2024 OMB.report | Privacy Policy