Supporting Statement A - final - OIRA_Final_12.20.19
Supporting Statement A - final - OIRA_Final_12.20.19.docx
NCCC Impact Studies
OMB: 3045-0189
Supporting Statement
Longitudinal Impact Evaluation of AmeriCorps National Civilian Community Corps
A. Justification
1. Circumstances of Information Collection
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and of each regulation mandating or authorizing the collection of information.
The Corporation for National and Community Service (CNCS) requests approval from the Office of Management and Budget (OMB) to conduct a longitudinal impact evaluation of its AmeriCorps National Civilian Community Corps program (NCCC). A national impact evaluation to identify how participation in NCCC promotes leadership qualities and skills among its members and strengthens the communities in which NCCC members served will contribute to CNCS’s efforts to fulfill its mission to improve lives, strengthen communities, and foster civic engagement through service and volunteering. This national impact evaluation of NCCC is closely linked to the CNCS’s strategic plan which aims to increase the impact of national service in CNCS-served communities (Goal 2) and strengthen national service so CNCS program participants consistently find satisfaction, meaning, and opportunity (Goal 1).
Goal of the Evaluation
The impact evaluation will consist of three studies:
Leadership Development: A mixed-methods longitudinal evaluation to examine how participation in NCCC impacts members’ leadership skills (professional skills, life skills, teamwork, and civic engagement) .
Member Retention: A mixed-methods evaluation to understand factors affecting retention of NCCC members at different stages and at different NCCC campuses.
Strengthening Communities: A mixed-methods exploratory and qualitative in-depth case study of a sample of NCCC service projects to gauge how these projects strengthen communities.
Need for the Evaluation
NCCC member outcomes in civic engagement, community engagement, and professional skills have been studied in prior AmeriCorps evaluations (Friedman et al., 2016; Jastrzab, et al., 2002; Epstein, 2009; Abt Associates, Inc. & CNCS, 2008; Westat, 2002), but there has not been a national impact evaluation specific to participation in NCCC. CNCS completed a comprehensive review of the literature and found no research studies that have examined either the leadership qualities and skills of NCCC members or how the service of NCCC members have strengthened the communities they served.
Retention of NCCC members through the end of their term of service has declined in the past five years. NCCC has also experienced a decline in the number of qualified applicants. These two trends have resulted in a higher than anticipated cost per graduating member. There is no underlying data or study to help NCCC understand the factors contributing to successful member retention or recruitment.
Strengthening communities is a core NCCC mission achieved through service projects to support local community needs. NCCC has extensive quantitative and qualitative data about the outputs and benefits of projects its members complete, yet there has not yet been a systematic effort to investigate and define what “strengthening communities” means in the context of NCCC, including what impact NCCC has had on the communities where projects were based. The strengthening community study adopts a primarily exploratory mixed methods approach that includes in-depth case studies of project sites combined with quantitative analysis of the existing service projects data to provide insight into the impact of Traditional NCCC and FEMA Corps teams on organizations and communities and to clarify the connection between quantified outputs and meaningful outcomes.
CNCS is required by the National and Community Service Act as amended under PUBLIC LAW 111–13—APR. 21, 2009, Section 164 (42 U.S.C. 12624) to conduct periodic evaluations of the National Civilian Community Corps Program and, upon completing each evaluation, transmit to Congress a report on the evaluation.
NCCC is a full-time, residential, team-based program whose mission is to strengthen communities and develop leaders through team-based national and community service. NCCC combines practices of civilian service with aspects of military service, including short-term on-site deployments, leadership development and team building. All members receive training in leadership, team building, disaster services and civic engagement.
Sponsors are responsible for matching NCCC support with in-kind resources, assisting teams in obtaining housing and meeting basic needs during their service, and supervising them during their project work. Sponsors are typically nonprofits who apply for support in a specifically designed project that can benefit from the addition of NCCC Corps members. Community beneficiaries are wide-ranging and include disaster survivors, seniors and people with disabilities, and people in low-income communities facing housing or food insecurity.
Both Traditional NCCC (founded in the early 1990’s) and FEMA Corps (created in 2012) promote Goal 1 by enrolling members in a full-time, residential, team-based service program in which young adults complete projects addressing essential community needs. NCCC promotes Goal 2 by involving members in structured training and other activities designed to enhance personal development, promote professional development, and foster leadership skills that shape program alumni for a lifetime.
CNCS completed three reports in 2018 for a feasibility and planning study to summarize the existing literature on youth leadership, factors affecting retention of youth in national service programs and the evidence on how national service programs have strengthened communities. All three reports indicate critical gaps to be filled in by the proposed collection of information. A succinct summary for each study is provided here.
Youth Leadership Development. The literature review on youth leadership development suggests that youth leadership qualities and skills can be learned, and that community service can be an appropriate means to develop these skills. However, scant research rigorously testing these models exist. Of the research that exists, it is found that national service positively impacts employment aptitude, employment outcomes, employability and career goals, and civic engagement. Limited research exists on teamwork and life skills, and on the long-term impacts of service on professional skill.
The leadership development study seeks to assess the impact of service and the training model via a longitudinal study that assesses variance across three measurement periods (baseline, close of service, one year follow-up) and selected demographics relative to matched control subjects.
In this model, specific to national service involvement, leadership development and service experience leads to:
Increased professional skills
Collaborative work practices,
Enhanced life skills and
Broader civic engagement activity,
These four dependent variables or outcomes will be used to assess the impact of NCCC training and service experience. The submitted survey, reviewed for this Information Collection Request, contains items theoretically linked to the four dependent variables. The items pertaining to each variable will be aggregated to create a composite score for each measure. These composite scores will be used to measure variance across demographic characteristics, experimental and control samples, and importantly, time.
Factors Affecting Retention. Youth volunteers are often motivated by a desire to acquire career-related benefits, experience social benefits, and have meaningful and substantive experiences. There is still relatively little knowledge available about the specific practices that contribute to volunteer retention, particularly with a focus on the retention of NCCC members. Some research exists indicating that the factors that contribute to retention include proper recruitment, selection, training, effective volunteer management, strong social networks, and the members’ satisfaction with tasks and roles.
Strengthening Communities. There is a gap in knowledge of how NCCC achieves its mission to strengthen communities. To assess how NCCC strengthens communities, there must first be a definition of the concept of community strengthening. Yet, the reviewed literature has no consensus on a definition and conceptual framework for community strengthening and, therefore, no consensus on methods of achieving or measuring community strengthening.
2. Purpose and Use of Information
Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
The data collection will be used to identify promising practices for recruiting and retaining Corps members and improving service projects to increase member leadership skills and strengthen the communities in which members serve. The primary users will be NCCC leadership and other stakeholders in the AmeriCorps program offices and in the CEO’s office to make decisions about program and service experience improvement for Corps members and communities served. NCCC headquarters and regional campuses will be able to review their programming to make meaningful changes to their interventions based on the evaluation report and results specific to their program. External stakeholders, such as researchers and federal partners, may also find value in the evaluation’s results as they endeavor to implement evaluations of similar programs, understand the community impact of service projects, leverage evaluation resources, and conduct focused research on national service interventions.
3. Use of Information Technology
State explicitly whether electronic submission, maintenance, or disclosure of information (including the electronic storage and filing by employers of information about their employees) would be practicable as a means of decreasing the burden and/or increasing the practical utility of the collection. Describe any special data collection procedures which are designed to reduce the burden to the respondent. Examples: submitting information on disk; electronic transmission of reports or applications; e-mail. Use of CAPI or CATI (Computer Assisted Personal/Telephone Interviews) technology should be reported here. Include a statement about the estimated percentage of responses that are expected to be submitted electronically.
Participants will primarily respond to the surveys electronically from any internet-connected computer. The survey can be completed using a mobile device, or laptop/desktop computer, enabling participants to complete them at a convenient time. If participants cannot complete the surveys electronically, an evaluator will follow up to complete the survey via phone.
To further reduce burden, the survey does not include questions about the characteristics of the service projects members were assigned during their term of service. Instead, the evaluator will use existing CNCS administrative data for analysis related to the impact of service project experiences on leadership skills, member retention, and the communities in which members serve. Furthermore, information that can be obtained from administration data including NCCC application data (e.g., campus where member is assigned, role the member served) are not asked in the survey but will be retrieved from the administrative data.
The focus groups and interviews for the three studies will be conducted with a subset of Corps members (approximately 476 of the 1,600 participants) and can be conducted online to reduce burden on participants. There will not be any focus groups or interviews with comparison group participants (non-Corps members), (See Selection Criteria in SSB).
4. Efforts to Identify Duplication
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in 2 above.
There are no other sources of information by which CNCS can meet the purposes described in A2. While CNCS surveys members about educational plans and their experiences in NCCC through the Member Experience Survey, that data on its own cannot be used to assess impact on leadership skills or reasons members leave early. The questions in the proposed survey combined with existing administrative data and data from the CNCS service projects database cover the outcome areas of interest in greater depth and breadth, and have been specifically tailored to assess the impact of NCCC on members and the communities served.
5. Involvement of Small Entities
If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.
The evaluation will not be administered to small businesses or other small entities.
6. Consequences If Information Collected Less Frequently
Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
The frequency of data collection from the participants will be held to the minimum necessary to meet the needs of the evaluation objectives. This evaluation is CNCS’s first opportunity to build rigorous impact evidence of NCCC on members and communities served. A consequence of not following through with data collection is that CNCS would lose the opportunity to be informed about the efficacy of NCCC as an intervention for supporting youth leadership, would not understand what factors contribute to attrition among its members, and could not adjust program operation and structure to minimize such attrition. In addition, although NCCC has extensive existing quantitative and qualitative data about the outputs and benefits of projects, there has not yet been a systematic effort to investigate and define what strengthening communities means in the context of NCCC and how NCCC has strengthened the communities in which its members serve, a central mission of NCCC. Given the lack of evidence via existing member and sponsor reports, it would likely be many years before CNCS would have a comparable level of information about the effectiveness of NCCC.
7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)
Explain any special circumstances that would cause an information collection to be conducted in a manner inconsistent with the general information guidelines in 5 CFR 1320.5.
This information collection fully complies with 5 CFR 1320.5(d)(2). There are no special circumstances that would require the collection of information in any other ways specified.
8. Consultation Outside the Agency
Identify the date, the volume number, and the page number of the publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on 05/03/2019, 84 FR 19056. Zero comments were received.
9. Payment to Respondents
Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
CNCS anticipates that the treatment group will fully cooperate to participate in the study, and will achieve a response rate of over 80 percent at baseline and the first follow-up, which are the two data collection points when these participants are in service at their assigned campuses. However, enrollment of the comparison group will be challenging. Because their participation is crucial for the planned impact analysis, maximizing response rates among the comparison group is a key priority for this evaluation to achieve adequate sample size and minimize sampling bias and error.
NCCC does not maintain a primary relationship with applicants who declined to serve (comparison group). Once these applicants have declined, they are uninterested and disengaged in communicating with NCCC. Consequently, it is anticipated they will be difficult to engage and enroll in the study. The timing of the baseline data collection is critical. If CNCS misses the window to collect the baseline due to multiple rounds of contacts and persuasion to participate, members term of service would have started, and this will reduce the ability to collect baseline data from the comparison group. As such, a delay in completing the baseline survey confounds CNCS’s ability to test the impact of the intervention, which is the main goal of the study.
Without an incentive, CNCS will need to engage in multiple rounds of contact to encourage participation. The cost of these multiple rounds outweighs the proposed participation incentive. Given the costs of the study and the potential uses of the findings, it is prudent to take steps to ensure a viable comparison sample.
CNCS carefully considered an incentive offer at the critical juncture when some target participants would be ‘hard to reach,’ as summarized in Table 1:
Table 1 Proposed Incentives for Target Participants
Target Participants |
Baseline survey |
Early Exit Survey |
First Follow-up Survey |
Second Follow-up Survey |
Corps members |
-- |
-- |
-- |
$20 |
Early exit members |
-- |
-- |
$20 |
$20 |
Comparison group |
$20 |
-- |
$20 |
$20 |
Sponsors/FEMA POC |
-- |
-- |
-- |
-- |
Community beneficiaries |
-- |
-- |
NCCC branded items |
-- |
Corps members will not receive any monetary incentive during their term of service. Once members complete their service or leave the program early, they will be more difficult to reach and will become disinterested in the study. As such, the opportunity cost for the time spent completing the survey increases. After the end of members term of service, CNCS will offer alumni a modest incentive of a $20 electronic gift card to complete the second follow-up (final) survey. CNCS will offer a $20 electronic gift card to members who leave early and do not complete their service to complete the first and second follow-up surveys. CNCS will offer a $20 electronic gift card to comparison group participants for each survey completed (a total of three surveys). Respondents will be asked to provide contact information where the electronic gift card can be sent upon completion of the survey. CNCS will provide in-kind NCCC-branded promotional items to beneficiaries who participate in focus groups or interviews in the community strengthening case studies.
A substantial body of research including experimental and meta-analyses supports the use of incentives to increase response rates (Brick et al. 2005; Church 1993; Edwards et al. 2002; James and Bolstein 1992; Shettle and Mooney 1999; Singer et al. 1999; Singer, Van Hoewyk, and Maher 2000; Yammarino, Skinner, and Childers 1991). Those studies demonstrate that the use of incentives has a positive impact on increasing response rates, with no adverse effects on reliability (Jäckle, & Lynn, 2008; Dillman,2000).
In longitudinal data collection, the use of incentive has been shown to be cost-effective due to the savings incurred by reducing the costs of follow-ups with non-respondents across waves of data collection. There are mixed findings about incentives in the research literature. Certain types, timing, and amounts of incentives may not be effective, but the overall recommendation is that an incentive is an effective means to increase response rate and reduce nonresponse bias. A study on the use of incentives in the Annals of the American Academy of Political and Social Science, Singer and Ye (2013) conclude that:
“Incentives increase response rates to surveys in all modes, including the Web, and in cross-sectional and panel studies;
Monetary incentives increase response rates more than gifts, and prepaid incentives increase them more than promised incentives or lotteries, though they are difficult to implement in Web surveys;
Incentives, thus, have clear potential for both increasing and reducing nonresponse bias. If they can be targeted to sample members who would otherwise fail to respond.”
The Singer and Ye article represents a balanced perspective in that it presents findings that suggest some types of incentives are not effective as well as findings that suggest incentives are effective. Nonetheless, the general conclusion based on research published in the most recent decade is that incentives are effective means to increase response rates.
The incentive will target participants at a critical juncture when they are disengaged with NCCC. Engaging in follow-up efforts to retain non-respondents would be costlier than the proposed incentive.
10. Assurance of Confidentiality
Describe any assurance of protection provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
Access and use of participants’ responses to this information collection will be limited to the contractor conducting the research, permitted agency staff, other federal agencies staff providing matching administrative data, and future contractor staff conducting further research and/or analysis. Additionally, data will be shared only were CNCS is legally obligated to do so. The evaluator will make clear when individual responses will be shared; these assurances will be provided using the survey invitation (below), consent forms, and surveys (see Attachments A and B). The Privacy Act may apply to some records used to complete this study (e.g., a participant’s responses from multiple studies will be linked together using a personal identifier in order to evaluate changes over time). All respondents will be assured that their participation is voluntary, that no adverse consequences will accrue to individuals who do not complete the surveys, and that their comments and opinions will be kept confidential.
Access to any data with identifying information will be limited to the contract staff conducting the research, permitted agency staff, other federal agencies staff providing matching administrative data, and future contractor staff conducting further research and/or analyses. Additionally, all analyses, summaries, or briefings will be presented at the aggregate level and it will not be possible to identify individual respondents in any material that is presented. The evaluator will provide CNCS a de-identified dataset that can be used for further analysis or shared with other research and evaluators for additional secondary analysis.
Privacy Statement: CNCS is required by the Privacy Act of 1974, as amended, (5 U.S.C. 552a) to tell you what personal information we collect and how it will be used: Authorities – Your personal information is requested pursuant to 42 U.S.C. Chapter 129 - National and Community Service and 42 U.S.C. Chapter 66 - Domestic Volunteer Services. Purposes – It will be used to (a) identify best practices for recruiting and retaining NCCC members and (b) improve service projects to increase member leadership skills and strengthen the communities in which members serve. Routine Uses – Routine uses of this information may include disclosure to (1) contractors hired to assist with this collection project or any related follow-up project, and (2) other Federal agencies to match your personal information with their data in order to complete additional research. Effects of Nondisclosure – This request is voluntary, but not providing a response may affect the results of the survey and your eligibility to receive a modest financial incentive. Additional Information – The current SORN, to include this collection, is pending modification and publication to the Federal Register.
11. Questions of a Sensitive Nature
Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
Questions of a sensitive nature embedded in the survey include homelessness, (27(f)), physical and mental impairment (28 (e) and (f)), foster care or runaway status (27(g) and 28(c)), and juvenile criminal history (28(d)). This information is collected at intake by NCCC to determine disadvantaged youth status as required by the Serve America Act (PUBLIC LAW 111–13—APR. 21, 2009). However, a dataset containing this information is not available for matching with survey data.
The survey contains the same items in order to determine disadvantage youth status. These items will be used in the analysis to determine differential recruitment and retention rates relative to non-disadvantaged youth status, and to assess relative levels of adjustment and satisfaction with service, and employment and education outcomes. Lastly these outcomes will assist the program in strengthening outreach, specialized programing, and services.
12. Estimates of Annualized Hour Burden
Table 2 Leadership Surveys Estimates of Hour Burden
|
Time/Respondent (minutes) |
Total Respondents |
Estimated Hours |
Baseline |
20 |
2740 |
913.33 |
Member early exit survey |
8 |
320 |
42.67 |
First follow-up |
20 |
2192 |
730.67 |
Second follow-up |
20 |
1654 |
551.33 |
|
|
|
|
Total Estimated Burden Hours |
|
|
2238 |
Table 3 Qualitative Data Collection Estimates of Hour Burden
|
Time (minutes) |
Total Respondents |
Estimated Hours |
Leadership Development/Member Retention |
|
|
|
Focus groups (Corps members and team leaders) |
90 |
240 |
360 |
Follow-up phone interviews (Corps members and team leaders) |
30 |
240 |
120 |
Phone interviews (early exit members) |
30 |
60 |
30 |
Phone interviews Sponsors/ FEMA |
30 |
48 |
24 |
|
|
|
|
Strengthening Communities (Comprehensive Case Studies) |
|
|
|
Focus groups (Corps members) |
90 |
96 |
144 |
Interviews (Team Leaders) |
60 |
6 |
6 |
Interviews (Sponsors/FEMA)1 |
60 |
24 |
24 |
Interviews (Community stakeholders) |
30 |
30 |
15 |
Strengthening Communities (Restricted Case Studies) |
|
|
|
NCCC Team Leader* |
30 |
12 |
6 |
Sponsors/FEMA POCs |
60 |
12 |
12 |
Community Stakeholders* |
30 |
24 |
12 |
|
|
|
|
Total Estimated Burden Hours |
|
792 |
753 |
*Up to 2 per case study/project. NCCC site supervisors and staff are not included in the calculation.
1Two time points.
13. Estimates of Annualized Cost Burden to Respondents
Provide an estimate of the total annual cost burden to respondents or recordkeepers resulting from the collection of information. (Do not include the cost of any hour burden shown in sections 12 and 14). The cost estimate should be split into two components: (a) a total capital and startup cost component (annualized over its expected useful life); and (b) a total operation and maintenance of services component. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information.
There are no direct costs to respondents other than their time to participate in the study.
14. Estimates of Annualized Cost to the Government
Estimate government costs for contracted data collection by adding the contract costs plus personnel costs of federal employees involved in oversight and analysis. This section should make it clear that the program "has planned and allocated resources for the efficient and effective management and use of the information to be collected, including the processing of the information in a manner which shall enhance, where appropriate, the utility of the information to agencies and the public."
This baseline year of the study involves a one-time cost to the Federal Government totaling approximately $436,132.81, which is the total contract cost for project planning and implementing the data collection for the three studies, and personnel costs of federal employees involved in oversight and analysis.
Cost Category |
Salary* |
% of Effort |
Cost |
Total Cost to Government |
Federal Oversight |
$85,000.00 |
45 |
|
38250 |
|
|
|
|
0 |
|
|
|
|
0 |
|
|
|
|
0 |
Contractor Cost |
|
|
Base Year - Direct Costs |
399,315.20 |
|
|
|
Base Year - Travel |
36,817.61 |
|
|
|
Total Base |
436,132.81 |
|
|
|
Option Year -Direct Costs |
843,341.43 |
|
|
|
Option Year -Travel |
116,836.77 |
|
|
|
Option Year One Total |
960,178.20 |
|
|
|
Option Year Two -Direct Costs |
647,799.10 |
|
|
|
Option Year Two -Travel |
65,196.48 |
|
|
|
Option Year Two Total |
712,995.58 |
|
|
|
Option Year Three - Direct Costs |
384,991.22 |
|
|
|
Option Year Three - Travel |
9,480.09 |
|
|
|
Option Year Three - Total |
394,471.31 |
Total |
|
|
|
2,542,027.90 |
15. Changes in Burden
Explain the reasons for any program changes or adjustments reported in sections 12.
This is an application for new data collection. There are no program changes.
16. Time Schedule, Publication and Analysis Plans
For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.
Time Schedule
The proposed online survey to be used for the studies will be administered at:
Baseline (first survey): prior to training, before members report to their campus. Recruitment of members and comparison group will occur at least two months prior to members reporting for their service.
Exit Survey (early exit members): members who leave early will be administered a short survey that includes four questions to measure their reasons for leaving as well as their perception about the service experience and how this experience may have contributed to their decision to leave. It is projected that members will begin to exit as early as March 2020 and up to the end of the term of service (10 months for Traditional NCCC and 12 months for FEMA Corps).
Follow-up (second survey): The second survey will be administered near the end of the term of service to members and comparison group respondents who completed the baseline survey.
Follow-up (third/final survey): Approximately one year following the end of the term of service. All participants who completed the second survey.
Qualitative Data Collection
The qualitative data for the leadership development and member retention studies will be collected through focus groups and semi-structured interviews from active Corps members, NCCC staff, NCCC Sponsors, and FEMA point of contacts. There will be two data collection points. The first will occur onsite at each campus near the end of members’ service year. The second data collection point will be a phone interview approximately one month after the term of service ended with the same members who participated in the onsite focus groups.
The data collection for the strengthening communities study will take place in site visits and follow-up phone interviews. Data collection that occurs during site visits will include in-depth interviews and focus groups with members, leaders, site sponsors and supervisors, and community stakeholders; observation; and a walking or windshield tour of the parts of the community in which NCCC is working. Follow-up data collection for comprehensive site visits will be conducted by phone with sponsors, FEMA point of contacts, site supervisors and community stakeholders. In addition, selected sites will also have ripple effects mapping workshops for combinations of sponsors, members, and community stakeholders already participating in interviews and focus groups.
Figure 1 shows the data collection timeline. Assuming enrollment into the study begins with Class 26 (Winter cycle), the data collection period for the baseline and first follow-up surveys will span 32 months. The baseline survey data collection period is proposed from December, 2019 to September 2020. The first follow-up survey is proposed from November 2020 to August 2021. The second follow-up survey is proposed from November 2021 to August 2022. The member early exit survey will be from March 2020 to June 2021. The qualitative data collection for the leadership development and member retention studies is proposed from September 2020 to September 2021. The first data collection point for the strengthening communities study is proposed from January 2021 to September 2021; the second data collection is proposed from January 2022 to September 2022. The analysis and report period spans 12 additional months from October 2020 to November 2023.
Figure 1 NCCC Longitudinal Evaluation Timeline
Publication Plan
Reporting and dissemination of results will be mainly in the form of issue briefs and reports with supporting graphs and tables on key data points. For internal audiences at CNCS and the NCCC campuses, these will be focused on information relevant to program improvement and enhancement. For external audiences, an evaluation report will be made available detailing the steps taken to conduct the evaluation and presenting the results; an emphasis will be made on the strengths and limitations of the data and corresponding analyses to ensure appropriate use of results. The data gathered from the evaluation may be used in analysis and planning work for other program evaluations and research projects conducted by CNCS as applicable.
The evaluator will analyze the data and provide CNCS a report for each study at the end of each data collection period (baseline, first and second follow-up). The reports will describe the evaluation findings for each round of data collection. The reports may be in the form of manuscripts intended for academic peer-reviewed journals. The evaluator will provide supporting materials that will include data files and a user’s guide for future research. CNCS will include the evaluation results in its Performance and Accountability Reporting to Congress and in a subsequent Congressional Budget Justification.
Analysis Plan
The Quasi Experimental Design will provide rigorous estimates of the impact of participation in NCCC on members’ leadership qualities and skills and factors affecting member retention. Implementing the QED involves:
Collecting surveys from treatment and comparison group participants at three time points. The measures to be collected through the surveys include background and demographic characteristics, prior volunteer service, and leadership qualities and skills.
Applying matching methods to construct equivalent comparison groups, which are comprised of individuals with the same observable characteristics as participants in the treatment group.
Producing rigorous estimates of program impacts through outcome comparisons between the treatment and the comparison groups.
Matching methods are a reliable approach for producing rigorous impact evaluations, particularly since a random assignment design is not feasible in this context. Matching methods provide credible impact estimates when: 1) the data include large samples of non-members; and 2) matching is performed based on rich information on member and non-member characteristics. Rich data collection on treatment and comparison group participants is necessary to ensure that observable characteristics influencing program participation can be sufficiently approximated.
Comparison group participants should ideally be as similar as possible to Corps members prior to statistical adjustments in observable background characteristics as Corps members prior to the start of service. To construct the matched comparison group, the evaluator will rely on survey responses and program administrative data which will provide information on the characteristics of all applicants who applied for membership in the NCCC program. The available sample for the comparison group will need to be large enough to locate matches for Corps members in the treatment group.
There are several ways researchers use propensity scores to reduce confounding factors, including matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting (IPTW) using the propensity score, and covariate adjustment using the propensity score (Austin, 2016; ; Austin & Stuart, 2015; Thoemmes & Ong, 2015; Stuart, 2010). For example, IPTW or full matching could be used if the decision based on the available data is to estimate the average treatment effect (ATE). Alternatively, given the possibility that there might not be as many comparison participants as treatment participants, appropriate matching choices might be subclassification and weighting by the odds to estimate the average treatment on treated (ATT).
Once matching is achieved, it is necessary to test if treatment and comparison individuals share similar characteristics. These tests involve comparisons of variable means and standard deviations between the treatment and the comparison group. If treatment-comparison differences in characteristics remain, the evaluator will modify the model specification used to match (e.g., include polynomials to capture nonlinearities or multiplicative terms to capture variable interactions, changing the method of matching) to eliminate such differences and ensure that a successful matching is achieved. This is a particularly necessary step for the internal validity of the impact analysis.
Impact Analysis. To estimate the impact of NCCC on members’ leadership qualities and skills, the evaluator will examine outcome differences between the treatment group and the matched comparison cases. To estimate the impact with increased statistical efficiency, the evaluator will use multivariate multiple regression models using the post-match sample, controlling for baseline characteristics. The dependent variable in this model will be each participant’s leadership qualities and skills measured by professional skills, life skills, team behavior, and community and civic engagement. The parameter constituting the ‘impact estimate’ in this model will be the regression-adjusted treatment effect of NCCC participation on each outcome of interest. It is anticipated that the outcome measures will likely not be normally distributed across participants. There are several regression models that can be explored such as Poisson, negative binomial or logistic, whereby adjustments can be made to the model estimation procedures such that the model makes fewer assumptions of outcome distribution normality.
The dependent variables in this model - professional skills, life skills, team behavior, and community and civic engagement will be used to assess each participant’s leadership qualities and skills development. The four dependent variables will be aggregated measures of survey item responses.
An exploratory factor analysis will be conducted to assess loadings of each item on the dependent variables, ostensibly to support the model framework, in particular the validity of the aggregation scores. Additionally, factor analysis will be used to assess the relative value of each measure for future modifications to the instrument.
There will also be an analysis to examine the impact of service projects on members’ leadership qualities and skills, and their impact on member retention. This analysis will include the sample of Corps members only and is a mixed-methods approach:
Merging Corps members survey responses with CNCS service projects database. The data on service projects will be important to gain exploratory insight into the heterogeneity of members’ experiences, and the association that the nature of participation activities shares with leadership qualities and skills, and member retention.
Triangulating Corps members survey responses with focus groups and semi-structured interviews data.
Content and Thematic Analysis. The quantitative analysis will be triangulated with the focus groups and semi-structured interview data. The evaluator will analyze the focus groups and semi-structured interviews data collected from members, NCCC staff, project sponsors and FEMA POCs to determine what factors contributed to leadership development in members while serving with NCCC. The narrative responses gathered from members will be used to complement the outcomes from the quantitative leadership surveys.
Table 4 summarizes the research questions, data sources and sample, and analysis approach for the leadership development study.
Table 4 Leadership Development Study: Research Questions, Data Sources and Sample, analysis
Table 4 Continued
Member Retention Study
Since the leadership development and retention studies share the same questionnaire, the measures for both studies are similar, except reasons for leaving which will be measured for those who leave service early. Like the leadership development study, the retention study will use a QED design. However, there will be two comparison groups. The treatment group will be Corps members who complete their term of service; one comparison group will consist of members who leave early and do not complete their term of service, and the second comparison group will be accepted applicants who declined to serve. The impact analysis will be conducted twice, at the end of service and again at one year following the completion of service. The QED will be implemented as previously described under the leadership study with one modification to the impact analysis since there are two comparison groups:
Impact Analysis. The evaluator will examine outcome differences between the treatment group (members who complete service), comparison group 1 (members who leave early), and comparison group 2 (accepted applicants who declined to serve). The parameter constituting the ‘impact estimate’ will be the regression-adjusted group membership effect of NCCC on each outcome of interest. To estimate the impact with increased statistical efficiency, the evaluator will use multivariate multiple regression models using the post-match samples, controlling for baseline characteristics.
To estimate the impact of the NCCC service experience on member retention, the evaluator will merge the survey responses with the data in the CNCS’s service projects database to examine outcome differences between the treatment group (members who complete service) and members who leave service early (comparison group 1). Comparison group 2 will not be included in this analysis regarding the impact of service experience. To estimate the impact with increased statistical efficiency, the evaluator will use multivariate multilevel regression models using the post-match sample, controlling for baseline characteristics. The dependent variable in this model will be a dichotomous variable measuring whether the member persisted through graduation.
Content and Thematic Analysis. Thematic analysis will be applied to the data collected from interviews and focus groups with members, NCCC staff, project sponsors and FEMA POCs to determine what factor played a role in member’s retention in the programs.
Table 5 summarizes the research questions, data sources and sample, and analysis approach for the member retention study.
Table 5 Member Retention: Research Questions, Data Sources and Sample, analysis
Research Questions |
Data sources / Sample |
Description of analysis |
Analysis |
|
Baseline and 1st follow-up surveys / sample of members only, divided in two groups: members who complete service compared to members who exit service early. |
Examine how the two groups of members differ in background characteristics, incoming leadership skills, motivation, sources where and how they learned about NCCC, knowledge and perception of NCCC (degree of mismatch in how they perceive the service experience). |
Descriptive analysis, chi-square test, t-test; latent class analysis to identify types of members base on motivation, sources where members learned about NCCC, differences in knowledge of NCCC. |
|
Baseline and 1st follow-up surveys / three groups: members who complete service, members who exit early, accepted applicants who do not serve (comparison group). |
Examine how the two groups of members (complete service, early exit) differ from comparison group (accepted applicants who do not serve) in background, interpersonal skills, and life skills; establish baseline equivalence between the two groups using propensity score methods to construct matched comparison cases.
|
Descriptive analysis, t-test or non-parametric test (e.g., Wilcoxon signed-rank test) to examine differences in observed characteristics; propensity score methods to construct matched comparison cases for members who leave early. |
Table 5 continued
Research Questions |
Data sources/Sample |
Description of analysis |
Analysis |
|
Baseline and 1st follow-up surveys; CNCS service projects database / sample of members only, two groups (complete serve, early exit). |
Examine personal and organizational/institutional factors that contribute to member retention. |
Multilevel logistic regression. |
|
Baseline, 1st and 2nd follow-up surveys; qualitative data from focus groups and interviews / two groups: members who complete service and those who exit early. |
Baseline and 1st follow-up: using the two groups of members (complete service, early exit) examine differences in change in leadership outcomes. Baseline, 1st and 2nd follow-up: using the two groups of members to examine the differences in growth in leadership outcomes over time. |
Impact analysis: multivariate multiple regression models; latent growth curve model. Content and thematic analysis of qualitative data. |
|
Baseline, 1st and 2nd follow-up surveys; qualitative data from focus groups and interviews / two groups: early exit and matched comparison cases. |
Baseline and 1st follow-up: using the sample of early exit and matched comparison cases examine difference in change over time in leadership outcomes. Baseline, 1st and 2nd follow-up: using the sample of early exit and matched comparison cases to examine the difference in growth in leadership outcomes over time. |
Impact analysis: multivariate multiple regression models; latent growth curve model. Content and thematic analysis of qualitative data. |
Strengthening Communities Study
The strengthening communities study uses an exploratory mixed-methods approach that includes quantitative analysis of the CNCS service projects database to document how NCCC has strengthened communities. This is supplemented with a qualitative approach that includes in-depth case studies of project sites. The evaluator will review data from the service projects database of all projects conducted since 2012. The database includes project characteristics and outputs as well as community benefit outcomes derived from project completion report narratives. The evaluator will use the results of analysis of the service projects database to select approximately 20 projects for case studies, which may be comprehensive or restricted. The evaluator will conduct comprehensive case studies with a minimum of six service projects. At least four case studies will focus on active Traditional NCCC projects from all four active campuses and representing different issue areas; two will focus on active FEMA Corps projects, including one steady-state project and one active or formerly active disaster site. The evaluator will conduct restricted case studies with up to 12 additional projects, including both active and past projects. The restrictive case studies will be smaller in scope relative to the comprehensive case studies and will be used to assess for congruence or divergence on prominent themes . The final number of projects selected for case studies will be determined based on the results of analysis of the service projects database and subsequent decisions about prioritized sponsor characteristics and project outcomes.
Analysis. The quantitative analysis of the service projects database will be primarily descriptive, though it may include some exploration of bivariate and multivariate relationships between sponsor and project characteristics and outcomes. For case studies, the evaluator will review project documentation and primary qualitative data collections (e.g., from site visits, phone interviews). The qualitative analysis will include a holistic overview of each case resulting in individual case descriptions, analysis and reporting of emergent themes within each case (within-case analysis), and thematic analysis across cases (cross-case analysis) (Creswell, 2005; Yin, 2003). The case studies will also include descriptive quantitative data relevant to each project using the service projects database, including project characteristics and outputs derived from sponsor applications, project completion reports, and other supporting documentation, and descriptive data about community characteristics obtained from the Census. Table 6 summarizes the research questions, data sources and sample, and analysis approach for the strengthening communities study.
Table 6 Strengthening Communities: Research Questions, Data Sources and Sample, analysis
17. Display of Expiration Date
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
The expiration date for OMB approval of the information collection will be displayed.
18. Exceptions to Certification Statement
Explain each exception to the certification statement identified in "Certification for Paperwork Reduction Act Submissions."
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Annie Georges |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |
© 2024 OMB.report | Privacy Policy