3045-NEW PHA Member outcome Surveys_SSA_FINAL
3045-NEW PHA Member outcome Surveys_SSA_FINAL.docx
Evaluation of Public Health AmeriCorps
OMB: 3045-0199
Corporation for National and Community Service (dba AmeriCorps)
Evaluation of Public Health AmeriCorps
OMB Control Number 3045-NEW
Part A. Justification
Overview of Information Collection
This is a request for approval of a New Collection. This information collection includes two survey instruments (one for members, one for grantees), a focus group protocol guide for grantees, an interview protocol guide for early exit members, an interview protocol guide for grantees’ partners, and a technical assistance and training form for grantees. Together, these will allow AmeriCorps to examine the extent to which Public Health AmeriCorps is progressing toward its three main goals:
Engage AmeriCorps members in addressing local public health needs
Advance equitable health outcomes for underserved communities
Create pathways to public health-related careers.
Need & Method for the Information Collection.
Public Health AmeriCorps is an investment from the American Rescue Plan of 2021, in support of President Biden’s Executive Order 13996 on Establishing the COVID-19 Pandemic Testing Board and Ensuring a Sustainable Public Health Workforce for COVID-19 and Other Biological Threats, Sec. 4, Establishing a Public Health Workforce Program. 86 Fed. Reg. 7197 (January 26, 2021). This funding established a partnership between AmeriCorps and the Centers for Disease Control (CDC) to enable recruitment and training of a workforce through national service that is ready to respond to the public health needs of the nation, while providing public health services in communities around the country. This partnership capitalizes on AmeriCorps’ experience managing national service and workforce development programs, and it benefits from CDC’s technical expertise as the country’s leading public health agency. The Public Health AmeriCorps program is opening the door for a career in public health to a new generation, bringing new talent to conduct urgent public health services. The program will also help communities address the broader public health needs of vulnerable communities exposed and exacerbated by the COVID-19 pandemic. This initiative expects to build on expertise, best practices, and lessons learned from existing AmeriCorps, CDC, and other public health programs. During their term of service, members are expected to develop core competencies of public health services and the ability to effectively serve their communities as well as build strong organizational and communication skills. These skills are expected to increase members’ knowledge of public health and provide them with training and real-world experience through national service to enter the public health field post-service.
The information collection is necessary to evaluate the success of Public Health AmeriCorps in meeting its goals; if this information collection is not conducted, AmeriCorps and CDC will not have the data upon which to measure the success of the program. AmeriCorps is conducting this collection and evaluation under the authority of Section 179 of the National and Community Service Act, as amended, 42 U.S.C. 12639.
Need for the evaluation
Public Health AmeriCorps is unique in its approach to training and developing a public health workforce. There are existing public health training and workforce development programs such as the HRSA Regional Public Health Training Center Network (PHTCN), the CDC Undergraduate Public Health Scholars Program (CUPS), and the CDC Public Health Associate Program (PHAP). None of these existing programs uses the national service model. Consequently, there is a knowledge and evidence gap in how effective the national service model is in building a public health workforce pipeline. The national evaluation of Public Health AmeriCorps will address this gap. The evaluation will generate credible, relevant, and actionable information about the effectiveness of Public Health AmeriCorps in facilitating and expanding the public health workforce in underserved communities. The evaluation will also examine the effectiveness of the CDC and AmeriCorps partnership, as well as the partnerships between Public Health AmeriCorps grantees, host sites, and state and local public health agencies. Together, these partnerships are intended to facilitate the implementation and impact of the program in creating a public health workforce pipeline to support public health infrastructure in underserved communities.
The information will be collected using the following instruments:
Member Alumni Survey (to be administered online):
Module 1: Core Questions (All Respondents) – how the member learned of PHA, the reasons they joined PHA, their prior experience in service, volunteering, and public health, challenges and barriers they faced during service, types of service activities engaged in, training and resources during service, and skills and knowledge gained from their PHA service, satisfaction with their service, whether national service helped fulfill their goals.
Module 2: Short-Term Plans and Current Employment and Career (All Respondents) – intentions post-service, education award use, current employment
Module 3: Early Exit Reasons (to be administered only to members who left before the end of their service term) – reasons for leaving service early
Module 4: Career Support and Career Impact (All Respondents)–effect of service on interest in a career in public health, use of career support during service, and impact of service on career path.
Module 5: Demographics (All Respondents) – demographics and background information
Grantee Survey (to be administered online):
Program design and implementation, communities served, recruitment, retention, and training of members, professional support, and potential career pathways for members, grantee partnerships.
Grantee Focus Group (virtual meeting):
Script for contractor to lead focus group discussion with grantees; the discussion includes how the grantee addressed Public Health AmeriCorps’ goals, successes, and challenges in implementing their project. There are three focus groups:
Focus Group on Member Recruitment and Retention – additional questions specific to how the grantees recruited, successes and challenges in recruitment, and how AmeriCorps could help with recruitment and retention.
Focus Group on Member Training and Support – additional questions specific to public health training provided and needed.
Focus Group on Partnerships – additional questions specific to whether grantees have worked with partnership in implementing their PHA grants, whether there have been additions or loss of any partnership, and challenges in building partnerships.
Partner Interview Guide (virtual meeting)
Script for contractor to lead interview with a sample of partners of Public Health AmeriCorps grantees; the discussion includes how the partnership was established, partner’s role and responsibilities, goals of the partnership, and outcome of the partnership to date.
Early Exit Interview Guide (virtual meeting)
Script for contractor to lead interview with a sample of early exit members; the discussion includes challenges that influenced the decision to end service, support and resources to overcome the challenges, recommendations on support that could mitigate early exit.
Grantee Technical Assistance and Training Form (to be administered online)
Requests grantee responses as to challenges they have faced in operating their program, what kind of training AmeriCorps could offer to members to enhance their impact, and any lessons learned in operating their program.
Consent forms each type of respondent (alumni, early exit members, and grantees and partners).
Use of the Information.
AmeriCorps will use the information for regulatory compliance as discussed under the response to question 1, above, as well as program administration, and to inform policy and guidance affecting the agency’s Public Health AmeriCorps grant funding. The information will assist AmeriCorps, in partnership with CDC, in establishing strategic goals and performance targets and informing program innovation, guiding program improvement, and demonstrating program impacts. Furthermore, beyond the legislative requirements to report to Congress on the program’s impact, in adhering to the 2019 Foundations for Evidence-Based Policymaking Act (FEBP) of 2018, Pub. L. 115-435, AmeriCorps has outlined a commitment to building an evidence base to inform decision-making and the allocation of resources in the agency’s strategic planning and Learning Agenda.
Beginning-to-end experience for respondents
Members will receive a link for the survey via email. No learning costs are anticipated, as the survey is fillable online. Any psychological costs will be minimal because the survey is optional to complete and there are minimal questions of an intrusive nature. For sensitive questions, members can elect to not provide a response to questions by selecting “prefer not to answer.” Any member who left service before their term expired could have a reason for doing so that causes discomfort, but the listed reasons for early departure are written in a non-judgmental and objective manner.
Grantees will receive a link to for the grantee survey and technical assistance and training form via email. No learning costs are anticipated, as the survey is fillable online. No psychological costs are anticipated, as the individual will be completing the information on behalf of their grantee organization.
Grantees opt into participating in the focus group and are not likely to experience any learning costs as the focus group is in a meeting format. No psychological costs are anticipated as the individual representatives may decline to respond to any question during the focus group.
AmeriCorps will share the information as an agency report to Congress and with the public through its website. The data collection will inform agency policy. All published reports will protect privacy as no personally identifiable information will be disclosed.
Specific Research Questions the Study will Address
AmeriCorps will conduct three concurrent national evaluations of Public Health AmeriCorps: a process evaluation, outcome evaluation, and impact evaluation.
Process evaluation focuses on the grantees and subgrantees (hereafter “grantees” for brevity) and their partner organizations. This evaluation is dynamic as it will be updated and reviewed each year during the contract period to gauge the progress Public Health AmeriCorps is making toward meeting its goals. The unit of analysis is the grantee. The process evaluation examines the successes and challenges in implementing the program; grantees’ measures of performance, progress, and effectiveness; types of training and support provided to members; and the partnership between grantees and local and state public health departments as well as private and other types of health institutions and clinics.
The research questions under the process evaluation cover four topic areas: grantee characteristics, program implementation, member recruitment, member training and support, and partnership and collaboration with other organizations. Grantee characteristics will consider commonalities and differences across programs. The evidence from the process evaluation will identify successes, challenges, and lessons learned in implementing Public Health AmeriCorps; the evidence will inform potential changes in policies and program guidance set forth by AmeriCorps to support and strengthen grantees' recruitment and retention efforts.
The evaluation questions under grantee characteristics and program implementation are as follows:
What are the commonalities across Public Health AmeriCorps programs? How do programs differ?
What are the successes and challenges of implementing Public Health AmeriCorps?
How well are Public Health AmeriCorps programs being implemented?
What measures do Public Health AmeriCorps grantees propose to use to gauge performance, progress, and program effectiveness? How reliable and valid are the proposed measures?
Regarding member recruitment, organizations often use a variety of strategies such as outreach events, social media campaigns, and word-of-mouth referrals. One of the goals of the evaluation is to assess the effectiveness of Public Health AmeriCorps in facilitating and expanding the public health workforce in underserved communities. A hypothesis of how Public Health AmeriCorps might achieve an increased public health workforce pipeline in underserved communities is by recruiting members from these communities, and ensuring the members serve in their own communities. Additionally, members that serve in the communities they are recruited from can help build trust and foster stronger engagement between the organization and the community. Member recruitment-related questions are as follows:
What strategies do Public Health AmeriCorps programs use to recruit members?
Are members recruited from the communities the grantees serve? Do members serve in the communities they are recruited from?
The questions on the training and support grantees provide include the type of training, certifications, and credentials that Public Health AmeriCorps members receive, and whether these trainings are evidence-based. The process evaluation will examine professional support, and training and workforce development support provided to members that can pave the way to a public health career. The process evaluation will examine the presence of systemic and structural barriers that might affect members’ experience and identify the services or resources that can ensure a successful transition to a public health career. The process evaluation also considers the broader context of members’ entry into the public health workforce. The evaluation questions related to the training and support grantees provide are as follows:
What types of trainings, certifications and credentials do members receive?
Are the trainings Public Health AmeriCorps members receiving evidence-based?
What kind of professional supports are provided to members?
What additional education, training, workforce development supports do members need to achieve their longer-term public health career goals?
What systemic and structural barriers do grantees anticipate members might experience? What services or resources have grantees put in place or identify to address these barriers?
How does AmeriCorps address the structural and systemic barriers that members experience? Are there additional supports and services that AmeriCorps could provide?
What ongoing health-related careers opportunities do grantees offer to broaden members’ career pathways in public health?
Partnership and collaboration with other organizations are important because the partners may serve as host sites where members serve, and these partners may provide members opportunities for a career in public health. The research questions for the process evaluation related to partnership and collaboration with other organizations are as follows:
What is the relationship, if any, between grantee organizations and local or state health departments?
How does a relationship with local or state health departments impact grantees’ programs and processes?
To what extent was the partnership between AmeriCorps and CDC successful or offered a model for cross agency collaboration, including at local, state, and federal levels?
What are the lessons learned from AmeriCorps and CDC collaboration and partnership effort that can be used to inform future cross agency partnership efforts?
At what stage(s) of the initiative’s life cycle was the cross-agency partnership most (or least) critical in terms of collaboration, communication, and coordination?
Outcome evaluation focuses on members and the communities served. The research questions for the outcome evaluation examine members’ public health career pathways as opposed to non-public health careers; core public health competencies members develop during their service; and members’ civic participation and commitment to national service. The evaluation’s research questions aim to investigate several aspects of members’ outcomes following service and the communities where the members served. The evidence from the outcome evaluation will assess how Public Health AmeriCorps strengthens capacity to address public health needs, especially in communities that have been historically underserved, and how recruiting members from the communities that grantees serve affects both the communities and members. The research questions examine members’ public health interaction, including the roles they serve and whether they perceive their service as impactful; and determine how the program strengthens capacity to address public health needs in underserved communities. The research questions examine how recruiting members from the communities that grantees serve affects both the communities served and the members themselves.
The research questions for the outcome evaluation cover four main themes. These themes include factors affecting members as they enter the service terms; the competencies, knowledge, and skills members develop through the service; the roles and support members provide to their grantees; and the extent members’ service activities address the communities' public health needs.
As for the factors affecting members entering the service terms, we aim to investigate the potential barriers and facilitators they encounter during their service terms. The research questions to examine the effect of Public Health AmeriCorps service on members’ competencies, knowledge, and skills development are:
To what extent do members find the training and skill building provided sufficient for supporting them in their service activities?
What types of competencies, knowledge, and skills do members develop?
What kinds of professional support helped members succeed in their career search and careers after their Public Health AmeriCorps service?
The outcome evaluation examines members’ roles and how their service activities support grantees and host sites. The research questions are:
What types of roles do members serve within their grantee organizations and/or host sites?
How do members interact with public health on a day-to-day basis?
What are members doing to support their grantees’ public health goals?
The outcome evaluation also explores how members’ service activities address the public health needs of underserved communities. The research questions guiding this objective are:
How do Public Health AmeriCorps alumni compare to the alumni of other public health training programs or experiences that offer similar opportunities to address community needs?
To what extent does Public Health AmeriCorps members build or strengthen capacity and support state and local public health settings to address the needs of underserved communities?
In what ways does participating in a Public Health AmeriCorps National Service program influence members’ civic participation?
Impact evaluation focuses on members and the communities served. The impact evaluation will use a benchmark comparison to measure the attainment of public health employment.
The research questions for the impact evaluation are designed to determine if participating in these programs leads members to public health-related careers by providing them with valuable onsite experience and training; whether participation influences members to pursue or obtain credentials, certifications, or degrees in a public health occupation. Additionally, the research questions address the impact of Public Health AmeriCorps on members recruited from within the communities in which they serve, and whether that influences members’ civic participation and commitment to national and community service.
By addressing these research questions in-depth, we can better understand the effectiveness of Public Health AmeriCorps programs, and examine members’ public health career pathways, civic participation, and commitment to national service. The research questions for the impact evaluation are:
What is the association between participation in Public Health AmeriCorps programs and the likelihood of pursuing public health-related careers?
What is the contribution of type of exposure through onsite public health service experience and training?
What is the association between participation in Public Health AmeriCorps program and the likelihood of pursuing credentials, certifications, or degrees in a public health occupation?
How do public health education and career outcomes differ between members recruited from the communities and members are not recruited from the communities they serve?
Strength and weaknesses
The information collection through this study demonstrates a strength: repeated cross sectional to capture emerging trends in the program’s early stages; the design also incorporates a longitudinal component that can capture medium-term outcomes particularly in expanding the public health workforce. The pre/post aspect of the data collection permits the analysis to strengthen the evidence of employment outcome and public health careers post-service. The ability to secure public health employment may take time, as some alumni may continue with higher education. It may also take longer to secure public health employment due to external barriers such as general labor market conditions, public health hiring and governmental public health careers, and opportunities in target communities by the public and private health sectors may be inelastic. The pre/post aspect of the data collection allows for a longer time frame to measure alumni’s employment outcomes.
The weakness is that the design cannot identify a comparison or control group to address selection bias. A selection bias may exist if individuals that enroll and complete their term of service were already predisposed to enter the public health workforce. To address this weakness by using benchmark analysis. The Public Health AmeriCorps survey data will be benchmarked with nationally available and comparable datasets to estimate program impacts. JBS has identified five potential data sources as shown in Table 1. These data sources include comparable programs that are nationally recognized public health training programs. As part of the benchmarking analysis based on programmatic characteristics, JBS plans to subset both internal Public Health AmeriCorps data and external benchmarking data based on member characteristics. To ensure valid comparison, JBS will use comparable measures from the member survey and the external data sources. The benchmarking data may be obtained through surveys, program reports, or publicly available sources. The early exit member survey will be used as a comparison to alumni. AmeriCorps Member Exit Survey (MES) is a potential source for benchmark analysis. The MES collects data from all AmeriCorps members on program experience, civic engagement, and commitment to national service and community service. The MES data can be used to compare Public Health AmeriCorps members to other AmeriCorps members on their civic engagement and commitment to national service.
Table 1. Proposed Benchmark and Comparison Groups.
Benchmark Data Source |
Potential Similarities in Scale |
Comparable Research Topic(s) |
Comparable Research Impact(s) |
HRSA Regional Public Health Training Center Network (PHTCN) |
National public health training program that provides field placements.
The program has young adult participants.
Some participants had a similar range of public health work experience before program participation.
Some participants had a similar range of public health skills /competencies before program participation. |
Individual employment in the public health workforce after program completion. |
Increasing the size and diversity of the public health workforce.
|
Association of Schools and Programs of Public Health (ASPPH) |
National program
A portion of the graduating classes are young adults.
Some graduates had a similar range of public health work experience before program participation.
Some graduates had a similar range of public health skills/competencies before program participation. |
Individual employment in the public health workforce after program completion. |
Increasing the size of the public health workforce. |
CDC Undergraduate Public Health Scholars Program (CUPS) |
National public health training program that provides field placements.
The program has young adult participants.
Participants had a similar range of public health work experience before program participation.
Participants had a similar range of public health skills/competencies before program participation. |
Individual employment in the public health workforce after program completion.
Individual enrollment in a public health graduate program after program completion.
|
Increasing the size and diversity of the public health workforce. |
CDC Public Health Associate Program (PHAP) |
National public health training program that provides field placements.
Young adult participants
Participants had a similar range of public health work experience before program participation.
Participants may have a similar range of public health skills/competencies before program participation. |
Individual employment in the public health workforce after program completion.
Individual enrollment in a public health graduate program after program completion.
|
Increasing the size and diversity of the public health workforce. |
Public Health Workforce Interests and Needs Survey (PH WINS) |
National sample
Potentially some young adult participants
Young adult participants may have a similar range of public health work experience before program participation.
Young adult participants may have a similar range of public health skills/competencies before program participation. |
Public health program areas participants are working/serving in. |
Public health program areas that need more professionals in the public health workforce pipeline. |
AmeriCorps Member Exit Survey (MES) |
National sample of AmeriCorps members
Some participants are young adults. |
Civic participation
Commitment to national service
|
AmeriCorps’ impact on members’ civic participation and commitment to national service and community service. |
Use of Information Technology.
The surveys will be administered online, via a link the members and grantees receive by email. The focus group will use online meeting tools to allow the participation of individuals in various geographic areas. JBS will use Alchemer, an online survey, for the survey administration. The use of technology will minimize burden in multiple ways. This online platform allows respondent to access the survey on any device, to pause and resume the survey at will. The interviews and focus groups with grantees, partners, and early exit members will use online meeting tools, to allow the participation of individuals in various geographic areas.
The consent form and first page of the survey will include an email and toll-free number to contact the research staff. In addition, respondents can contact the research staff via email or phone to complete the survey by telephone if this is the respondent’s preferred method to participate.
Non-duplication.
This information is not collected by any other entity. No similar information is available, as the Public Health AmeriCorps program is new and unique.
There are no other sources of information by which AmeriCorps can meet the purposes described in the response to question 2, above. The proposed evaluation will three primary objectives:
Identify successes, challenges, and lessons learned in implementing Public Health AmeriCorps (process evaluation)
Assess how Public Health AmeriCorps strengthens capacity to address public health needs, especially in communities that have been historically underserved, and how recruiting members from the communities that grantees serve affects both the communities and members (outcome evaluation)
Examine members’ public health career pathways, civic participation, and commitment to national service (impact evaluation)
There is currently no evidence-based data on using the national service model to train and develop a public health workforce as Public Health AmeriCorps is unique in this approach.
Burden on Small Business.
This information collection does not impact small businesses, but AmeriCorps has developed the questions to impose as little burden as possible in obtaining the information.
Less Frequent Collection.
If this collection is not conducted, then AmeriCorps will lack any data on members’ and grantees’ experiences with Public Health AmeriCorps.
Paperwork Reduction Act Guidelines.
This information collection fully complies with 5 CFR 1320.5(d)(2). There are no special circumstances that would require the collection of information in any other ways specified.
Consultation and Public Comments.
AmeriCorps published a notice in the Federal Register on September 7, 2023, providing 60 days for public comment; see 88 FR 61575. In response, AmeriCorps received one comment on the collection. The comment period ended November 6, 2023. One public comment was received from this Notice. The comments that were received inquired about 1) data collection and data ownership, 2) the purpose and need for the Public Health AmeriCorps Technical Assistance and Training Form, and 3) the length of the Public Health AmeriCorps Outcome Evaluation Draft survey.
The concern regarding data collection and data ownership is valid. The contractor, JBS International will manage and securely store all data collected throughout the duration of the contract. During the contract period, data will be provided to the Office of Research and Evaluation in aggregate form. Once the contract ends, the contractor will deliver a de-identified public use data file and a restricted use data file to the Office of Research and Evaluation. The data will be securely stored in the Office of Research and Evaluation and AmeriCorps programs will not have access to any files. If program team representatives submit inquiries about the evaluation, they will be able to receive information in aggregate. The Public Health AmeriCorps Technical Assistance and Training form was a requested instrument from the program team. This instrument will allow for the contractor to distill any information that wasn’t already collected in the grantee survey or focus group guide. The instrument will be used as a supplemental tool to capture any additional information that wasn’t collected in other activities. The evaluation team has been mindful of the length and burden of the member survey. The survey has gone through several iterations and has been shortened. The contractor piloted the instrument with up to nine members including cognitive interviews after the members completed the survey. The version the contractor piloted took pilot participants on average less than 30 minutes and did not express dissatisfaction with the length of the survey. The reason for keeping questions that are also found in the Member Exit Survey (MES) is that participants in the Public Health AmeriCorps evaluation may not complete the MES, and consequently it would not be possible to merge the two surveys. It is important for the Public Health AmeriCorps survey to include complete data to address the evaluation questions, and to create more complete datasets across the agency. We want to ensure we close any gaps (intake information and member surveys) and confirm information already received. The purpose of reaching out to members that exited early is to better understand their experience and challenges they encountered in completing their service. This data may inform programmatic change to minimize attrition among members. We will also benchmark the Public Health AmeriCorps survey to other similar public health program to see how Public Health AmeriCorps is achieving the goal of creating a pathway to public health careers.
In addition, AmeriCorps consulted with six Field Working Group members consisting of one AmeriCorps State and National (ASN) program impact specialist, one Office of Diversity, Equity, Inclusion, and Accessibility (DEIA), two Office of Regional Operations (ORO) portfolio managers, and two ORO senior portfolio managers. The consultations were to obtain their views on the availability of data, the administration and frequency of data collection and instruments, and the approach to enrolling members in the study.
AmeriCorps also consulted with six Technical Working Group members who bring a diverse range of experience and expertise in research and evaluation on public health workforce, data collection, sampling, analysis, and dissemination of scientific studies.
AmeriCorps published a second notice in the Federal Register on February 6, 2024, providing 30 days for public comment. See 89 Fed. Reg. 8184.
Gifts or Payment.
AmeriCorps anticipates that Public Health AmeriCorps members and grantees will fully cooperate to participate in the study. However, enrollment of the alumni (those who completed their service term and have not re-enrolled for another service term) and Early Exits (members who did not complete their term of service and have not re-enrolled for another service either with Public Health AmeriCorps or other AmeriCorps national service program) will be challenging. Public Health AmeriCorps does not maintain a primary relationship with former members. Once these members leave the program, they are disengaged in communicating with Public Health AmeriCorps. Consequently, it is anticipated there will be challenges with engaging and enrolling them into the study. The data collection includes a communication strategy to engage during their service, at the time of completing their service or early exit to maximize response rate. Particularly, Public Health AmeriCorps program staff, portfolio managers have set up processes to gather and update members’ contact information (e.g., current email and phone number). The mailing distribution is critical in reaching the respondents for the study. The data collection instruments are short and respondents only answer questions that apply to them; at the time of exit (whether because the member completed their service or ended their service early), each member and grantee will receive a two-page summary (Appendix A) describing the value and relevancy of the data collection; in addition program staff and ORE staff will frequently update grantees about the study through recurring monthly grantee meetings. A toll-free number and email will be available for all respondents to contact the research team to answer questions or to complete the survey by phone.
The information collection also requests an incentive for members. In prior information collection with AmeriCorps members, the use of incentives has been extremely effective resulting in over 75 percent among members serving with AmeriCorps NCCC, and over 80 percent among AmeriCorps Seniors volunteers. The cost-effectiveness of the incentive is that it reduces multiple rounds of contacting respondents, hence reducing overall labor cost to the government. The incentive also increases the response rate among those who are the most difficult to consent to participate thus increasing the most difficult-to-reach respondents which minimize non-response bias. The cost of these multiple rounds and the bias that could be introduced due to non-response outweigh the proposed incentive. Respondents will receive an electronic MasterCard gift card which can be redeemed nationally.
Table 2 Proposed Incentives for Target Participants
Respondents |
Incentive per Member |
|
|
Alumni |
$40 |
Early Exit member |
$40 |
Interview– early exit members only |
$40 |
AmeriCorps will offer a modest incentive of $40 to survey respondents; and $40 to the subset of respondents sampled to participate in an interview.
The use of incentives has a positive impact on increasing the response rate with no adverse effects on reliability. The use of incentives to increase the response rate is particularly important when collecting data to assess outcomes of current volunteers from participants who no longer serve as AmeriCorps Seniors volunteers. Respondents will receive payment in form of a gift card. The payment will be mailed directly to the respondents upon returning the completed survey. A substantial body of research including experimental and meta-analyses supports the use of incentives to increase response rates (Brick et al. 2005; Church 1993; Edwards et al. 2002; James and Bolstein 1992; Shettle and Mooney 1999; Singer et al. 1999; Singer, Van Hoewyk, and Maher 2000; Yammarino, Skinner, and Childers 1991). Those studies demonstrate that the use of incentives has a positive impact on increasing response rates, with no adverse effects on reliability (Jäckle, & Lynn, 2008; Dillman,2000).
In data collection, the use of incentive has been shown to be cost-effective due to the savings incurred by reducing the costs of follow-ups with non-respondents. There are mixed findings about incentives in the research literature. Certain types, timing, and amounts of incentives may not be effective, but the overall recommendation is that an incentive is an effective means to increase response rate and reduce nonresponse bias. A study on the use of incentives in the Annals of the American Academy of Political and Social Science, Singer and Ye (2013) conclude that:
“Incentives increase response rates to surveys in all modes, including the Web, and in cross-sectional and panel studies;
Monetary incentives increase response rates more than gifts, and prepaid incentives increase them more than promised incentives or lotteries, though they are difficult to implement in Web surveys;
Incentives, thus, have clear potential for both increasing and reducing nonresponse bias. If they can be targeted to sample volunteers who would otherwise fail to respond.”
The Singer and Ye article represents a balanced perspective in that it presents findings that suggest some types of incentives are not effective as well as findings that suggest incentives are effective. Nonetheless, the general conclusion based on research published in the most recent decade is that incentives are effective means to increase response rates.
Privacy & Confidentiality.
Participants’ responses to this information collection will remain private to the extent permitted by law. The independent contractor will make clear that individual responses will not be shared; the basis for the assurance of privacy is from the privacy statement in the survey invitation, consent forms, and surveys. All respondents will be assured that their participation is voluntary, that no adverse consequences will accrue to individuals who do not complete the surveys, and that their comments and opinions will be kept private.
The focus group and interview scripts include an assurance of privacy because the notes/recording from these conversations will not include any personally identifiable information, and results will be compiled into a summary that does not identify any specific grantee or organization, their partners, or the members.
No personally identifiable information, proprietary trade secrets, or other confidential information is being collected from grantees and partner organizations The personally identifiable information that is collected from members/alumni is covered by the System of Record Notice CNCS-04-CPO-MMF-Member Management Files (MMF), 85 Fed. Reg. 3896 (January 23, 2020). Access to any data with identifying information will be limited only to the independent contractor directly working on the information collection. The independent contractor will de-identify the data prior to analysis, so that individual responses cannot be linked to a specific individual. Additionally, all analyses, summaries, or briefings will be presented at the aggregate-level, and it will not be possible to identify individual respondents in any material that is presented. The independent contractor will provide AmeriCorps a de-identified dataset that can be used for further analysis or shared with other research and evaluators for additional secondary analysis.
Sensitive Questions.
Of the proposed questions, the demographic questions on gender identity and sexual orientation may be of sensitive nature. These questions are asked to gauge whether the Public Health AmeriCorps program is meeting its diversity goals when recruiting members. For both questions, respondents are offered an option of “prefer not to answer”. Any member who left service before their term expired could have a reason for doing so that causes discomfort, but the listed reasons for early departure are written in a non-judgmental and objective manner.
Burden Estimate.
The total number of responses is 2,198; and the estimated total burden is 2,095.9 hours for the information collection (see Table 3a and Table 3b on page 17). The estimate differs from the estimate in the 60-day Federal Register Notice which was then estimated at a total burden of 1,004 hours. The reason for the difference is that through discussion with scholars in the technical working group, AmeriCorps added the follow-up survey so that alumni can be surveyed twice to strengthen the evidence of employment outcome and public health careers post-service. The ability to secure public health employment may take time, as some alumni may continue with higher education. It may also take longer to secure public health employment due to external barriers such as general labor market conditions, public health hiring and governmental public health careers, and opportunities in target communities by the public and private health sectors may be inelastic. The longitudinal aspect of the data collection allows for a longer time frame to measure alumni’s employment outcomes.
The number of respondents and responses broken down by form and each phase of the information collection are listed in the next three tables:
Phase 1: June 2024 – October 2024 Burden Estimates
Form |
Responses |
Burden Hr Per Response |
Total Burden Hours |
Member Alumni Survey |
609 |
0.5 |
304.5 |
Early exit survey |
238 |
0.5 |
119.1 |
Grantee Survey |
82 |
0.5 |
41 |
Training and TA Form |
82 |
0.25 |
20.5 |
Grantee Focus Group |
15 |
1.5 |
22.5 |
Early exit interviews |
5 |
0.5 |
2.5 |
Partner interview |
4 |
0.5 |
2 |
TOTAL |
1,035 |
-- |
512.1 |
Phase 2: January 2025 – May 2025
Form |
Responses |
Burden Hr Per Response |
Total Burden Hours |
Member Alumni Survey |
953 |
0.5 |
476.3 |
Early exit survey |
205 |
0.5 |
102.6 |
Early exit interviews |
5 |
0.5 |
2.5 |
TOTAL |
1,163 |
-- |
581.4 |
Phase 3 Follow-up post survey with respondents from Phases 1 and 2: November 2025 – March 2026
Form |
Responses |
Burden Hr Per Response |
Total Burden Hours |
Member Alumni Survey |
1562 |
0.5 |
780.8 |
Early exit survey |
443 |
0.5 |
221.7 |
TOTAL |
2,005 |
-- |
1,002.5 |
Table 3a and Table 3b show the three phases combined; the total estimated burden is as follows:
Table 3a. Total Estimated Burden for the Information Collections
Form |
Responses |
Total Burden Hours (rounded) |
Member Alumni Survey |
1562 |
1562 |
Early exit survey |
443 |
443 |
Grantee Survey |
82 |
41 |
Training and TA Form |
82 |
21 |
Grantee Focus Group |
15 |
23 |
Early exit interviews |
10 |
5 |
Partner interview |
4 |
2 |
TOTAL |
2,198 |
2,097 |
Table 3b. Total Estimated Burden for the Information Collections
|
Requested |
Program Change Due to New Statute |
Program Change Due to Agency Discretion |
Change Due to Adjustment in Agency Estimate |
Change Due to Potential Violation of the PRA |
Previously Approved |
Annual Number of Responses |
2,198 |
0 |
2,198 |
0 |
0 |
0 |
Annual IC Time Burden (Hour) |
2,097 |
0 |
2,097 |
0 |
0 |
0 |
Annual IC Cost Burden (Dollars) |
99,267 |
0 |
99,267 |
0 |
0 |
0 |
|
Details of each IC follow. The $47.36 cost per hour is based on the Office of Personnel Management (OPM) General Schedule (GS) hourly rate of $34.76 for a GS-11, step 1, employee (Salary Table 2024-RUS for calendar year 2023 for the rest of the U.S.) plus the 36.25% civilian personnel full fringe benefit rate from OMB memorandum M-08-13 ($34.76 + $12.60 = $47.36, rounded to the nearest dollar).
Member Alumni Survey (1,562 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Early Exit Survey (443 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Grantee Survey (82 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Training and Technical Assistance Survey (82 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Grantee Focus Group (15 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Early Exit Interview (10 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Partner Interview (4 Respondents)
Burden per Response:
|
|
||||||||||||||||||||
Annual Burden:
|
|||||||||||||||||||||
Estimated nonrecurring costs.
No total capital and start-up costs, operation and maintenance costs, or purchase of services beyond customary and usual operating costs are required by this information collection.
Estimated cost to the Government.
The estimated cost to the Government is shown in the following table. It is estimated that it will take the Government 1.05 hour(s) to review each response. This estimate was developed by staff involved in the management of current activity.
Estimation of Cost to the Government |
|
Number of responses |
2,198 |
Hours per response |
1.05 |
Total estimated hours (number of responses multiplied by hours per response) |
2,307.09 |
Cost per hour (hourly wage) |
$52.24 |
Annual burden (estimated hours multiplied by cost per hour) |
$120,522 (rounded) |
Note: The cost per hour is based on the average of eight 2024 OPM GS hourly rates (base + locality) for a GS-11, step 1 employee living in our eight regions of operation, plus the 36.25% civilian personnel full fringe benefit rate from OMB memorandum M-08-13. The average cost per hour is $38.34 (average hourly rate) + $13.90 ($38.34 average hourly rate x 36.25% fringe) = $52.24/hour (rounded to the nearest penny).
The eight regions of operation and cities used are West (Sacramento - $38.44), Mountain (Denver - $38.65), North Central (Des Moines, IA - $35.02), South Central (Houston - $40.09), Midwest (Chicago - $38.81), Southeast (Atlanta - $36.74), Mid-Atlantic (Washington, DC - $39.66), and Northeast (Boston - $39.27). The average is $38.34 ($306.68/8).
Reasons for changes.
This new collection is due to a “Program Change Due to Agency Discretion” because it is necessary to implement Executive Order 13996, Establishing the COVID-19 Pandemic Testing Board and Ensuring a Sustainable Public Health Workforce.
Publicizing Results.
Time Schedule
The proposed information collection with members, grantees, and partners will be administered at three time points during the contractor’s period of performance. Figure 1 shows the time schedule for the process evaluation is June 1, 2024 – March 30, 2025. The information collection with grantees includes a survey to all active grantees, focus group with a sample of grantees, and interviews with a sample of the grantees’ partners. The timeline for the data collection occurs from June 2024 through October 2024, following by analysis of the data then dissemination.
Figure 1. Process Evaluation Information Collection Timeline
Figure 2 shows the time schedule for the outcome evaluation is June 1, 2024 – March 30, 2026. The information collection proposes three rounds of data collection. The first two rounds are cross sectional consisting of recent alumni and early exit members. The third round is a follow-up with respondents who participated in the first two rounds of the information collection.
Phase 1 information collection with alumni and early exit members begins in June 2024 and ends in October 2024. In Phase 1 includes members from the program inception through January 2024. During that period, some members completed their service term and are considered alumni, and some members who began service exited early and did not complete their service term, these members are considered early exit members. In Phase 1, early exit members consist of those who did not complete their term of service from the program inception through January 2024, did not reapply to serve and did not serve in another AmeriCorps program; alumni are members who completed their service terms from the program’s inception through January 2024. The information collection includes survey with alumni and early exit members, and interviews with a sample of early exit members. The timeline for the information collection occurs from June 2024 through October 2024, followed by analysis then dissemination.
Phase 2 information beginning in January 2025 and ends in May 2025. In Phase 2 includes members from February 2024 through January 2025. In Phase 2, early exit members early exit members consist of those who did not complete their term of service, did not reapply to serve and did not serve in another AmeriCorps program; alumni are members who completed their service term. The information collection includes survey with alumni and early exit members, and interviews with a sample of early exit members. The timeline for the information collection occurs from January 2025 through May 2025, followed by analysis then dissemination.
For the third and final Phase, the time schedule is November 2025 through March 2026, JBS will administer a follow-up survey to alumni and early exit members who participated in Phases 1 and 2. JBS will clean and analyze the collected data between April 2026 and September 2026. JBS will report findings between October 2026 and May 2027.
Figure 2. Outcome and Impact Evaluation Information Collection Timeline
Publication Plan
In discussion and collaboration with AmeriCorps, JBS identified five primary target audiences for the Public Health AmeriCorps evaluation’s reports and communications products. Although these are the priority audiences, it is important to note that other stakeholders and audiences, such as the communities where Public Health AmeriCorps members serve, other community services and public health entities, Congress, and the White House are also of interest in this evaluation. JBS will continue the discussion and collaborate with the Contracting Officer Representative (COR) on how best to support disseminating critical information from the Public Health AmeriCorps evaluation to these additional audiences.
To ensure the evaluation’s products reach the above-mentioned prioritized audiences, JBS identified the most effective means of delivery. The main means of dissemination will be:
Email
SharePoint
Newsletters
Internal AmeriCorps Audience-Specific Recurring Meetings/Phone Calls
Webinars/Presentations
Blogs
Social Media Messaging
Public Health AmeriCorps Grantee Exchange
OMB Not to Display Approval.
The expiration date for OMB approval of the information collection will be displayed.
Exceptions to "Certification for Paperwork Reduction Submissions."
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
Surveys, Censuses, and Other Collections that Employ Statistical Methods.
This request includes survey and will use statistical estimation technique; Part B supporting statement is provided.
Appendix A Public Health AmeriCorps Two-Pager
OMB Control Number: XXX (Expires XX/XX/XX)
National Process, Outcomes, and Impact Evaluation of
Public Health AmeriCorps Study Fact Sheet
Summary
Public Health AmeriCorps, a partnership between AmeriCorps and the Centers for Disease Control and Prevention, has invested since Spring 2022 in programs across the country to (1) engage AmeriCorps members in addressing local public health needs, (2) advance equitable health outcomes for underserved communities, and (3) create pathways to public health-related careers. To document implementation, facilitate “real-time” learning and improvement, and assess progress toward goals, outcomes, and impacts, AmeriCorps contracted with JBS International to conduct a five-year evaluation of Public Health AmeriCorps.
Study Background
Public Health AmeriCorps is a partnership between AmeriCorps and the Centers for Disease Control and Prevention (CDC) that includes a $400 million investment, over five years, from the American Rescue Plan Act workforce funding that will enable the recruitment, training, and development of a new generation of public health leaders who are ready to respond to the needs of the nation by serving their communities. Public Health AmeriCorps has two main goals:
Address public health needs of local communities by providing support in state and local public health settings and advancing equitable health outcomes for underserved communities.
Create pathways to public health-related careers through onsite experience and training, and recruiting AmeriCorps members who reflect the communities they serve.
In Spring 2022, Public Health AmeriCorps began investing in programs across the country to support COVID-19 pandemic recovery and meet public health needs of local communities. Public Health AmeriCorps members provide needed capacity and support in local public health settings to advance more equitable health outcomes, emphasizing historically underserved communities.
In August 2022, AmeriCorps contracted with JBS International, an independent research and evaluation firm, to conduct a multi-year national evaluation of this Initiative. The purpose is (1) to document implementation, facilitate “real-time” learning and process improvement and (2) to assess the Initiative’s progress toward goals and capture program outcomes and impacts. The Office of Research and Evaluation at AmeriCorps, in consultation with the CDC, is managing the national evaluation.
Study Goals
The national Public Health AmeriCorps evaluation will help AmeriCorps and its stakeholders understand the extent to which Public Health AmeriCorps is progressing toward its goals to address public health needs and develop the next generation of public health leaders. The Public Health AmeriCorps evaluation has three primary objectives:
Assess how Public Health AmeriCorps strengthens capacity to address the public health needs, especially in communities that have been historically underserved, and how recruiting members from within the communities grantees serve affects both communities and members;
Identify successes, challenges, and lessons learned in implementing Public Health AmeriCorps;
Examine members’ public-health-related career pathways, civic participation, commitment to national service, and sense of community.
Study Design
The mixed methods evaluation will use quantitative and qualitative data to answer the following evaluation questions:
What are the overall successes and challenges of implementing Public Health AmeriCorps? How do grantees measure their performance and program effectiveness? What are the outreach and retention strategies grantees use?
What education, training, and workforce development supports do Public Health AmeriCorps members receive that allow them to serve communities and prepare them to achieve their public health career goals?
What is the nature of the relationship and partnership between grantees and their local and state health departments?
Does participation in Public Health AmeriCorps affect civic and community engagement?
Does participation in Public Heath AmeriCorps lead to placement in public health-related careers? Does participation in Public Health AmeriCorps lead members to pursue or obtain credentials, certifications, or degrees in a public health occupation?
JBS International will collect data from the following sources: surveys of grantee/subgrantees and member; grantee documents and literature reviews expert stakeholder discussions; and focus groups and interviews with AmeriCorps and CDC staff, grantees/subgrantees, members, and community members.
Key Evaluation Activities & Timelines in 2023
January – March 2023 Literature Review
January – June 2023 Identify and secure existing secondary data sources
February – June 2023 AmeriCorps and subject expert advisory group meetings
February – April 2023 Develop instruments
April – May 2023 Pilot test instruments
April – June 2023 Draft evaluation design
May – August 2023 Submit OMB Paperwork Reduction Act (PRA) packet
June – August 2023 Focus groups and interviews with grantees
January – December 2024 OMB approval, implement data collection, and conduct analysis
Between June and August 2023, JBS staff, in collaboration with the Office of Research and Evaluation, will gather data to inform how best to implement the evaluation
Interviews and focus groups. JBS staff will contact AmeriCorps, CDC, and grantee staff to participate in virtual interviews and focus groups to discuss successes, opportunities, challenges and other experiences implementing Public Health AmeriCorps programs.
Pilot surveys. JBS staff will request a small number of grantee staff and members to participate in piloting online member survey, focus group, and interview protocols designed to assess the Initiative’s influence on members, grantees, and communities.
Study Contacts
Belayeth Hussain, PhD |
Annie Georges, PhD |
Ashley Lederman, DC, MPH |
Project Director |
Principal Investigator |
Project Officer |
JBS International |
JBS International |
AmeriCorps |
|
650-373-4977 |
650-373-4938 |
202-594-0612 |
If you have questions, please contact us at [email protected] |
||
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2024-07-21 |
© 2024 OMB.report | Privacy Policy