Supporting Statement_ECA_Career Connections_v4

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

Career Connections Evaluation
OMB Number 1405-XXXX

1. JUSTIFICATION

1. Why is this collection necessary and what are the legal statutes that allow this?

The Department of State’s Bureau of Educational and Cultural Affairs (ECA) regularly monitors and evaluates its programs through the collection of data about program accomplishments in order to enable program staff to assess the impact of its programs, where improvements may be necessary, and to modify/plan future programs.

The Career Connections program is managed by the Bureau of Educational and Cultural Affairs (ECA) Office of Alumni Affairs (OAA). Started in 2019, the Career Connections program brings together American alumni (18-35 years old) of U.S. Government-sponsored exchange programs with expert career coaches, professionals from diverse fields, and international leaders to help alumni market their international exchange experiences. Delivered as two-day seminars across the country, the Career Connections program provides invaluable networking opportunities for U.S. alumni with leaders in their communities with activities including: resume-building, developing a personal brand, translating skills gained through the exchange experience, developing an online presence, and networking to develop connections with fellow alumni and expert speakers alike. ECA’s Evaluation Division will undertake an evaluation of the Career Connections program. The purpose of this evaluation is to inform the next iteration of the award with participant-driven recommendations on how to strengthen the Career Connections program. The evaluation will be completed by September 2021, in time for recommendations to be implemented in the new award.

Legal authorities and administrative requirements that necessitate the collection of these data can be found below:

1. Government Performance and Results Act of 1993 (GPRA)

2. Government Performance and Results Modernization Act of 2010 (GPRAMA)

3. Mutual Educational and Cultural Exchange Act of 1961, as amended, 22 U.S.C. 2451 et seq. (also known as the Fulbright-Hays Act)

2. What business purpose is the information gathered going to be used for?

The evaluation will provide evidence to inform programmatic decision-making by the OAA, who will be the primary users of the evaluation results. Information collected will inform the design and implementation of the Career Connections program for future ECA alumni, as well as make any necessary adjustments to strengthen the program’s utility. The final report will also be made available to the public as part of ECA’s responsibility to be accountable for the use of its funds and performance of its mission.

3. Is this collection able to be completed electronically (e.g. through a website or application)?

Yes, data collection will take place electronically. The ECA Evaluation Division will coordinate the survey construction, sampling strategy, piloting, and troubleshooting prior to data collection. It will be administered via the online platform Qualtrics. The estimated time for survey completion is 20 minutes. Key Informant Interviews (KIIs) will be conducted remotely via phone or a secure platform such as Zoom or Microsoft Teams. The estimated time for KIIs is 60 minutes.

4. Does this collection duplicate any other collection of information?

This will not be a duplication of effort. The purpose of the data collection, and therefore the focus of the questions asked, is to strengthen professional development opportunities offered to American alumni. ECA has not conducted an external evaluation of the Career Connections program in the past and existing project monitoring data does not cover the topics being investigated in sufficient breadth or depth.

5. Describe any impacts on small business.

We do not expect there to be any impacts on small businesses.

6. What are consequences if this collection is not done?

The purpose of this evaluation is to strengthen professional development opportunities offered to ECA’s American alumni. The evaluation will provide evidence to inform programmatic decision-making to ECA’s OAA, who will be the primary user of the evaluation results, in order to inform the design and implementation of the Career Connections program for future seminar participants. If data collection does not take place, the program will be unable to modify the future scope and procurement to take into account these learnings – thereby negatively impacting the effectiveness, efficiency, and/or impact of the program.

7. Are there any special collection circumstances?

This data collection involves no special circumstances, as it is a one-time data collection and does not require submission of any information that is not OMB-approved.

8. Document publication (or intent to publish) a request for public comments in the Federal Register

The 60-day Federal Register Notice was published on January 4, 2021 (86 FR 174). No comments were received in that period. The Department will publish a notice in the Federal Register soliciting public comments to OMB for a period of 30 days.

9. Are payments or gifts given to the respondents?

No payments or gifts are proposed for respondents.

10. Describe assurances of privacy/confidentiality

ECA and its external contractors follow all procedures and policies stipulated under the Privacy Act of 1974, as amended. Informed consent will be obtained from all respondents. Adults must provide consent with full awareness that participation is entirely at will and can be declined at any time. The survey and KIIs will alert all respondents at the start on the nature of the Career Connections evaluation, intended data use, and all aspects of data privacy, storage, and management. Each survey includes the following language (KIIs have similar consent/introductory language):

The Evaluation Division in the Bureau of Educational and Cultural Affairs (ECA), in coordination with the Office of Alumni Affairs, is currently undertaking an evaluation of the Career Connections program. The purpose of this evaluation is to improve the professional development programming for U.S. citizen alumni of government sponsored exchange programs. This is an opportunity for you to share your experiences and let ECA know what you are looking for from professional development opportunities.

The survey contains 36 questions and should take 20 minutes (average) to complete. You do not have to have participated in a Career Connections seminar to take this survey.

Please note that your participation in this survey is voluntary, and you are free to end the survey at any time. By clicking the “Consent and enter survey” button below, you are consenting to the following terms:

• Your participation in this evaluation is voluntary. We do not anticipate that participating in this evaluation will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the Career Connections Program—and, thereby, the general public. You may skip any questions you are not comfortable answering.

• The information that you provide in the survey will be used to write a report. This report will be shared with the U.S. Department of State and other stakeholders for comment and will eventually be made public. Any responses you provide may be reported in the final report as part of the anonymized aggregated quantitative analysis or the qualitative analysis from open-ended responses, with all personal identifying information removed.

• The U.S. government will take reasonable measures to protect privacy data, personally identifiable information, and other sensitive data obtained from the survey. Responses to questions may be reported by demographic category (i.e., field of study, employment status), country, or cohort year. The only identifying information used will be the demographic information collected in the beginning of the survey.

• As this evaluation requires us to speak with a broad range of program alumni, we may ask you to share contact information for the connections that you mention, in the cases where we do not already have updated or valid contact information for these parties. As with all other questions, you may skip or decline to answer any questions you are not comfortable answering. Updated contact information may be shared with the U.S. Department of State upon completion of this survey.

• The data you provide may be reanalyzed at a later date for a follow-up study or other purpose approved by the U.S. Department of State. The data may be made available to third parties as required by law.

• You may withdraw your consent at any time by contacting [email protected].

We ask that you complete this survey by [DEADLINE].

If you have any questions about the survey, please contact Mary Ann Aabye at [email protected]. If you have any questions about the Career Connections program, please contact Paul Garr at [email protected].

CONSENT TO PARTICIPATE

By clicking the button to begin the survey below, you are giving your consent to participate in this evaluation. If you do not wish to participate, please click the exit survey link below.

• Consent and enter survey

• Decline and end survey now

In line with ECA policy, individual names will not be reported, but responses will be used in the aggregated analysis, and may be disaggregated by variables of interest, such as state, seminar, etc. For qualitative data, information will be coded and anonymized to remove personally identifiable information.

11. Are any questions of a sensitive nature asked?

There is one question that has been deemed to be of a sensitive nature on the mentor survey: sex of the respondents. This question has been included because the evaluation will apply a gender and social inclusion lens as part of its analysis and employ data triangulation—an analysis strategy in which data sources are first analyzed independently, then in parallel—to understand if and how findings differ by gender, cohort, and other key variables. This step is important as alumni may have divergent experiences of how they are benefiting from the professional development opportunities provided by ECA.

12. Describe the hour time burden and the hour cost burden on the respondent needed to complete this collection

The total estimated hour burden for this data collection is 9,635 minutes (160.58 hours), broken down as follows in Table 1.

Table 1. Hour Time Burden for Career Connections Evaluation Respondents

Instrument	Estimated Number of Responses	Average Time per Response	Total Estimated Burden Time
Alumni Survey	313	20 minutes	6,260
Alumni Key Informant Interview	45	60 minutes	2,700
Workshop Presenter Key Informant Interview	15	45 minutes	675
Total			9,635 min

The average times were calculated based on estimated times from similar survey and KII question lengths. The estimated number of survey responses was calculated by adding anticipated responses from a stratified sample of two groups: (1) Career Connections participants (625 American alumni multiplied by the acceptable minimum survey response rate of 30% for n = 188) and (2) American alumni that have not participated in Career Connections seminars (sample of 2,500 alumni multiplied by the acceptable minimum response rate of 5% for n = 125).

Time Cost to Respondents

The total estimated cost burden for this data collection is $278,289. Most respondents will respond as individuals, and the cost to them is the opportunity cost if they were to undertake paid work for the time that they spend responding to surveys or participating in KIIs. Therefore, the cost estimate for these groups is based on the Bureau of Labor Statistics May 2019 National Occupational Employment and Wage Estimates.¹ For alumni, given their diverse sectors and career levels, the average for all occupations of $25.72 was used and increased by 3% to $26.49 to handle overhead. For workshop presenters, the average for management occupations of $58.88 was used and increased by 3% to $60.65 to handle overhead. Table 2 highlights the estimate of overall respondent group hour and cost burden.

Table 2. Estimate of Respondent Hour and Cost Burden

Respondent Group	Total Average Hourly Cost Rate	Total Estimated Hours	Total Cost Burden
Alumni Survey	$ 26.49	6,260	$ 165,827
Alumni Key Informant Interview		2,700	$ 71,523
Workshop Presenter Key Informant Interview	$ 60.65	675	$ 40,939
Total			$ 278,289

13. Describe the monetary burden to respondents (out of pocket costs) needed to complete this collection.

There are no costs incurred by respondents.

14. Describe the cost incurred by the Federal Government to complete this collection.

The only costs to the USG for the Career Connections evaluation as related to this collection are associated with wages for Federal employees at DOS, and are estimated at $44,718. This estimate includes employees’ labor for the design, data collection, and analysis activities. The wage rates of Federal employees at DOS were estimated using Steps 1 for Grades 13 ($49.19/hour at 360 hours) and 14 ($58.13/hour at 80 hours) of the General Schedule in the Washington-Baltimore-Arlington, DC-MD-VA-WV-PA locality area.² The Department multiplied the hourly wage rate by 2 to account for a fringe benefits rate of 69 percent³ and an overhead rate of 31 percent.⁴ There are no additional costs for equipment, supplies, or travel associated with this data collection.

Table 3. Total Cost to Federal Government

Federal Staff Costs	Total
GS-14, Step 1 equivalent – $58.13/hour x 2 @ estimated 80 hours total	$ 9,301
GS-13, Step 1 equivalent – $49.19/hour x 2 @ estimated 360 hours	$ 35,417
Total	$ 44,718

15. Explain any changes/adjustments to this collection since the previous submission

This is a new collection.

16. Specify if the data gathered by this collection will be published.

Once data have been collected and analyzed, the ECA Evaluation Division will produce a final report for publication on its website (https://eca.state.gov/impact/eca-evaluation-division). However, the raw data or those of individual respondents will not be published in any way with attribution or any personally identifiable information.

17. If applicable, explain the reason(s) for seeking approval to not display the OMB expiration date. Otherwise, write “The Department will display the OMB expiration date.”

The Department will display the OMB expiration date.

18. Explain any exceptions to the OMB certification statement below. If there are no exceptions, write “The Department is not seeking exceptions to the certification statement”.

The Department is not seeking exceptions to the certification statement.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This collection will employ statistical methods for each survey and interview.

1. Potential Respondent Universe and Respondent Selection Methods

Sampling Strategy

The respondent universe for this evaluation includes all American alumni that both have and have not participated in Career Connections seminars. Given the large universe of respondents, the Evaluation Division will conduct a stratified sample of two groups. Group 1 consists of Career Connections participants, of which the Evaluation Division will conduct a census. Within this group is the 625 American alumni that have participated in the Career Connections program, with the acceptable minimum survey response rate set at 30% for n = 188. Group 2 consists of American alumni that have not participated in Career Connections seminars but are eligible to. Given the large number of potential respondents in this group (n = 50,000), the Evaluation Division has chosen to survey a sample of 2,500 alumni, with the acceptable minimum response rate set at 5% for n = 125.

Key informants, both alumni and workshop presenters, will be identified through sampling methods that reflect stakeholder diversity and ensure key demographics are adequately represented in the samples.

2. Procedures for the Collection of Information

The Evaluation Division will administer the survey via the online survey platform Qualtrics. Alumni will be sent a link to the survey that will remain open for four weeks.

Quality Control

The Evaluation Division will monitor survey responses daily and flag any potential problems with the survey platform, survey completion rates, suspected burnout, etc. These flags will be investigated and corrective actions (including requesting feedback from the respondent) will be taken the following day, if deemed necessary.

Security and Data Management

Data will be protected on the password-secured website during survey administration. All data utilized, captured, or acquired for this evaluation will be securely stored and segregated from any other Department data. These procedures will include segregated project folders used only in the execution of this evaluation, and which will only be accessible by personnel engaged in evaluation activities. Full data access will be limited to project staff and higher to limit those who have access to the full range of data captured and utilized throughout completion of the project.

Methods for Validating Data

As the survey will be administered via an online survey platform, data will already be formatted for analysis and can be exported to SPSS or Excel for cleaning. Closed-ended questions will have already been assigned codes in the Qualtrics platform.

During data cleaning, the team will analyze question responses for outliers, implausible values, and digit preference (on numeric responses), and for unexpected distributions on categorical or ordinal variables. These issues are then addressed, replacing clearly errant values with missing values, imputing the value based on available information, or merely flagging the issue in the top-line analysis report. Significantly skewed ordinal or interval data may be transformed to approach normality.

3. Maximization of Response Rates and Nonresponse

ECA considers nonresponse to be a risk. Wherever possible, questions and scales were selected for parsimoniousness. As much as possible, response categories were limited to yes/no and no more than four-point scale options to reduce the time required to respond to the questions.

It is likely that responses will be most forthcoming from respondents with more recent engagement with the program, and limited from respondents with no connection to the program. Halfway through the survey administration, data on total number of responses will be reviewed by ECA for acceptability.

The Evaluation Division will send on open survey link to respondents through the OAA. The invitation to complete the survey will include an estimated completion time, which should encourage participation. Once the survey has been launched (via an introductory email including the survey link), the OAA will send periodic reminders to boost the response rate, including posting on Social Media. Based on prior experience, we expect the survey to be open for 4 weeks, requiring two to three reminders.

4. Data Analysis and Reporting

Quantitative Data Analysis

The Evaluation Division will analyze survey data using the Qualtrics data analytics tools.

Qualitative Data Analysis

During data collection, the ECA Evaluation Division will take detailed notes and following data collection, the recordings of KIIs will be transcribed. These will become the basis of qualitative data analysis and cleaning. Once data collection is completed, the contractor will use NVivo qualitative data coding and analysis software to conduct content, trend, contribution and pattern analysis to identify response categories and emergent themes and contextual factors, upon which to draw conclusions and develop evidence-based recommendations.

5. Relevant Contacts

This evaluation is being conducted internally by ECA’s Evaluation Division: Natalie Donahue, Chief of Evaluation (202-632-6193).

1 USDOL Bureau of Labor Statistics, May 2019 National Occupational Employment and Wage Estimates, https://www.bls.gov/oes/current/oes_nat.htm#25-0000, last updated March 31, 2020, accessed November 24, 2020.

2 Source: Office of Personnel Management, “2020 General Schedule (GS) Locality Pay Tables,” https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2020/general-schedule/

3 Source: Congressional Budget Office, “Comparing the Compensation of Federal and Private-Sector Employees, 2011 to 2015” (April 2017), https://www.cbo.gov/publication/52637. The wages of Federal workers averaged $38.30 per hour over the study period, while the benefits averaged $26.50 per hour, which is a benefits rate of 69 percent.

4 Source: U.S. Department of Health and Human Services, “Guidelines for Regulatory Impact Analysis” (2016), https://aspe.hhs.gov/system/files/pdf/242926/HHS_RIAGuidance.pdf. On page 30, HHS states, “As an interim default, while HHS conducts more research, analysts should assume overhead costs (including benefits) are equal to 100 percent of pretax wages….” To isolate the overhead rate, the Department subtracted the benefits rate of 69 percent from the recommended rate of 100 percent.

8