Supporting Statement_ECA_TechWomen_30 day ECA_11.18.2020

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

TechWomen Evaluation
OMB Control Number 1405-XXXX

A. JUSTIFICATION

1. Why is this collection necessary and what are the legal statutes that allow this?

The Department of State’s Bureau of Educational and Cultural Affairs (ECA) regularly monitors and evaluates its programs through the collection of data about program accomplishments in order to enable program staff to assess the impact of its programs, where improvements may be necessary, and to modify/plan future programs. ECA is currently conducting an evaluation of the ECA’s TechWomen Program. TechWomen empowers, connects and supports the next generation of women leaders in science, technology, engineering and mathematics (STEM) from Africa, Central and South Asia, and the Middle East by providing them the access and opportunity needed to advance their careers, pursue their dreams, and inspire women and girls in their communities. Since 2011, more than 700 women have participated in TechWomen. ECA contracted Social Impact (SI) in early 2020 to conduct an evaluation of the TechWomen program. The purpose of this evaluation is to determine the strength and sustainability of professional networks created by the program and the extent to which these networks have been leveraged for collaborations between Alumnae to enact change. The evaluation will provide evidence to inform programmatic decision-making by the ECA program team, who will be the primary users of the evaluation results, in order to inform the design and implementation of the TechWomen program for future cohorts, as well as make any necessary adjustments. The evaluation will be completed by May 2021 to incorporate lessons learned for the next iteration of the program.

Legal authorities and administrative requirements that necessitate the collection of these data can be found below:

1. Government Performance and Results Act of 1993 (GPRA)

2. Government Performance and Results Modernization Act of 2010 (GPRAMA)

3. Mutual Educational and Cultural Exchange Act of 1961, as amended, 22 U.S.C. 2451 et seq (also known as the Fulbright-Hays Act)

2. What business purpose is the information gathered going to be used for?

The evaluation will provide evidence to inform programmatic decision-making by the ECA program team and its implementing partners, who will be the primary users of the evaluation results. Information collected will inform the design and implementation of the TechWomen program for future cohorts, as well as make any necessary adjustments to strengthen the program’s utility. A high-level version final report will also be made available to the public as part of ECA’s responsibility to be accountable for the use of its funds and performance of its mission. The ECA Evaluation Division, in partnership with Social Impact (who is conducting the evaluation) will be responsible for collecting and analyzing the data.

3. Is this collection able to be completed electronically (e.g. through a website or application)?

Data collection will take place electronically. SI will coordinate the survey construction, sampling strategy, piloting, and troubleshooting prior to data collection. It will be administered via the online platform ONASurveys.com, a platform specifically designed for social network analysis. ONASurveys.com supports , individualized surveys, multiple network questions in one survey and utilizes a searchable roster to identify contact thus enhancing data validation. Key Informant Interviews (KIIs) will be conducted in-person or remotely via a secure platform such as Zoom or Microsoft Teams if in-person data collection is not feasible due to the COVID-19 pandemic.

4. Does this collection duplicate any other collection of information?

This will not be a duplication of effort. The purpose of the data collection, and therefore the focus of the questions asked, is to determine the strength and sustainability of professional networks created by the TechWomen program and the extent to which these networks have been leveraged for collaborations between Alumnae to enact change. ECA has not conducted an external evaluation of the TechWomen program in the past.

5. Describe any impacts on small business.

We do not expect there to be any impacts on small businesses.

6. What are consequences if this collection is not done?

Approval is being sought for a one-time collection. As the TechWomen program has been operating since 2011, ECA deems it critical to conduct an independent evaluation to determine the strength and sustainability of professional networks created by the program and the extent to which these networks have been leveraged for collaborations between Alumnae to enact change. The evaluation will provide evidence to inform programmatic decision-making to the ECA program team, who will be the primary user of the evaluation results, in order to inform the design and implementation of the TechWomen program for future cohorts as well as make any necessary mid-course adjustments prior to the launch of the second decade of TechWomen. Absent this data collection, ECA will be unable to modify the future scope and procurement to take into account these lessons.

7. Are there any special collection circumstances?

This data collection involves no special circumstances, as it is a one-time data collection and does not require submission of any information that is not OMB-approved. Consent procedures include obtaining providing the notices outlined in paragraph 10, prior to collection of any data.

8. Document publication (or intent to publish) a request for public comments in the Federal Register

The 60-day Federal Register Notice was published on August 14, 2020 (85 FR 49712). Two

comments were received but did not provide feedback on the evaluation data collection tools

and was not deemed relevant. The Department will publish a notice in the Federal Register

soliciting public comments for a period of 30 days.

9. Are payments or gifts given to the respondents?

No payments or gifts are proposed for respondents.

10. Describe assurances of privacy/confidentiality

ECA and its external contractors follow all procedures and policies stipulated under the Privacy Act of 1974 to guarantee the privacy of the respondents. Informed consent will be obtained from all respondents. The survey and Key Informant Interviews (KIIs) will alert all respondents at the start on the nature of the TechWomen evaluation, intended data use, and all aspects of data privacy, storage, and management. Each survey includes the following language (KIIs have similar consent/introductory language):

Social Impact is working on behalf of the TechWomen Program, an initiative of the U.S. Department of State’s Bureau of Education and Cultural Affairs. Social Impact is not affiliated with IIE. The goal of this survey is to understand the relationships you have built as a result of your participation in the TechWomen program. The survey will also ask questions about your life after participation in the program. 

Please take this survey on a computer or laptop, if possible.  

Taking this survey should take about 20-30 minutes.  There are no risks to participating in the survey, and there are no direct benefits, though your participation may benefit those who participate in future programs. Participating in this survey does not influence your ability to participate in future TechWomen programming.  

All the responses you provide during this survey will be kept confidential to the extent authorized by law. Only the researchers directly involved in this study will have access to your personal information.  

Your participation in this study is voluntary and you are under no obligation to participate. If you start the survey and wish to stop at any time for any reason, or if you do not want to answer any questions, you may do so without penalty.  

Please complete the survey by date.

If you have any questions or concerns, you may contact the Social Impact Institutional Review Board at [email protected] or +1 703 465 1884 with questions about the study or results. 

By clicking the click “Proceed to next step” button below, you are consenting to the above terms.

In line with ECA policy, individual names will not be reported, but responses will be used in the aggregated analysis, and may be disaggregated by variables of interest, such as country, program year, etc. For qualitative data, information will be coded and anonymized to remove personally identifiable information.

11. Are any questions of a sensitive nature asked?

There is one question that has been deemed to be of a sensitive nature on the mentor survey: sex of the respondents. This question has been included because the evaluation will apply a gender and social inclusion lens as part of its analysis and employ data triangulation—an analysis strategy in which data sources are first analyzed independently, then in parallel—to understand if and how findings differ by gender, cohort, and other key variables. This step is important as Mentors may have divergent experiences of how networking is occurring and who is benefitting.

12. Describe the hour time burden and the hour cost burden on the respondent needed to complete this collection

The total estimated hour burden for this data collection is 23,670 minutes (394.5 hours), broken down as follows in Table 1.

Table 1. Hour Time Burden for TechWomen Evaluation Respondents

Respondent Instrument	Estimated Number of Respondents	Average Time per Response	Total Estimated Burden Time
Mentor Survey	709	30 minutes	21,270 minutes (354.5 hours)
Mentor Key Informant Interview (KII)	40	60 minutes	2,400 minutes (40 hours)
Total Estimated Burden Time			23,670 minutes (394.5 hours)

The average times were calculated based on estimated times from similar survey question lengths.

Time Cost to Respondents

Most respondents will respond as individuals, and the cost to them is the opportunity cost if they were to undertake paid work for the time that they spend responding to surveys or participating in KIIs. Therefore, the cost estimate for these groups is based on the Bureau of Labor Statistics May 2019 National Occupational Employment and Wage Estimates¹. Given the diverse sectors and career levels of the mentors, a variety of representative occupations were selected from the Bureau of Labor Statistics’ data as proxies to represent potential respondent careers and sectors. Compensation estimates for proxies were then averaged. Compensation estimates for proxies were then averaged for an hourly wage of $52.99 and inflated by 3% to $54.58 . Table 2 highlights the estimate of overall respondent group hour and cost burden. Table 3 highlights the occupation proxies selected for this analysis.

Table 2. Estimate of Respondent Hour and Cost Burden

Respondent Group	Total Average Hourly Cost Rate	Total Estimated Hours	Total Cost Burden
Mentor Survey	$54.58	354.5 hours	$ 19,348.61
Mentor Key Informant Interviews (KII)		40 hours	$2,183.20
Total		394.5 hours	$21,531.81

Table 3. Wage Estimate Proxies

Respondent Group	Occupation Proxies	Hourly Compensation (includes costs for wages, salaries, and benefits)	Total Average Hourly Cost Rate
Mentor Survey	Architectural and Engineering Managers	$73.52	$52.99
	Engineers	$48.45
Mentor Key Informant Interviews (KII)	Software and Web Developers, Programmers, and Testers	$51.44
	Project Management Specialists and Business Operations Specialists, All Other	$38.57

13. Describe the monetary burden to respondents (out of pocket costs) needed to complete this collection.

There are no costs incurred by respondents.

14. Describe the cost incurred by the Federal Government to complete this collection.

The estimated cost to the USG for the TechWomen Evaluation as related to this collection is $89,756.00. This estimate includes all direct and indirect costs of the design, data collection, and analysis activities. In Table 3 below, Personnel and Fringe Benefit costs are for the contractor (Social Impact) personnel who conduct the data collection. The wage rates of Federal employees at DOS were estimated using Steps 1 for Grades 13 ($49.19/hour at 40 hours) and 14 ($49.19 at 120 hours) of the General Schedule in the Washington-Baltimore-Arlington, DC-MD-VA-WV-PA locality area. ² The Department multiplied the hourly wage rate by 2 to account for a fringe benefits rate of 69 percent³ and an overhead rate of 31 percent.⁴

Table 3. Total Cost to Federal Government

Cost Item	Total
Federal Staff Costs [indicate grade and step and percentage of time] GS-14, Step 1 equivalent – $58.13/hour @ estimated 40 hours total GS-13, Step 1 equivalent - $49.19/hour×2 @ estimated 120 hours	$ 16,456.00
Personnel	$15,994.00
Fringe Benefits	$ 2,577.00
Travel	$37,960.005
Equipment	NA
Supplies	NA
Total Direct Costs for Domestic Data Collection and Analysis	$ 72,987.00
Indirect Charges [include Overhead, Fee, and G&A]	$16,769.00
Total	$ 89,756.00

15. Explain any changes/adjustments to this collection since the previous submission

This is a new collection.

16. Specify if the data gathered by this collection will be published.

Once data have been collected and analyzed, the evaluation contractor will produce a final report for publication. The ECA Evaluation Division will publish the external version of the report, and infographic on its website (https://eca.state.gov/impact/eca-evaluation-division). However, the raw data or those of individual respondents will not be published in any way with attribution or any personally identifiable information.

17. If applicable, explain the reason(s) for seeking approval to not display the OMB expiration date.

The Department will display the OMB expiration date.

18. Explain any exceptions to the OMB certification statement below..

The Department is not seeking exceptions to the certification statement.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This collection will employ statistical methods for each survey and interview.

1. Potential Respondent Universe and Respondent Selection Methods

Sampling Strategy

The respondent universe for this evaluation includes the 946 individuals who participated as mentors in the TechWomen Program. To gather sufficient data for a whole of network analysis, the acceptable minimum survey response rate is set at 75% (n = 709). As a result, it was determined that a census was the most effective way to achieve the required sample size.

The contractor will identify key informants through sampling methods that reflect stakeholder diversity and ensure key demographics are adequately represented in the samples.

2. Procedures for the Collection of Information

The contractor will administer the survey via the online research platform ONAsurveys.com. Prior to the survey being deployed to mentors, an email will be sent to all mentors by the TechWomen implementing partner, IIE, with the option for mentors to opt out of survey participation. Mentors that opt-out will not be sent individualized links to the mentor survey. Mentors that do not opt out of survey participation will be sent an individualized link to the mentor survey and must utilize the individualized link for participation. The surveys will each remain open for 30 days.

Quality Control

The Evaluation Team (ET) will monitor survey responses daily and flag any potential problems with the survey platform, survey completion rates, suspected burnout, etc. These flags will be investigated and corrective actions (including requesting feedback from the respondent) will be taken the following day, if deemed necessary.

Security and Data Management

Data will be protected on the password-secured website during survey administration. All data utilized, captured, or acquired for this evaluation will be securely stored and segregated from any other company data. These procedures will include segregated project folders used only in the execution of this evaluation, and which will only be accessible by personnel engaged in evaluation activities. Full data access will be limited to the ET’s management staff and higher to limit those who have access to the full range of data captured and utilized throughout the execution of the contract.

Methods for Validating Data

As the survey will be administered via an online survey platform, data will already be formatted for a network study and can be exported to SPSS or Microsoft Excel for cleaning. Closed-ended questions will have already been assigned codes in the ONA platform.

During data cleaning, the team will analyze question responses for outliers, implausible values, and digit preference (on numeric responses), and for unexpected distributions on categorical or ordinal variables. These issues are then addressed in consultation with the ET, replacing clearly errant values with missing values, imputing the value based on available information, or merely flagging the issue in the top-line analysis report. Significantly skewed ordinal or interval data may be transformed to approach normality.

Data Ownership

The ET understands that all data utilized, captured, or acquired within the implementation of an evaluation is solely owned by the end-client, unless explicitly and contractually noted. As such, the contractor will not utilize the data for any other pursuits other than for those specifically required to perform the work under contract, without express written consent.

3. Maximization of Response Rates and Nonresponse

ECA considers nonresponse to be a risk. Wherever possible, questions and scales were selected for parsimoniousness. As much as possible, response categories were limited to yes/no and no more than four-point scale options to reduce the time required to respond to the questions.

Although the survey will be sent as a census, it is possible that responses will be most forthcoming from respondents with more recent engagement with the program. Halfway through the survey administration, data on total number of responses will be compiled and presented by the contractor for ECA’s review and consultation on acceptability.

The contractor will send individualized survey links to respondents via the ONA surveys platform. The invitation to complete the survey will include an estimated completion time, which should encourage participation. Once the survey has been launched (via an introductory email including the survey link), the contractor will send period reminders boost the response rate. Based on prior experience, we expect the survey to be open for 4 weeks, requiring two to three reminders. If response rates remain low despite reminders, the contractor may request that the TechWomen implementing partner, IIE, assist with promotion of the survey since they have established relationships with many of the respondents.

4. Data Analysis and Reporting

Social Network Analysis

The contractor will use data from the online survey to provide visual representations and analytic data of the TechWomen network structure. The ET will analyze network data using UCINET, a software package specifically for the analysis of social network data, and visualized with Gephi, an open-source network analysis and visualization software package. The ET will use clustering methods to identify any discernable groups that emerge from the network, such as by age, program year, region, or area of interest.

Qualitative Data Analysis

During data collection, the contractor will take detailed notes and following data collection, the recordings of KIIs will be transcribed. These will become the basis of qualitative data analysis and cleaning. Once data collection is completed, the contractor will use Dedoose qualitative data coding and analysis software to conduct content, trend, contribution and pattern analysis to identify response categories and emergent themes and contextual factors, upon which to draw conclusions and develop evidence-based recommendations

5. Relevant Contacts

This evaluation was contracted through a competitive process. A number of ECA staff reviewed and approved the proposed methodology: Natalie Donahue, Chief of Evaluation (202-632-6193). Social Impact is the contractor that was selected to carry out the evaluation. Social Impact’s technical evaluation experts developed the original design in response to the solicitation, contact names and numbers for Social Impact group provided review and refinement of the proposed design upon receipt of the project files. Social Impact’s evaluation team will collect and analyze the data on behalf of ECA.

1 USDOL Bureau of Labor Statistics, May 2019 National Occupational Employment and Wage Estimates, https://www.bls.gov/oes/current/oes_nat.htm#25-0000, last updated March 31, 2020, accessed April 9, 2020.

2 Source: Office of Personnel Management, “2020 General Schedule (GS) Locality Pay Tables,” https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2020/general-schedule/

3 Source: Congressional Budget Office, “Comparing the Compensation of Federal and Private-Sector Employees, 2011 to 2015” (April 2017), https://www.cbo.gov/publication/52637. The wages of Federal workers averaged $38.30 per hour over the study period, while the benefits averaged $26.50 per hour, which is a benefits rate of 69 percent.

4 Source: U.S. Department of Health and Human Services, “Guidelines for Regulatory Impact Analysis” (2016), https://aspe.hhs.gov/system/files/pdf/242926/HHS_RIAGuidance.pdf. On page 30, HHS states, “As an interim default, while HHS conducts more research, analysts should assume overhead costs (including benefits) are equal to 100 percent of pretax wages….” To isolate the overhead rate, the Department subtracted the benefits rate of 69 percent from the recommended rate of 100 percent.

5 Based on travel to the San Francisco/Bay Valley area for 15 days for 2-person evaluation teams.

6