Revised Supporting Statement Gender Surveys (6-2012)

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

Bureau of Educational and Cultural Affairs

Office of Policy and Evaluation

Evaluation Division:

Gender Assessment Surveys

OMB Control Number: 1405-XXXX

SV2011-0027, SV2011-0028, SV2011-0029, sv2011-0030

1. Justification

1. The Department of the State’s (DoS) Bureau of Educational and Cultural Affairs (ECA), Office of Policy and Evaluation, Evaluation Division (ECA/P/V) is requesting a new information collection to conduct a new ECA formative evaluation. ECA evaluations to date have provided evidence of the personal and professional achievement of participants on ECA programs and changes in the institutions they represent. However, to date ECA evaluations have not focused on how our programs are experienced and utilized by women and men specifically.

This new Gender Assessment evaluation is a formative, descriptive and exploratory study which will investigate the similarities and/or differences that men and women express in terms of their program experiences, professional development, and any application of what they learned while on the ECA programs once they return home. . The study is not geared towards proving or testing a hypothesis, and includes the following four ECA exchange programs: International Leaders in Education Program (ILEP), Fortune/U.S. State Department Global Women’s Mentoring Partnership (Fortune), Institute for Representative Government (IRG), and American Fellows Program (Fellows). This information collection will include four similarly structured surveys, one tailored to each program and that will be sent to that program’s participants. Program participants took part in these programs between 2003 and 2010. Although the four programs of study are rather small in terms of the number of participants, these programs were specifically selected for an exploratory study on gender differences and/or similarities because they are all professional programs with a very targeted professional development focus (i.e. they are not academic programs or cultural immersion programs etc.). The similar focus of the programs enables the study to achieve a broad degree of standardization of study topics (e.g. professional learning, professional application, professional networking etc.) which facilitates the design of a formative, exploratory and descriptive study. None of these programs are geared towards gender impact, so to reiterate, this is not an impact evaluation but purely an exploratory study.

This study will specifically focus on the professional development of men and women, on their individual learning and skills obtained, on their communication with peers and colleagues and on their application by men and women, at their places of work, in their communities, and at their respective organizations and institutions. Because the programs are vastly different each program will be analyzed separately and not aggregated under one finding. The findings will also not be compared against each other, or used to generalize beyond the group represented in the survey.

The data captured will help DOS and ECA Bureau successfully meet organizational performance and accountability goals established through the following mandates. Because a prime mission goal of DoS is to expand our reach to underserved groups, particularly women, this study will help DoS and ECA Bureau report on our organization performance by specifically collecting data disaggregated by gender. The information gathered in this explorative study potentially will provide insight which will enable our ECA programs offices to better understand the needs of women specifically. In this way our exploratory study will address the OMB directive (listed below) to expand information made available to the public of evaluations underway that study alternative approaches for achieving outcomes and/or determine effective strategies for the programs in this study. That said, there are some potential limitations to this particular data collection; consequently, this study will not be used to report on program wide outcomes or program performance in terms of meeting stated program objectives/goals.

Further information can be found at the following links:

• Mutual Educational and Cultural Exchange Act of 1961, as amended (also known as the Fulbright-Hays Act) (22 U.S.C. 2451 et seq.)

http://www2.ed.gov/about/offices/list/ope/iegps/fulbrighthaysact.pdf

• Government Performance and Results Act of 1993 (GPRA)

http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html

• OMB Increased Emphasis on Program Evaluations specified Memo

http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-01.pdf

As stated in the memo, “OMB will work with agencies to make information readily available online about all Federal evaluations focused on program impacts that are planned or already underway” as part of a three-pronged effort to strengthen government-wide program evaluation efforts. The guidance noted that public availability of program evaluation information will promote transparency, since agency program evaluations will be made public regardless of the results.

2. The primary purpose of this information collection is to enable ECA/P/V to further examine how these four exchange programs have affected women and men similarly or differently. This is an area of study in which ECA has never before gathered data. The data collected will provide a starting point for discussion in ECA Bureau regarding what we have learned in terms of what men and women say about our programs and their experiences. This may in turn, enable some Program Offices to consider future program design issues or adjustments, information dissemination and outreach initiatives.

As with all of ECA evaluations this assessment will examine ECA participants in highly, complex environments. The current study will allow for summary and descriptive statistics across a wide array of topics and will provide State Department leadership, ECA senior management, and program officers with data they currently do not have, and data that can potentially be used to improve extant programs, and inform on-going and future activity design with an eye towards gender issues (and similarities).

The table below lists the major research questions developed for this Evaluation, the outcome measures, the type of analysis that may be conducted, and provide contextual information for understanding the affects of these four exchange programs on men and women specifically. The data source that will be used to answer all of the major research questions are from the set of on-line survey questionnaires.

Table 1

Major Research Question	Outcome Measure	Type of Analysis
What knowledge, skills or experience did women and men gain in the program? Are there differences between men and women in the knowledge, skills or experience that they gained?	Answers to questions addressing knowledge, skills and experience that program participants gained (e.g., in relation to gaining new knowledge and skills, building relationships, and various aspects of gaining cross-cultural understanding).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
How do women and men apply what they have learned? Are there differences in approach to application?	Answers to questions addressing the ways program participants apply what they have learned, both in their work/organization and outside of their work environment, the activities they engage in both environments, and those that had the most impact (e.g., whether they have been able to apply their new program-specific new knowledge and skills, new skills in building relationships and collaborations, and new or enhanced cross-cultural knowledge and understanding; whether they engaged in particular new activities at work and outside of work, and which of those activities had the greatest impact and why).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
What new or different professional opportunities are now available to women and men, which they would attribute to participation in the program?	Answers to questions addressing the new or different professional opportunities that are available to program participants as a result of their participation in the program (e.g., given access to new resources that they can use for their work, expanded professional development opportunities, favorably changed the way their supervisor(s) view them, helped them gain recognition as an expert, and other).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
Are women and men participants now in leadership positions at work, in their communities, or organizations/institutions?	Answers to questions addressing the impact of participation in the program on participants' career outlook as well as impact on their work and non-work environment, the types of activities they engaged in both environments upon their return, their serving as a mentor to others in their work place and outside of their work environment, and their changed understanding of what it means to be a leader in their profession (e.g., whether their participation in the program contributed to their taking new leadership roles and responsibilities, led to promotion within their organization, improved their organization's prestige/recognition in their community, helped them obtain a new job in another organization, and other).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
What barriers have been encountered in applying knowledge and skills by women and men in their home environments?	Answers to questions addressing the factors that impeded participants' ability to implement the new knowledge and experience (e.g., lack of access to resources, lack of availability of educational opportunities, social norms, cultural values, and other).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
Can we determine outcome “multiplier effects,” with respect to women’s and men’s participation in an ECA program?	Answers to questions addressing the new activities that were undertaken by participants in their work and outside of work environments, and the extent and ways in which participants shared their newly acquired knowledge, skills and experiences with various groups/audiences (e.g., who did they share their new knowledge, skills and experience with, how and via what channels and activities).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
Do men and women participants now have access to key resources necessary to professional development and advancement, such as: training opportunities, networking opportunities, mentoring, and technology required by their work? To what effect?	Answers to questions addressing the factors that facilitated or impeded participants' ability to implement the new knowledge and experience upon their return home (e.g., access to resources, availability of educational opportunities, being mentored by someone in their organization or outside of their work environment, access to U.S. Department of State exchange program alumni activities in their country, and other).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
Do men and women participants maintain long-term contacts with different groups of people they meet while participating in the program?	Answers to questions on the frequency of contacts maintained by program participants with various stakeholders since their return home (e.g., contacts with other program alumni from their country and from other countries, contacts with their US mentors and hosts, and other).	Basic descriptive statistics Cross-tabular analysis by demographics as needed
How has program participation changed their personal lives?	Answers to questions addressing the ways in which program participation changed program participants' personal lives (e.g., whether it helped them better understand people from different cultures, establish long-lasting friendships with other participants and/or with American counterparts, bring positive change to their communities, and other)	Basic descriptive statistics Cross-tabular analysis by demographics as needed

3. The information collection surveys will be web-based; via an online surveying application Survey Gizmo to ease any burden on the participant. We anticipate that there will be a limited number of respondents for one program who may require a hard copy questionnaire.

4. Currently, no duplicative information exists as ECA has never conducted this type of exploratory study before, and there is no other reliable method for ECA to collect the information needed to fulfill the requirements of the Department’s annual strategic planning and reporting process and the annual Congressional budget process as part of the GPRA, PART and PMA mandates.

5. Information collected under this collection will have no impact on small businesses and other small entities.

6. If the information is not collected, ECA will be unable to complete this study or gather the data requested by ECA leadership, in order to document how these four programs affect women and men participants, which will provide insight into future programming. Moreover, the Department will be unable to comply fully with its congressional and executive mandates, including OMB’s mandate to evaluate and report the results of its exchange programs as mentioned under section 1. There are no technical or legal obstacles to reducing burden.

7. There are no special circumstances.

8. ECA/P/V has solicited public comments on this collection via a 60-day Notice published in the Federal Register (Vol. 76, No. 221, Nov. 16^th 2011) seeking public comments. One comment was received. Upon reviewing the comment, ECA/P/V determined that the comment was unrelated to the information collection and instead addressed broader Department wide policy and budget regarding the programs. ECA/P/V has consulted with an external contractor, Research Solutions International (RSI) about the surveys design, methodology, analysis, and data collection approach.

9. No gifts or payments will be made to the respondents.

10. Agency will keep the information private to the extent permitted by law.

11. No questions of a sensitive nature are asked in the survey.

12. It is estimated that the annual hour burden will be 404 hours for 778 respondents. The table below breaks down the burden and respondent numbers by each survey. Because this survey will only be conducted once, the three year total is the same as the annual total.

Table 2

Respondent Burden per Survey

	ITEM	ANNUAL TOTAL	3 YEAR TOTAL
FORTUNE Survey	Estimated Number of Respondents	146	146
	Estimated Number of Responses	146	146
	Average Hours Per Response	35 Minutes	35 Minutes
ILEP Survey	Estimated Number of Respondents	257	257
	Estimated Number of Responses	257	257
	Average Hours Per Response	35 Minutes	35 Minutes
IRG Survey	Estimated Number of Respondents	200	200
	Estimated Number of Responses	200	200
	Average Hours Per Response	20 Minutes	20 Minutes
American Fellows Survey	Estimated Number of Respondents	175	175
	Estimated Number of Responses	175	175
	Average Hours Per Response	35 Minutes	35 Minutes
Total Hours		404	404
Total Respondents		778	778

The Fortune, ILEP and Fellows surveys were pre-tested prior to this submission using anywhere between 2 and 5 current or past participants. Burden hours took this into account, as well as the total number of questions and the number of open-ended questions, as well as experience on previously conducted evaluations.

13. There are no costs incurred by respondents.

14. The data collection budget for this Gender assessment is approximately $149,551. This includes contractor labor for drafting surveys as well as the actual survey administration/data collection and software/server expenditures. The analysis and reporting budget for the data collected through this collection is approximately $89,334 and will include contractor labor for analysis, report writing and materials, and briefings

It is therefore estimated that the annualized cost to the Federal government for this collection will be $238,885

15. This is a new collection.

16. Survey data collection is estimated to begin immediately after OMB approval is received. It is estimated the data collection period will take at least 6 weeks. Once the data are collected and cleaned, an analytic data file will be developed, and data will be examined for use in descriptive analysis. A report will be developed, with several iterations submitted to ECA for review and approval. Once approved by the Assistant Secretary of the Bureau, ECA/P/V will release the final evaluation report to the public. Results for this evaluation will be tabulated by the external contracting firm (RSI), and is estimated to conclude about 9 months after the data collection period has ended.

17. ECA/P/V will display the OMB expiration date.

18. There are no exceptions requested for this collection.

2. Collections of Information Employing Statistical Methods

1. There is no sampling for this information collection, as the potential respondent universe for this information collection will be all 778 exchange participants from the ILEP, Fortune, IRG, or Fellows programs for the years between 2003 and 2010. The anticipated response rate for this collection is 60-65%.

2. This information collection will consist of four similarly structured surveys that will be administered to participants from one of the following programs: FORTUNE, IRG, American Fellows, or ILEP that were conducted between the years of 2003 and 2010. The statistical methodology used will be via census. This information collection will only be conducted one time as part of this effort.

3. All ECA/P/V data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are located. Successfully contacting and achieving the highest possible response rates are the goals of survey administration. Our methods will include:

• Updating Contact Information Prior: Contact information for program participants will be updated through online searches. All research methods in this effort will be as non-invasive as possible and only information which is publicly available on the Internet will be used. Personal websites, blogs, public message boards such as Yahoo Groups, and 3rd party directories will be used, although we recognize that some of them might contain obsolete information. These sites usually list the time of their most recent update. If the update was over 2 years ago, we will disregard the information or look for additional evidence of its relevance, such as a current news article listing the same person and organization.

• Customized Intro: Customized lead emails will be sent at the start of survey administration to encourage respondent cooperation. The email will inform them about the evaluation, and explain that the survey is voluntary. The email will also provide ways for respondents to contact either the evaluation’s contractor or the Evaluation’s Project Manager here in ECA/P/V with any concerns or questions about the evaluation.

• Informing the Grantee Organizations Prior: Many program participants continue to be in communication with the grantee organization that administered their exchange program long after the program has ended. Informing the grantee organizations in advanced will allow the grantees to vouch for the survey requests that get sent out by the contractor. Doing this will only serve this purpose in the event any of the participants contact the grantee regarding any doubt as to the legitimacy of the initial introductory email that will be sent by RSI. No other information about the participants themselves will be provided to the grantee. A general email communication will be sent out by the grantees encouraging program participants to respond to the survey questionnaire if necessary.

• Non-English Approach: Having multiple langauges will allow respondents who do not have the English capability to both understand and participate in this evaluation. We will translate the American Fellows survey into Spanish and Brazilian Portuguese in addition to offering it in English. Because the IRG program does not have an English speaking requirement the IRG survey in addition to being offered in English will also be translated into no more than 10 foreign languages. These languages are Dari, Arabic, Brazilian Portuguese, Khmer, Spanish, Russian, Bahasa, Malay, Serbian, and Turkish. These languages were selected because a large number of the program’s participants speak these languages and offering this set of 10 foreign languages beyond just English potentially ensures adequate representation across the various regions of the World (as defined by the State Department’s regional structure). Surveys will be translated and back-translated to ensure the meaning of the questions does not vary across the multiple languages offered. Open-ended responses to these two surveys likewise will be translated back into English.

The ILEP and Fortune program participants have strong English skills as the respective programs require and thus these two surveys will only be conducted in English.

• Multiple Attempts and Reminders: A limited number of follow-up reminders will be sent periodically to non-respondents to encourage them to respond.

• Pre-testing Survey: Pre-testing the surveys is extremely useful for clarifying instructions and questions, refining the response categories, as well as ensuring clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live. This in turn will allow minimum burden to respondents and encourage them to complete their survey. The Fortune, ILEP and Fellows surveys were pre-tested prior to this submission using between 2 and 5 current and/or past participants.

Using such methods as listed above has in our previous experiences increased response rates.

Additionally, the data collected is only representative of the evaluation’s respondents and therefore all analysis of results and future reports will be clearly linked to only the universe that was surveyed.

4. In order to help develop and refine survey instruments, formative interviews and cognitive interviews were conducted with past program participants for three of the program surveys. Each survey had 5 or fewer formative and/or cognitive interviews conducted.

One-on-one formative, phone interviews were conducted in order to better understand participants' experiences with the program and garner potential ways in which they could implement the newly acquired knowledge and skills upon return home. The point of this series of interviews was to assist with development of survey questions and appropriate response categories.

Cognitive interviews were one-on-one interviews where participants were probed about their interpretations and answers generated in response to the questionnaire. The point of these voluntary and confidential interviews was to get enough information about the respondents’ comprehension and preparation of a response to assess whether they understood the questions and responded as the researchers intended. A secondary goal of this exercise was to garner information to improve the construction and administration of the questions. This research activity included the participants’ evaluation, review, and completion of the survey questionnaire prior to the interview, followed by participation in a telephone or Skype interview for the “think-aloud” exercise. Individual questions were tested for clarity, relevance and comprehensibility. Typical probes in this research included questions such as: How easy or difficult was it to respond to this survey? Did you have difficulty with the meaning of any of the questions? What, if anything, was confusing? Did these response categories make sense to you? What, if anything, was missing from this list? Did you have any difficulties answering this question? The information and feedback from the respondents were used to refine the survey instrument, identify additional response categories, and pinpoint potential changes needed to make the instrument more precise in its language and meaning for the broader group of target respondents.

5. The ECA/P/V individual managing this evaluation’s external contractor (RSI) who will be collecting the data and analyzing the information is Melinda Pitts, 202-632-6314.

7