ECA Language Supporting Statement PART B Revised (9-05-12)

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION

Bureau of Educational and Cultural Affairs

Office of Policy and Evaluation

Evaluation Division:

English Language Evaluation Surveys

OMB Control Number: 1405-XXXX

SV2011-0031, SV2011-0032, SV2011-0033,

2. Collections of Information Employing Statistical Methods

1. There is no sampling for this information collection, as the potential respondent universe for this information collection will be all 3,400 program participants from the E-Teacher, ETA, or English Language Specialist Programs. E-Teacher participants are foreign citizens living in their home countries, while ETA and English Specialist are U.S. citizens. Participants surveyed will cover the years between 2004 and 2009. The participants from these program years have never been evaluated in regards to these research questions before. The anticipated response rate for this entire collection is 40%. This number is based on experience with previous DoS studies that have been completed, the response rates in a currently underway study, and on contacts with the program office and the grantee organization.

2. This information collection will consist of three electronic surveys. Only the applicable survey will be administered to participants from that program. Because of the duration that has passed since participation in the program, the importance of obtaining sufficient responses for each program year, and the low total N in the programs we’re surveying, sampling would likely yield an insufficient number of responses. For the one program in which sampling may be appropriate (the ETA program, with N>2000) probability sampling would be difficult due to potential concerns with contact information for participants from earlier cohorts. As a result, the statistical methodology used will be via census. This information collection will only be conducted one time as part of the English Language Evaluation.

3. All ECA/P/V data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are located. Successfully contacting and achieving the highest possible response rates are the goals of survey administration. Our methods will include:

• Customized Intro Email: A customized intro email will be sent at the start of survey administration to encourage respondent cooperation. This email will inform them about the evaluation and will also provide ways for respondents to contact the evaluation’s contractor with any concerns or questions about the evaluation.

• Participant Contact Information Verification: Extensive contact lists for each program were requested from the respective administering grantee organizations and State Department program offices to establish baseline participation in each program over the 2004-2009 period and to obtain an initial set of contact data. In addition, ECA/P/V queried the State Department’s Alumni databases to obtain any additional or updated contact information in order to ensure that the contact lists are as accurate as possible.

• Informing the Grantee Organizations Prior: Many program participants continue to be in communication with the grantee organization that administered their exchange program long after the program has ended. Informing the grantee organizations in advance of the start of the evaluation’s data collection period will allow the grantees to vouch for the survey requests that get sent out by the contractor. Doing this will only serve this purpose in the event any of the participants contact the grantee regarding any doubt as to the legitimacy of the initial intro email that will be sent by EurekaFacts. No other information about the participants themselves will be provided to the grantee.

• Survey Reminders: Besides the initial intro email, three follow-up reminders will be sent to non-respondents to encourage them to respond over the course of the administration period, including a final reminder as the survey comes to a close that will indicate the urgency. Response rates and survey user feedback will be monitored and recorded upon each biweekly reminder to ensure a satisfactory response. ECA/P/V will also be ready to make a judgment call based on response rate status throughout the administration period to both extend the administration period as deemed fit, as well as send an additional reminder.

• Pre-testing Survey: Pre-testing the survey was extremely useful for clarifying instructions and questions, refining the response categories, as well as ensuring clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live. This in turn allowed the survey’s questions to be designed in a way in which to minimize the burden to respondents and encourage them to complete their survey.

Using such methods has in our previous experiences stimulated response rates.

This data collected is only representative of the evaluation’s respondents and all analysis of results and future reports will be clearly linked to only the universe that was surveyed. We will monitor the potential for non-response bias, including tracking response rates by cohort over the collection period and reviewing both respondent and non-respondent demographics. These factors will be taken into account in our analysis and reporting of results, especially when disaggregating the data according to key demographics for which the number of respondents may be less than ideal.

4. To enhance each questionnaires design, a small number of formative interviews were conducted. For each survey, five (5) former program participants were interviewed prior to the survey development phase. Each program’s questionnaire included a distinct set of questions, thus complying with the PRA information collection requirement that identical questions were not asked of 10 or more respondents. These interviews increased each questionnaire designers’ level of understanding in regard to program participants’ experiences, particularly in terms of identifying the full range of activities, interactions, roles, and outcomes associated with program participation. In addition to formative interviews prior to questionnaire design, a small number of cognitive/pre-test interviews were conducted upon completion of the questionnaire design phase. As part of these interviews a small number of past program participants, completed a test version of the on-line survey and were later de-briefed through telephonic interviews or via e-mail to identify any needed modifications to the instrument prior to OMB submission. The debriefing interviews focused on determining whether question wording was clear, conveyed its intended meaning, contained realistic and mutually exclusive response options, and presented scaling of magnitude, agreement/disagreement, etc. that is relevant and understandable to the respondents.

5. The ECA/P/V individual managing this evaluation’s external contractor (EurekaFacts) who will be collecting the data and analyzing the information is Julien Kreuze, 202-632-6317.

2