Part B

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This collection will employ statistical methods for each survey and interview.

Respondent Universe: The respondent universe for this survey includes 171 host institution supervisors (44 of which participated in a reciprocal exchange to visit the foreign fellow) and 1,373 host institution point of contacts (487 of which participated in a reciprocal exchange) who interacted with a PFP fellow between 2013 and 2018.

Given that there is no additional cost or effort to sending emails to the entire universe and the desirability of having as large sample as possible with which to demonstrate the impact of the PFP on American stakeholders, we recommend a census approach, rather than a sampling approach for this survey. Based on past experience implementing web surveys with respondent groups for whom contact information is not regularly updated, we anticipate an overall response rate of 20%.

The contractor will employ the same census approach for the Homestay Host Survey as for the survey discussed above. There are approximately 855 individuals who hosted foreign fellows who participated in the PFP between 2013 and 2018 that will be invited to participate in the survey. We anticipate an overall response rate of approximately 10%.

Data collection methodology and maximizing response rates: To protect privacy (and to facilitate sending repeat reminders), the contractor will utilize Survey Monkey for data collection. The survey will be structured so that it follows a logic based on each respondent group, excluding any non-relevant questions and maximizing question applicability to each group. As soon as the contractor has entered the survey into Survey Monkey, we will conduct a pre-test by sending the link to internal contractor staff, to the Evaluation Division, and to Program staff to check for spelling and correct skip patterns. Any deficiencies will then be remedied.

The contractor will send a general survey link to the census of respondents with contact information. The invitation to complete the survey will include an estimated completion time, which should encourage participation. Once the survey has been launched (via an introductory email including the survey link), the contractor will send period reminders boost the response rate. Based on prior experience, we expect the survey to be open for 6 – 8 weeks, requiring two to three reminders. If response rates remain low despite reminders, the contractor will ask implementing partners to assist with promotion of the survey since those organizations have established relationships with many of the respondents. The contractor will track emails that are not deliverable, and will not send the reminders to these individuals.

For both surveys, analysis will be detected to determine if there are any significant differences between the survey respondents and non-respondents with respect to program year, program theme, program implementer, and respondent type. As the data presented in the final report will be descriptive of the program as a whole, any such non-response differences (biases) will be duly noted in the methodological section of the report. The goal is to demonstrate the extent to which the respondents are representative of the stakeholder population, to set the context for the results.

Relevant Contacts: This evaluation was contracted through a competitive process. General Dynamics Information Technology (GDIT) was selected to carry out the evaluation. GDIT’s technical evaluation experts (Dr. Marta Muco and Dr. Karen Aschaffenburg) developed the original design in response to the solicitation. A number of ECA staff reviewed and then approved the proposed methodology: Natalie Donahue, Chief of Evaluation (202-632-6193), Elizabeth Botkin (202-632-6423) and Linnéa Allison (202-632-6060). GDIT’s evaluation team will collect and analyze the data on behalf of ECA.