Supporting Statement for Network Scale Up Method (NSUM) online survey to estimate foreign born scientists and engineers (FBSE) w

Memorandum

Date: June 16, 2023

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval of a Network Scale Up Method (NSUM) online survey to estimate foreign born scientists and engineers (FBSE) who remain long-term in the US

The purpose of this memorandum is to inform you that the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF) plans to conduct an online survey under the generic clearance for improving survey projects (OMB control number 3145-0174). “As more countries offer their students reasons to stay in their own country for their education or to return home after earning a degree, the U.S. could face a shortage in a critical segment of its workforce” (National Science Board, 2020). To address this concern, NCSES intends to build evidence to understand the availability and demand for global science and engineering training and talent. Specifically, NCSES must support policy-relevant questions such as the return on investment on US training of foreign-born scientists and engineers (FBSE) or how many FBSE go on to gain permanent resident status in the US. Through America’s DataHub Consortium, which brings together infrastructure to securely fill information gaps and take on key analytic questions and evidence building challenges, NCSES has integrated relevant data across a dozen different sources from six different Federal agencies to estimate the size of in administrative data and that surveys may provide better estimates of this hard-to-count population at lower cost. NCSES intends to explore innovative methods to improve upon the accuracy, cost, and speed associated with estimating hard-to-count populations, using FBSE as a use-case that can be validated against other methods already employed for estimating the FBSE population in the US.

The current effort will investigate a novel approach for estimating this population, employing a network scale up method (NSUM) survey on two different platforms, Amazon Mechanical Turk (AMT) and Facebook. The NSUM is a proven approach used to estimate hard-to-count populations when traditional survey methods may have high levels of response bias, recall error, or cognitive burden. The online implementation of NSUM may also serve to lower the cost and burden to the public for data collection. To the best of our knowledge, the NSUM has not been previously fielded via AMT or Facebook, so NCSES seeks to validate these modalities prior to considering them for any future data collection. This approach may improve the cost, speed, impact on public burden and reduce bias over current methods.

This effort will examine the effectiveness and utility of online convenience samples and novel survey methods by examining three research questions:

1. Does an online non-probability sample combined with the Network Scale-Up Method (NSUM) allow the derivation of reliable, unbiased estimates of the foreign-born scientist and engineer population in the U.S.?

2. Does the use of an online NSUM improve cost and reduce public burden compared to existing methods without degrading quality?

3. Do response rates and time to complete survey data collection vary across platforms and compensation levels (e.g., AMT versus Facebook versus traditional methods)?

The NSUM will be fielded using two different online platforms: Facebook and Amazon Mechanical Turk (AMT). NCSES will derive point estimates of the FBSE population from Facebook and AMT and compare them to estimates derived from other methods to include administrative data. The ability to collect this data at lower cost and burden to the public with greater speed, accuracy and less bias would not only be a benefit for estimating the FBSE population but also validate its use for other hard-to-sample populations.

Background

The NSUM is an established statistical sampling approach designed to avoid sampling bias and response bias when social desirability is a concern or access to a target population is challenging. It extends the seminal works of noted anthropologists Russ Bernard, and colleagues (2010) on response bias to improve survey response for hard-to-count populations and is explained in further detail below. NSUM has been effectively used to estimate the size of populations at higher risk of HIV (Johnsen et al, 1995) as well as other public-health relevant populations (Bernard et al, 2010). It has been used to estimate the size of homeless populations and unreported crime (Killworth et al, 1998). Typically, NSUM is conducted as interviewer- administered surveys, which is time consuming and resource intensive. Given the rise of online convenience samples, the effective use of an online-delivered NSUM remains an open research question.

Two online delivery platforms will be used to deploy the NSUM. Amazon Mechanical Turk is a crowdsourcing platform that can reach a large population sample relative to other platforms (Chandler, 2018; Chandler et al, 2017). In addition to online crowdsourcing platforms, social media platforms such as Facebook provide an alternate source for non-probability sampling. Facebook is one of the most widely used social media platforms in the US with demographic distributions similar to the national population (Auxier and Anderson, 2022).

Purpose of research

This survey is in support of a research effort to validate the use of an online NSUM to estimate population size in hard-to-count populations, specifically for the FBSE population that remains in the US long term.

This effort extends prior research with NSUM, convenience samples, and online surveys in three ways: 1) by determining if an online-delivered NSUM can be used to answer research questions in a faster, more cost-effective manner with less burden on the public; 2) determining if using this approach will allow more accurate estimation of more hard-to-enumerate subpopulations; and 3) increasing understanding how cost, response rates, and time to complete survey data collection may vary across platforms and compensation levels (e.g. AMT versus Facebook versus traditional methods).

The NSUM estimates hard-to-count or hidden populations by asking respondents to report on the number of people they know from countable subpopulations, such as the number of people they know that had a baby in the last year, have the first name Michael, or are employed as a primary school teacher. This allows creation of a respondent-specific estimate of the fraction of the population known to the respondent. This estimate can be used with questions such as the number of foreign-born scientists known to the respondent to estimate the likely population size. The currently used model for NSUM is:

where there are L known subpopulations. (Here i is the individual, who knows m_ij in subpopulation j.) Network size is (the sum of all the people reported as known in subpopulations of known size, divided by the total size of those subpopulations) multiplied by the population within which the subpopulations are embedded. Countable populations should be those that make up 1-5% of the full population to avoid respondent recall bias (McCormick et al, 2010).

The goal is to validate whether it is possible to reliably estimate foreign born scientists and engineers in five categories: 1) life science, 2) physical science, 3) computer and mathematical science, 4) social science, and 5) engineering; via a fast and inexpensive online convenience sample NSUM. Through this evaluation, we hope to obtain an increased understanding of the measurement error associated with estimates of the FBSE population collected through an online non-probability sample.

We will also test the use of Facebook as an alternative approach for recruiting respondents. Seventy-six percent of the US population has a Facebook account and is similar in characteristics like age, race, and ethnicity to the US population (Auxier & Anderson, 2022). While the anticipated response rate is extremely low, the cost to deliver invitations is also extremely low and may result in a more cost-effective platform for recruiting respondents. A key limitation of this approach is that online samples, in general, and Facebook in particular, is a convenience sample and not representative of the general population. We intend to measure how results from this convenience sample compare with other estimates in a similar fashion to that conducted by Mullinix and colleagues (2015) that support the use of online convenience samples in other use-cases.

Methodology

The NSUM survey will be conducted online, with one set of participants recruited from AMT and another set of participants recruited from Facebook. Samples obtained from online NSUMs will be studied to examine the internal validity of NSUM questions as delivered on different online platforms. The results from the survey will be compared with estimates from federal surveys and administrative data to provide insight on whether an online convenience sample NSUM delivered via AMT or Facebook can shed light on hard-to-count populations, specifically FBSE that reside in the US.

The NSUM questionnaire is included in Attachment B and consists of 6 response questions and 24 control questions grouped into four different categories: occupations, health-related, crime-related, and names. The control questions consist of known, countable populations used to estimate the fraction of the total population known to the respondent. As stated earlier, the size of the countable populations should be approximately 1-5% of the total population.

The survey will be delivered only to AMT workers or Facebook users who reside in the United States, which is a delivery setting offered by the respective platforms. Specific to Facebook, a paid, survey delivery will be set to reach a diverse set of US users on the platform to ensure a random sample of delivery to US based platform users. Once participants are recruited, they will be given a unique link to the online survey instrument, which will be hosted specifically for this research allowing a single response per user. The platform will also restrict duplicate responses. The recruitment material consists of the paragraph at the beginning of the survey in Attachment A.

Analysis Plan

Analysis will first focus on validation of the online NSUM as a method and secondarily on estimating the size of the FBSE population that remains long-term in the US. For the research questions outlined above, NCSES intends to perform the following analyses:

• Comparing estimates between the AMT and Facebook NSUM samples against federal surveys and administrative data on the number of foreign-born scientists and engineers that reside in the US as described above.

• Comparing response distributions among the 24 control questions grouped by four different categories (occupations, health, crime, names) to evaluate the type of control questions that perform best in an online NSUM.

• Evaluate the reliability of NSUM estimates across different sample sizes using a cross-fold validation. Cross-fold validation is a statistical method that will leave out a random subset of data (e.g., 10% of respondents) when estimating findings to create a distribution of final estimates that allow statistical inference across methods and design decisions.

• Evaluate the response rate to inform future costs to deliver online NSUM instruments.

Participants

NCSES will use AMT to recruit up to 1,000 participants from the population of adult US-based master turkers. Master turkers are a specialized group of crowdsource workers that consistently demonstrate high accuracy and quality in responding to tasks, such as survey response. NCSES will also deliver sufficient links via Facebook to recruit an estimated 1,000 participants from the general population of Facebook users in the U.S. assuming a typical response rates of 0.4%.

Burden Hours

NCSES estimates that it will take approximately 1 minute to read the Facebook announcement (for potential Facebook respondents) and 6 minutes to complete the survey. We plan to target 1,000 AMT respondents and 1,000 Facebook respondents. Facebook estimates that for every 20 individuals who will spend 1 minute reading the announcement, 1 will completing the survey. This results in 534 burden hours as shown in Table 1 below.

Table 1. Burden Hours by Survey Modality

Modality	Time Burden	Participants	Total Burden
Read Facebook Announcement	1 min	20,000	334 hours
Complete Survey on Facebook	6 min	1,000	100 hours
Complete Survey on AMT	6 min	1,000	100 hours
Total			534 hours

Incentives to Participants

All AMT participants will receive a $4.00 payment amount that is typical for similar survey-taking tasks on AMT. The funds for respondent compensation are paid to Amazon as part of an existing firm fixed price contract. Amazon then pays the individual participants through their Amazon payment account.

Informed Consent

At the beginning of the survey, participants will be informed of the OMB control number, the expected survey completion time, and the voluntary nature of the study, as stated in the first two paragraphs at Attachment B.

Survey Schedule

The tentative schedule for the survey is as follows:

Proposed Date	Activity/Deliverable
June 20, 2023	OMB submission for approval
July 11, 2023	OMB clearance
July 11, 2023	Launch survey
August 11, 2023	Survey due date
October 6, 2023	Final report

Contact Person

John Finamore

Chief Statistician

National Center for Science and Engineering Statistics

National Science Foundation

[email protected]; 703-292-2258

Attachment A: NSUM Survey Recruitment Ad

Attachment B: NSUM Survey Questionnaire

References

Auxier, B. Anderson, M. (2022). Social media use in 2021. Pew Research Center: Internet, Science and Tech. Retrieved May 13, 2022 from https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/

Bernard, H.R., Hallett, T., Iovita, A., Johnsen, E.C., Lyerla, R., McCarty, C., Mahy, M., Salganik, M.J., Saliuk, T., Scutelniciuc, O., Shelley, G.A., Sirinirund, P., Weir, S., Stroup, D.F. (2010) “Counting hard-to-count populations: the network scale-up method for public health” Sexually Transmitted Infections, 2010 86: ii11-ii15.

Chandler, J., Poznyak, D., Sinclair, M., and Hudson, M. (2017) “Use of Online Crowdsourcing and Online Survey Sample Providers for Survey Operations.” Mathematica Policy Research Report for the National Center for Science and Engineering Statistics.

Chandler, J. (2018) “ A Feasibility Study of Using Mechanical Turk to Test Survey Questions” Mathematica Policy Research Report for the National Center for Science and Engineering Statistics.

Johnsen, E., Bernard, H.R., Killworth, P.D., Shelley, G.A., and McCarty, C. (1995) “A Social Network Approach to Corroborating the Number of AIDS/HIV+ Victims in the US.” Social Networks 17:169-187 (1995).

Killworth, P.D., McCarty, C., Bernard, H.R., Shelley, G.A., and Johnsen, E. (1998). “Estimation of Seroprevalence, Rape and Homelessness in the U.S. Using a Social Network Approach.” Evaluation Review 22:289-308 (1998).

McCormick, T. H., Salganik, M. J., & Zheng, T. (2010). How many people do you know?: Efficiently estimating personal network size. Journal of the American Statistical Association, 105(489), 59-70.

Mullinix, K.J., Leeper, T.J., Druckman, J.N. and Freese, J. (2015) ‘The Generalizability of Survey Experiments’, Journal of Experimental Political Science, 2(2), pp. 109–138. doi: 10.1017/XPS.2015.19.

  

5