AmeriCorps Social Network Analysis_Supporting Statement B_5.12.15

This section provides supporting statements for each of the five points outlined in Part B of the OMB guidelines, in order to collect information for a social network analysis (SNA) study of select sites of AmeriCorps State and National (ACSN) grantees and other organizations in the communities they serve. As noted in Part A, the study will be carried out in five communities served by ACSN grantees. This submission seeks clearance for conducting the Partnership and Collaboration (PAC) survey as part of the AmeriCorps State & National Social Network Study, with respondents from organizations in five AmeriCorps grantee communities.

B.1 Potential Respondent Universe and Sampling Methods

The PAC survey aims to understand the degrees of collaboration among organizations operating in the communities served by ACSN grantees, and that work on common goals and focus areas. It should be noted, the definition of community is based on geography, not population demographics or group membership. The specific geographic boundary of a community was not defined, but rather allowed to be defined by the grantees. This was done because ACSN grantees target different levels of geographies depending on their specific circumstances, including intervention, resources, and the area being served, among other factors. Determining the respondent universe is done in two stages, first being the identification of the sites for the survey administration followed by identification of the final respondent population.

Identifying Sites for the Study

A list of ten candidate AmeriCorps grantees was identified by the CNCS Office of Research & Evaluation, in conjunction with ACSN program officers. ACSN grantees typically operate in a number of sites or service locations within a community, and this study focuses on collaboration at the site level. The grantees were contacted to provide recommendations for their service locations to administer the PAC survey. Interviews were then conducted with these sites, following a semi-structured interview protocol. One of the grantee sites was dropped from the list for being unresponsive, leaving a list of nine candidate sites. In general, interviews with sites asked about:

The number of organizations working on the grantee’s issue area and serving the same geographic community.
The grantees’ perceptions of the amount of collaboration among the identified organizations.
The grantees’ perceptions of their own role in the network of organizations.

After completing the interviews, the results were analyzed together with administrative data and public records (US Census, IRS 990 filings for non-profits, etc.) to select the final five grantee sites in which to administer the PAC survey and subsequently conduct the social network analysis. Criteria for site selection included:

The grantee explicitly uses collaboration with other organizations in the local community as a strategy to meet its objectives.
The network within which the grantee operates is expected to be bounded, meaning there are a set number of network actors.
The grantee identifies at least 2 and no more than 50 core partners.
The selected grantees as a group represent different geographical areas, including rural/urban, and region.
The selected grantees as a group work in at least three service focus areas.

Exhibit 1 shows the list of nine grantee sites, how they rated against the criteria, and the final inclusion.

Exhibit 1. Candidate Sites for Administering the Partnership and Collaboration Survey

Grantee	Site	Uses collaboration	Bounded network	More than 2, less than 50 partners	Urban/Rural	Census Division	Focus area
Sites Selected
National Association of Community Health Centers	Heart of Texas Community Health Center, Waco, TX	Yes	Yes	Yes	Rural	West South Central	Healthy Futures
Reading Partners	Tulsa Public Schools, Tulsa, OK	Yes	Yes	Yes	Urban	West South Central	Education
Jesuit Volunteer Corps Northwest	Gorge Grown Food Network, Hood River, OR	Yes	Yes	Yes	Rural	Pacific	Healthy Futures/Capacity Building
Vermont Housing and Conservation Board	Burlington, VT	Yes	Yes	Yes	Rural	New England	Economic Opportunity
FoodCorps	Detroit Black Community Food Network, Detroit, MI	Yes	Yes	Yes	Urban	East North Central	Healthy Futures
Sites Not Selected
Project Homecoming	New Orleans, LA	Somewhat	YES	YES	Urban	West South Central	Economic Opportunity
Citizen Schools	Orchard Gardens School, Boston, MA	Yes	No	No	Urban	New England	Education
National Association of Community Health Centers	Family Health Care Network, Visalia, CA	Yes	Yes	No	Rural	Pacific	Healthy Futures
American Legion Auxiliary	Volunteers of America Indiana, Fort Wayne, IN	Yes	No	No	Rural	East North Central	Veterans and Military Families

The four sites not selected did not fully meet at least one criterion. The five selected sites met all criteria, and respondents from the sites were enthusiastic about the study.

Respondent Universe for PAC Survey

Because network analysis methods focus on relations among organizations, organizations cannot be sampled independently to be included as observations. If one organization happens to be selected, then all the other organizations it is connected to will need to be included in the sample. As a result, network approaches tend to study whole populations by means of census, rather than by sample. The participants for this study will consist of all organizations within an AmeriCorps grantee community that work on the same focus area as the grantee. Because a census will be conducted, no complex sampling design will be used and survey weights will not be calculated.

Within the five sites chosen, the potential respondent universe is defined as all organizations in the geographic community providing services addressing the focus area of the ACSN grantee, including organizations in the private, public, non-profit, and religious sectors. Defining the network boundary around a set of actors (which in the case of this study is the population of organizations in the grantee communities) is a key step in designing a SNA.¹ Various approaches have been recommended.² Our method will combine the positional and relationship-based approach. The positional approach defines network membership depending on the presence of a specific attribute – in our case, the organization must exist in the geographical area and in some way contribute to the objective of the grantees’ AmeriCorps interventions or services. The relationship-based approach defines membership based on specific social relationships – in our case through snowball sampling, we will identify organizations that have relationships with others in the geographic and focus area. This overall approach will exclude organizations that do not work in the geographical area, such as national support organizations like funders or umbrella organizations. It will also exclude organizations that are in the same geographic area but have no relationship with any other organization. This approach was selected to ensure that a manageable number of organizations would be included, because as previously stated, collecting primary data for SNA requires a very high response rate and cannot be based on a sample.

It is anticipated that in each of the five sites there will be at 50 organizations that work on the same focus area as the AmeriCorps grantee; the final number of organizations surveyed is expected to be 250 respondents across the five sites. The sample for each community should represent, as close as possible, the universe of all possible respondents in order to get the most complete picture of the relationships among all of the organizations within the grantees’ networks. Omitting any organizations from the network analysis could result in an inaccurate portrayal of the network characteristics as a whole.

Respondents to the PAC survey will be an individual within each organization, authorized to speak on behalf of the organization, that is most knowledgeable about that organization’s collaborations, partnerships and interactions with other organizations in the geographic community working on the grantee’s focus area.

The universe of respondents with potentially relevant knowledge is difficult to estimate and hence snowball sampling methods will be used to identify respondents for the PAC survey. An initial list of organizations will be put together through initial calls with the ACSN grantees and also through an environmental scan of the organizations in the community that work on the same focus area as the ACSN grantee. Then, snowball sampling will be used where existing respondents help identify and recruit other respondents from organizations with which they interact the most. We anticipate conducting three rounds of sampling, the first being with the grantees themselves, the second with the organizations identified by the grantee, and the third being with the organizations identified in the second round. This sampling technique is often used in social network analysis where it is hard to establish a sampling frame and it is assumed that cases are affiliated through links that can be used to locate other respondents based on existing ones.

Potential Bias from Using Snowball Sampling

Potential biases can occur in social network analysis when snowball sampling is used.³^,⁴^,⁵Some of the major limitations are: 1) it does not yield a random sample and hence the results are not generalizable to the population of organizations; 2) the initial group of respondents selected via a convenience sample may introduce bias; 3) there is no guaranteed way to find all organizations in a population, 4) actors who are isolated or not as well connected are less likely to be mentioned or referred to us and hence omitted from the network. Therefore there is bias towards inclusion of individuals with relationships that overemphasize connectedness. These biases will be discussed as limitations in any publications or presentations produced using the study data.

B.2 Information Collection Procedures

Initial interviews with the ACSN grantees in each site will be held to identify organizations they interact with, and contact information from key individuals from those organizations will be obtained. Using snowball sampling, a list of 50 organizations per grantee will be compiled and a total of 250 respondents will be sent a pre-notification letter and a formal email invitation. The formal invitation will explain the survey, including the voluntary nature of survey completion, confidentiality, and the risks, benefits, and rights as respondents. Non-responders will receive two email reminders and a postcard reminder, a paper copy of the survey with return postage will be sent to them and finally the survey will be completed on the telephone with respondents that don’t complete the paper copy. (Attachment F: Email Reminder 1; Attachment G: Email Reminder 2; and Attachment H: Post Card Reminder).

B.3 Maximizing Response Rates

We will employ a number of strategies to maximize response rates, including keeping all survey instruments short and simple to minimize respondent burden. All participants will receive a pre-notification letter, a hyperlink to the survey, and then multiple reminders as necessary. Strategies that will be used to enhance the response likelihood include the following:

Many respondents will have been contacted in the snowball sampling, which will provide an initial contact regarding the survey. This will increase identification with the survey and respondent buy-in, as they will have contributed to the development of the sample.
E-mail requests will be individualized by respondent name.
The request will include friendly and inviting language and include an estimate of the short time required to complete the survey.
These e-mail requests will convey the potential value of results to respondents, and will indicate that the respondents’ identities will not be identified to any government agency.
Web-based survey respondents also will be provided with a “resume” capability that allows them to break off the session mid-survey and then return to the survey at a later time to complete it without losing previously entered data.
Reminder e-mail notices will be sent to non-responders beginning 3 days after the initial invitation. A total of 2 reminders e-mails (1 week apart), which include a hyperlink to the Web-based survey, will be sent to non-responders. Respondents will also receive a post-card reminder. If they still do not respond, a paper copy of the survey with return postage will be sent to them and finally the survey will attempt to be completed on the telephone with respondents that don’t complete the paper copy.
Respondents will be provided a copy of the final report when it is complete, which will have general assessments of the extent of collaboration in the network, as well as recommendations on how collaboration can be improved.
Through the snowball sampling procedure, we hope to identify leaders in the community who will be willing to be champions of the survey effort. The status of these leaders may serve to increase the response rate by validating the survey and raising awareness of the benefits of the survey to the community.

We expect to achieve at least an 80 percent response rate, and hope to achieve over a 90 percent response rate given the intensity of contact and buy-in we plan on obtaining from respondents. Although we will not calculate non-response weights, we will conduct a non-response bias analysis. The analysis will be based on a limited amount of information available on organizations identified in the snowball sampling that can be obtained from publicly available sources. We anticipate having at a minimum:

Organizational sector.
Approximate organization size.
Organization funding (based on 990 records for non-profits and religious organizations; public financial reporting for businesses required to do so; publicly-available budgets for government entities; non-existent for small businesses and non-profits that do not file public financial reports).

B.4 Tests of Procedures

The survey instruments have been reviewed extensively by the Office of Research & Evaluation, the research team at ICF International, and outside experts in social network analysis.

The survey was subjected to cognitive testing conducted by ICF International. The recruitment materials, consents, and PAC survey were piloted over the telephone with nine respondents from a CNCS grantee site from December 15, 2014 to January 15, 2015. The overall aim of the cognitive interviews was to test respondent comprehension of the survey protocol, to assess difficulty of the proposed questions, ensure clarity of the survey language, and identify complex conceptual issues that may not be readily obvious from simple reading of the questionnaire. The cognitive interviews were kept to no more than an hour to avoid respondent fatigue.

Overall, respondents reacted positively to the survey and flow of the questions. Relevant comments from respondents included:

General agreement that the survey was intuitive and the instructions were clear.
Agreement that the qualitative, free-text questions were important and would provide a more in-depth understanding of the networks in their communities.
Use of the term “your organization” was unclear. Some respondents did not know if this meant the entire organization, or just the specific office or department dealing with the target focus area. This was clarified in the final instrument.
Wording was slightly modified on a number of items to ensure greater clarity in meaning.
Respondents agreed that the length of the entire survey was reasonable. Several respondents expressed concern about the length of time required to respond to additional questions on each organization that they interacted with. In addition, many respondents believed the definition used in the pilot questionnaire of “an organization you interact with” was too vague, and it was likely different respondents would interpret differently. Rather than a binary “yes/no” question, as was piloted, the final survey asks a Likert scale of increasing collaboration based on the Hogue mode discussed in Part A of this justification. This is meant to A) provide greater anchoring to respondents on the nature of the relationship; and B) reduce respondent burden. The piloted survey asked five additional items related to any organization the respondent said “yes” to interacting with. The final survey reduces the additional items to two, and only asks them for organizations where the level of collaboration is of “coordination” or higher, which we believe will be more restrictive than the piloted instrument. The three items that were removed were seen as redundant with the question on the Hogue model.

B.5 Names and Telephone Numbers of Individuals Consulted.

The organization responsible for study design and data collection activities is ICF International. Several individuals at CNCS Office of Research and Evaluation, led by Robin Ghertner, MPP also participated in conceptualizing the design, methods and instruments. CNCS will be primarily responsible for data analysis.

CNCS Staff:

Robin Ghertner, MPP

Senior Research Analyst

Office of Research and Evaluation

Corporation for National and Community Service

Telephone Number: 202-606-6772

Email Address: [email protected]

Anthony Nerino, MA

Research Associate

Office of Research and Evaluation

Corporation for National and Community Service

Telephone Number: 202-606-3913

Email Address: [email protected]

Joseph Breems

Research Fellow

Office of Research and Evaluation

Corporation for National and Community Service

Telephone Number: 202-606-6992

Email Address: [email protected]

ICF International

Bhuvana Sukumar, Ph.D

Senior Manager

ICF International

Telephone Number: 404-592-2122

Email Address:[email protected]

Bryan Higgins, Ph.D

Technical Specialist

ICF International

Telephone Number: 703-934-3498

Email Address:[email protected]

Yisong Geng, Ph.D

Technical Specialist

ICF International

Telephone Number: 203-504-8748

Email Address:[email protected]

1 Knoke, D., and Yang, S. Social Network Analysis, Second Edition. Thousand Oaks, California. Sage, 2008.

2 Laumann, E.O., Marsden, P.V., and Prensky, D. The Boundary Specification Problem in Network Analysis. In L.C. Freeman, D.R. White, and A.K. Romney (eds.), Research Methods in Social Network Analysis. Fairfax, Va.: George Mason University Press, 1989.

3 Griffiths, P., Gossop, M., Powis, B. and Strang, J. (1993). Reaching hidden populations of drug users by privileged access interviewers: methodological and practical issues, Addiction, vol. 88, 1617-1626.

4 Biernacki, P. and Waldorf, D. (1981). Snowball Sampling: Problems and Techniques of Chain Referral Sampling. Sociological Methods & Research. November 1981 10: 141-163.

5 Van Meter, K. (1990) Methodological and Design Issues: Techniques for Assessing the Representatives of Snowball Samples, NIDA Research Monograph, 31-43.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Robin
File Modified	0000-00-00
File Created	2021-01-25

AmeriCorps Social Network Analysis_Supporting Statement B_5.12.15