Digital Inclusion Corps Pilot Project Section B --FINAL

Digital Inclusion Pilot Project Evaluation
Section B. Description of Statistical Methodology
B.1. Respondent Universe and Response Rate Estimation B.1.1. Universe
The target universe for the Digital Inclusion Pilot Project Evaluation is approximately 30 organizations in
five pilot sites. Each pilot site will be led by the designated Anchor Institution (2 museums, 3 libraries total)
and include and up to 5 community partner organizations per site (social service, education, public health,
employment centers, etc.) totaling 25 community partners in all.

A. 1.2 Estimated Response Rate
Our goal is to obtain 100% response rate of anchor institutions and 80% community partners at each pilot
site. Participation in survey activities is expected to be high because Digital Inclusion Corps members will
be in close contact with community partners identifying their organizations’ needs and creating solutions to
address those needs. In addition to a small sample size, each organization will have input on the areas of
focus for digital inclusion activities and setting priorities for the local project.

B.1.3 Respondent Selection
Anchor institutions serve as the lead implementing organization and each will designate a respondent for
survey activities. Digital Inclusion Corps members will identify and recruit community organizations to
partner in local Digital Inclusion Pilot Project activities. Partner organizations activity engaged in the
project will be informed and invited to participate in this research study and will provide contact information
to Digital Inclusion Corps members to be forwarded along to the research team. Respondents from these
partner community organizations will be engaged with the anchor institutions in digital inclusion activities
through this pilot project.

B.1.4 Prior Data Collection
The Digital Inclusion Corps Project is a national pilot project; there have not been prior data collection
activities.

B.2. Procedures for the Collection of Information
B.2.1 Design
The summative evaluation in five pilot sites uses a mixed methods approach drawing on data produced
from respondent surveying and project administrative sources. (e.g., quarterly status reports, local digital
inclusion plans, digital inclusion corps members’ blog posts and other materials).. In each pilot site, a
respondent from the anchor institution will be surveyed pre-project, post-project as well as doing a selfevaluation at project’s end. Pre-post project surveying additionally will be done in each pilot site for one
respondent per community partner organization, maximum of five community partner organizations.
At each of the five pilot sites, a respondent for the anchor institution and one for up to 5 community
partners will be invited to participate and complete an online survey. A paper version of this survey will be
available upon request. Appendix A contains a copy of the questionnaires.
The first eight questions in the questionnaires have survey respondents provide background information
(Q1-8). (Appendix A contains a copy of the questionnaires). The rest of the survey items correspond to
the evaluation’s research questions as outlined below:

1

Anchor Institution

Pre Test Question

Post

End of Project Self
Evaluation

Build institutional
capacity in identifying
community digital
inclusion needs
Strengthen internal
capacity to provide
digital inclusion
services to the
community
Engage the community
regarding digital
inclusion needs
Services provided to
community
organizations in regard
to their digital inclusion
needs
Capacities of anchor
institutions and/or
partner organizations
Activities, services or
resources provided to
the community or
constituent groups

Q: 9, 9d, 9e, 12, 13,
14c, 15, 15a

Q:1, 8, 8a 4, 7, 10, 11,
11a

Q: 1, 2, 3, 4, 5

Q: 9a, 13, 14, 14a, 14b,
14c, 15

Q: 1a, 7, 9, 9a, 9b, 10,
11

Q: 1, 2, 3, 4, 5, 6, 7

Q: 9, 9c, 9d, 10, 10a,
11, 11a, 12

Q: 1, 4, 8, 3, 3a, 2a, 4,
5, 5a

Q: 1, 2, 3, 4, 5, 7

Q: 9b, 10a, 11a, 12a

Q: 6, 3a, 2, 3, 3a, 4, 5,
5a

Q: 4, 5, 6, 7

Q: 9b, 10a, 11a, 12a,
14, 14a, 14b, 14c, 15

Q: 6, 3a, 2, 3, 3a, 4, 5,
5a, 9, 9a, 9b, 10, 11

Q: 5, 6, 7

Q: 9b, 9c, 10a, 12a,
14c, 15

Q: 6, 4, 5, 3a, 5a, 10,
11

Q: 3, 4, 5, 7

Q: 12, 13, 15, 16, 17,
19, 20, 20b, 21, 22, 23

Q: 1, 1a, 2, 4, 5, 7, 8,
10, 12, 13, 14

Q: 10, 11, 12, 21

Q: 1, 1a, 12

Q: 10,11,12, 13, 14, 15,
16, 17, 18, 19, 19a, 20,
20a, 20b, 21, 22, 23,
Q: 13, 17, 18, 19, 19a,
20, 20a, 21, 22

Q:1, 1a, 2, 3, 4, 5, 6, 7
8, 9, 10, 11, 13, 14

Q:12, 13, 14, 15, 16,
17, 18, 22, 23

Q: 1, 1a, 2, 3, 4, 5, 6,
13, 14

Q: 12, 13, 14, 15, 16,
19a, 20, 20a, 20b, 21,
22, 23

Q: 1, 1a, 2, 3, 4, 5, 7, 8,
9, 10,11, 12, 13, 14

Community Partners
Build institutional
capacity to identify
community digital
inclusion needs
Strengthen internal
capacity to provide
digital inclusion
services to the
community
Engage the community
regarding digital
inclusion needs
Services provided to
community
organizations in regard
to their digital inclusion
needs
Capacities of anchor
institutions and/or
partner organizations
Activities, services or
resources provided to
the community or
constituent groups

Q: 2, 5, 6, 7, 8, 9, 11,
13

2

B.2.2 Communications and Access
Harkin Consulting Services will release the pre-project surveys to collect baseline activities on digital
inclusion activities for the 5 respondents in anchor institutions from March 20-April 17. This firm will begin
pre-project surveying of community partners two weeks following OMB approval
A maximum of two follow-up emails will be sent to participants as a reminder to non-respondents and
respondents who have only partially completed the survey. A paper questionnaire will be made available if
a participant specifically requests this method.

B.3. Methods to Secure Cooperation, Maximize Response Rates, and Deal with NonResponse
B.3.1 Sample and Contact Validation, Emailing and Tracking
On-site anchor institution staff, including Digital Corps members will supply a contact list of unique email
addresses that will be imported into SurveyMethods’® survey engine, or equivalent, allowing Harkin
Consulting Services to track individual response rates and schedule reminders for non-respondents and
incomplete responders.
A maximum of two reminders will be sent prior to the closure of the survey.
The post-project implementation and anchor institution self-evaluation surveys will be administered at the
end of the 2017 calendar year.

B.3.2. Gaining Cooperation
Harkin Consulting Services will invite DICPP participants to complete the online survey by email. This
email includes consent language, survey instructions, and appropriate contact information. (Appendix B:
Survey Invitation and Reminder Communications)
The survey invitation email will be from NDIA Executive Director, Angela Siefer. On-site Digital Corps
members will also communicate with community partners to ensure they are aware of the survey release.
Follow up reminder emails (a maximum of two) will be sent to participants as a reminder to nonrespondents and those respondents who have only partially completed the survey.
No monetary incentives or personal benefits are offered in exchange for participation in online surveys.

B.3.3. Technical Methods to Maximize Response Rates
PAST Foundation will host the online survey using SurveyMethods® (www.SurveyMethods.com), or
equivalent, and Harkin Consulting Services will use the software’s design engine to develop an instrument
that will include multiple-choice, and Likert scale questions. The survey will incorporate headings,
sections, and conditional logic branching to optimize the user experience and thereby decrease their
reporting burden.

Survey Support. In addition to the field-level online help provided by the SurveyMethods® website, or
equivalent, content and technical support will be provided by email or phone by Harkin Consulting and/or
PAST Foundation.

3

Skip Routines. Data quality will be enhanced through a set of user feedback mechanisms such as logical
relationship validation checks and skip logic. Harkin Consulting has determined the pattern of skip logic
required during instrument development. By employing skip routines, participants will only be asked to
respond to questions that are relevant to their situation.
Progress Bar. The survey will be presented in section-level and/or sub-section level pages, with data
validation and response storage occurring on each individual page. Survey respondents will be able to
save their responses at any time and return to complete the survey over multiple visits. A visual feedback
mechanism will indicate the progress a user has made through the survey. These mechanisms set
expectations throughout the survey and help participants anticipate completion of their task.
Confidentiality and Data Security. All data transferred to and from the SurveyMethods® website, or
equivalent, is protected through encryption protocols such as Secure Sockets Layer (SSL). The survey
URL will begin with “https”, which will encrypt the data being transmitted. The survey specific to this
project is password protected.
All data downloaded from SurveyMethods®, or equivalent, will be in Microsoft Excel (.xls) format for data
analysis. All data files will be kept safe and protected according to protocols established at PAST under
the guidance of their Internal Review Board.
After publication of the summative evaluation by Harkin Consulting Services, LLC, the raw data will be
destroyed further protecting participants.
Only members of the evaluation team will have access to the data, unless otherwise required by law. No
personally identifiable information will be included in raw data that is publicly disseminated, unless
otherwise required by law.
Response Rate Monitoring and Reminders. Harkin Consulting Services will monitor response rates on
a weekly basis and provide status reports to the program manager. Over the collection period, reminder
emails generated through SurveyMethods®, or equivalent, will be sent to non-responders and partial
responders. In the event response rates are lower than expected, Harkin Consulting may decide to extend
the data collection period after consultation with both IMLS and the PAST Foundation.
Paper Survey. If requested, DICPP participants may receive a paper-version of the survey. This
document will contain the exact same questions as those found online. Instructions and skip-logic
directions will be included. Harkin Consulting Services will manually enter this information and merge the
data with that collected online. Any raw data produced by paper and pencil will be stored in a locked
environment and protected according to protocols established at PAST under the guidance of their
Internal Review Board. After publication of the summative evaluation by Harkin Consulting Services, LLC,
the raw data will be destroyed further protecting participants.

B.4. Tests to Minimize Burden
Additionally, the online survey mechanism (on SurveyMethods®, or equivalent) has been tested to assess
the survey validation routines; skip patterns, and overall time it takes to enter survey response, as well as
specific timing for sections of the questionnaire. Harkin Consulting Services and PAST Foundation staff
4

has conducted this testing, essentially test-driving the survey mechanism with dummy data. Harkin
Consulting Services will make any further mechanism adjustments to the web-based survey prior to data
collection.

B.5. Individuals Responsible for Study Design and Performance
The following individuals are responsible for the study design and the collection and analysis of the data for
Digital Inclusion Corps Pilot Project Evaluation.

Personnel Involved with the DICPP Evaluation
(*Only Angela Siefer and Kathleen Harkin will have access to the raw data)
Person
Address

Email / Phone

Angela Siefer*
National Digital Inclusion Alliance

1001Kinnear Road
Columbus, OH 43212

[email protected]

Monica Hunter, Ph.D.
PAST Foundation

1001Kinnear Road
Columbus, OH 43212

[email protected]
614-340-1208

Maria Cohen
PAST Foundation

1001Kinnear Road
Columbus, OH 43212

[email protected]
614-340-1208

614-537-3057

Matthew Birnbaum, Ph.D.
955 L’Enfant Plaza North, SW,
Supervising Social Science Researcher, Institute of Suite 4000
Museum and Library Services
Washington, D.C. 20024

[email protected]
202-653-4760

Kathleen Harkin*
Harkin Consulting Services, LLC
Principal

[email protected]
614-886-9084

123 E. Whittier St
Columbus, OH 43206

5