Measures that Matter -- Assessing Public Libraries’ Activities Related to Workforce Development
Part A. Justification
A1. Necessity of the Information Collected
Public libraries support workforce and business development activities in their communities. Examples include job search support services, computer access for online job search activities, and formal classes on starting a business. However, not much is known at the national level about the full range of library activities in this area, including how these library systems manage these types of services across their multiple outlets, how the pandemic may be affecting their responses to community workforce and business development needs, the impact of library workforce and business development services on the broader workforce and business development ecosystem, and the consistency of libraries’ ability to meaningfully measure and communicate their workforce and business development outcomes.
To better understand the role public library systems (“libraries”) play in supporting and measuring these services, the Institute of Museum and Library Services (IMLS) is supporting a study to investigate different models of workforce and business development activities in a variety of local contexts. Mt Auburn Associates will conduct the study in partnership with the Chief Officers of State Libraries (COSLA) as part of IMLS’s Measures that Matter (MtM) Initiative.1
The pilot project is designed to 1) develop a framework that articulates libraries’ roles in workforce and business development; 2) explore libraries’ design and delivery of library workforce and business development services, engagement with partners, and outcome assessment; and 3) identify potential tools to measure the impact of libraries on workforce development outcomes to help build library capacity to develop innovative programming.
Mt. Auburn Associates has developed research questions to be used in creating ten case studies of libraries engaged in innovation related to workforce development or business development service delivery and measurement (see Part B. Supporting Statement). The proposed case studies are intended to build hypotheses to support further research about how libraries organize around these workforce and business development activities independently or in conjunction with other partners, their role in the broader workforce development ecosystem, and the feasibility and utility of program measurement and communication. These purposive and non-representative case studies are intended to be illustrative of good examples of workforce and economic development service design and delivery to inform future research and practitioner dialogue.
A2. Purposes and Uses of the Data
The purpose of this project is to investigate the range and provision of library-delivered workforce and business development services, how the pandemic impacted the delivery of these services, and how libraries measure and communicate their contribution toward workforce and business development individual- and community-level outcomes. The case studies will allow us to identify hypotheses in these areas that can be further tested in future research and practitioner dialogue.
To investigate the proposed research questions, the Mt. Auburn team plans to conduct ten case studies of “influential” public library systems related to workforce development or business development service delivery. These case studies will focus on:
the role the library is playing in workforce and business development;
the types of outcomes associated with the library’s activities;
how libraries coordinate and administer workforce and business development services across their system, including the role that outlets play in multi-outlet systems;
the capacity challenges of the library to take on efforts to measure the outcomes of its work;
the partnerships the library has with other local organizations;
how other stakeholders in the workforce development and business development ecosystems perceive the libraries;
how state-level resources, supporting policies, and other contextual factors are influencing the library’s approach to workforce and business development; and
the factors that impeded or facilitated library elasticity to the dynamic environment of the COVID-19 public health crisis.
The data collection will inform the proposed study’s two key deliverables: 1) a final report with case studies and cross-site findings on key themes and hypotheses and 2) three targeted research briefs that provide public libraries and other interested groups with measurement tools for helping build monitoring capacity in this area of work.
The primary stakeholders likely to be interested in the findings from this research include COSLA, IMLS, American Library Association (ALA), Public Library Association (PLA), state and local libraries, state and local workforce development agencies, policy makers, and small business support providers. Mt. Auburn will structure the project to be responsive to this diverse stakeholder audience and focus on creating high-utility products, such as the research briefs, that identify libraries’ contributions and build library capacity for outcome measurement and communication.
The project may inform broader field learning and build research hypotheses for further investigation related to effective service delivery practices, libraries’ role in the workforce and business development ecosystem, and measurement and communication efforts regarding the impact of libraries in economic development. Libraries may identify promising practices that may better serve the needs of their communities, and more traditional workforce and small business development stakeholders, including education providers and state and local workforce development agencies, may glean learnings that influence potential partnerships with libraries.
A3. Use of Information Technology
The team will capture in-person and telephone interviews with notetaking (a rough transcript) via laptops and audio recordings with interviewee agreement. Mt. Auburn Associates will store all Microsoft Word and audio files securely per federal data security guidelines, including NIST 800-209 as well as other federal and local regulations, and will destroy all audio files upon transfer to COSLA/IMLS at the conclusion of the project. Only the Mt. Auburn Associates team will have access to the transcript-related data files.
A4. Efforts to Identify Duplication
During a landscape scan and literature review, Mt. Auburn Associates collected detailed information on the type, format, and frequency of library outcome measurement efforts currently underway that are germane to the proposed project in minimizing the need for collecting new data:
IMLS Public Libraries Survey: The public libraries survey is an online survey that librarian respondents complete. Data collected through this survey include content on the number of library visits, circulation, collection size, service hours, staffing, electronic resources, operating revenues, expenditures, and number of service outlets. There are limited elements in the survey related to workforce development; however, several states have added data elements covering this area.
Edge: Edge is a collaborative effort led by the Urban Libraries Council that involves an online assessment tool that facilitates evaluation of libraries’ community value, engagement with the community and decision-makers, and organizational management. Related to education and workforce development, Edge asks libraries about their provision of technology classes, content and e-resources for workforce development support, and their use of resources and technology to assist in patron education.
Project Outcome: Project Outcome is a data collection, utilization, and communication effort available to libraries, led by the Public Library Association (PLA) with primary funding from the Bill & Melinda Gates Foundation. It contains standardized patron-facing surveys for across seven content areas: civic/community engagement, digital learning, early childhood literacy, economic development, education/lifelong learning, job skills, and summer reading.
Impact Survey: The Impact Survey was initially part of the University of Washington Information School and now sits within the PLA. The survey collected patron feedback on their use of technology in several program areas, including education, employment, entrepreneurship, health and wellness, civic engagement, and social inclusion.
In addition to these larger data collection efforts, the Mt. Auburn team identified other related studies or activities that were currently underway that had relevance to the proposed research project:
Understanding the Social Wellbeing Impact of the Nation’s Libraries and Museums is an IMLS effort, in partnership with Reinvestment Fund, Social Impact of the Arts Project, and HR&A Advisors, to gain a better understanding of the conditions in which libraries contribute to community quality of life and wellbeing. The outcome framework and case studies, including community indicators, attribution of library impact, and proposed outcome communication models.
Larra Clark, deputy director, ALA Public Policy and Advocacy Office & Public Library Association, has engaged in ongoing research focusing on partnerships between libraries and workforce development agencies.
The ALA and Google Libraries Build Business initiative focuses on developing or enhancing libraries’ small business and entrepreneurship training to increase access for low-income and underrepresented communities. The project includes a framework with common metrics to evaluate program progress, a playbook of learnings and program recommendations, and a peer-learning network for librarians involved in small business and entrepreneurship programs. This project also includes a small sample survey to participating libraries related to their small business and economic development services.
ALA is conducting multiple and potentially ongoing surveys to collect information from libraries related to their COVID-19 responses and changes to operations and implementation.
These related surveys and data collection activities form the basis of the Mt. Auburn Associates team’s request for case study data collection. Their learnings were reviewed in avoiding duplication of data requests with this new proposed investigation. To the extent possible, Mt. Auburn Associates will be integrating information available from these existing data to reduce unnecessary questions and reduce respondent burden as well as attempt to avoid conducting the case studies concurrent with other library data collection activities or overwhelming target respondents with redundant research questions.
A5. Methods Used to Minimize Burden on Small Businesses
Mt. Auburn Associates anticipates that some interview respondents will be from small libraries, and some interview respondents may also be from smaller workforce agencies/businesses or related partner organizations.
To minimize the burden on all libraries and partner agencies, but particularly on small libraries and partner organizations with limited staff capacity, data collection processes will be clear and straightforward and limit the need for additional follow-up information. Mt. Auburn anticipates that phone and in-person interviews will take 45 minutes or less. Interviews will take place at the convenience of the interviewee.
The Project Advisory Committee reviewed the case study interview protocols to verify the relevance of questions with project scope and ensure clear, accessible language.
A6. Consequences of Less Frequent Data Collection
As evidenced by the literature review the Mt. Auburn team conducted in the first stage of the project, and confirmed by interviews with nine library stakeholders during theory of change development, without this study, COSLA/IMLS, the library and workforce development fields, in general lack a framework for building hypotheses about libraries' contributions to economic vitality within communities. Additionally, the data collection will provide critical information on library outcome measurement capacity and practices that will contribute to the development of field-building toolkits to improve library data collection and measurement. The case studies will allow for a more in-depth understanding of the research questions and reveal promising practices or common challenges related to workforce development and measurement. The project incorporates rigorous qualitative methodology with input from a variety of stakeholders, providing unique depth to build on field learning.
Since the data collection is a one-time effort, opting to forgo this data collection would hinder the ability to uncover important learnings on the issues to be addressed in this investigation. Furthermore, the pandemic-related insights will help inform future adaptations made to improve these programs and services during a time of particular need as communities begin to recover.
This voluntary data collection effort will take place over several months. The study will provide unique information on how libraries contribute to the workforce and small business development ecosystems, how libraries' role in workforce and small business development has changed as a result of the pandemic, and how libraries collect and communicate workforce and business development data and outcomes.
The proposed data collection will comply with the guidelines established in 5 C.F.R. § 1320 (Controlling Paperwork Burdens on the Public). There are no special circumstances that require deviation from these guidelines.
A Federal Register Notice published on August 20, 2020 (85 FR 51496) solicited public comment. IMLS received two comments, each requesting copies of the data collection instrument. The agency provided the requested document on November 4, 2021.
The Mt. Auburn research team, including senior methodologists, in frequent communication with the study advisory committee and client stakeholders, developed the study design and data collection instruments. These stakeholders provided input on the utility of the data collected through the instruments, the clarity of questions and instructions, possible additional data elements, and the anticipated level of respondent burden. The team revised data collection instruments based on feedback received through several facilitated sessions with the advisory committee.
Mt. Auburn Associates does not anticipate payments to case study participants. Because participation in efforts to improve library and/or workforce and business development services and outcomes are generally part of library and partner staff job responsibilities, the study will not compensate library or partner organization staff for their participation in the study (e.g., interview participation).
The team will begin all case study interviews with an overview of the study and an explanation of how Mt. Auburn will use the information.
Interviewers will inform respondents that the information they provide will be anonymized, only accessible to Mt. Auburn research staff involved in the project, and they may opt-out of any question throughout the interview. Although responses will be anonymized, certain libraries may be inferentially identifiable based on their responses, and study products may identify specific libraries to highlight promising practices and lessons learned. In these cases, Mt. Auburn will contact respondents for permission prior to including any quotes or identifying information from interview responses in study reports or products. Mt. Auburn will only provide identifying examples in study products if the respondent and the library have granted permission.
None of the data collected for this study will contain personally identifiable information (PII). Mt. Auburn will pay close attention to the security of collected data. The Mt. Auburn team will store all data in secure electronic files that are password protected and comply with federal data security guidelines, including NIST 800-209, as well as other federal and local regulations and standards. Mt. Auburn will restrict data access to team staff directly involved in the project and will destroy all data at the conclusion of the project upon transfer to COSLA/IMLS. Data sets provided to COSLA/IMLS at the end of the study will not contain any PII but will contain organizational (library) identifiers. Mt. Auburn will suppress data as appropriate in instances where small respondent pools may unintentionally expose PII.
Mt. Auburn will ensure to minimize any potential risks to participants in the project. The burden of participation is limited to the time required to participate in interviews.
No questions of a sensitive nature will be included.
Table A.1 below presents the projected burden hour estimates for case study data collection. Table A.1 estimates the time respondents will need to participate in case study interviews. Mt. Auburn estimates 130 case study respondents. The team has used an average wage rate of $27.48 for library staff respondents2 and a wage rate of $27.07 for partner organizations3 based on average national wage data. Data collection instruments will be single-use; the team will only ask respondents to participate in one interview during case study data collection.
Mt. Auburn is planning to conduct between four and 12 library staff interviews at each case study site for an estimated 80 library staff interviews, and four to six interviews with partner organization staff at each case study site for an estimated 50 partner organization interviews. Each interview will be approximately 45 minutes for a total of a 60-hour burden estimate for library staff ($1,648.80) and a 37.5-hour burden estimate for partner organization staff ($1,015.13).
Table
A.1. Burden Estimates4
Instrument |
Predicted or Target Responses |
Burden Hour Per Response |
Total Burden Hours (Predicted or Targeted Responses) |
Hourly Cost per Response |
Cost (Predicted or Targeted Responses) |
Interviews with case study library staff respondents |
80 |
0.75 |
60 |
$27.48 |
$1,648.80 |
Interviews with other case study stakeholders |
50 |
0.75 |
37.5 |
$27.07 |
$1,015.13 |
Total |
130 |
|
97.5 |
|
$2,663.93 |
Mt. Auburn does not expect any additional cost burdens to respondents beyond the labor cost of burden-hours described in item A12 above.
Costs to the Federal Government are estimated to be $170,260.00, representing $149,040.00 in salaries and wages and $21,220.00 in travel expenses, paid through a contract with Mt. Auburn Associates as part of a cooperative agreement between IMLS and the Chief Officers of State Library Agencies.
This is a new information collection request.
Activity |
Expected Completion Date |
Selection of ten “influential” case study sites* with 5 back up options |
December 2021 |
Baseline research |
January 2022 |
Plan case studies – initial call and interviews |
January 2022 |
Site visits |
April 2022 |
Final report/Research briefs |
* See Part B, Question 2 for selection criteria related to the determination of universe of influential sites
COSLA/IMLS will display the expiration date of OMB approval and OMB approval number on all instruments associated with this information collection, including case study protocols.
No exceptions are requested for this information collection.
1 Measures that Matter (MtM) is a collaborative venture between IMLS and COSLA to improve the library field’s ability to collect, report, and communicate public library outcomes and contributions to the community. MtM's identified goal is to build bridges among public library data collection activities to reduce duplication of efforts; develop a comprehensive approach to library data that includes input, output, and outcome measures; and address long-term data access.
2 “Occupational Employment Statistics: Occupational Employment and Wages, May 2021 – 25-4022 Librarians and Media Collections Specialists,” Bureau of Labor Statistics, accessed June 9, 2021, https://www.bls.gov/oes/current/oes254022.htm#nat.
3 “Occupational Employment Statistics: Occupational Employment and Wages, May 2021 – 00-000 All Occupations, Bureau of Labor Statistics, accessed June 9, 2021, https://www.bls.gov/oes/current/oes_nat.htm.
4 “Predicted or targeted responses” represents the total estimate of expected responses. For the interviews, the table presents the responses as targeted estimates (e.g.,, Mt. Auburn will conduct interview outreach to achieve the targeted number of respondents, understanding that there may be slight variation in final participation by respondent group (e.g., one site may have five library staff respondents and three partner stakeholder respondents based on the unique context of that site).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Eldridge, Matthew |
| File Modified | 0000-00-00 |
| File Created | 2021-11-11 |