MDL Part A Justification

Museums for Museums for Digital Learning Project Evaluation

Part A. Justification

A.1. Necessity of the Information Collection

The Indianapolis Museum of Art at Newfields (IMA) received a National Leadership Grant for Museums, for a project called Museums for Digital Learning (MDL)—a two-year collaborative pilot project bringing together K12 educators and partner museums to build and pilot a shared, open source digital platform through which museums can leverage their digital collections-related data and assets to serve the educational needs of the nation’s educators and students.

Two lead museum content partners, The Field Museum and History Colorado, will contribute significant digital collections data, assets, and interpretative content to the project while co-creating the technical strategy for the pilot project and leading the educational content creation efforts.

Working collaboratively, these three museums in consultation with IMLS will identify ten K-12 educators from around the country to serve as co-creators and pilot testers of the digital educational products in the first year of the project. In the second year of the project, up to ten additional museum partners from various disciplines will test the platform, contributing their collections-based content using the standard templates and shared platform model created by the project partners. Testing of the platform and products will be undertaken with test users at key points in the project timeline. Formative and impact evaluation will be carried out by a third-party evaluator named HG&Co, an evaluation firm that possesses extensive experience in evaluating digital education platforms produced by the cultural heritage community.

About IMLS:

The Institute of Museum and Library Services is the primary source of federal support for the nation's libraries and museums. We advance, support, and empower America’s museums, libraries, and related organizations through grantmaking, research, and policy development. Our vision is a nation where museums and libraries work together to transform the lives of individuals and communities. IMLS legislative authority is located at 20 U.S.C. 9101, et seq.

About Newfields:

The Newfields Lab (formerly IMA Lab)—a department within the IMA—is a team of software developers, designers, data managers, and digital strategists that creates technology solutions for institutions in the cultural sector. Their skill set includes web development, enterprise applications, systems and data integration, search engines, metadata design and management, data-driven applications, and APIs. On the design side, Newfields Lab members are skilled in information architecture, digital graphic design, and user experience design. Most importantly, they are familiar with museum metadata standards, as well as with many of the sector-specific systems used to manage collections, documents, and digital assets.

The primary beneficiaries of this project are museums and K-12 educators.

HG&Co will conduct formative and impact evaluation of the Museums for Digital Learning (MDL) project, using a developmental approach. This evaluative approach that supports adaptation in an evolving environment and is well-suited to iterative design. The evaluator works collaboratively with the project team and community organizations to provide quick, iterative feedback to ideas and prototypes during the formative phase, including facilitating on-going, data-based decision-making. This approach also allows for iterative testing and refinement over time, resulting in greater opportunities to surface potential design solutions across multiple stakeholders and sites. The design for the MDL project is mixed methods, containing both quantitative and qualitative elements across multiple points in time to reduce bias and inform the team as the project evolves.

The evaluation research questions are outlined below.

1. What elements of the program support stakeholders? What types of changes are needed for the product to be scaled up to a larger group of museums and educators?

Museum Stakeholders:

1. To what extent does the platform allow museums to easily create and configure content for collections-based educational materials?

2. Does the platform increase institutions’ ability to serve K-12 educator needs?

K-12 Educator Stakeholders:

3. Has this platform increased educators' knowledge, access, and ability to use museum-based collections?

4. Do educators feel this platform support and enhance learning within the classroom? If yes, in what ways?

2. How effectively have the team members collaborated in designing and then testing the product?

   1. How and in what ways does the collaborative process support each stakeholder's needs?

   2. Has the team articulated shared goals, expectations, and understanding of roles and process?

   3. In what ways could the process of co-creation be improved?

A.2. Purposes and Uses of the Data

Results will be used by Newfields, History Colorado, The Field Museum, IMLS, and HG&Co to support lead museums and museum partners throughout the co-created and collaborative process, support the process of input into the design of the template, and outline the implementation challenges that may need to be addressed prior to any scale-up after MDL’s successful launch.

The final evaluation report will be shared with project stakeholders and in a publicly releasable form to the public.

We anticipate the final report will include the following sections: executive summary; introduction; evaluation research questions and goals; methodology; summary of findings; recommendations; and appendices.

A.3. Use of Information Technology

All interview data will be collected via telephone and by survey during working hours at a time that is convenient for interviewees. Screen recorders will be used to record telephone interviews.

Data collection activities will proceed after consent to participate has been secured from participants.

Once interview data is securely shared with a professional transcription service and transcribed, all audio and video data will be destroyed.

Additionally, the evaluation team will create an online survey, in collaboration with IMLS and Newfields, to simplify the data collection process. K-12 educators, as well as representatives from each of the lead content partners and additional museum partner sites will access a survey that will be developed and administered through data-protected survey software.

Data collection will proceed after consent to participate has been secured from participants. Newfields and HG&Co will confirm email addresses of the 10 K-12 educators as well as staff participant representatives from each of the lead content partners and additional museum partner sites to ensure that a final contact list is valid.

A hard copy of the online survey will be provided for those educators and museum participants that are unable to utilize the electronic process.

A.4. Efforts to Identify Duplication

As this is a new program model, this is also a new data collection effort.

This evaluation will generate formative evaluation findings specific to a co-created program model and its various implementations across the lead content partners and additional museum partner sites.

A.5. Methods Used to Minimize Burden on Small Businesses

For individuals at all institutions, participation in the MDL project development is voluntary.

A.6. Consequences of Less Frequent Data Collection

This is a one-time data collection.

A.7. Special Circumstances

No special circumstances require the collection to be conducted in a manner inconsistent with the guidelines in 5 CFR 1320.6.

A.8. Consultations Outside the Agency

Public comments solicited through Federal Register

IMLS published a notice in the Federal Register with a 60-day public comment period to announce this proposed information collection on April 2, 2019, 84 FR 12636.

IMLS published a notice in the Federal Register with a 30-day public comment period to announce this proposed information collection being submitted to OMB on July 25, 2019, 84 FR 35889.

No public comments have been received from either announcement.

Consultants outside the agency

As part of the cooperative agreement referenced above, IMLS has closely consulted with Newfields, History Colorado, and The Field Museum and external evaluation firm HG&Co in the development of the evaluation plan, data collection and instruments.

A.9. Payments or Gifts to Respondents.

None.

A.10. Assurance of Confidentiality.

This particular project will not be submitted for IRB, as all of the participants will be either project partners or electing to participate as part of their professional duties. 

However, any personally identifiable data collected (e.g., the name of the person who responded on behalf of the lead content partner or additional museum partner site) will be removed prior to analysis.

Any identifiable information collected as part of coordinating the data collection effort will be securely stored and destroyed once data collection is complete.

Any personal data associated with published work (e.g. Quotes in the final report or presentation) will be used only if approved by the participant.

No data will be collected from individual youth participants in this study.

Informed consent will be strictly followed.

Assurances of how the information will be used will be conveyed in the consent form and the consent section at the beginning of the MDL facilitator survey instrument and at the beginning of the interview protocols.

A.11. Justification for Sensitive Questions

No sensitive questions will be included within the evaluation instruments or interview protocols.

A.12. Estimates of Hour Burden to Respondents/Table

Survey	# of respondents	Estimated response time	Total burden hours
Collaboration Survey (t1)	23	30 minutes	11.5
Think Alouds with K-12 Educators (t1)	10	60 minutes	10
K-12 Educator Prototype Testing Survey and Classroom Observations	10	60 minutes	10
Think Alouds with K-12 Educators (t2)	10	60 minutes	10
Collaboration Survey (t2)	23	30 minutes	11.5
Additional Museum Partners Questionnaire	24	30 minutes	12

A.13. Estimates of Cost Burden to Respondents

The estimated cost to applicants is $1755. The average cost per hour is based on an average of $27.00, the Bureau of Labor Statistics average mean hourly wage of a museum employee ($25.62), and primary/secondary teachers ($28.15).

The estimated total burden hours are 65 hours.

This survey does not require respondents to purchase equipment, software, or services beyond those normally used in schools or museums as part of customary and usual business.

A.14. Estimates of Cost to Federal Government

There is no extra cost to the federal government

A.15. Reason for Program Changes or Cost Adjustments

This is a new submission.

A.16. Project Schedule

Year 1:

Mar-June 2019	July-August 2019	Sept 2019
MDL partners finalize technical approach, delivery framework and functionality as well as themes, framework and templates Evaluators develop and refine Collaboration Survey for OMB approval	Submit Collaboration Survey for OMB clearance, respond to questions, receive clearance by August Mid-August: Collaboration Process Survey (Instrument #3): (subject to OMB approval)	Deliver Memo Report on Year 1 to Newfields and IMLS

Year 2:

Sept 2019	Jan-Feb 2020	Mar 2020	April 2020	June-July 2020
Think-Alouds + semi-structured interview (Instrument # 1): Conduct Think- Alouds and semi- structured interviews w/10 K-12 educators to examine MDL template (via videocall) Synthesize memo report of findings and recommendations	Implementation (Instrument # 2A and 2B): K-12 educators receive prompts to aid in classroom observations, and answer survey questions designed to assess ease of implementation	Collaboration Process Survey (Instrument #3): Conduct collaboration survey for MDL Project Team: K-12 educators, Newfields, lead content partners. Deliver Memo Report	Think-Alouds + semi-structured interview (Instrument # 1) Conduct Think- Alouds and semi-structured interviews w/10 K-12 educators to examine MDL template (via videocall) Follow-Up Questionnaire (Instrument #4): Conduct questionnaire with professionals at (up to) 10 additional museum partner sites to assess implementation of MDL’s content templates	Synthesize and draft report, focused on MDL product utility for K-12 educators and museums Deliver Final Report by July 2020

A.17. Request to Not Display Expiration Date

No exemption from the requirements to display the expiration date for OMB approval of the information collection is being requested for this evaluation. The OMB approval number and expiration date will be displayed on all data collection materials and documentation.

A.18. Exceptions to the Certification

No exceptions to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” apply to the MDL Evaluation.

8