STEM Maker Ed Support Part A Revised 2017-10-13

Section A. Justification

A.1. Necessity of the Information Collection

The Institute of Museum and Library Services (IMLS) funds the Maker/STEM Education Support for 21^st Century Community Learning Centers (21^st CCLCs) through a Cooperative Agreement with the New York Hall of Science (NYSCI). The Cooperative Agreement in the amount of $750,000 is for the period of January 9 – December 31, 2017 (and is supported through an interagency agreement with the U.S. Department of Education (ED). NYSCI is the lead museum/science center for the 21^st CCLC Maker/STEM program NYSCI will oversee six additional museums/science centers selected by IMLS to participate in this Maker/STEM program.

As part of the program, each of the six museums/science centers is partnering with five geographically proximal 21^st CCLC sites to implement a six-activity curriculum with 21^st CCLC upper elementary and middle school students during the summer of 2017. These approximately 30 21^st CCLC sites were identified by ED, IMLS and NYSCI. The full parameters of the 21^st CCLC Maker/STEM program and the obligations of each party are outlined in the Cooperative Agreement.

The proposed 21^st CCLC Maker/STEM Evaluation is budgeted at $150,000. Under the Cooperative Agreement, NYSCI is working with an external evaluation partner, Education Development Center (EDC). NYSCI and EDC propose to conduct a formative evaluation of the program model and its components in order to understand the implementation of NYSCI’s Maker/STEM activities in the 21^st CCLC sites. The formative evaluation will document feedback from museum/science center partners, 21^st CCLC facilitators, and 21^st CCLC youth participants on their experiences with the program. The results of this evaluation are intended to benefit the museum field as well as the public. The information collected through the evaluation will also identify implementation successes, challenges and other improvements and provide information that might be useful for potential scale-up efforts by the field, NYSCI, IMLS or others.

About IMLS

The Institute of Museum and Library Services (IMLS) is the primary source of federal support for the nation's 123,000 libraries and 35,000 museums. IMLS' mission is to create strong libraries and museums that connect people to information and ideas. IMLS works at the national level and in coordination with state and local organizations to sustain heritage, culture, and knowledge; enhance learning and innovation; and support professional development. IMLS legislative authority is located at 20 U.S.C. 9101, et seq.

About NYSCI:

NYSCI was founded at the 1964–65 World’s Fair and has evolved into New York’s premiere center for interactive science, serving a half million students, teachers, and families each year. New York Hall of Science presents 450 exhibits, demonstrations, workshops and participatory activities that explain science, technology, engineering, and math (STEM). In 2012, NYSCI unveiled a new permanent Maker Space and has since launched a comprehensive series of educational programs where aspiring Makers of all ages may tinker, experiment, create, share ideas and work with physical and digital tools, year-round. Today, NYSCI’s Maker programs provide young people (ages 3 to 18) multiple pathways to learning STEM that foster imagination, creative critical thinking, entrepreneurship and innovation. Since its public opening, Maker Space has hosted more than 20,000 visitors, including children, parents, students, teachers, and community residents, through afterschool programs, weekend workshops, summer camps, and community events.

About the Maker/STEM Education Support for 21st Century Community Learning Centers:

“Making” is a grassroots movement that celebrates creativity and innovation through the process of designing and making physical objects that blend high and low technologies. “Making” activities can be a means to engage people in exploring STEM concepts, practices and phenomena. NYSCI has developed a six activity Making curriculum for implementation with upper elementary and middle school students in 21^st CCLC programs. As a train-the-trainer program model (figure 1), NYSCI first will train the six museum/science center partners on this curriculum. After the NYSCI training, each participating museum/science center will train facilitators from five geographically proximal 21^st CCLC sites to implement the curriculum with youth. Once trained on the activities, the 21^st CCLC facilitators and the museum/science center partner staff will co-implement the Making curriculum with up to 15 youth at each of the approximately 30 21^st CCLC sites.

Figure 1: Program model

The Making activities will be implemented in approximately 30 21^st CCLC sites in six states in the summer of 2017. These states are Pennsylvania, Texas, Oregon, Florida, Arkansas, and Wisconsin. In addition to NYSCI, the museums/science centers involved include: the Children’s Museum of Houston (TX), Frost Museum (FL), Betty Brinn Children's Museum (WI), Franklin Institute (PA), ScienceWorks (OR), and the Scott Family Amazeum (AR).

A.2. Purposes and Uses of the Data

NYSCI and its external evaluation partner, EDC, propose to conduct a formative evaluation of the Maker/STEM Education Support for the 21^st CCLC program based on the implementation of the program model represented above. For the purposes of the evaluation, this program model includes the following three core components:

1. the training NYSCI develops and delivers to the museum/science center partners on the Making activities;

2. the trainings the museum/science center partners develop and implement with their local 21^st CCLC staff to facilitate the Making activities with youth; and

3. the activities the museum/science center partners and 21^st CCLC facilitators co-implement with youth at 21^st CCLC sites.

For the purposes of this document, the three programmatic components listed above will be known as the program model.

As a formative evaluation, quantitative and qualitative data will be collected and analyzed via a mixed methods approach. Data will be gathered from each of the key groups within the program model: 1) museum/science center partners, 2) 21^st CCLC faciliators, and 3) youth participating in the program at 21^st CCLC sites. Specifically, the formative evaluation will be guided by the following questions:

1. What are the experiences of the museum/science center partners, 21^st CCLC facilitators and youth participants during the implementation of STEM Making programs in 21^st CCLC sites?

2. What factors do each of the groups (i.e., museum/science center partners, 21^st CCLC facilitators and youth participants) feel contribute to the quality of STEM Making programs in 21^st CCLC sites?  

3. In what ways, if at all, do NYSCI’s model, curriculum materials, professional development and technical assistance support 21^st CCLC sites’ readiness and capacity to implement STEM Making programs and associated program quality?

4. What are the opportunities and challenges faced:

   • By museums/science centers in training 21^st CCLC sites to implement Maker/STEM education programs?

   • By 21^st CCLC sites in implementing Maker/STEM education programs?

Based on these questions, the evaluation will focus on the following five objectives:

1. Document the implementation of the program model (i.e., the museum/science center partner training, the 21^st CCLC site staff trainings, and the 21^st CCLC site implementations) and its adaptations

2. Gather feedback from museum/science center partner staff on the development and implementation of trainings and on 21^st CCLC site implementation

3. Gather feedback from 21^st CCLC site facilitators on their experiences in the trainings and their implementations of the Making activities

4. Gather feedback from participating youth on their experiences with the program as implemented

5. Understand the opportunities and challenges of program model implementation at the museum/science center and 21^st CCLC site level, and investigate the elements of the program model that appear promising for replication during future scale-up

Data collection methods are summarized in Table 1. To document the program model from the perspective of the museum/science centers and gather feedback on their experiences, interviews will be conducted with all museum/science center partners (n=6). The evaluation will also document the role of the 21^st CCLC facilitators and gather their reactions to training and implementation via online survey (n=approximately 30 corresponding to all expected sites participating in the project).

Table 1: Data collection summary

Participant group	Data collection method	Sample size
Museum/science center staff	Interview	6 (one per participating museum/science center)
21st CCLC Facilitators	Online survey	30 (one per site, all sites)
	Interview	12 (one per site for the 12 selected sites)
21st CCLC Youth	Interview	48 (four per site for the 12 selected sites)

In addition to gathering perspectives from all participating museum/science center project partner center staff via interviews and all 21^st CCLC facilitators via online surveys, the evaluation team will use a non-probability purposive sampling¹ procedure, called maximum variation sampling (Schwandt, 2007), to gather more in-depth information of facilitators and participating youth across a sample of twelve of the thirty 21^st CCLC sites. The focus of the in-depth investigation will be on examining the implementation of the program model across the 21^st CCLC sites and observations of differences in the 21^st CCLC facilitator trainings and youth experiences with the curriculum.

Based on the short implementation timeframe (with evaluation to begin in the summer of 2017) and the number of total sites (approximately 30), EDC, NYSCI and IMLS determined that it is prohibitive to collect in-depth information (i.e., 21^st CCLC facilitator and youth interviews) from all implementing sites. Instead, maximum variation sampling will allow the evaluation team to select specific “cases” or sites based on their characteristics in order to ensure the data collected explores the maximum variability in the implementation of the program model. As the program model for this evaluation is designed to allow for variation in how 21^st CCLC sites are trained and how the program is then implemented with youth, a maximum variation sampling procedure will allow the evaluation team to gather additional in-depth feedback from a range of 21^st CCLC facilitators and youth participating in the program.

Specifically, the evaluation team will select 12 of the approximately 30 21^st CCLC sites (approximately two sites per museum/science center partner) via this sampling procedure. Selection criteria for using the maximum variation sampling will include the following characteristics:

• Assigned museum/science center partner

• Prior experience implementing Making programs at the 21^st CCLC site

• Format of the training provided by the museum/science center partner to the 21^st CCLC facilitators (e.g., one full day training, trainings just prior to activity implementation)

• Format/dosage of the 21^st CCLC site implementation (e.g., one activity per session, multiple activities per session)

• Age of youth participating at the 21^st CCLC site

Once sites are selected, in-person or telephone interviews will be conducted with one facilitator at each of these 12 selected sites to understand their training and experiences with implementation more deeply. Additionally, in-person interviews with a selection of youth at the 21^st CCLC sites will be conducted to gather feedback on their experiences with the program. In consultation with the museum/science center partners and 21^st CCLC facilitators, youth will be selected for interviews based on the following criteria: 1) signed parent/guardian consent form; 2) distribution of age; 3) distribution of gender; and 4) range of participation dosage in the program. The evaluation team will summarize survey data by calculating descriptive statistics and coding qualitative responses, which will be then organized and presented by evaluation question in the final report and presentations. Quantitative data from surveys will be displayed in table and/or graphical form and qualitative data will be displayed thematically. Interviews, using semi-structured protocols, will be audio recorded by evaluation staff and transcribed for analysis. Using the evaluation objectives as the framework, interview transcripts will be coded using both iterative and open coding procedures to capture themes from the qualitative data.

As this is not an impact evaluation, results are not intended to support causal inferences about the effect of the program. Instead as a formative evaluation, results will be used by NYSCI, IMLS, and ED to understand the program model including: 21^st CCLC staff and participating youth experiences with the activities; the role of the museum/science centers in the program; and the implementation challenges that may need to be addressed prior to any scale-up. The final evaluation report will be shared with project stakeholders and in a publicly releasable form to the public. We anticipate the final report will include the following sections: executive summary; introduction; formative evaluation research questions and goals; methodology; documentation of program activities, summary of findings; program recommendations; and appendices.

A.3. Use of Information Technology

The 21^st CCLC facilitator and museum/science center partner staff interview data will be collected in-person or via telephone during working hours at a time that is convenient for the interviewee. Youth participant interview data will be collected in-person during the final regularly scheduled 21^st CCLC Maker program activity. Audio recorders will be used to record in-person and telephone interviews. Data collection activities will proceed after consent to participate has been secured from adult participants and written permission from the parents/guardians of youth participants. Once interview data is securely shared with a professional transcription service and transcribed, all audio data will be destroyed.

Additionally, the evaluation team will create an online survey, in collaboration with IMLS and NYSCI, to simplify the data collection process with the entire population of 21^st CCLC facilitators. The 21^st CCLC facilitator survey will be developed and administered through Qualtrics © survey software. Data collection will proceed after consent to participate has been secured from participants. NYSCI and EDC will confirm email addresses of 21^st CCLC facilitators to ensure that a final contact list is valid. A hard copy of the online survey will be provided for those 21^st CCLC site staff unable to utilize the electronic process.

A.4. Efforts to Identify Duplication

As this is a new program model, this is a new data collection effort. This evaluation will generate formative evaluation findings specific to the program model and the various implementations across the 21^st CCLC sites.

A.5. Methods Used to Minimize Burden on Small Businesses

For individuals at all institutions, participation in the 21^st CCLC STEM/Maker Evaluation is voluntary.

A.6. Consequences of Less Frequent Data Collection

The Maker/STEM Education Support for 21^st CCLCs aims to provide youth access to Making activities at their 21^st CCLC sites. This is a one-time data collection.

A.7. Special Circumstances

No special circumstances require the collection to be conducted in a manner inconsistent with the guidelines in 5 CFR 1320.6.

A.8. Consultations Outside the Agency

Public comments solicited through Federal Register

IMLS published a notice in the Federal Register with a 60-day public comment period to announce this proposed information collection on February 13, 2017 (FR vol. 82, No. 28, pgs. 10501-10502). A copy of the Federal Register Notice is provided. No comments were submitted.

IMLS published a notice in the Federal Register on June 12, 2017 (Volume 82, Number 111, page 26958-26959), with a 30-day public comment period to announce forwarding of the information collection request to OMB for approval. IMLS received one comment on the notice which is attached in the Supporting Materials.

Consultants outside the agency

As part of the cooperative agreement referenced above, IMLS has closely consulted with NYSCI and external evaluation firm, EDC, in the development of the formative evaluation plan, data collection and instruments.

A.9. Payments or Gifts to Respondents.

None

A.10. Assurance of Confidentiality.

All data collection activities have been submitted for Institutional Review Board (IRB) Education Development Center, Inc. review and approval (FWA 00000038; IRB 00000865). Per our IRB, we strictly follow 45cfr46 (Code of federal regulations, Protection of Human Subjects). Any personally identifiable data collected (e.g., the name of the person who responded on behalf of the museum/science center, 21^st CCLC site, etc.) will be removed prior to analyses. Any identifiable information collected as part of coordinating the data collection effort will be securely stored and destroyed once data collection is complete. Relevant identifiable data (e.g., state of the 21^st CCLC) will only be used in aggregate. Any personal data associated with published work (e.g.,quotes in the final report or presentation) will be used only if approved by the participant. No identifiable information outside of participant age and gender will be collected from youth participants. Per Human Protections regulations, informed consent will be strictly followed. Assurances of how information will be used will be conveyed in the parent/guardian consent form and the consent section at the beginning of the 21^st CCLC facilitator survey instrument and youth, facilitator and museum/science center interview protocols.

The evaluation team is seeking to gather this information to better understand the range of youth experiences that are shared through the interviews. Based on the goals of this evaluation and our past experiences with gathering data from youth in out-of-school STEM programs, participant age and their gender can play a role in how they experience a program. Specifically, the purpose of capturing age is to help us understand the program experiences and feedback on the activities from various ages, since we know that the museums will be implementing the model with a wide range of ages both between and within sites. As for gender, we know from our experience with evaluating out-of-school STEM programs, that girls and boys can experience these types of STEM programming differently. These differences will be important to capture from the youth.

A.11. Justification for Sensitive Questions

To confirm the program is serving the intended program target, the age of program participants will be collected during youth interviews. Outside of this demographic question and the question about gender, no other sensitive questions will be included within the various evaluation instruments.

A.12. Estimates of Hour Burden to Respondents

	# of respondents	Estimated response time	Total burden hours
Museum/Science center partner interview	6	60 minutes	6
21st CCLC facilitator survey	30	15 minutes	7.5
21st CCLC facilitator interview	12	45 minutes	9
21st CCLC youth interview	48	10 minutes	8

A.13. Estimates of Cost Burden to Respondents

The estimated cost to applicants is $584.38. The average cost per hour is based on $19.16, the Bureau of Labor Statistics average mean hourly wage of a museum employee ($27.40), primary/secondary teachers ($24.35) and youth ($0.00). The estimated total burden hours is 30.5 hours.

This survey does not require respondents to purchase equipment, software, or services beyond those normally used in museums/science centers or 21^st CCLC sites as part of customary and usual business.

A.14. Estimates of Cost to Federal Government

The annualized cost to IMLS is estimated at $4,760 based on 40 hours at $60.00 for IMLS Museum Services Staff and $59.00 based on 40 hours for IMLS Miscellaneous Staff. See section A.1. for the other costs.

A.15. Reason for Program Changes or Cost Adjustments

There are no changes from the OMB Form 83-I. This is a new submission.

A.16. Project Schedule

Project activity	Timeframe
Seek OMB clearance	February – July 2017
Secure IRB approval	May 2017
Implement Maker activities at 21st CCLC sites	July – August 2017
Collect youth and 21st CCLC facilitator surveys	August – September 2017
Conduct 21st CCLC facilitator interviews	August – September 2017
Conduct museum/science center partner interviews	August – September 2017
Clean and analyze data	September – November 2017
Submit report	December 2017

A.17. Request to Not Display Expiration Date

No exemption from the requirements to display the expiration date for OMB approval of the information collection is being requested for the 21^st CCLC STEM/Maker Evaluation. The OMB approval number and expiration date will be displayed on all data collection materials and documentation.

A.18. Exceptions to the Certification

No exceptions to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I apply to the 21^st CCLC STEM/Maker Evaluation.

1 “In the logic of sampling based on a theoretical or purposive strategy, units are chosen not for their representativeness, but for their relevance to the research question . . .relevance may be a matter of choosing multiple places, cases, or sites to facilitate comparisons either because these different units are likely to yield predictable contrasts in understanding the definition of social action or because they are likely to show the same or similar definition of social action.” (p. 269 – 270) Schwandt, T.A. (2007). The Sage Dictionary of Qualitative Inquiry, 3^rd ed. Sage Publications, Thousand Oaks, CA.