Data Collection Instruments

Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning

230915_OMB Package_Appendix C Data Collection Instruments

Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning

OMB: 1850-0989

Document [docx]

Download: docx | pdf

REL NW Professional Learning Course

Appendix C: Data Collection Instruments

Course Instructor Application for Course, Consent, and Follow-up Survey
Instructor Focus Group Protocol
Structured Course Observation of Materials Protocol
Student Survey and Consent
Administrator Interview Protocol and Consent
Facilitator Interview Protocol and Consent
Observation Protocol and Fidelity Rubric for Course Synchronous Sessions

1. Course instructor application, consent, and follow-up survey

Thank you for your interest in the professional learning course Using Technology to Support Postsecondary Student Learning! Developed by the Regional Educational Laboratory Northwest (REL NW), the course engages community college instructors in five evidence-based practices to improve how they use technology and digital tools in their teaching to support communication, collaboration, and active learning. On behalf of REL NW, the Community College Research Center (CCRC) is conducting a study to evaluate the implementation and efficacy of the professional learning course at community colleges in the state of Oregon, including yours. To participate in the course and research study, instructors should respond to the following questions to determine eligibility. In total, this application should take approximately 20 minutes to complete. Space in the course is limited to 120 instructors; if more than 120 instructors apply, participants will be selected at random.

Half of the instructors who are selected to participate in the course and research study will be randomly assigned to participate in the professional learning course in summer 2024 [INSERT DATES]. The other half will be invited to participate in the course in [INSERT MONTH/DATES] 2025. All study participants will be asked to complete two surveys and will be asked to encourage their students to complete a survey in 2024. A subset of instructors will be asked to participate in a focus group and to share syllabi, assignments and other course materials with the research team.

Screening Question

Are you scheduled to teach at least one credit-bearing college-level or developmental-level course during fall 2024?

If yes, at which college will you be teaching? [drop-down]

Course Selection

To participate in the research study, instructors must select a “focal course” to try out the tools and practices that are taught in the professional learning course. Focal courses can be offered in any delivery format (online, hybrid, in-person, or other). Continuing education courses, courses in noncredit programs, and courses that exclusively enroll dually enrolled students cannot serve as the focal course. When selecting a focal course, applicants should select:

A course that you are scheduled and expect to teach in fall 2024 at [COLLEGE NAME]
A credit-bearing college-level or developmental/remedial course in any discipline
A course that does not enroll ONLY dually enrolled high school students
A course that you are interested in reflecting on and improving as part of your participation in the professional learning course

Focal course selections cannot be changed after applications are submitted.

Please provide the following for your selected focal course:

Title: ___________
Prefix (department abbreviation): __________
Course number: _____________
Course section(s): _______________
College offering course: ___________

What best describes the delivery format of your selected focal course?

In-person
Online and asynchronous (no required meeting times)
Online with synchronous components (required meetings held using video conferencing)
Hybrid (some online components and some in-person components)

Are you currently scheduled to teach the above-named course during fall 2024?

I understand that my focal course cannot be changed after my application has been submitted.

Thank you for confirming your eligibility to participate in the study of the course Using Technology to Support Postsecondary Student Learning. To proceed with the application, please review the Informed Consent Form that describes the purpose of the study, risks and benefits to participation, available monetary incentives, and procedures for confidentiality. If you consent to participate in the study, you will be directed to a brief survey about your experiences using instructional technology in your teaching.

INFORMED CONSENT

Protocol Title: Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning, IRB Protocol Number 23-393

Principal Researcher: Dr. Thomas Brock, Community College Research Center, Teachers College, 212-678-3091, [email protected]

INTRODUCTION You are invited to participate in this research study called the “Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning.” This study is being conducted by the Community College Research Center (CCRC) on behalf of the Regional Education Laboratory Northwest (REL NW), which is funded by the Institute of Education Sciences.

CCRC has initiated a request for applications for community college instructors to participate in a research-based online professional learning course entitled, Using Technology to Support Postsecondary Student Learning. Instructors may qualify to take part in this research study if they are scheduled to teach an eligible course at a participating college during fall 2024. Eligible courses include all courses taught at the college except for continuing education courses, courses in non-credit programs, and courses that exclusively enroll dually enrolled students.

Enrollment in the professional learning course is limited to eligible instructors who apply to participate in this research study. Professional learning course applicants will be grouped into five broad academic disciplines. Half of eligible applicants from each broad academic discipline will be randomly assigned to participate in the course during the summer 2024 (treatment group). The other half will be randomly assigned to participate in the course during 2025 (comparison group). Within each broad academic discipline, an instructor’s odds of being placed into each study group are approximately 50/50. Instructors will be notified of their study group assignment in May 2024. Approximately 120 instructors will participate in this study.

All study participants will be asked to (1) attend the research-based course as assigned, (2) complete two surveys, and (3) share information about a student survey with their students. Additionally, a subset of 15 participating instructors assigned to the treatment group and a subset of 15 participating instructors assigned to the comparison group will be randomly selected to participate in a focus group and asked to share their course in the college learning management system with the CCRC researchers for a course review. The purpose of the course review is to observe and document what digital tools/techniques instructors employ in their courses and how. The subset of instructors selected for these final two research activities will be notified in October 2024.

It is expected that the total amount of time for participation in this study, including the online professional learning course, will be up to 18.5 hours including all of the activities listed above (not all study participants will be part of the focus groups or course reviews).

WHY IS THIS STUDY BEING DONE? This study is being done to evaluate the efficacy and implementation of the professional learning course, Using Technology to Support Postsecondary Student Learning. The course engages community college instructors in the use of five evidence-based practices to improve how instructors use technology and digital tools in their teaching to support communication, collaboration, and active learning. Findings will be used to improve professional learning for community college instructors.

WHAT WILL I BE ASKED TO DO IF I AGREE TO TAKE PART IN THIS STUDY? Instructors who decide to participate in the study will be randomly assigned to participate in the online professional learning course in either 2024 (treatment group) or 2025 (comparison group). The professional learning course consists of approximately 16 hours of professional learning (8 hours of synchronous online sessions and 8 hours of asynchronous learning).

Participants in both the treatment and comparison groups will complete 2 surveys about the use of technology in instructional practices. Each survey will take approximately 20 minutes to complete. The first survey follows this consent form. The second survey will be emailed to participants at the end of fall 2024.

At the end of the Fall 2024 term, instructors will be asked to remind students to check their college email for information about a student-level survey to the students who were ever enrolled in their course. The researchers will provide instructors with a template email and script for this task which should take no more than 15 minutes over the course of two weeks.

In fall 2024, a randomly selected subset of participants will be asked to participate in a virtual focus group that will last up to one hour and will be attended by up to four community college instructors who are participating in the research study. In the focus group instructors will be asked to discuss their use of instructional technology and process for selecting, evaluating, and implementing technology as well as their experience with and perceptions of the professional learning course. The focus group will be audio-recorded. After the audio recording is written down (transcribed) the audio recording will be deleted. If an instructor does not wish to be audio-recorded, they will still be able to participate. In the transcript each instructor will be given a pseudonym or false name (or de-identified code) in order to keep their identity confidential. Additionally, this subset of participants will be asked to share their focal course with CCRC researchers using the college learning management system. Members of the research team will view the course syllabus, activities, assignments and assessments (no student data will be collected).

Following the conclusion of the study the research team will produce a publicly accessible, restricted use data file containing administrative and survey data collected for the study. The final dataset will not include any personal information or fields that may indirectly identify participants.

WHAT POSSIBLE RISKS OR DISCOMFORTS CAN I EXPECT FROM TAKING PART IN THIS STUDY? The risks associated with this study are minimal. Instructors will be asked questions about their experiences and opinions, which may be a source of discomfort if an instructor wants to express what they perceive to be an unpopular opinion. To minimize this discomfort, the researchers will de-identify all data prior to conducting any analyses. Study participants do not have to answer any questions or share anything they do not want to talk about. The primary researcher is taking precautions to keep each participant’s information confidential and prevent anyone from discovering or guessing their identity, such as using a pseudonym instead of their name and keeping all information on a password protected computer and locked in a file drawer.

WHAT POSSIBLE BENEFITS CAN I EXPECT FROM TAKING PART IN THIS STUDY? Instructors who take part in this study will have the opportunity to participate in a free online professional learning course focused on teaching with technology either in summer 2024 or in 2025. In addition, some participants may find it interesting or constructive to respond to questions related to teaching with technology through surveys and a focus group. The goal of the study is to inform improvements in instruction and student success in higher education.

WILL I BE PAID FOR BEING IN THIS STUDY? Instructors who participate will receive the following incentives for participation in this study:

$250 in summer 2024 after completing the application with survey
$250 after completing the fall 2024 survey
$50 gift card after an instructor notifies CCRC that they have reminded students about the student survey two times.

There are no costs to participants for taking part in this study.

WHEN IS THE STUDY OVER? CAN I LEAVE THE STUDY BEFORE IT ENDS? The data collection activities will conclude by the end of January 2025. Participants can leave the study at any time even if they haven’t completed all of the activities. Participation in this study is voluntary and will have no effect on an instructor’s employment at their institution. Instructors can stop participating in the study at any time without penalty.

PROTECTION OF YOUR CONFIDENTIALITY Only the research team will have access to data that is linked to any identifying information. The research team will strip the data of any identifying information (name, email address) prior to beginning analysis. Electronic data will be stored on a server within the Teachers College network, behind the College firewall, and will be accessible only to the research team via on-campus computers or off-campus networked computers that have been authenticated using the College’s Virtual Private Network (VPN). Regulations require that research data be kept for at least ten years; however, the data will be destroyed at the conclusion of that period. For quality assurance, the research team, the study sponsor (grant agency), and/or members of the Teachers College Institutional Review Board (IRB) may review the data collected from a participant as part of this study. Otherwise, all information obtained from an instructor’s participation in this study will be disclosed only with their permission or as required by U.S. or State law.

HOW WILL THE RESULTS BE USED? The results of this study may be published in journals or reports and presented at academic conferences. The identity of participants will be removed from any data provided before publication or use for educational purposes. A participant’s name or any identifying information about them will not be published.

WHO CAN ANSWER MY QUESTIONS ABOUT THIS STUDY? If participants have any questions about taking part in this research study, they should contact the primary researcher, Thomas Brock of CCRC, [email protected] or at (212) 678-3019.

If a participant has questions or concerns about their rights as a research subject, they should contact the Institutional Review Board (IRB) (the human research ethics committee) at 212-678-4105 or email [email protected] or write to the IRB at Teachers College, Columbia University, 525 W. 120^thStreet, New York, NY 10027, Box 151. The IRB is the committee that oversees human research protection for Teachers College, Columbia University.

PARTICIPANT’S RIGHTS

I have read the Informed Consent Form and have been offered the opportunity to discuss the form with the researcher.
I have had ample opportunity to ask questions about the purposes, procedures, risks and benefits regarding this research study.
I understand that my participation is voluntary. I may refuse to participate or withdraw participation at any time without penalty.
The researcher may withdraw me from the research at the researcher’s professional discretion.
If, during the course of the study, significant new information that has been developed becomes available which may relate to my willingness to continue my participation, the researcher will provide this information to me.
Any information derived from the research study that personally identifies me will not be voluntarily released or disclosed without my separate consent, except as specifically required by law.
My data will not be used in further research studies.
I should receive a copy of the Informed Consent Form document.

By providing my name below I agree to participate in this study:

Name: ___________________________________________________________ Date: _____________________

Instructional use survey (part of instructor application)

Instructional Technology Knowledge and Experiences:

The questions that follow ask about your knowledge of and experiences using technology tools for teaching.

A “technology tool” is a digital resource that you use to enhance your teaching. How strongly do you agree with the following statements? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

I find technology tools useful.
Using technology tools helps me accomplish tasks more quickly.
Using technology tools increases my productivity.
Using technology tools enhances my effectiveness.
Using technology tools makes it easier to do my work.

To what extent has incorporating technology into your teaching contributed to the following? (Decreased, Slightly decreased, No change, Slightly increased, Increased)

Students learning collaboratively
The quality of interactions among students
The quality of interactions between students and instructor(s)
Students receiving personalized instruction or content
Student understanding of course content
Students setting personal learning goals
Students seeking help in my courses
Students evaluating and correcting their own work
Students receiving timely feedback about their performance
Students receiving personalized feedback
Students using simulations (e.g., computer simulations, game-based learning, virtual reality) to engage in problem-solving
Students using simulations to practice newly acquired skills
Students using simulations to engage in problem-based learning

The following 2 questions refer to the list of items immediately following.

How familiar are you with uses of the following technologies to enhance or support teaching in your subject area(s)? (Not familiar at all, Somewhat unfamiliar, Moderately familiar, Very familiar, Extremely familiar)

Which do you currently use in your classroom? Check all that apply.

Learning management systems (LMS) (e.g., Canvas, Blackboard, D2L, Moodle, Google Classroom)
Video conferencing tools (e.g., Zoom, Microsoft Teams, Google Hangouts)
Lecture capture software (e.g., Canvas Studio, Panopto, Kaltura)
Presentation software (e.g., PowerPoint, Google Slides)
Educational content platforms (e.g., Khan Academy, TED-Ed, textbook publisher content and courseware, Open Educational Resources)
Online assessment tools (e.g., Quizlet, Google Forms)
Classroom poll response systems (e.g., Poll Everywhere, Zoom polling, Kahoot!)
Collaborative learning tools (e.g., Google Apps, Microsoft Office 365, Padlet)
Accessibility tools (e.g., ReadSpeaker, JAWS)
Simulation technologies (e.g., computer simulations, game-based learning, virtual reality)
Collaborative whiteboard software (e.g., Jamboard, LiveBoard)

How strongly do you agree with the following statements? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

I am clear about the purpose of technology tools for communication and collaboration in my courses.
I design relevant digital discussion tasks that are integrated with course material.
I monitor digital communication boards and follow up with students.
I leverage technology tools to vary the formats for delivering my courses.
I use technology to provide multiple ways for students to access and study course materials.
Delivery formats for my courses are chosen based on the content being covered.
I use technology tools to support students’ self-monitoring of their learning.
I use technology tools to support students with time planning and management.
I use technology tools to support students in my courses with finding tutoring or mentoring.
I use automated scoring of online homework or quizzes to provide immediate feedback to students.
I use adaptive learning features such as built-in assessments in online textbooks to provide timely feedback.
I use data collected from student response systems to inform my instruction.
The content of my course lends itself to students engaging with authentic problems.
I use technology tools to simulate problems.
I use simulated problems to support my teaching.

How strongly do you agree with the following statements? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

Responding to electronic communications with students is overwhelming.
Using multiple technologies to support asynchronous and synchronous social interactions is overwhelming.
The privacy risks associated with digital communication and collaboration make me uncomfortable.
I find it difficult to adapt course materials to incorporate technology tools.
Learning how to use new technology tools distracts students from focusing on the content of lessons.
Self-regulated learning (i.e., thinking about one’s thinking and taking strategic actions to learn) is not something I should have to teach college students.
I do not know how to incorporate opportunities for students to build self-regulation skills into my courses.
Adding real-time checks for understanding and follow-up discussion is too time-consuming.
Adapting instruction during a lesson based on student responses is too difficult.
I’d rather keep learning activities I am familiar with than switch to new activities that use new technologies.

[NOTE: If this following information can be obtained through administrative records, these questions will NOT be included in the survey. If the college does not have the capacity to provide this information about participating instructors through administrative records, this background information will be gathered through the survey.]

Background Questions:

What best describes your gender identity?

Woman
Man
Prefer to self-describe: __________

Are you Hispanic or Latino/a?

Select one or more of the following races:

American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or Other Pacific Islander
White

What best describes your position at the college?

Full-time instructor or faculty member
Part-time instructor or faculty member

How many years have you taught at any college/university, not including graduate teaching assistant positions? If this is your first year teaching at any college/university, please enter "1 year."  (Drop-down choices: 1 year–20 or more years)

How many years have you taught at this college? If this is your first year teaching at this college, please enter "1 year."  (Drop-down choices: 1 year–20 or more years)

Public Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx. Public reporting burden for this collection of information is estimated to average 20 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this individual collection, or if you have comments or concerns regarding the status of your individual form, application or survey, please contact (Program Office address/contact information) directly.

REL NW Professional learning Course Follow-up Instructor Survey

Thank you for your participation in the research study to evaluate the professional learning course Using Technology to Support Postsecondary Student Learning. This is the second of two surveys about your experiences using instructional technology in your teaching. This survey should take approximately 20 minutes to complete. As a participant in the study, you previously completed an Informed Consent Form that describes the purpose of the study, risks and benefits to participation, and procedures for confidentiality. To review this information and your rights as a research participant, you may access the study consent form here: [INSERT LINK TO CONSENT FORM].

Many of the questions in this survey relate to the course you identified as a “focal course” when you applied to the professional learning course earlier this year. As a reminder, your focal course is [COURSE TITLE, SECTION NUMBER]. When responding to questions about your focal course, please answer based on your experiences in that course section during the current term.

Did you teach [FOCAL COURSE TITLE, SECTION NUMBER] this term?

What best describes the delivery format of this course?

In-person
Online and asynchronous (no required meeting times)
Online with synchronous components (required meetings held using video conferencing)
Hybrid (some online components and some in-person components)

Instructional Technology Knowledge and Experiences

This first set of questions asks you about your knowledge of and experiences using technology tools for teaching.

I find technology tools useful.
Using technology tools helps me accomplish tasks more quickly.
Using technology tools increases my productivity.
Using technology tools enhances my effectiveness.
Using technology tools makes it easier to do my work.

To what extent has incorporating technology into your teaching contributed to the following? (Decreased, Slightly decreased, No change, Slightly increased, Increased)

Students learning collaboratively
The quality of interactions among students
The quality of interactions between students and instructor(s)
Students receiving personalized instruction or content
Student understanding of course content
Students setting personal learning goals
Students seeking help in my courses
Students evaluating and correcting their own work
Students receiving timely feedback about their performance
Students receiving personalized feedback
Students using simulations (e.g., computer simulations, game-based learning, virtual reality) to engage in problem-solving
Students using simulations to practice newly acquired skills
Students using simulations to engage in problem-based learning

Learning management systems (LMS) (e.g., Canvas, Blackboard, D2L, Moodle, Google Classroom)
Video conferencing tools (e.g., Zoom, Microsoft Teams, Google Hangouts)
Lecture Capture Software (e.g., Canvas Studio, Panopto, Kaltura)
Presentation Software (e.g., PowerPoint, Google Slides)
Educational Content Platforms (e.g., Khan Academy, TED-Ed, textbook publisher content and courseware, Open Educational Resources)
Online Assessment Tools (e.g., Quizlet, Google Forms)
Classroom Poll Response Systems (e.g., Poll Everywhere, Zoom polling, Kahoot!)
Collaborative Learning Tools (Google Apps, Microsoft Office 365, Padlet)
Accessibility Tools (e.g., ReadSpeaker, JAWS)
Simulation technologies (e.g., computer simulations, game-based learning, virtual reality)
Collaborative whiteboard software (e.g., Jamboard, LiveBoard)

How strongly do you agree with the following statements? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

Responding to electronic communications with students is overwhelming.
Using multiple technologies to support asynchronous and synchronous social interactions is overwhelming.
The privacy risks associated with digital communication and collaboration make me uncomfortable.
I find it difficult to adapt course materials to incorporate technology tools.
Learning how to use new technology tools distracts students from focusing on the content of lessons.
Self-regulated learning (i.e., thinking about one’s thinking and taking strategic actions to learn) is not something I should have to teach college students.
I do not know how to incorporate opportunities for students to build self-regulation skills into my courses.
Adding real-time checks for understanding and follow-up discussion is too time-consuming.
Adapting instruction during a lesson based on student responses is too difficult.
I’d rather keep learning activities I am familiar with than switch to new activities that use new technologies.

The following questions refer to your use of technology in [focal course title, section number] during this term.

Which technology tools are you currently using in [FOCAL COURSE TITLE, SECTION NUMBER]? Check all that apply.

Learning management Systems (LMS) (e.g., Canvas, Blackboard, D2L, Moodle, Google Classroom)
Video conferencing tools (e.g., Zoom, Microsoft Teams, Google Hangouts)
Lecture capture software (e.g., Canvas Studio, Panopto, Kaltura)
Presentation software (e.g., PowerPoint, Google Slides)
Educational content platforms (e.g., Khan Academy, TED-Ed, textbook publisher content and courseware, Open Educational Resources)
Online assessment tools (e.g., Quizlet, Google Forms)
Classroom poll response systems (e.g., Poll Everywhere, Zoom polling, Kahoot!)
Collaborative learning tools (e.g., Google Apps, Microsoft Office 365, Padlet)
Accessibility tools (e.g., ReadSpeaker, JAWS)
Simulation technologies (e.g., computer simulations, game-based learning, virtual reality)
Collaborative whiteboard software (e.g., Jamboard, LiveBoard)

How strongly do you agree with the following statements in relation to [FOCAL COURSE TITLE, SECTION NUMBER] this term? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

I am clear about the purpose of technology tools for communication and collaboration in my courses.
I design relevant digital discussion tasks that are integrated with course material.
I monitor digital communication boards and follow up with students.
I leverage technology tools to vary the formats for delivering my courses.
I use technology to provide multiple ways for students to access and study course materials.
Delivery formats for my courses are chosen based on the content being covered.
I use technology tools to support students’ self-monitoring of their learning.
I use technology tools to support students with time planning and management.
I use technology tools to support students in my courses with finding tutoring or mentoring.
I use automated scoring of online homework or quizzes to provide immediate feedback to students.
I use adaptive learning features such as built-in assessments in online textbooks to provide timely feedback.
I use data collected from student response systems to inform my instruction.
The content of my course lends itself to students engaging with authentic problems.
I use technology tools to simulate problems.
I use simulated problems to support my teaching.

Focal Course Student Experiences

The following questions ask you to reflect on the students enrolled in your focal course this term.

About what percent of students in [FOCAL COURSE TITLE, SECTION NUMBER] this term do the following? (Less than 25%, between 26–50%, between 51–75%, more than 75%)

Ask questions in class or contribute to class discussions
Work hard to master course material
Complete readings and assignments in a timely manner
Communicate with you directly (either asynchronously—e.g., email or via a learning management system—or synchronously—e.g., office hours) at least occasionally

How strongly do you agree with the following statements in relation to [FOCAL COURSE, SECTION NUMBER] this term? (Strongly disagree, Somewhat disagree, Neither agree nor disagree, Somewhat agree, Strongly agree)

When students encounter challenges related to the course, they come to me for help.
Students communicate with one another about course content.
Students use the technology tools I expect them to use in this course.

Professional Learning Experiences

These final questions ask about your experiences with professional learning and support for using technology in teaching.

How strongly do you agree with the following statements? (Strongly disagree, somewhat disagree, neither agree nor disagree, somewhat agree, strongly agree)

When I have questions about how to implement a new technology tool in my teaching, I can access the help I need.
I have professional relationships with other faculty that help me improve my teaching.
In the past six months, I have learned about new promising instructional approaches that are relevant to my teaching practice.
In the past six months, I have learned about new promising instructional approaches using technology that are relevant to my teaching practice.
In the past six months, I have improved my teaching as a result of a professional learning opportunity.

Within the past year, did you participate in [XXX PROFESSIONAL LEARNING INTERVENTION]? (Yes, No, Unsure)

Note: this question will be included if college administrators identify any recent professional learning activities at the college with similar goals to the professional learning course.

Please describe any [ADDITIONAL] professional learning opportunities related to teaching with technology that you have participated in in the last year.

Name of activity [open field]
Purpose and topic [open field]
Approximate number of hours of engagement [open field]

Note: respondents will be able to enter multiple activities

Thank you for your time spent taking this survey. Your response has been recorded.

Public Burden Statement

2. Instructor focus group protocol

Part 1. Uses of instructional technology and their purposes in the focal course

Tell me about the technology tools you use in your [FOCAL] courses.
How do those tools help you create the learning environment you are trying to achieve?
How do those tools help students meet the learning outcomes for the course?
Probe for and/or code for Practice Guide tools/practices:
- Communication tools
- Multiple pathways to learn content
- Organizational tools (calendars, to do lists)
- Nudges/self-monitoring prompts
- SRL strategy modeling
- Adaptive learning, tutoring, or mentoring
- Pedagogical agents
- Assessments to monitor progress
- Automated scoring/feedback
- Polling
- Simulation technologies
- Reflective discussion

Part 2. Processes for selecting, implementing, and evaluating technology

When and how did you come to incorporate those technological tools into your course? What factored into your decision?
How do you communicate with students about these technology tools and their purposes?
What challenges do you see students encountering as they engage with the technology tools in your course?
What kinds of support do you find students need to engage with these tools?
How do you assess how well technology is working in your course? Can you offer any examples of how you have made adjustments to your approach to using technology recently?
Relevant concepts from the Practice Guide:
- Consider student preferences
- Identify learning objectives first
- Select technology aligned to content and objectives
- Communicate expectations to students
- Provide supports for using technology
- Monitor student technology use
- Assess and evaluate technology

Part 3. Student engagement

Tell me about student engagement in your focal course. What opportunities do students have to connect and engage with one another?
- What opportunities do students have to connect and engage with you?
What are the most effective things you do to improve student engagement in the course and with the content? (Probe for uses of instructional technology.)
What are the challenges and barriers to student engagement in the course?

Part 4. Professional learning course perceptions and experiences (for treatment group instructors only)

Do you recall what learning goals you identified for yourself at the beginning of the professional learning course? Do you feel you met those goals?
Tell me about your experience as a learner in the course.

Do you have any reflections on the way the professional learning course was organized, and the content was delivered? (Probe for facilitators’ use of practice guide recommendations in the delivery of the modules.)

In Module 4, instructors were asked to share a part of their teaching practice that embodies one of the practice guide recommendations. Do you recall learning something by preparing for and watching those presentations? If yes, can you share an example?
What recommendations do you have for improving the professional learning course?

Part 5. Learning resources and opportunities

How do you learn about new instructional technology tools?
How do you learn how to incorporate technology tools into your course design/instructional practice?
Can you tell me about your participation in any [other] professional learning experiences within the past year that focus on the use of instructional technology?
- Probe for: Purpose, content, format, duration, and intensity

3. Structured course observation of materials protocol

(Note: This protocol will be used by CCRC researchers to conduct a passive review of 30 randomly selected focal courses.)

Part 1. General course characteristics

Enter disciplinary area and course title
What is the delivery format?
1. Online (asynchronous)
2. Hybrid
3. Face-to-face/in-person
4. Remote (online and synchronous)

How many students are enrolled?

Part 2. Practice Guide recommendations

Describe any communication and collaboration tools to increase interactions:
1. Between instructor and students
2. Among students
3. How are expectations for these tools articulated?
4. What is the prevalence or frequency of this tool use? (Once during the course, Two to four times during the course, More than five times)

Describe the presence of personalized digital resources that provide multiple pathways to deliver/learn content.

How are expectations for student use of these resources articulated?
What is the prevalence or frequency of this resource use? (Once during the course, Two to four times during the course, More than five times)

Describe technology tools that model or foster self-regulated learning strategies, including:

Organizational tools: calendars, to dos
Nudges/self-monitoring prompts
Adaptive learning, tutoring, or mentoring
Pedagogical agents
How are expectations for student use of these tools articulated?
What is the prevalence or frequency of this tool use? (Once during the course, Two to four times during the course, More than five times)

Describe the use of technology tools to provide timely and targeted feedback on student performance, including:

Assessments to monitor progress
Automated scoring/feedback
Polling
How are expectations for student use of these tools articulated?
What is the prevalence or frequency of this tool use? (Once during the course, Two to four times during the course, More than five times)

Describe the use of simulation technologies that help students engage in complex problem solving

Describe any reflective discussion associated with simulation technologies
How are expectations for student use of these tools articulated?
What is the prevalence or frequency of this tool use? (Once during the course, Two to four times during the course, More than five times)

4. Student survey and consent

Thank you for your interest in completing a 10-minute survey about your experiences in your college course this term. Click the link to review an Informed Consent Form that describes the purpose of the study, risks and benefits to participation, available monetary incentives, and procedures for confidentiality. If you consent to participate in the survey, you will be directed to the survey.

INFORMED CONSENT

Student Survey

Protocol Title: Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning, IRB Protocol Number 23-393

Principal Researcher: Dr. Thomas Brock, Community College Research Center, Teachers College, 212-678-3091, [email protected]

INTRODUCTION You are invited to participate in this research study about how you experienced technology in one of your courses this fall. You may qualify to take part in this research study because you are or were enrolled in a course that qualifies for this study and you are 18 years old or older. Approximately 2880 students will participate in this study. This study is being conducted by researchers at the Community College Research Center (CCRC) on behalf of the Regional Education Laboratory Northwest, which is funded by the Institute of Education Sciences.

WHY IS THIS STUDY BEING DONE? This study is being done to evaluate the effectiveness of a professional learning course for community college instructors about using technology in teaching. The instructor of a course you are enrolled in is participating in the study. Findings from the study may be used to improve professional learning for instructors, which may also improve teaching and the student experience in community colleges.

WHAT WILL I BE ASKED TO DO IF I AGREE TO TAKE PART IN THIS STUDY? You will be asked to complete an online survey about your experience in a course you enrolled in this fall which will take approximately 10 minutes of your time. As part of the study, your responses will be linked to other study data using a unique identification code assigned by your college. This data includes your grades for fall 2024 and your enrollment for 2025. This study is voluntary. Participating or not participating in the study will have no impact on your standing (i.e., grades or performance) at your institution.

WHAT POSSIBLE RISKS OR DISCOMFORTS CAN I EXPECT FROM TAKING PART IN THIS STUDY? The risks associated with this study are minimal; however, there are some risks to consider. The survey will ask you questions about your experience in one of your courses this term. It is possible that some of the questions may make you uncomfortable. You may decline to answer any question that might make you uncomfortable.

WHAT POSSIBLE BENEFITS CAN I EXPECT FROM TAKING PART IN THIS STUDY? There is no direct benefit to you for participating in this study, although some participants may find it interesting or constructive to reflect on their learning experiences. The goal of the study is to inform improvements in instruction and student success in higher education.

WILL I BE PAID FOR BEING IN THIS STUDY? If you complete the survey, you will be eligible to receive one of 15 gift cards worth $100 each. Approximately one in 200 students will receive a gift card.

WHEN IS THE STUDY OVER? CAN I LEAVE THE STUDY BEFORE IT ENDS? The study is over when you complete the survey. You can leave the study at any time.

PROTECTION OF YOUR CONFIDENTIALITY: Your name will not be collected or appear in any public reports or documents. Your survey response will be associated with a unique code to facilitate linking with other data collected for this study. Electronic data will be stored on a server within the Teachers College network, behind the College firewall, and will be accessible only to the research team via on-campus computers or off-campus networked computers that have been authenticated using the College’s Virtual Private Network (VPN). Regulations require that research data be kept for at least ten years; however, the data will be destroyed at the conclusion of that period. For quality assurance, the study team, the study sponsor (grant agency), and/or members of the Teachers College Institutional Review Board (IRB) may review the data collected from you as part of this study. Otherwise, all information obtained from your participation in this study will be disclosed only with your permission or as required by U.S. or State law.

HOW WILL THE RESULTS BE USED?

The results of this study may be published in journals or reports and presented at academic conferences. Your name or any identifying information about you will not be published.

WHO CAN ANSWER MY QUESTIONS ABOUT THIS STUDY?

If you have any questions about taking part in this research study, you should contact Dr. Thomas Brock, at [email protected] or at (212) 678-3019.

If you have questions or concerns about your rights as a research subject, you should contact the Institutional Review Board (IRB) (the human research ethics committee) at 212-678-4105 or email [email protected] or you can write to the IRB at Teachers College, Columbia University, 525 W. 120^thStreet, New York, NY 10027, Box 151. The IRB is the committee that oversees human research protection for Teachers College, Columbia University.

PARTICIPANT’S RIGHTS

I have read the Informed Consent Form and have been offered the opportunity to discuss the form with the researcher.
I have had ample opportunity to ask questions about the purposes, procedures, risks and benefits regarding this research study.
I understand that my participation is voluntary. I may refuse to participate or withdraw participation at any time without penalty.
If, during the course of the study, significant new information that has been developed becomes available which may relate to my willingness to continue my participation, the researcher will provide this information to me.
Any information derived from the research study that personally identifies me will not be voluntarily released or disclosed without my separate consent, except as specifically required by law.
Identifiers may be removed from the data. De-identifiable data may be used for future research studies by the study researchers.
I should receive a copy of the Informed Consent Form document.

By checking the “I agree” box and typing your name, you are electronically signing this consent form to participate in this study. You affirm that an electronic signature has the same effect as a written signature. You also confirm that you are 18 years or older.

To agree, check the “I agree” box and type in your name. If you do not wish to participate in this study, close out of this browser window.

[ check box ] [ participant full name ]

Student Survey

Most questions in this survey focus on one course you enrolled in this term: [COURSE NAME]. Please think about that course and its instructor when you respond to the questions. Please complete the survey to the best of your ability even if you are no longer enrolled in or attending the course.

Are you currently enrolled in [COURSE NAME]?

Yes, I am still enrolled and attending/participating in this course.
Yes, I am still enrolled but I am no longer attending/participating.
No, I have withdrawn from or dropped this course.

What best describes the modality of this course?

In-person
Online and asynchronous (no required meeting times)
Online with synchronous components (required meetings held using video conferencing)
Hybrid (some online components and some in-person components)

Is this course required for your degree program or major?

Yes
No
Not sure
Not applicable because I am not enrolled in a degree program

COURSE EXPERIENCES

To what extent do the following behaviors, thoughts, and feelings describe you in this course?

(Not at all like me, Not really like me, Moderately like me, Like me, Very much like me)

I find ways to make the course material relevant to my life.
I apply course material to my life.
I find ways to make the course interesting to me.
I think about the course between class meetings.
I really desire to learn the material.

I make sure to study on a regular basis.
I go to the professor’s office hours or contact him/her to review assignments or tests or to ask questions.
I take good notes on class lectures, discussions and other materials.
I stay up to date on the readings.
I complete assignments on time.
I look over class notes between classes to make sure I understand the material.
I ask questions when I don’t understand the instructor.
I have fun in class.
I help fellow students.

To what extent do the following behaviors, thoughts, and feelings describe you in this course? (Not at all like me, not really like me, moderately like me, like me, very much like me, NOT APPLICABLE)

I raise my hand or answer questions in class
I participate actively in small group or discussion board discussions

To what extent do the following behaviors, thoughts, and feelings describe you in this course? (Not at all like me, not really like me, moderately like me, like me, very much like me)

I regularly work with other students on course areas that I find challenging
I regularly study with other students
I feel part of a group of students committed to learning

How strongly do you agree with the following statements about the instructor of this course? (Strongly disagree, somewhat disagree, neither agree nor disagree, somewhat agree, strongly agree)

The instructor makes a real effort to understand any difficulties I am having in the course
The instructor takes an interest in my progress
The instructor gives helpful feedback on my progress
The instructor is usually available to discuss my questions

What types of technology tools did you use to support your learning in this course? Check all that apply.

Learning management systems (LMS) (e.g., Canvas, Blackboard, D2L, Moodle, Google Classroom)
Video conferencing tools (e.g., Zoom, Microsoft Teams, Google Hangouts)
Lecture capture software (e.g., Canvas Studio, Panopto, Kaltura)
Presentation software (e.g., PowerPoint, Google Slides)
Educational content platforms (e.g., Khan Academy, TED-Ed, textbook publisher content and courseware, Open Educational Resources)
Online assessment tools (e.g., Quizlet, Google Forms)
Classroom poll response systems (e.g., Poll Everywhere, Zoom polling, Kahoot!)
Collaborative learning tools (Google Apps, Microsoft Office 365, Padlet)
Accessibility tools (e.g., ReadSpeaker, JAWS)
Simulation technologies (e.g., computer simulations, game-based learning, virtual reality)
Collaborative whiteboard software (e.g., Jamboard, LiveBoard)

How often did you experience technology hardware or software problems that interfered with your participation in this course? (Very often, Often, Occasionally, Never)

How satisfied are you with the following elements of this course? (Very satisfied, Somewhat satisfied, Somewhat dissatisfied, Very dissatisfied)

My access to the technology resources I need to succeed
Instructor's preparation
Quality of course content
Quality of instruction
My overall learning

On average, approximately how many hours do you spend on this course each week? Include time attending class, studying and reviewing materials, and completing homework and assignments.

Less than 5 hours
6–10 hours
11–15 hours
More than 15 hours

BACKGROUND CHARACTERISTICS
What was your high school GPA?

4.0 (or higher); 93-100 (or higher); A+/A
3.7–3.99; 90–92; A-
3.3–3.699; 87–89; B+
3.0–3.299; 83–86; B
2.7–2.99; 80–82; B-
2.3–2.699; 77–79; C+
2.0–2.299; 73–76; C
1.7–1.99; 70–72; C-
1.3–1.699; 67–69; D+
1.0–1.299; 65–66; D
0.0–0.99; Below 65; F
Unknown
Not applicable

The following personal information will ONLY be used to link your survey responses to your course grades and your course enrollments, as provided by your college. Once the research team links the survey to your course enrollment and grades, this personal information will be deleted and will not be used for any other purposes.

What is your first name? [open field]

What is your last name? [open field]

What is your date of birth? [date input]

What is your student ID? [open field]

Public Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx. Public reporting burden for this collection of information is estimated to average 10 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this individual collection, or if you have comments or concerns regarding the status of your individual form, application or survey, please contact (Program Office address/contact information) directly.

5. Administrator interview protocol

INFORMED CONSENT

Protocol Title: Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning, IRB Protocol Number 23-393

Principal Researcher: Dr. Thomas Brock, Community College Research Center, Teachers College, 212-678-3091, [email protected]

You are invited to participate in a one hour virtual interview to discuss your college’s experience implementing the professional learning course.

WHAT WILL I BE ASKED TO DO IF I AGREE TO TAKE PART IN THIS STUDY?

If you decide to participate, a member of the research team will interview you individually. The interview will occur in person or via phone or a web conference platform. The interview will be scheduled at a time and place that is convenient to you. The interview will be audio-recorded. After the audio recording is written down (transcribed) the audio recording will be deleted. If you do not wish to be audio-recorded, you will still be able to participate. The researcher will take handwritten notes. The interview will take approximately one hour.

In the interview, you will be asked to discuss perceived instructor learning needs related to instructional technology, available professional learning opportunities for instructors related to instructional technology, perceptions of the professional learning course, as well as related institutional priorities, initiatives, and investments.

WHAT POSSIBLE RISKS OR DISCOMFORTS CAN I EXPECT FROM TAKING PART IN THIS STUDY? The risks associated with this study are minimal. Administrators will be asked questions about professional learning and opinions about the course, which may be a source of discomfort if an administrator wants to express what they perceive to be an unpopular opinion. To minimize this discomfort, the researchers will de-identify all data prior to conducting any analyses. Study participants do not have to answer any questions or share anything they do not want to talk about. The primary researcher is taking precautions to keep each participant’s information confidential and prevent anyone from discovering or guessing their identity, such as using a pseudonym instead of their name and keeping all information on a password protected computer and locked in a file drawer.

WHAT POSSIBLE BENEFITS CAN I EXPECT FROM TAKING PART IN THIS STUDY? Participants may find it interesting or constructive to respond to questions related to professional learning opportunities regarding teaching using technology. The goal of the study is to inform improvements in instruction and student success in higher education.

WILL I BE PAID FOR BEING IN THIS STUDY? There is no compensation for participating in the virtual interview. There are no costs to participants for taking part in this study.

WHEN IS THE STUDY OVER? CAN I LEAVE THE STUDY BEFORE IT ENDS? This study is voluntary. You do not need to answer any question that might make you uncomfortable. You can exit this study interview at any time without penalty. Participating or not participating in the study will have no impact on your standing at your institution.

PROTECTION OF YOUR CONFIDENTIALITY Only the research team will have access to data that is linked to any identifying information. The research team will strip the data of any identifying information (name, email address) prior to beginning analysis. Electronic data will be stored on a server within the Teachers College network, behind the College firewall, and will be accessible only to the research team via on-campus computers or off-campus networked computers that have been authenticated using the College’s Virtual Private Network (VPN). Regulations require that research data be kept for at least ten years; however, the data will be destroyed at the conclusion of that period. For quality assurance, the study team, the study sponsor (grant agency), and/or members of the Teachers College Institutional Review Board (IRB) may review the data collected from a participant as part of this study. Otherwise, all information obtained from an instructor’s participation in this study will be disclosed only with their permission or as required by U.S. or State law.

PARTICIPANT’S RIGHTS

I have read the Informed Consent Form and have been offered the opportunity to discuss the form with the researcher.
I have had ample opportunity to ask questions about the purposes, procedures, risks and benefits regarding this research study.
I understand that my participation is voluntary. I may refuse to participate or withdraw participation at any time without penalty.
The researcher may withdraw me from the research at the researcher’s professional discretion.
If, during the course of the study, significant new information that has been developed becomes available which may relate to my willingness to continue my participation, the researcher will provide this information to me.
Any information derived from the research study that personally identifies me will not be voluntarily released or disclosed without my separate consent, except as specifically required by law.
My data will not be used in further research studies.
I should receive a copy of the Informed Consent Form document.

To agree, check the “I agree” box and type in your name. If you do not wish to participate in this study, close out of this browser window.
[ check box ] [ participant full name ]

Do you give consent to be audio-recorded? You may still participate in the study even if you do not wish to be audio-recorded. By checking the “YES” box, you are consenting to be audio-recorded.
[ ] YES [ ] NO

Part 1. Perceived instructor learning needs related to instructional technology

What student learning and success challenges might be improved through instructor use of course-based instructional technology?
What do instructors grapple with related to instructional technology?
What makes adopting/implementing instructional technology challenging for instructors?
Do you notice differences in adoption and implementation of technology across departments/disciplinary areas?

Part 2. Related institutional priorities, initiatives, and investments

Please describe any strategic initiatives or ongoing efforts related to improving teaching and learning, including culturally responsive or equity minded teaching/curriculum efforts.
Please describe any strategic initiatives or current efforts related to improving, teaching, course quality and course outcomes using instructional technology

Part 3. Available professional learning opportunities for instructors related to instructional technology

Tell us about the learning opportunities and supports available to instructors to adopt and implement instructional technology.
- Probe for:
  - Purpose and content of professional learning activities
  - Format, duration, and intensity of professional learning activities
  - Incentives for instructor participation
  - Inclusion of adjunct instructors
  - Uptake or participation levels
- If administrators describe any interventions with similar goals to the REL NW professional learning course, the research team will request relevant documentation (e.g., syllabi, lists of learning objectives or topic areas)

Part 4. Perceptions of the professional learning course

What do you think motivated instructors to want to participate in this opportunity?
Do you anticipate that the college will offer the professional learning course again?
- What would determine whether it would continue to be offered beyond this study?
- What might be the barriers?

6. Facilitator interview protocol and consent

INFORMED CONSENT

Protocol Title: Evaluation of the Professional Learning Resources of the REL NW Toolkit for Using Technology to Support Postsecondary Student Learning, IRB Protocol Number 23-393

Principal Researcher: Dr. Thomas Brock, Community College Research Center, Teachers College, 212-678-3091, [email protected]

You are invited to participate in a one hour in-person interview to discuss your experience as a facilitator of the professional learning course.

WHAT WILL I BE ASKED TO DO IF I AGREE TO TAKE PART IN THIS STUDY?

If you decide to participate, a member of the research team will individually interview you. The interview will occur in person or via phone or a web conference platform. The interview will be scheduled at a time and place that is convenient to you. The interview will be audio-recorded. After the audio recording is written down (transcribed) the audio recording will be deleted. If you do not wish to be audio-recorded, you will still be able to participate. The researcher will take handwritten notes. The interview will take approximately one hour.

In the interview, you will be asked to discuss your experience preparing and delivering the professional learning course, facilitating the course and using the course materials. You will also be asked to share observations of learner engagement with the course and materials, as well as how and whether the course could be supported by the institution for implementation among all faculty. Finally, you will be asked to describe other professional learning opportunities at the institution regarding instructional technology and how the course in this study compares.

WHAT POSSIBLE RISKS OR DISCOMFORTS CAN I EXPECT FROM TAKING PART IN THIS STUDY? The risks associated with this study are minimal. Facilitators will be asked questions about their experiences and opinions, which may be a source of discomfort if a facilitator wants to express what they perceive to be an unpopular opinion. To minimize this discomfort, the researchers will de-identify all data prior to conducting any analyses. Study participants do not have to answer any questions or share anything they do not want to talk about. The primary researcher is taking precautions to keep each participant’s information confidential and prevent anyone from discovering or guessing their identity, such as using a pseudonym instead of their name and keeping all information on a password protected computer and locked in a file drawer.

WILL I BE PAID FOR BEING IN THIS STUDY? There is no compensation for participating in the virtual interview. There are no costs to participants for taking part in this study.

PROTECTION OF YOUR CONFIDENTIALITY Only the research team will have access to data that is linked to any identifying information. The research team will strip the data of any identifying information (name, email address) prior to beginning analysis. Electronic data will be stored on a server within the Teachers College network, behind the College firewall, and will be accessible only to the research team via on-campus computers or off-campus networked computers that have been authenticated using the College’s Virtual Private Network (VPN). Regulations require that research data be kept for at least ten years; however, the data will be destroyed at the conclusion of that period. For quality assurance, the study team, the study sponsor (grant agency), and/or members of the Teachers College Institutional Review Board (IRB) may review the data collected from a participant as part of this study. Otherwise, all information obtained from an instructor’s participation in this study will be disclosed only with their permission or as required by U.S. or State law.

PARTICIPANT’S RIGHTS

I have read the Informed Consent Form and have been offered the opportunity to discuss the form with the researcher.
I have had ample opportunity to ask questions about the purposes, procedures, risks and benefits regarding this research study.
I understand that my participation is voluntary. I may refuse to participate or withdraw participation at any time without penalty.
The researcher may withdraw me from the research at the researcher’s professional discretion.
If, during the course of the study, significant new information that has been developed becomes available which may relate to my willingness to continue my participation, the researcher will provide this information to me.
Any information derived from the research study that personally identifies me will not be voluntarily released or disclosed without my separate consent, except as specifically required by law.
My data will not be used in further research studies.
I should receive a copy of the Informed Consent Form document.

To agree, check the “I agree” box and type in your name. If you do not wish to participate in this study, close out of this browser window.
[ check box ] [ participant full name ]

Part 1. Preparing and delivering the Workshop

What planning and preparation activities did you undertake to prepare to deliver the professional learning course?
- Did you encounter any challenges during the preparation period?
What recommendations would you offer to ensure future facilitators are prepared to deliver the course?

Part 2. Facilitating the Professional learning Course

Tell me about your experience using the materials for the professional learning course, Using Technology to Support Postsecondary Student Learning including slide decks and facilitation materials provided in the REL NW toolkit.
- Did you encounter any barriers or challenges to using the materials as they were intended?
Tell me about your perceptions of how well the professional learning course went.
- What did you notice about how instructors engaged as learners?
- Did you notice any differences in instructor engagement across the four sessions?
- Did you see evidence of participants forming a sense of community? If yes, please describe.
When instructors had choices on which recommendations to focus on (i.e., in modules 1 and 2), did you notice any patterns in their interest?
What challenges did you encounter as a facilitator during the course?
What recommendations would you offer to refine the REL NW toolkit content and professional learning course structure for the future?

Part 3. Institutionalizing support activities

What did you hear from participants about the availability of institutional supports to implement the practice guide recommendations?
Did you observe differences in instructor capacity (and institutional capacity) across disciplines?
Do you anticipate that the college would offer the professional learning course again?
- What would determine whether it would continue to be offered beyond the research study?
- What might be the barriers?

Part 4. Other available professional learning opportunities

Are you familiar with other similar learning opportunities available to instructors to adopt and implement instructional technology?
How is the professional learning course similar to or different from these other opportunities

7. Observation protocol and fidelity rubric for professional learning course synchronous sessions

Part 1. Context

Date; start/end time
Session number [1, 2, 3, 4]
Number of facilitators and their roles/affiliations
Number of instructor participants present
- Total: ___
- On time: ___
- Arrived >5 mins late: ___
- Departed >5 mins early: ___

Part 2. Running fieldnotes (researcher uses this section to capture chronological notes on the session content, activities, and discussions)

Part 3. Practice Guide recommendations

Use communication and collaboration tools to increase interaction among students and between students and instructors (example strategies or tools: use of shared documents such as Google Docs or Slides, shared whiteboards, discord, Jamboard, Flip)

Specific strategies/approaches/technology tools discussed: ___
Participants’ rationale/purposes for employing the recommendation (what problems of practice will it solve?): __
Challenges/barriers/concerns raised by participants: __

Use varied, personalized, and readily available digital resources to design and deliver instructional content (example strategies or tools: instructor created or curated videos or audio files)

Instructor participants’ rationale/purposes for employing the recommendation (what problems of practice will it solve?): __
Challenges/barriers/concerns raised by participants: __

Incorporate technology that models and fosters self-regulated learning strategies (example strategies or tools: use of planning lists (e.g., Gmail tasks; Todoist), use of progress bar for content in LMS)

Specific strategies/approaches/technology tools discussed: __
Instructor participants’ rationale/purposes for employing the recommendation (what problems of practice will it solve?): __
Challenges/barriers/concerns raised by participants: __

Use technology to provide timely and targeted feedback on student performance (example strategies or tools: use of clickers, polls, or surveys)

Specific strategies/approaches/technology tools discussed: __
Instructor participants’ rationale/purposes for employing the recommendation (what problems of practice will it solve?): __
Challenges/barriers/concerns raised by participants: __

Use simulation technologies that help students engage in complex problem-solving (example strategies or tools: simulation websites like Desmos, iCivics, etc.)

Specific strategies/approaches/technology tools discussed: __
Instructor participants’ rationale/purposes for employing the recommendation (what problems of practice will it solve?): __
Challenges/barriers/concerns raised by participants: __

Part 4. Indicators of Instructor Engagement

What proportion of participant attendees contribute to discussions verbally or via chat?
Do instructor participants offer explicit connections to their own courses/teaching practices? If yes, provide examples.
Is there evidence that instructors are building community (i.e., responding to one another’s comments, making reference to shared experiences)? If yes, provide examples.

Part 5. Fidelity Rubric

Observers should use Session Materials provided in the toolkit to rate the extent to which each content area was covered by the facilitators.

0 = Content was not introduced or covered

1 = Content was partially covered

2 = Content was fully covered as described in the Session Materials

Record the total time spent in small groups (i.e., in Breakout Room that are not shaded). Use the notes column to explain deviations from the Session Materials.

Session 1 Materials	Score 0 \| 1 \| 2	Minutes in Breakout Rooms	Notes
Welcome!
Where are we in the course?
Session Goals
Session Agenda
Our Values/Our Norms
Small Group Meet & Greet (12 mins)
Using simulations to engage in complex problem solving
Small Group (15 mins)
Whole Group conversation
Break
Recommendation 5
Small Group (15 mins)
Reflecting on Communication & Collaboration (2 slides)
Next Steps & Wrap-up (2 slides)
Casual Q&A
TOTAL SCORE / TOTAL MINS	/30	/42

Session 2 Materials	Score 0 \| 1 \| 2	Minutes in Breakout Rooms	Notes
Welcome!
Where are we in the course?
Session 1 Feedback (2 slides)
Session Goals
Session Agenda
Our Values/Our Norms
Introductions & the “Digital Divide” (15 mins)
Digital Divide Takeaways
Solving design and delivery challenges
Small Group (25 mins)
Break
Addressing common obstacles (2 slides)
Small Group (15 mins)
Reflecting on the Learning Experience (3 slides)
Next Steps & Wrap-up (2 slides)
Casual Q&A
TOTAL SCORE / TOTAL MINS	/32	/55

Session 3 Materials	Score 0 \| 1 \| 2	Minutes in Breakout Rooms	Notes
Welcome!
Where are we in the course?
Session 2 Feedback (2 slides)
Session Goals
Session Agenda
Our Values/Our Norms
Introductions & discussing the traits of self-regulated learners (20 mins)
Traits of self-regulated learners (2 slides)
Supporting students to develop their learning skills (2 slides)
Small Group (25 mins)
Break
Providing timely, targeted feedback
Potential obstacle to providing timely, targeted feedback (2 slides)
Small Group (18 mins)
Next Steps & Wrap-up (2 slides)
Casual Q&A
TOTAL SCORE / TOTAL MINS	/32	/63

Session 4 Materials	Score 0 \| 1 \| 2	Minutes in Breakout Rooms	Notes
Welcome!
Where are we in the course?
Session 3 Feedback (2 slides)
Session Goals
Session Agenda
Our Values/Our Norms
Small Group Introductions (9 mins)
Workshopping Redesign Plans
Small Group (30 mins)
Break
Workshopping Redesign Plans--continued
Small Group (30 mins)
Virtual White Board Appreciation
Next Steps & Wrap-up (2 slides)
Casual Q&A
TOTAL SCORE / TOTAL MINS	/30	/69

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Ellen Wasserman
File Modified	0000-00-00
File Created	2023-09-30