Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics Monitoring System (TUES)

APPENDIX A

TUES Monitoring System Screenshots – Mockup Survey

The screenshots are not yet available for this new system. Mockups are provided for the survey instrument. Depending on the type of award, respondents answer a subset of questions tailored to their circumstances. We have provided a mockup of a full set of questions.

TUES Monitoring System (for Lead PI on a Collaborative Project)

For the [2009-2010] Academic Year and Preceding Summer

Note to reviewers: The web system will include a feature that allows the respondent to add a comment to any question. To make a comment, the respondent will click a comment icon near the question, and then a free-form text field will display.

1. About This Award

1. NSF Award ID: _________________________________

2. Grantee Institution: _________________________________

3. Lead PI’s Name: _________________________________

4. Lead PI’s Discipline: _________________________________

5. Project Title: _________________________________

6. Phase or Type:

 Phase 1/Type 1

 Phase 2/Type 2

 Phase 3/Type 3

 Central Resource Project

7. Award Start Date: _________________________________

8. Award End Date: _________________________________

9. Year of Project:

 Year 1

 Year 2

 Year 3

 Year 4

 Year 5 or more

 Project has ended

10. Single Award or Collaborative Project:

 Single Award Project. There is only one awardee institution for this project.

 Collaborative Project. There are multiple awardee institutions associated with this project.

Lead institution for this collaborative project: ________________________________

Other awardee institutions in this collaborative:

___________________________________

___________________________________

___________________________________

2. About the Collaborative Project as a Whole

The questions in this section pertain to your CCLI/TUES Collaborative Project as a whole; when answering, consider the efforts of all of the institutions involved.

Project Overview

Disciplinary Focus

1. What is the primary disciplinary focus of the project? (mark one)

 Multidisciplinary or Interdisciplinary

 Astronomy

 Biological Sciences

 Chemistry

 Computer Sciences

 Engineering

 Engineering Technology

 Geosciences

 Mathematical Sciences

 Physics

 Social and Behavioral Sciences

 Other, please specify: _______________________

2. What are the additional disciplinary focuses of the project, if any? (mark all that apply)

 Multidisciplinary or Interdisciplinary

 Astronomy

 Biological Sciences

 Chemistry

 Computer Sciences

 Engineering

 Engineering Technology

 Geosciences

 Mathematical Sciences

 Physics

 Social and Behavioral Sciences

 Other, please specify: _______________________

 None

TUES Program Components

3. What is the primary way that your project contributes to the development of exemplary undergraduate STEM education? (mark one)

 Assessing and evaluating student achievement

 Creating learning materials and strategies

 Conducting research on undergraduate STEM education

 Developing faculty expertise

 Implementing new instructional strategies

4. What other ways does your project contribute to the development of exemplary undergraduate STEM education? (mark all that apply)

 Assessing and evaluating student achievement

 Creating learning materials and strategies

 Conducting research on undergraduate STEM education

 Developing faculty expertise

 Implementing new instructional strategies

5. Does a significant aspect of the project include the introduction of new disciplinary knowledge into undergraduate curricula?

 Yes  No

Target Audiences

6. What course levels does your project address? (mark all that apply)

 Upper division

 Lower division

 Developmental

7. What groups are the primary focus of your project? (mark all that apply)

 Faculty (e.g., developing faculty expertise)

 Pre-service teachers

 Instructional staff (e.g., graduate students, lab assistants, lab technicians, adjunct faculty)

 Undergraduate STEM majors

 Undergraduate majors other than STEM

 Other, please specify: ____________________

8. Does the project foster scientific and/or technical literacy among students who are in majors other than STEM majors?

 Yes  No

Project Activities, Outputs, and Outcomes during the Current Reporting Period

Summary

9. Did the project develop, adapt, adopt, or use any of the following during the current reporting period:

 Instructional materials

 Instructional strategies

 Assessment instruments or processes to assess student achievement

 None of the above

10. Please list any publicly available assessment instruments (e.g., SALG, FCI, NSSE) that your project administered during the current reporting period:

__________________________________________________

__________________________________________________ __________________________________________________

11. Did the project include a faculty professional development component (e.g., seminar, workshop, meeting session) that shared project approaches with other faculty members during the current reporting period?
    
     Yes  No

12. Did the project conduct research on undergraduate STEM education during the current reporting period?
    
     Yes  No

Instructional Materials (for respondents choosing this in response to 2.9)

13. Which of the following are true for your project during the current reporting period? (mark all that apply)

 We developed or revised instructional materials for our project

 We adapted or adopted materials created by others

 We used our project’s instructional materials with students or faculty

14. If you marked any of the above, what types of instructional materials did your project develop, adapt, adopt, or use? (mark all that apply)

 Case studies  Demonstrations  Educational Games  Field activity guides or manuals  Instructor materials for adapting project activities or procedures to new settings  Instructor guides or manuals to guide student activities  Laboratory guides or manuals  Lesson plans  Practice problems or exercises for students  Simulations

 Software tools  Student handbooks or workbooks  Textbooks  Tutorials  Visualizations  Web-based collaborative tools  Web-enhanced tools  Workshop materials for training faculty  Workshop materials for training students  Other instructional materials (please specify)

Note to reviewers: The system will show the user the list of instructional materials above. The user will choose the materials from the list that apply to their project. For each material chosen, the system will display the following questions:

Sample Instructional Material	 Developed or revised materials created by our project  Adopted or adapted materials created by others
	 Used with undergraduate STEM majors  Used with undergraduates not majoring in a STEM field  Used with graduate students to develop expertise  Used with faculty members to develop expertise
	Formats:  print  online  other electronic format

Instructional Strategies (for respondents choosing this in response to 2.9)

15. Which of the following are true for your project during the current reporting period? (mark all that apply)

 We developed or revised new instructional strategies for our project

 We adapted or adopted strategies created by others

 We used instructional strategies with students or faculty

16. What types of instructional strategies did your project develop, adapt, adopt, or use? (mark all that apply)

 Case-based learning strategies

 Cooperative or collaborative learning strategies

 Design activities

 Field studies

 Hands-on activities (e.g., building or testing objects)

 Integrated courses

 Intelligent tutoring systems

 Laboratory experiences

 Learning communities (e.g., student cohorts)

 Lectures

 Linked courses

 Real time modification strategies based on student performance (e.g., Just-in-Time Teaching)

 Peer instruction strategies (e.g., peer tutoring, peer mentoring)

 Personal response systems (e.g., “clickers”)

 Problem-based learning strategies

 Quantitative computational approaches

 Strategies involving real-world examples, problems, or contexts

 Strategies involving remote laboratories

 Strategies involving formal research experience in classroom laboratories

 Service learning strategies

 Studio-based strategies

 Web-enabled homework submission and grading systems

 Web-enabled instruction strategies

 Web-enabled repositories

 Web-enabled student collaboration / social networking tools

 Other instructional strategies (please specify):

17. For each instructional strategy marked above, please indicate its role in your project during the current reporting period:

Sample Instructional Strategy	 Developed or revised strategies created by our project  Adopted or adapted strategies created by others
	 Used with undergraduate STEM majors  Used with undergraduates not majoring in a STEM field  Used with graduate students to develop expertise  Used with faculty members to develop expertise
	Formats:  print  online  other electronic format

Assessment Instruments and Processes (for respondents choosing this in response to 2.9)

18. Which of the following are true for your project during the current reporting period? (mark all that apply)

 We developed or revised new assessment instruments or processes

 We adapted or adopted assessment instruments or processes created by others

 We used assessment instruments or processes with students or faculty

19. What types of assessment instruments or processes did your project develop, adapt, adopt, or use during the current reporting period? (mark all that apply)

 Assessment of general knowledge or skill (e.g., critical thinking skills, problem-solving skills)

 Cognitive protocol interviews

 Concept inventories

 Course exams

 Standardized tests (e.g., Fundamentals of Engineering; American Chemical Society examinations)

 Performance assessment of skills (e.g., observation and rating of skills)

 Students’ perceptions of their own learning

 Expert review

 Other instruments and processes to determine outcomes (please specify):

20. For each assessment instrument or process marked above, please indicate its role in your project during the current reporting period:

Sample Assessment Instrument or Process	 Developed or revised instruments or processes created by our project  Adopted or adapted instruments or processes created by others
	 Used with undergraduate STEM majors  Used with undergraduates not majoring in a STEM field  Used with graduate students to develop expertise  Used with faculty members to develop expertise
	Formats:  print  online  other electronic format

Research on Undergraduate STEM Education (for respondents answering Yes to 2.12)

21. What is the topic of your research on undergraduate STEM education?

__________________________________________________

__________________________________________________ __________________________________________________

22. What type of research does your project employ?

 Quantitative research design

 Qualitative research design

 Mixed method research design

Faculty Professional Development Component (for respondents answering Yes to 2.11)

23. With whom were project faculty professional development approaches shared during the current reporting period? (mark all that apply)

	Graduate students and Post-docs	Faculty	Instructional staff
At the lead, collaborating, or subawardee institutions			
At other participating institutions			
At other institutions across the U.S.			
At international institutions			

24. Which disciplines were the focus of the project’s faculty professional development efforts during the current reporting period? (mark all that apply)

 Multidisciplinary or Interdisciplinary

 Astronomy

 Biological Sciences

 Chemistry

 Computer Sciences

 Engineering

 Engineering Technology

 Geosciences

 Mathematical Sciences

 Physics

 Social and Behavioral Sciences

 Other, please specify: _______________________

 None

24. What types of institutions were represented at these efforts during the current reporting period? (mark all that apply)

 2-year Institution

 Bachelors

 Masters

 Doctoral

 Other, please specify: ____________________

 Don’t know

26. Did the institutions represented at the project’s faculty professional development efforts have any special institutional designations? If yes, what types? (mark all that apply)

 Asian/Pacific Islander Serving College

 Hispanic Serving Institution (more than ___% Hispanic students)

 Institution Primarily Serving Students with Disabilities

 Minority Serving Institution

 Native American Serving Nontribal College

 Predominantly Black College

 Historically Black College or University

 Tribal College

 None

 Not Reported / Don’t Know

27. We are interested in whether participants who receive funding to attend faculty professional development activities are more or less likely to implement what they learn. Please estimate how many faculty professional development participants received funds from the project or their institution to attend. For each group, please also indicate approximately how many are implementing the ideas shared.

	Participants funded by the project	Participants funded by their institution (if known)	All other participants
Number of faculty members
Number of institutions represented by those faculty members
Number of faculty implementing what they learned

Note: Your response to this question is optional this year, but may be required in future years.

Publications and Presentations

28. Please provide the number of project-related outputs that were created during the current reporting period for each of the publication types listed below. If a publication was created in multiple formats, it should be counted once under “unique publications,” and then counted again for each format in which it was published.

Publication Types	Total number of unique publications	Number of publications by format
		Print	Online	Electronic – Not Online
Peer-reviewed journal articles that were accepted for publication during the current reporting period.
Peer-reviewed conference papers or abstracts that were accepted for inclusion in conference proceedings during the current reporting period
Conference papers or abstracts that were accepted during the current reporting period without peer review.
Book chapters in edited volumes that were accepted for publication during this reporting period.
Monographs or books that were accepted for publication during this reporting period.
Other publications (e.g., newsletters, magazines) accepted for publication during the current reporting period

28. How many presentations or workshops on the project were delivered during the current reporting period? (Include presentations that were conducted by individuals trained by the project)

Presentation Types	Number of presentations/workshops
Peer-reviewed conference presentations
Invited conference or meeting presentations
Invited faculty development workshops

Honors and Awards

28. How many honors or awards did the project receive during the current reporting period (include individuals recognized for their contributions to the project)?

Top of Form



Bottom of Form

Enter 0 if no honors or awards were received.

Please complete the following form for each honor or award the project received during the current reporting period:

Title: __________________________________________________

Year: _____

Presenting Organization: __________________________________________________

Reason for the honor or award: ________________________________________

Institutionalization

31. Are there plans to sustain the project at your institution or any of the collaborating institutions after NSF funding ends?

 Yes  No (if no, skip to next section)

31. During the current reporting period, were steps taken to sustain the project’s activities after NSF funding ends?

 Yes, some steps to sustain the project were taken.

 Yes, all of the necessary steps to sustain the project were completed.

 No, no steps to sustain the project were taken during the current reporting period.
If no, when does the project plan to start? (enter month and year) ____ / ____

Dissemination

33. Where else are the project’s materials, strategies, faculty development activities, or assessment instruments/processes being used?

 At lead, collaborative, or subawardee institutions by people not directly involved with the project

 At the project’s other participating institutions by people not directly involved with the project

 At other institutions across the U.S.

 At international institutions

 None of the above

2. About Your Institution

The following questions are about your institution specifically, not the collaborative as a whole.

Demographics

1. Type of Institution: (mark one)

 2-year Institution

 2-year Institution on a four-year college or university campus

 Bachelors

 Masters

 Doctoral

 Nonacademic institution (if so, skip to Institutionalization)

 Other U.S. Department of Education IPEDs type:____________________

2. Special Institutional Designations: (mark all that apply)

 Asian/Pacific Islander Serving College

 Hispanic Serving Institution (more than ___% Hispanic students)

 Institution Primarily Serving Students with Disabilities

 Minority Serving Institution

 Native American Serving Nontribal College

 Predominantly Black College

 Historically Black College or University

 Tribal College

 None

 Not Reported / Don’t Know

4. About Your Subawardees and Other Participating Organizations

The questions in this section are specific to your CCLI/TUES Award. They ask about 1) your subawardees and 2) any other organizations that participated in the project through your institution (regardless of whether they received project funds). DO NOT include the other CCLI/TUES awardees in your Collaborative Project, or their subawardees.

Subawardees

1. Were subaward funds expended or transferred from this award to any subawardees during the current reporting period? (mark all that apply)

 Yes, we disbursed subawards from this award to one or more U.S. Degree-granting academic institutions during the current reporting period.

 Yes, we disbursed subawards from this award to one or more non-academic organizations during the current reporting period.

 No, we disbursed no subawards from this award during the current reporting period. (Skip to 4.2)

For each U.S. Degree-granting Academic institution receiving a subaward from this award, please answer the following:

Institution Name: _______________________________________________

Type of institution (mark one):

 2-year Institution

 2-year Institution on a four-year college or university campus

 Bachelors

 Masters

 Doctoral

 Nonacademic institution (if so, skip to Institutionalization)

 Other U.S. Department of Education IPEDs type: ____________________

Special Institutional Designations (mark all that apply):

 Asian/Pacific Islander Serving College

 Hispanic Serving Institution (more than ___% Hispanic students)

 Institution Primarily Serving Students with Disabilities

 Minority Serving Institution

 Native American Serving Nontribal College

 Predominantly Black College

 Historically Black College or University

 Tribal College

 None

 Not Reported / Don’t Know

Number of faculty from this institution that were involved in the project: _________

Number of students from this institution that were involved in the project: _________

For each non-academic institution receiving a subaward from this award, please answer the following:

Organization Name: _______________________________________________

Organization type:

 Business

 Consultant

 National Laboratory

 Museum

 Nonprofit

 Professional society

 Other (please specify): __________________________

Other participating organizations

2. Did any other organizations participate significantly in your project during the current reporting period (not including your subawardees or other awardees in the collaborative project)?

 Yes  No  Don’t know (If No or Don’t Know, skip to next page)

If yes, indicate how many of each type of organization participated in each of the categories below. It is okay to count the same organization or individual in multiple categories:

	Number of organizations participating significantly	Number of faculty or professionals participating	Number of students participating
U.S. degree-granting academic institutions
2-year Institutions
2-year Institutions on a four-year college or university campus
Bachelors-granting institutions
Masters-granting institutions
Doctorate-granting institutions
Other U.S. Department of Education IPEDs type:____________________
Institutions with Special Designations
Asian/Pacific Islander Serving College
Hispanic Serving Institution
Institution Primarily Serving Students with Disabilities
Minority Serving Institution
Native American Serving Nontribal College
Predominantly Black College
Historically Black College or University
Non-academic organizations
Businesses
Consultants
National Laboratories
Museums
Non-profits
Professional Societies
Other type of non-academic organization

4. Student, Faculty, and Staff Involvement in Project Activities

The questions in this section ask you to report the number of people involved in your project’s work. As the Lead PI, you are asked to report both:

• the number of people who participated in independent project activities (those where yours is the only CCLI/TUES award involved), and separate from that,

• the number of people who participated in collaborative activities (those where multiple CCLI/TUES awardees are involved).

The PIs at the other awardee institutions in your CCLI/TUES Collaborative Project report only their independent project activities.

1. Do you have data about the approximate numbers of students, faculty, and staff engaged in or impacted by project activities conducted during the current reporting period?

 Yes  No (If no, skip to Reflection)

Instructional Materials, Strategies, and Assessment Components

2. Student Counts

   During the current reporting period approximately …	Through independent project activities supported by your CCLI/TUES award (and its subawards)	Through collaborative project activities supported by multiple CCLI/TUES awards
   How many undergraduate students used the project’s instructional materials?
   How many undergraduate students were taught using the project’s instructional strategies?
   How many undergraduate students’ knowledge, skills, or attitudes were assessed?

3. Faculty and Staff Counts

During the current reporting period approximately …	Through independent project activities supported by your CCLI/TUES award (and its subawards)	Through collaborative project activities supported by multiple CCLI/TUES awards
How many project faculty worked on the project’s instructional materials?
How many project faculty worked on the project’s instructional strategies?
How many project faculty worked on the project’s assessment instruments?
How many project faculty used the project’s materials, strategies, or assessment instruments with students?
How many project faculty conducted discipline-based educational research?
How many non-instructional professional staff worked on the project? (e.g., lab technicians, professional staff associated with Centers for Teaching and Learning, and Centers for Institutional Research; Centers for Institutional Evaluation professional staff)
How many non-instructional professional staff used the project’s materials, strategies, or assessment instruments with students?

Faculty Professional Development Events

4. Did the project conduct any faculty professional development events during the current reporting period?

 Yes  No (If no, skip to Reflection)

If yes …

During the current reporting period approximately …	As independent events supported by your CCLI/TUES award (and its subawards)	As collaborative events supported by multiple CCLI/TUES awards
How many faculty professional development events were conducted?

The following questions ask for the number of people involved in the project’s faculty professional development events. When one individual is involved in the same way in multiple events (for example, participating as learners at multiple workshops), count that person only once in your response to that question. Similarly, be sure to count each person only once even if they participated in multiple venues.

5. Faculty involvement in professional development events

During the current reporting period approximately …	At independent events supported by your CCLI/TUES award (and its subawards)	As collaborative events supported by multiple CCLI/TUES awards
How many faculty were involved in creating the project’s faculty professional development events?
How many faculty were involved in leading the project’s faculty professional development events?
How many faculty participated as learners in the project’s faculty professional development events?

6. Graduate student and post-doc involvement in professional development events

During the current reporting period approximately …	At independent events supported by your CCLI/TUES award (and its subawards)	As collaborative events supported by multiple CCLI/TUES awards
How many graduate students and post-docs were involved in creating the project’s faculty professional development events?
How many graduate students and post-docs were involved in leading the project’s faculty professional development events?
How many graduate students and post-docs participated as learners in the project’s faculty professional development events?

7. Other measures of involvement in your institution’s faculty professional development events

During the current reporting period approximately …	At independent events supported by your CCLI/TUES award (and its subawards)	As collaborative events supported by multiple CCLI/TUES awards
How many representatives from nonacademic organizations (e.g., professional societies, industry, community organizations) participated as learners?
How many institutions were represented?

4. Other Uses of Funds from this Award

The questions in this section are specific to this CCLI/TUES award and any of its subawards.

Student Support

1. During this reporting period, were funds from this award used to support students (undergraduate, graduate, or post-docs)?

 Yes  No (If no, skip the next question)

2. If yes, how many students were paid to work on the project using funds from this award?

Number of undergraduate students paid: ___

Number of graduate students paid: ___

Number of post-docs paid: __

Laboratory Instrument Purchases

3. Did you purchase scientific instruments for laboratory improvement with this award during this reporting period?

 Yes  No

3. Number of instruments you purchased with this award that cost more than $5,000: _____

3. Total price of all laboratory instruments purchased with this award this period: $__________

3. Where are these new laboratory instruments housed? (Or if they have not been received yet, where will they be housed?) (mark all that apply)

 At [name of respondent’s institution]

 At [name of lead institution]

 At [name of collaborating institution 1]

 At [name of collaborating institution 2]

[… list all collaborating institutions …]

 At [name of subawardee institution 1]

 At [name of subawardee institution 2]

[… list all subawardee institutions …]

 At other participating institutions

 Other, please specify: ___________________

4. Project Management and Evaluation

The questions in this section are specific to this CCLI/TUES award and any of its subawards.

1. Which of the following evaluation and assessment activities did you conduct during the current reporting period? (mark all that apply)

 Project evaluation for improving the project

 Project evaluation for characterizing project accomplishments

 Assessments of changes in knowledge, skills, or attitudes of participants for improving the project

 Assessments of changes in knowledge, skills, or attitudes of participants for characterizing project accomplishments

 None of the above (If so, skip the next question)

2. If you conducted evaluations or assessments, did the results of these influence the direction of the project?

 Yes  No (If no, skip to the next page)

If yes, list the actions you took in response to what you learned, and describe the results of your actions:

__________________________________________________

__________________________________________________ __________________________________________________

Note: Your answers to this question will be analyzed by NSF so that a program-wide set of choices can be offered in subsequent data collection cycles. Please describe the actions you took with this in mind.

3. Did you use any project management tools during the current reporting period? If yes, please provide the name(s) of the tools, and describe any best practices. (Your answers to this question will be used to advise other awardees).

__________________________________________________

__________________________________________________ __________________________________________________

4. Were the project activities supported by this award implemented in courses or course-related laboratory sections?

 Yes  No (If no, skip the rest of this question)

If yes, in how many courses or course-related laboratory sections were project activities implemented? ______  Don’t know

Approximately how many students were enrolled in these courses or course-related laboratory sections? ______  Don’t know

Approximately how many students successfully completed these courses or course-related laboratory sections with a grade of C or better? ______  Don’t know

8. Reflection

1. What is the SINGLE most important project accomplishment for the reporting period?

(You may enter up to 500 characters, including spaces.)

____________________________________________________________

____________________________________________________________

____________________________________________________________

2. What is the SINGLE most significant challenge that the project faced during the reporting period? (You may enter up to 500 characters, including spaces.)

____________________________________________________________

____________________________________________________________

____________________________________________________________

3. What changes were made to the project, as outlined in the proposal, during the reporting period? (You may enter up to 500 characters, including spaces.)

____________________________________________________________

____________________________________________________________
____________________________________________________________

9. Final Submit

Congratulations! You have met all of the requirements of the survey. We have one last question.

Approximately how many person hours were required to complete the survey?
________ person hours

Data Collection Personnel

Note to reviewers: These questions may be embedded in the interface, not the survey because they are required for administering the data collection

Use this section to authorize additional project personnel to use the data collection system and to update information about the project, and/or to access the results of the annual survey for the TUES program as whole and the project’s individual reporting. The system will automatically send an e-mail message with a user name and password to each person to whom you grant data collection privileges.

For your convenience, the system has preloaded the personnel you authorized for the last data collection. Please delete any personnel that should no longer have credentials to use the system.

Person

Name: (only first and last names are required)

Honorific: ____________________

First: ____________________

Middle: ____________________

Last: ____________________

Suffix: ____________________

o Mark this box to delete this person from the list of authorized data collection personnel.

(The system will complete the section if the box is marked)

E-mail Address: ______________________________

Phone Number: ____________________ Extension: __________

What is this person’s primary role in your project? (mark one)

___ Lead Primary Investigator (Lead PI) (if a new PI is indicated, the system presents the next question)

___ Co-Primary Investigator (Co-PI)

___ Project Coordinator (PC)

___ Data Entry Personnel

___ Other, please specify: ____________________

4. You have indicated that this award has a new lead PI. Please indicate the new lead PI’s discipline: (mark one)

 Multidisciplinary or Interdisciplinary

 Astronomy

 Biological Sciences

 Chemistry

 Computer Sciences

 Engineering

 Engineering Technology

 Geosciences

 Mathematical Sciences

 Physics

 Social and Behavioral Sciences

 Other, please specify: _______________________

Privileges

Note: The Lead PI’s privileges are automatically set to “Yes” for all privileges.

Will this person have the authority to enter data?

___ Yes ___ No

Will this person have the authority to use this screen to authorize additional personnel?

___ Yes ___ No

Should this person receive announcements and reminders that are sent to your projects PIs from this system and from the data collection’s technical support team?

___ Yes (System will automatically mark “Yes” for PCs.) ___ No

Page 3 of 40