Survey of Higher Education Instructors to Support the Development of Teaching Materials Based on OPRE's Research and Evaluation

Develop Teaching Materials: Faculty and Instructor Survey

Environmental Scan

June 1, 2020

Language: English

Mode: web only

Survey distribution: open survey link

Record meta-data: IP address

Back Button: include

Soft check on questions throughout by default

Note: Question numbers and titles are for internal use only and will not be displayed to participants

Section A: Stakeholder group characteristics

A00 Introduction

The Office of Planning, Research, and Evaluation (OPRE) is a division of the Administration for Children and Families (ACF). OPRE is responsible for studying ACF programs and the populations they serve through rigorous research and evaluation projects. These studies include program evaluations, research syntheses, and descriptive and exploratory studies. OPRE-funded research uses a variety of methods and disciplinary perspectives to investigate topics like adoption and foster care, early childhood care and education, home visiting, youth services, family strengthening, welfare, and employment.

OPRE is interested in creating and disseminating free teaching materials based on this research. We are seeking input from US college and university instructors about the types of materials they would be most interested in incorporating into their own courses. This survey is being administered by MEF Associates and Mathematica on behalf of OPRE.

This survey consists of three parts: questions about your teaching experience; questions about how you develop teaching materials in general; and questions about a specific course you taught. Your responses will help us understand what instructors want and will contribute to setting priorities at OPRE.

All responses will be kept private. Institution name and other identifying information will be removed during analysis. All responses will be aggregated for reporting, except direct quotes of written comments. Comments may be shared publicly but will remain anonymous.

This collection of information is voluntary and will be used to develop teaching materials. Public reporting burden for this collection of information is estimated to average 15 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0531, Exp: 7/31/2022. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Mathematica; 220 East Huron Street, Suite 300, Ann Arbor, MI 48104-1912, Attention: Jesse Chandler.

By proceeding, you give your consent to participate in this survey.

(Programming Note: If a person attempts to exit the survey, have a soft check and message: Please complete the survey now if you can. We greatly value the time you can provide to help us understand your perspective. Unfortunately, if you exit the survey, you will be unable to return to where you left off.)

<page break>

A0 Section introduction

First, we would like to learn more about your teaching background.

A1.0 Screener

Have you ever been an instructor of record for a course at an institution of higher education?

Select one answer.

 Yes 1

 No 2

[Programming: include hover text definition as follows: By ‘instructor of record’ we mean able to authorize credit for the successful completion of a course.]

<IF A1.0 = 0, skip to F2.0: >

<page break>

A1.1 Amount of teaching experience

How many years have you been teaching higher education courses?

If you have been teaching for less than one year, please round up to 1.

Click or tap here to enter text.

YEARS (1-50)

<page break>

A2.1 Primary institution name

At what institution of higher education do you teach? If you teach at more than one institution, report the name of the institution you teach at most.

Click or tap here to enter text.

PRIMARY EMPLOYER (STRING 500)

A2.2 Primary institution zip code

What is the zip code for this institution of higher education?

Click or tap here to enter text.

ZIP CODE (NUMBER 5)

 My institution is not located in the USA 0

<IF A2.2 = 0, terminate with message: Thank you for your interest. This survey is specifically for US higher education instructors. You can learn more about OPRE at https://www.acf.hhs.gov/opre>

<page break>

A3 Discipline and department

Please indicate the department(s) or school(s) in which you have teaching appointments.

Select all that apply

 Economics 1

 Education 2

 Human Development/Family Studies 3

 Psychology 4

 Public Policy 5

 Public Health 6

 Social Work 7

 Sociology 8

 Statistics 9

 Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

<page break>

A4.1 Rank

What is your academic rank? If you teach at more than one institution, please report your rank at the institution you teach at most.

Select one answer.

• Professor (including titles such as Research Professor, Clinical Professor) 1

• Associate Professor (including titles such as Research Associate Professor, Clinical Associate Professor) 2

• Assistant Professor (including titles such as Research Assistant Professor, Clinical Assistant Professor) 3

• Instructor/Lecturer with a fixed term appointment 4

• Adjunct Professor, Adjunct Instructor, or Adjunct Lecturer 5

• Postdoc 6

• Graduate Student 7

 Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

<page break>

Adapted from the AERA-NAEd's Assessment of Education Research Doctorate Programs Faculty Que

A5 Number of courses

Between May 2019 and April 2020 (or the prior year if you were on sabbatical or leave during May 2019 - April 2020), what is the total number of higher education courses you taught?

Please include courses taught in the summer. If the same course is taught more than once, count each time separately.

Number of graduate courses

Click or tap here to enter text.

(NUMBER 0-18)

Number of undergraduate courses

Click or tap here to enter text.

(NUMBER 0-18)

Number of courses with a mix of graduate and undergraduate students

Click or tap here to enter text.

(NUMBER 0-18)

<HARD CHECK: SUM OF A5 < 20 >

<SOFT CHECK, SUM OF A5 > 10: “please confirm that you have taught X classes last year”>

<page break>

A6 Design of courses

Have you ever developed, revised or selected materials for a higher education course? Include courses that you worked on in collaboration with someone else. Select one answer.

 Yes 1

 No 2

<page break>

A7 Design of methods courses [Display if A6= 1]

Have you had significant input into the development, revision, or selection of teaching materials for at least one research methods course in the last three years?

Select all that apply.

 No 0

 Yes, undergraduate methods course(s) 1

 Yes, graduate-level methods course(s) 2

<page break>

Section B: Familiarity with OPRE research and evaluations

B0 Section B intro

Now we would like you to tell us about your experience and knowledge of federal agencies, including the Office of Planning, Research, and Evaluation (OPRE).

B1 Sources of federal funding

Have you ever conducted research supported by federal funding from any of the following agencies?

Select all that apply.

 Never conducted research supported by federal funding 0

 OPRE at the Administration for Children and Families (ACF) 1

 Administration for Children and Families (ACF; offices other than OPRE) 2

 Department of Defense (including DARPA) 3

 Department of Housing and Urban Development (HUD) 4

 Department of Labor (DOL) 5

 Health Resources and Services Administration (HRSA) 6

 National Institutes of Health (NIH) 7

 National Science Foundation (NSF) 8

 Substance Abuse and Mental Health Services Administration (SAMHSA) 9

 U.S. Department of Agriculture (USDA) 10

 U.S. Department of Education (including IES) 11

 Other Federal Sources (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

 Don’t Know 98

<IF 0 IS SELECTED UNSELECT EVERYTHING ELSE: IF >1 IS SELECTED, UNSELECT 0>.

<page break>

B2.1 Familiarity with ACF/OPRE

How familiar are you with the mission, research activities, or materials of the following?

	Select one answer for each item
	Very familiar	Familiar	Somewhat familiar	Not familiar
a. Department of Health and Human Services (DHHS)	4 	3 	2 	1 
b. Administration for Children and Families (ACF) within DHHS	4 	3 	2 	1 
c. Office of Planning, Research, and Evaluation (OPRE) within ACF	4 	3 	2 	1 

<page break>

B2.3 Familiarity with OPRE [Display if B2.1.c > 2]

Regarding activities involving the OPRE, to the best of your knowledge, have you ever:

Select all that apply.

 Incorporated OPRE-funded research into your course materials (e.g., sending students to OPRE’s website, including an OPRE report as reading material, or used an OPRE data set in class) 1

 Read an OPRE-funded report or publication 2

 Cited an OPRE-funded report or publication 3

 Analyzed OPRE-funded data 4

 Worked on an OPRE-funded project (including as a consultant) 5

 Attended an OPRE-funded conference such as ACF’s National Research Conference on Early Childhood (NRCEC), (Hover text: formerly Head Start National Research Conference), the Child Care and Early Education Policy Research Consortium (CCEEPRC), (Hover text: formerly Child Care Policy Research Consortium) or the Research and Evaluation Conference on Self-Sufficiency (RECS) (Hover text: formerly Welfare Research and Evaluation Conference) 6

• None of the above 97

• Don’t know 98

<IF 97 IS SELECTED UNSELECT EVERYTHING ELSE: IF <97 IS SELECTED, UNSELECT 97>.

<page break>

B2.4 Using OPRE materials [Display if B2.1.c > 1 AND B.2.3.1 is not checked]

Please explain why you do not currently use OPRE-funded research in your course materials.

Click or tap here to enter text.

(STRING 2000)

<page break>

B3 Perceptions of OPRE-funded research

Please indicate the extent to which you agree with the following statements:

Usually, research produced by [IF B2.1.c = 1 FILL “federal agencies like”] OPRE:

	Select one answer for each item.
	Strongly agree	Agree	Disagree	Strongly disagree	Don’t know
a. Is rigorous	4 	3 	2 	1 	98 
b. Has practical relevance	4 	3 	2 	1 	98 
c. Advances theory	4 	3 	2 	1 	98 
d. Uses cutting edge research methods	4 	3 	2 	1 	98 
e. Is objective	4 	3 	2 	1 	98 
f. Is easy to understand	4 	3 	2 	1 	98 
g. Produces useful data sets for the field	4 	3 	2 	1 	98 

<RANDOMIZE RESPONSE ORDER>

<page break>

Section C: Selecting and developing teaching materials

C0 Section C intro

In the next section, we would like you to tell us about how you usually evaluate, select, and develop teaching materials for the courses you teach. By teaching materials, we mean all materials you use in your courses (e.g., slide decks, syllabi, readings, activities, assessments, etc.).

C3 Use of technology in teaching

Which of the following technological tools are important for supporting your teaching?

Please do not include temporary accommodations made in response to the COVID‑19 pandemic.

Select all that apply.

 Apps for mobile devices 1

 Polling tools (e.g., “clickers”) 2

 Online assessments to gauge student understanding 3

 Online video lectures 4

 Videos or audio clips 5

 Simulations/animations 6

 Online homework or virtual labs 7

 Online discussion boards 8

 Podcasts 9

• Websites for students to browse/search for materials 10

• Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

• None of the above 97

<page break>

C3.1 Use of apps [Display if C3.1 is selected]

Which apps are important for supporting your teaching?

Click or tap here to enter text.

APPS (STRING 500)

<page break>

C4 Criteria used to evaluate assignment features of potential teaching materials

How important are the following features when evaluating potential assignments?

	Select one answer for each item.
	Important	Somewhat important	Somewhat unimportant	Unimportant
a. Visually appealing graphics	4 	3 	2 	1 
b. Access to primary source materials (e.g., data, transcripts)	4 	3 	2 	1 
c. Opportunities for student participation	4 	3 	2 	1 
d. Ease of grading	4 	3 	2 	1 
e. Easy to understand language with limited jargon	4 	3 	2 	1 
f. Adaptable to different levels or types of students	4 	3 	2 	1 

<RANDOMIZE QUESTION ORDER>

<page break>

C5 Sources consulted to find new teaching materials [Display if A6 =1]

Have you used any of the following resources to develop or find teaching materials?

Select all that apply.

 Online search engines (for example, Google) 1

 Books, articles, or online resources about teaching 2

 A bibliography database (for example, ERIC, PsycINFO) 3

 Government websites that share empirical research 4

 Additional websites that share empirical research (for example, research firms and professional organizations) 5

 Conferences 6

 Online repositories for sharing teaching materials 7

 Open-access or open educational resources (OER) sites 8

 Social media groups who discuss teaching 9

 News media 10

• Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 500)

<page break>

C5.1 Specific sources consulted for developing and finding new teaching materials – websites [Display if C5 = 4 and/or 5]

What websites do you find most useful for developing and finding teaching materials?

Click or tap here to enter text.

(STRING 500)

C5.2 Specific sources consulted for developing and finding new teaching materials – online repositories [Display if C5 = 7]

What online repositories do you find most useful for developing and finding teaching materials?

Click or tap here to enter text.

(STRING 500)

C5.3 Specific sources consulted for developing and finding new teaching materials – open-access [Display if C5 = 8]

What open-access or open educational resources do you find most useful for developing and finding teaching materials?

Click or tap here to enter text.

(STRING 500)

C5.4 Specific sources consulted for developing and finding new teaching materials – social media [Display if C5 = 9]

What social media platforms do you find most useful for developing and finding teaching materials?

Click or tap here to enter text.

(STRING 500)

<page break>

C9 Needs or gaps in teaching materials

What are the most pressing needs or gaps in teaching materials for your courses?

Click or tap here to enter text.

(SRING 2000)

<page break>

Section D How and when materials would be used

D0 Section D introduction

In the next section, we would like you to tell us about your interest in different types of teaching materials.

D9 Interest in OPRE course materials

If OPRE developed or provided any of the following as course materials, which would you be interested in incorporating into a course that you teach?

Select all that apply.

 Presentation slides 1

 Infographics 2

 Case studies 3

 Discussion guides 4

 Assignments, projects, and activities applying information 5

 Sample questions that can be answered from analysis of OPRE datafiles 6

 Video or other multimedia sources 7

 Test or quiz questions 8

 A course syllabus 9

 A module covering a specific topic and providing readings, slides, and supporting discussion questions or assignments 10

 An entire course including suggested readings, slides, discussion questions, and assignments 11

 Data visualizations drawn from studies 12

• Briefs 13

 Executive summaries of reports 14

 Full-length reports produced by or for government agencies 15

 Peer-reviewed journal articles 16

 Case studies 17

• Data sets produced by or for a government agency 18

• Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

 I am not interested in using OPRE teaching materials 97

[Programming Note: If 97 is selected, all other items in D9 should be unselected]

<page break>

D12 Interest in OPRE substantive topics

If OPRE developed course materials on the following topics, which would be relevant to courses that you teach?

Select all that apply.

 Abuse, neglect, adoption, and/or foster care 1

 Early care and education 2

 Home visiting 3

 Youth services 4

 Self-sufficiency 5

 Welfare 6

 Employment and the labor market 7

• Preventing family violence 8

 Healthy marriage and relationship education 9

 Responsible fatherhood 10

 Behavioral science 11

 These substantive topics are not relevant to my courses 97

[Programming Note: If 97 is selected, all other items in D12 should be unselected]

[Programming Note: INCLUDE HOVER TEXT DEFINITIONS AS FOLLOWS:

Abuse, neglect, adoption, and/or foster care = OPRE partners with ACF’s Children’s Bureau to conduct research on children who are maltreated or who are at risk for child maltreatment and children and families who come to the attention of child protective services such as children in foster care or who exit foster care through adoption. This research covers a broad array of topics, including identification of antecedents and consequences of child maltreatment, strategies for its prevention, and service needs and outcomes for children who experience it, and building evidence for child welfare interventions.

Early care and education = OPRE partners with ACF program offices to study early care and education including child-care, Early Head Start, and Head Start to inform program improvements and support early childhood development. Research in this area focuses on early learning education standards, improving quality in care settings, innovative interventions, and supporting parental employment through access to high quality care.

Home visiting = OPRE in collaboration with the Health Resources and Services Administration is managing a number of evaluation activities related to the Maternal, Infant, and Early Childhood Home Visiting program. This research includes topics such as understanding the implementation of services, analysis of programmatic costs, evaluating the impacts associated with home visiting models, and needs assessments of states and territories.

Youth services = OPRE in collaboration with ACF partner offices examines approaches for supporting positive youth development and transitions to adulthood. Efforts include evaluations related to teen pregnancy prevention and sexual risk avoidance strategies, teen romantic relationships, transitions from foster care, runaway and homeless youth, and employment and self-sufficiency for at-risk youth.

Self-sufficiency = Designed and implemented in collaboration with other HHS and federal partners, OPRE’s portfolio addresses innovative approaches for increasing economic self-sufficiency and reducing public assistance dependency, including sponsoring rigorous research on the effectiveness of education and training strategies for improving employment and earnings for TANF recipients and other low-income individuals.

Welfare = Designed and implemented in collaboration with other HHS and federal partners, OPRE’s research examines the local implementation of TANF pro-grams and policies, the intersection of TANF with other safety net programs, and other relevant issues related to TANF program operations, TANF recipients, and TANF-eligible families.

Employment and the labor market= Designed and implemented in collaboration with other HHS and federal partners, OPRE’s research focuses on strategies for helping TANF recipients and other low-income individuals find jobs, maintain employment, and advance in the labor market.

Preventing family violence = In collaboration with ACF partner offices, OPRE’s portfolio addresses identifying, preventing, and addressing the consequences of family violence.

Healthy marriage and relationship education = In partnership with the Office of Family Assistance, OPRE studies services to support and strengthen romantic relationships and co-parenting.

Responsible fatherhood = In partnership with the Office of Family Assistance, OPRE studies programs designed to help fathers overcome barriers to effective and nurturing parenting, support their family formation and healthy relationships, and improve economic outcomes for themselves and their families.

Behavioral science = OPRE is building a portfolio of projects focused on learning and applying insights from social and behavioral sciences.]

<page break>

D13. Breadth and Depth of OPRE assignment materials

If OPRE developed any of the following course materials, which would you be interested in incorporating into a course that you teach?

I would be interested in using materials that…

	Select one answer for each item.
	Interested	Somewhat interested	Somewhat uninterested	Uninterested
a. Are interdisciplinary, providing different perspectives on a research topic	4 	3 	2 	1 
b. Are specialized, illustrating the research of a specific discipline	4 	3 	2 	1 
c. Are brief, providing a high-level summary of a program or topic	4 	3 	2 	1 
d. Are in-depth, providing a deep dive into a program or topic	4 	3 	2 	1 
e. Are focused, discussing only a single study, project, or policy	4 	3 	2 	1 
f. Are integrative, synthesizing findings from many studies	4 	3 	2 	1 

<page break>

D0.1 Course-specific introduction

For the following questions, we would like you to think of a specific [IF A7> 0 FILL "research methods"] course. Please pick a course for which OPRE materials could be useful.

D1 Course name

Write the name of this course in the space below.

Click or tap here to enter text.

(STRING 500)

(Programming Note: If a person attempts to skip this item, have a soft check and message: Please enter the name of a course that you recently taught.)

<page break>

D2 Course type (course-specific)

This course is a(n):

Select one answer.

 Undergraduate lecture 1

 Undergraduate seminar 2

 Graduate lecture 3

 Graduate seminar 4

 Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

(Programming Note: Please add hover text to define lecture as “by ‘lecture’ we mean instructor-directed, typically for a large group” and define seminar as “by ‘seminar’ we mean a discussion-based course, typically for a small group”.)

<page break>

D7 Typical course assignments

What kinds of assignments do you typically use in [FILL D1]?

Select all that apply.

 Multiple choice tests or exams 1

 Essays 2

 Term/research papers 3

 Response/reaction papers 4

 Discussion board/blog posts 5

 Infographics or other visual assignments 6

 Problem sets, observation reports, or lab exercises/reports 7

 Student presentations 8

 Multimedia projects 9

 Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 250)

<page break>

D8 Interest in specific kinds of content (course-specific)

How interested are you in using the following kinds of content in [FILL D1/IF D1 IS BLANK, FILL “the course”]?

	Select one answer for each item.
	Strong interest	Moderate interest	Little interest	No interest
a. Program evaluations	4 	3 	2 	1 
b. How research informs policies or federal regulations	4 	3 	2 	1 
c. How research informs programs and practice (e.g., program performance standards)	4 	3 	2 	1 
d. Studies that elevate diverse perspectives and identities	4 	3 	2 	1 
e. Literature reviews or systematic reviews that summarize a field or topic	4 	3 	2 	1 
f. Studies that apply research in “real world” settings	4 	3 	2 	1 
g. Studies that illustrate the application of a specific research method	4 	3 	2 	1 
h. Implementation research	4 	3 	2 	1 
i. Quantitative studies	4 	3 	2 	1 
j. Qualitative studies	4 	3 	2 	1 
k. Mixed method studies	4 	3 	2 	1 

<page break>

Section E: Demographics

E0 Section Introduction

Now we have a few questions so we can better understand our sample. This information will help us describe, in a general way, the instructors who complete this survey.

Taken from U.S. Department of Health and Human Services Implementation Guidance on Data Collection Standards for Race, Ethnicity, Sex, Primary Language and Disability Status.

E2a Ethnicity

Are you Hispanic, Latino/a, or Spanish Origin?

Select one answer.

 No, not of Hispanic, Latino/a, or Spanish origin 1

 Yes, of Hispanic, Latino/a or Spanish origin 2

 Decline to answer 97

<page break>

Taken from U.S. Department of Health and Human Services Implementation Guidance on Data Collection Standards for Race, Ethnicity, Sex, Primary Language and Disability Status.

E2b Race

What is your race?

Select all that apply.

 American Indian or Native Alaskan 1

 Asian or Asian American 2

 Native Hawaiian or Pacific Islander 3

 White or Caucasian 4

 Black or African American 5

 Other (SPECIFY) 99

Specify

Click or tap here to enter text.

(STRING 500)

 Decline to answer 97

<page break>

Source: Adapted from the Williams Institute’s Best Practices for Asking Questions to Identify Transgender and Other Gender Minority Respondents on Population-Based Surveys. This is a one-step item recommended by the FIWG on Improving Measurement of SOGI in Federal Surveys. individuals,

E3 Gender identity

How do you describe yourself?

Select one answer.

 Male 1

 Female 2

 Transgender 3

 Do not identify as female, male, or transgender 4

 Decline to answer 97

<page break>

Section F: Survey End

F1 Closing Comments

Before you submit your responses, is there anything else you want to tell us?

Click or tap here to enter text.

F2.1 Closing [display if A1.0 = 1]

Thank you for completing this survey. Your responses have been recorded.

The Office of Planning, Research, and Evaluation (OPRE) makes <a href = https://www.acf.hhs.gov/opre/research> research </a> available to the general public. OPRE also regularly <a href = https://www.acf.hhs.gov/opre/research/topic/overview/archived-data> archives research and evaluation data </a> for secondary analysis by researchers.

Based on your field, you may be interested in these examples of OPRE research and evaluations:

[Insert urls to relevant OPRE materials]

The results from the survey you completed will be used to set priorities for developing higher education teaching materials that will be made freely available. These materials are part of a larger effort to increase engagement with the academic community and bridge the gap between the activities and findings of academic and federal research.

To stay up-to-date on OPRE research and materials:

• Sign up to get updates here: https://www.acf.hhs.gov/opre/opre-newsletter

• Follow @OPRE_ACF on Twitter.

• Like us on Facebook https://www.facebook.com/OPRE.ACF

• Follow us on Instagram https://www.instagram.com/opre_acf/

F2.2 Closing [display if A1.0 = 0]

Thank you for your interest. This survey is specifically for higher education instructors of record.

The results from the survey you completed will be used to set priorities for developing higher education teaching materials that will be made freely available. These materials are part of a larger effort to increase engagement with the academic community and bridge the gap between the activities and findings of academic and federal research.

To stay up-to-date on OPRE research and materials:

• Sign up to get updates here: https://www.acf.hhs.gov/opre/opre-newsletter

• Follow @OPRE_ACF on Twitter.

• Like us on Facebook https://www.facebook.com/OPRE.ACF

• Follow us on Instagram https://www.instagram.com/opre_acf/