Supporting Statement A - Survey of Higher Education Instructors to Support the Development of Teaching Materials Based on OPRE’s

Survey of Higher Education Instructors to Support the Development of Teaching Materials Based on OPRE’s Research and Evaluation

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part A

JUNE 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:
Alysia Blandon

Shirley Adelstein

Wendy DeCourcey

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request: This is a new information collection request for a web survey of higher education instructors to inform the development of teaching materials based on the Administration for Children and Families Office of Planning, Research, and Evaluation’s (OPRE’s) portfolio of research and evaluation. Data collected in this survey are not intended to be generalized to a broader population. Rather, survey findings from a broad range of instructors will help the project team set priorities for creating teaching materials in the near term. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: Because this is a survey for higher education instructors, our timeline is linked to an academic schedule. To get the minimum necessary number of responses, we will need to administer the survey during the academic year, when faculty are most readily available. Taking our project timeline into consideration, we have identified the fall 2020 semester (August – November) as our target survey administration period.

A1. Necessity for Collection

This data collection is needed to provide information from higher education instructors to the Office of Planning, Research and Evaluation (OPRE) to support the creation and dissemination of postsecondary teaching materials based on OPRE’s research and evaluation. There are no legal or administrative requirements that necessitate this collection. The Administration for Children and Families (ACF) is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0351):

• Planning for the provision of research and evaluation-related training or technical assistance (T/TA).

The purpose of the information collection is to gather data to: (1) contribute to the Office of Planning, Research and Evaluation’s (OPRE) understanding of the academic audience for teaching materials based on OPRE’s research and evaluation; (2) to describe the instructional needs of this audience; and (3) to inform how to meet the instructional needs of this audience. The survey will collect information about whether instructors currently use OPRE materials in their teaching, how they evaluate teaching materials, who is most likely to use OPRE teaching materials, and what types of OPRE teaching materials they would be likely to use.

The study team will use this information in planning and setting priorities for creating the teaching materials. For example, survey results will help the team identify topic areas for which there is substantial demand for materials among respondents. OPRE will also use the survey to guide the development of a dissemination plan for the teaching materials. Survey data will not be used by anyone outside of OPRE or the project team¹, and it will not be used for any purpose other than supporting planning for the development of teaching materials based on OPRE’s research and evaluation.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

The instructor survey aims to inform the following research questions:

1. How familiar are instructors with OPRE’s research and evaluation work?

2. What resources do instructors use when developing teaching materials?

   1. What are instructors’ motivations for developing teaching materials?

   2. What process do instructors use for developing teaching materials?

   3. What is the demand for different types of teaching materials?

3. How do instructors assess if teaching materials meet their instructional goals?

4. Who is most likely to use OPRE teaching materials?

5. For what courses and in what ways would instructors want to use OPRE teaching materials?

Study Design

We will field a one-time web survey with at least 180 higher education instructors concentrated in 9 targeted disciplines (social work, public policy, psychology, economics, education, human development, sociology, public health, and research/evaluation). We will collect survey responses in a single data collection during the fall 2020 academic semester (August to November 2020). We will recruit respondents in two ways. First, we will share an open survey link with instructors using broad messages on online communication channels, including electronic mailing lists, online forums (e.g., relevant Facebook groups), and social media (e.g., Twitter). We will then supplement these responses with direct emails to instructors to ensure that we have a diverse group of respondents from a variety of disciplines, institutions, and career stages. See Supporting Statement B, section B2 for additional information.

This approach is intended to uncover a diverse set of perspectives to draw upon when creating teaching materials, particularly for faculty interested in using OPRE materials in their classes. Participating instructors will not be representative of our target population, and results are not intended to be generalized to this population. Indeed, given the diversity of potential users of OPRE teaching materials, it would not be feasible to identify and recruit a representative sample of the intended audience, nor is it necessary to achieve the project’s goals. However, collecting survey responses from a diverse, though not representative, group of instructors will help us determine where interest is strong and features that are of interest to academic instructors.

Data Collection Activity	Instrument	Respondent, Content, Purpose of Collection	Mode and Duration
Web survey	Develop Teaching Materials: Faculty and Instructor Survey (see Instrument 1)	Respondents: Higher education instructors in 9 targeted disciplines to which OPRE research is most relevant Content: Familiarity with OPRE research and evaluations Motivation for selecting or developing teaching materials Process of selecting or developing teaching materials and resources used Demand for different teaching materials How instructors assess if teaching materials meet instructional goals Characteristics of course(s) taught Potential uses for OPRE teaching materials Institutional characteristics Individual characteristics Respondent demographics Purpose: To inform planning for the development of teaching materials based on OPRE’s research and evaluation. Specifically, information about instructors’ needs and preferences will help us craft an approach that results in materials we are confident will be relevant and useful to instructors (e.g., if we see high numbers of instructors indicating they would like materials tailored to the online learning environment, we may prioritize the development of materials suited to distance learning).	Mode: Online Duration: 15 minutes (average)

Other Data Sources and Uses of Information

No other data sources will be used.

A3. Use of Information Technology to Reduce Burden

We will recruit participants and conduct the survey entirely online. This approach maximizes our reach, prioritizes participant convenience, and minimizes burden for both participants and the research team.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

We are unaware of any existing data that would meet the goals of this information collection. The purpose of this information collection is quite specific—to inform planning for the development of teaching materials based on OPRE’s research and evaluation.

A5. Impact on Small Businesses

This information collection does not affect small businesses.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. A subsequent notice, updated with more specific information, was published on June 18, 2019, Volume 84, Number 117, page 28307, and provided a thirty-day period for public comment. During the notice and comment periods, no substantive comments were received.

Consultation with Experts Outside of the Study

We consulted with nine higher education instructors across several disciplines in one-on-one calls. We discussed priorities for the project and the design of the survey for the purposes of refining the instrument and outreach approach. Consultations were held with the following instructors:

• Marah Curtis, Professor, University of Wisconsin – Madison, School of Social Work

• Daniel Eisenberg, Professor, University of Michigan, School of Public Health

• Yoonsook Ha, Assistant Professor, Boston University, School of Social Work

• Carolyn Heinrich, Professor, Vanderbilt University, Peabody College of Education and Human Development

• Heather Hill, Associate Professor, University of Washington, Evans School of Public Policy

• Sacha Klein, Associate Professor, Michigan State University, School of Social Work

• Eric Lindsey, Professor, Pennsylvania State University

• Henry May, Associate Professor, University of Delaware, School of Education

• Cynthia Osborne, Associate Dean for Academic Strategies, University of Texas at Austin, Lyndon B. Johnson School of Public Affairs

A9. Tokens of Appreciation

We will not offer any tokens of appreciation.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

While none of the survey questions solicit personally identifiable data, any participant’s responses, when considered in combination, may be identifiable. For example, we will ask instructors to identify their institution and provide the name of a course they teach for which OPRE teaching materials would be most relevant. Taken together, this information could make an individual identifiable. However, it should be noted that the survey does not collect any sensitive information (see A11) and information (aggregate or individual) will not be shared beyond the study team and OPRE.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, contractor and subcontractor will comply with all Federal and Departmental regulations for private information. The contractor has secured expedited IRB approval through Health Media Labs (See Appendix C).

Data Security and Monitoring

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security Plan.. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract are trained on data privacy issues and to comply with the above requirements.

As specified in the evaluator’s contract and detailed in the approved data security plan, the Contractor uses Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor uses encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor incorporates these standards into their property management/control system including procedures to account for all devices and portable media that store or process sensitive information. Any data stored electronically is secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. All data files are stored electronically only. Paper records or raw data will not be produced or maintained.  

A11. Sensitive Information ²

We are collecting no sensitive information.

A12. Burden

Explanation of Burden Estimates

We will recruit a minimum of 180 higher education instructors to complete the survey. Because we will use an open recruitment strategy (see B2, Recruitment), we estimate burden with a projected maximum of 350 respondents.

The one-time survey will take place between August 2020 and November 2020 and will take no more than 15 minutes to complete, on average. We have pilot tested and user tested the instrument to achieve a 15-minute average completion time. Testing was completed with 8 individuals.

Estimated Annualized Cost to Respondents

The average respondent hourly wage has been calculated using salary averages reported in results of the 2018-2019 Faculty Compensation Survey conducted by the American Association of University Professors. The average salary for full-time instructors, inclusive of all ranks and across institution types, is $100,800.³ While using this average lacks some precision, we believe it captures the range in salary differences among our potential respondents, who will come from varied postsecondary settings, disciplines, and career ranks. To determine an hourly wage, we assumed that a full-time appointment extends through the academic year (9 months) and entails 1,560 hours of work per year, dividing the average salary by this figure. We estimate the average respondent’s hourly wage to be $64.62 (see below).

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Develop Teaching Materials: Faculty and Instructor Survey	350	1	.25	88	$64.62	$5,686.56
Total	350			88		$5,686.56

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total/annual cost for the data collection activities under this current request will be $152,944.

Activity	Detail	Estimated Cost
Survey Design Instrument Development Pilot and User Testing	FTE time Operational expenses (such as equipment, overhead, printing, and staff support), A Other expenses which would not have been incurred without this collection of information	$97,904
Survey administration	FTE time Operational expenses (such as equipment, overhead, printing, and staff support), A Other expenses which would not have been incurred without this collection of information	$28,008
Analysis and initial dissemination	FTE time Operational expenses (such as equipment, overhead, printing, and staff support), A Other expenses which would not have been incurred without this collection of information	$27,032
Total/Annual costs		$152,944

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

The following table articulates the timeline for the information collection:

Activity	Targeted Start*	Anticipated Duration
OMB Approval of Generic Clearance package	Prior to recruitment	N/A
Recruitment	August 2020	4 months
Conduct Survey	August 2020	4 months
Data Analysis	November 2020	2months
Preparation of Internal Report	December 2020	2 months
Internal Findings Report Presented to OPRE	January 2021	N/A

*All dates are dependent on the date OMB approval is received.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument 1: Develop Teaching Materials: Faculty and Instructor Survey

Appendix A: Survey Outreach and Invitation Language

Appendix B: Examples of Open Recruitment Sources and Online Communication Channels

Appendix C: IRB Approval

1 OPRE has contracted with MEF and Mathematica to complete the work for this project.

2 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

3 Survey Report Table 1, p. 15. American Association of University Professors. (May 2020). The annual report on the economic status of the profession, 2017-2018. Available at: https://www.aaup.org/sites/default/files/2019-20_ARES.pdf.