ExCELS_Descriptive Study_SSA_NSC_Clean 05192022

Early Care and Education Leadership Study (ExCELS) Descriptive Study

OMB Information Collection Request

New Collection

Supporting Statement

Part A

AUGUST 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer: Nina Philipsen, Ph.D.

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a new data collection. We are requesting two years of approval.

• Description of Request:

The Office of Planning, Research, and Evaluation within the Administration for Children and Families in the U.S. Department of Health and Human Services contracted with Mathematica and its subcontractor, the Institute for Early Education Leadership and Innovation at the University of Massachusetts Boston, to conduct the Early Care and Education Leadership Study (ExCELS). The purpose of ExCELS is to learn about leadership in center-based early care and education settings, and better understand how leadership might improve the quality of care and education centers provide and outcomes for staff, children, and families. We are seeking approval to collect data for the ExCELS descriptive study which will take place in spring 2022. We will recruit 120 center-based child care settings that receive funding from Head Start or the Child Care and Development Fund and serve children whose ages range from birth to age 5 (but who are not yet in kindergarten) to participate in the data collection activities. Data collection will include interviews with each center’s primary site leader and surveys for select center managers and all teaching staff. We do not intend for this information to be used as the principal basis for public policy decisions.

A.1. Necessity for collection

Leadership is widely recognized as an essential driver of organizational performance and improvement, but little is known about its role in driving the quality of care and education centers provide and outcomes for staff, children, and families. In launching the Early Care and Education Leadership Study (ExCELS), the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) seeks to fill the definitional and measurement gaps to help the early childhood field understand how effective leadership can improve quality in early care and education (ECE). The descriptive study will test hypothesized associations between leadership constructs and outcomes in the study’s theory of change (Appendix A) and will support the production of a final short-form measure of ECE leadership. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

While theories, models of change, and research findings from business and other disciplines, including K–12 education, identify links between strong leaders and better outcomes¹, how these results apply to early childhood care and education (ECE) agencies, centers, and the systems that support them is the subject of current debate.^2,3 There are currently limited measures available that (1) capture who ECE leaders are or the leadership structure within a center, and (2) are able to distinguish what different center staff (such as center managers and teachers) do as leaders.⁴ Furthermore, little is known about how ECE leadership may be effective in promoting quality and providing positive experiences for children and families that can lead to positive outcomes.⁵ In October 2017, OPRE hosted a meeting of experts from a range of disciplines to review existing definitions and approaches to measuring leadership in ECE. Although the experts made progress toward identifying definitions and potential measures, gaps remain and new questions emerged about the key features, strategies, and practices of effective leadership. The information collected from ExCELS will fill this research gap.

A.2. Purpose

1. Purpose and use

The purpose of the ExCELS descriptive study is to support research efforts by developing a new measure of ECE leadership that has strong psychometric properties and examine the associations among key constructs and outcomes on how ECE leadership could support quality improvement.

The results of the study could be used by researchers, program administrators, and technical assistance providers to advance their understanding of ECE leadership in center-based settings and to identify ways to measure leadership and how it might produce positive outcomes for staff. The de-identified data will be made available as part of a restricted-use data file at the Child and Family Data Archive for secondary analysis by qualified researchers. ACF published reports can enable centers or technical assistance providers to understand what leadership looks like within a center and see where there might be opportunities for growth and improvement. Reports from this study can also help guide federal program administrators, researchers, and technical assistance providers as they pursue next steps in policy, research, and practice to support ECE leadership and its measurement for program accountability and improvement purposes. The study will also attempt to answer a new research question for the ECE field on the minimum number of teaching staff needed to obtain a reliable center-level estimate of leadership based on teaching staff reports. The study information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

2. Research questions or tests

The study team will interview a center’s primary site leader (the person in the building who is responsible for oversight of all that happens in the center on a daily basis) and survey select center managers and all teaching staff. The information collected will help address the following four research questions:

1. How can we develop a technically sound summary of leadership at the center-level across three elements—who leaders are (whether formal or informal), what those leaders bring, and what those leaders do—while reflecting the perspectives of managers and teaching staff?

2. What does leadership look like in center-based ECE settings across the three leadership elements? How does it vary by staff and center characteristics?

3. What is the staffing structure of formal leadership roles? How does this formal structure relate with the three leadership elements?

4. Is there empirical support for the associations among the leadership elements and a center’s culture, climate, and communication and the outcomes depicted in the ExCELS theory of change?

See the study design section (A3) and tokens of appreciation section (A9) below, and Supporting Statement Part B for more details.

3. Study design

The ExCELS descriptive study will include a purposive sample of 120 ECE centers from four states, selected to achieve variation that is critical for assessing the conceptual modeling and for psychometric analyses. States will be selected to achieve variation on administrator qualifications; quality rating and improvement system (QRIS) participation levels; stringency of licensing requirements to reflect the regulatory environment; funding to support access and quality; and geographic diversity. Centers will be selected to achieve variability in funding sources and center size. The study will include ECE centers serving children whose ages range from birth to age 5 (but who are not yet in kindergarten) who are supported by Head Start grants and/or Child Care and Development Fund (CCDF) subsidies.

OPRE contracted with Mathematica and its subcontractor, the Institute for Early Education Leadership and Innovation at the University of Massachusetts Boston, to conduct this study. The study team will begin recruiting centers by sending hard copy letters and emails to all centers on their sample list and then follow up with more targeted letters and emails to schedule a phone conversation with the primary site leader. Once the primary site leader is reached by phone, liaisons will conduct the recruitment call (Instrument 1) with them requesting their center’s participation in the study. Some centers may need approval from their program office or their umbrella organization to participate, in which case, the study team will contact the program office or umbrella organization and obtain approval to recruit the center (Instrument 2). Recruitment follow-up will be conducted strategically to target centers to maximize center variation on the characteristics described above. The study team will repeat the center recruitment process until they have 120 centers for data collection. See B.4 for more information on the recruitment protocol and Appendix B for the center recruitment materials.

The study team will conduct the following activities with the recruited centers:⁶

• An engagement interview with the primary site leader to collect center characteristics and confirm center eligibility for the study (Instrument 3).

• Remote data collection, where the study team conducts a staffing structure and leadership positions (SSLP) interview with the primary site leader to understand the staffing structure and formal leadership positions at the center (Instrument 4), and obtains a list of teaching staff that will be invited to complete a survey (Instrument 5).

• Surveys of the center manager and teaching staff to understand the three leadership elements (who leaders are, what leaders bring, and what leaders do), the center’s culture, climate, and communication (such as culture of respect, shared growth, and learning; collaboration among staff), and outcomes for staff (job satisfaction, staff well-being and mental health, and staff retention) (Instrument 6 and Instrument 7, respectively).

Table A.1 includes each of the data collection activities, respondents, mode, duration, and purpose.

Table A.1. Data collection activities for the ExCELS descriptive study

Data collection activity	Respondent(s)	Mode	Duration	Purpose
Center recruitment call (Instrument 1)	Primary site leader	Telephone	20 minutes	Discuss the study and recruit center to participate
Umbrella organization recruitment approval call script (Instrument 2)	Program or umbrella organization administrator	Telephone	20 minutes	Discuss the study and obtain approval to recruit center to participate
Center engagement interview (Instrument 3)	Primary site leader	Telephone	20 minutes	Collect center characteristic information
SSLP interview (Instrument 4)	Primary site leader	Telephone with CADE on the web	30 minutes	Collect information on the number and roles of all center staff, and collect a list of potential respondents for the center manager survey
Teaching staff roster (Instrument 5)	Primary site leader	CADE on the web	15 minutes	Collect a list of potential respondents for the teaching staff survey
Center manager survey (Instrument 6)	Select center managers	Web with paper option	25 minutes	Gather information on leadership elements and outcomes that ECE leadership can potentially influence
Teaching staff survey (Instrument 7)	Teaching staff	Web with paper option	60 minutes	Gather information on leadership elements and outcomes that ECE leadership can potentially influence and collect information on the center culture, climate, and communication

CADE = computer-assisted data entry.

To ensure high response rates the study team will employ several engagement tactics including: offering the center manager survey and teaching staff survey online or on paper, offering tokens of appreciation for participating, and sending paper and email reminders to survey non-responders. If the teaching staff survey response rate at a given center remains below 70% after all the planned follow-up is conducted, the study team will offer to conduct a site visit with the center to support survey completion. See A.9 for more information on the tokens of appreciation amounts and Supporting Statement Part B for additional information on the survey respondent materials.

4. Other data sources and uses of information

This data collection is one component of the information that the study will use. The study team intends to access publicly available data to support state selection including administrator qualifications, QRIS participation, licensing requirements, and funding supports. Within the selected states, the study team will obtain lists of CCDF centers from state administrators or from public access sites, and lists of Head Start centers from the Head Start Enterprise System or Early Childhood Learning & Knowledge Center website. To support site selection, the study team will access publicly available information about funding, center size, and center type. No burden will be required to access this data, but it will allow for the selection of sites with the variation in center characteristics needed to conduct analysis.

A.3. Use of information technology to reduce burden

Center managers and teaching staff can complete surveys on the web or on paper. This will accommodate respondent preferences and the varying schedules of center staff and reduce respondent burden. If respondents choose to complete the surveys on the web, they will be able to complete the survey over more than one sitting. The web survey will save the respondents’ progress so they can return to finish the survey at a later time.

A.4. Use of existing data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

None of the study instruments will ask for information that can be reliably obtained from alternative data sources. Furthermore, the design of the study instruments ensures no duplicate data are collected through each instrument. Other OPRE data collection efforts, such as the Head Start Family and Child Experiences Survey (OMB 0970-0151) or the National Survey of Early Care and Education (NSECE) (OMB 0970-0391), do not collect surveys from the entire teaching staff (lead and assistant teachers) needed for ExCELS to support the psychometric analyses.

A.5. Impact on small businesses

The study team will recruit some small organizations, including community-based organizations and other nonprofits, to participate. To minimize the burden on these centers, the study team will carefully schedule telephone interviews with the primary site leader at times that are most convenient for them and when it will not interfere with the care of children. Center managers and teaching staff will be able to complete the surveys (on the web or on paper) when it is convenient for them.

A.6. Consequences of less frequent collection

This is a one-time data collection.

A.7. Now subsumed under 2(b) above and 10 below

A.8. Consultation

1. Federal register notice and comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 7, 2021, Volume 86, Number 87, pages 24626-24627, and provided a 60-day period for public comment. During the notice and comment period, no comments were received.

2. Consultation with experts outside of the study

The study team consulted with experts outside of the study to complement their knowledge and experience (Table A.2). Experts included researchers with knowledge in ECE measures development, study design, and psychometric analysis.

Table A.2. ExCELS descriptive study expert consultation

Name	Affiliation
Sally Atkins-Burnett	Mathematica
Abby Copeman Petig	Center for the Study of Child Care Employment
Stacy Ehrlich	NORC at the University of Chicago
Robert Goerge	Chapin Hall, University of Chicago
Lynn Karoly	RAND Corporation
Teri Talan	National Louis University, McCormick Center for Early Childhood Leadership

A.9. Tokens of appreciation

To support a successful data collection with high response rates, the study team will offer a $25 gift card to respondents of the 25-minute center manager survey. Because the teaching staff survey is longer, the study team proposes offering those respondents a $40 gift card for completing the 60-minute survey. The study team will distribute the $40 to teaching staff in the form of a $5 physical gift card as part of the respondent invitation packets, and a $35 electronic gift card after completing the survey.

High response rates are key to ensuring the quality of the information gathered from each center and mitigating nonresponse bias. Tokens of appreciation are one piece of a broader plan to address response rates and nonresponse bias; see Supporting Statement Part B for more details on the techniques the study team will use to increase response rates and mitigate nonresponse bias.

Prior research has shown that offering tokens of appreciation help improve survey response rates regardless of mode and mitigate nonresponse bias across different respondent populations including low-income and minority respondents.⁷ For the Project LAUNCH Cross-Site Evaluation (OMB number 0970-0373, expired October 31, 2019), the study initially did not offer a token of appreciation to parents who completed a web-based survey and then OMB approved a $25 post-pay token of appreciation. It found that early respondents (pre-tokens of appreciation) were not representative of their communities. Minorities, individuals with lower incomes and education levels, and those who worked part-time or were unemployed were underrepresented. Completion rates and representativeness both improved following the added token of appreciations.⁸ Further research has shown that offering a respondent a small token of appreciation with survey invitation materials (pre) and an additional token of appreciation after completing the survey (post) can be more effective in improving response rates compared to only offering a token of appreciation after completing the survey.^9,10 A recent study with the RAND American Teacher Panel found that pre-tokens of appreciation improved response rates compared to post-tokens of appreciation among K-12 public school teachers.¹¹

A.10. Privacy: Procedures to protect privacy of information while maximizing data sharing

1. Personally identifiable information

To distribute the center manager survey and teaching staff survey, the study team will collect names, titles, and email addresses of center staff. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

2. Assurances of privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, that their information will be kept private to the extent permitted by law, and that the information the study collects will be used only for research purposes and in ways that will not reveal who they are or identify the center or its staff. As specified in the contract, the contractor will comply with all federal and departmental regulations for private information.

Due to the sensitive nature of this research (see A.11 for more information), the evaluation will gain approval from an institutional review board and obtain a Certificate of Confidentiality. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

3. Data security and monitoring

The study team has developed a data safety and monitoring plan that assesses all protections of respondents’ personally identifiable information. Mathematica will ensure that all of its employees and consultants who perform work under this contract are trained on data privacy issues and comply with the above requirements. Upon hire, every Mathematica employee signs a Confidentiality Pledge stating that any identifying facts or information about individuals, businesses, organizations, and families participating in projects conducted by Mathematica are private and are not for release unless authorized.

As specified in OPRE’s contract, Mathematica will use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. Mathematica will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. Mathematica will (1) ensure that this standard is incorporated into the company’s property management and control system; and (2) establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, Mathematica must submit a plan for minimizing, to the extent possible, the inclusion of sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive or personally identifiable information to ensure secure storage and limits on access.

The study team will create a restricted use data set based on this data collection. Disclosure analyses will be done prior to releasing the data file, and masking of data will occur to ensure privacy of respondents. The data will be archived at the Child and Family Data Archive for future research and analyses by qualified researchers.

A.11. Sensitive information

To understand what leadership looks like in center-based ECE settings, the study team will ask teaching staff to answer questions about themselves and the actions of their center management. A few sensitive questions will be asked of all participants, such as questions on job stress, to understand how leadership practices may influence staff outcomes as a proximal outcome toward quality improvement. Survey invitations will be sent to center managers and teaching staff to participate and will inform them that the survey will ask these questions, that they do not have to answer any questions that make them uncomfortable, and that the responses they provide will not be reported to the center leader or other center staff.

A.12. Burden

1. Explanation of burden estimates

Table A.3 presents the current request for data collection activities. The estimates include time for respondents to review instructions, search data sources, complete and review the responses, and transmit or disclose information. This information collection request covers a period of two years. The study team expects the total annual burden to be 1,092 hours for all of the instruments in this information collection request. Figures are estimated as follows:

1. Center recruitment call script (Instrument 1). The study team expects to conduct recruitment calls with 500 centers in order to recruit 150 centers into the study and secure the participation throughout data collection of the 120 centers necessary for this study. They anticipate each recruitment call (with the primary site leader for each center) to take about 20 minutes.

2. Umbrella organization recruitment approval call script (Instrument 2). The study team anticipates that 75 percent of centers that agree to participate (113 of 150 centers that initially agree), will ask the team to speak with a program office, or an administrator of a larger umbrella organization which the center is affiliated with to fully obtain agreement for the center’s participation in the study. This discussion will be similar to the center recruitment call and will take about 20 minutes to complete, on average.

3. Center engagement interview (Instrument 3). The study team expects to conduct engagement interviews with about 215 centers in order to recruit 150 that agree to participate in the study and to have eligibility confirmed as part of the engagement interview. Each interview is estimated to take about 20 minutes to complete (with the primary site leader for each center).

4. SSLP interview (Instrument 4). The study team anticipates that they will need to conduct the 30-minute staffing structure and leadership positions interview with the primary site leader for 135 centers eligible for the study in order to complete data collection activities with 120 centers for use in the analysis. This assumes some attrition of centers after the SSLP interview, yet prior to release of the surveys.

5. Teaching staff rosters (Instrument 5). The study team will work with the primary site leader to obtain a roster with contact information for all the teaching staff targeted for the teaching staff survey. The team expects it will take about 15 minutes for the primary site leader to obtain this information. The team plans to gather the rosters from the primary site leader at 135 centers eligible for the study in order to complete data collection activities with 120 centers for use in the analysis. This assumes some attrition of centers after obtaining the teaching staff roster, yet prior to release of the surveys.

6. Center manager survey (Instrument 6). The study team expects to administer surveys to two center managers in half the centers and three center managers in the other half of centers, based on center size, for a total of 300 center managers. The team expects the center manager survey to take 25 minutes to complete the new leadership measure and outcome and background questions.

7. Teaching staff survey (Instrument 7). The study team will target the teaching staff survey to all teaching staff at each of the 120 centers (with an average of 14 teaching staff per center according to the NSECE, for a total of 1,680 teaching staff). The team expects the survey to take 60 minutes to complete including 40 minutes of content related to the new leadership measure and outcome and background questions, and 20 minutes of content added from a validated existing measure on center’s culture, climate, and communication (such as culture of respect, shared growth, and learning; collaboration among staff), .

Table A.3. Total burden requested under this information collection

Instrument	No. of respondents (total over request period)	No. of responses per respondent (total over request period)	Avg. burden per response (in hours)	Total burden (in hours)	Annual burden (in hours)	Average hourly wage rate	Total annual respondent cost
Center recruitment call script	500	1	.33	165	83	$26.41	$2,192.03
Umbrella organization recruitment approval call script	113	1	.33	37	19	$26.41	$501.79
Engagement interview guide	215	1	.33	71	36	$26.41	$950.76
SSLP interview guide	135	1	.50	68	34	$26.41	$897.94
Teaching staff roster	135	1	.25	34	17	$26.41	$448.97
Center manager survey	300	1	.42	126	63	$26.41	$1,663.83
Teaching staff survey	1,680	1	1	1,680	840	$17.57	$14,758.80
Total Annual Burden and Cost:					1,092		$21,412.12

2. Estimated annualized cost to respondents

The study team expects the total annual cost to be $21,412.12 for all of the instruments in the current information collection request.

The study team used the Bureau of Labor Statistics, Occupational Employment Statistics (2021)¹² to estimate the average hourly wage and derive total annual costs. For each instrument included in Table A.3, the study team calculated the total annual cost by multiplying the annual burden hours by the average hourly wage.

The mean hourly wage of $26.41 for education administrators of preschool and child care centers or programs (occupational code 11-9031) is used for primary site leaders and center managers and applies to all data collection tools except the teaching staff survey. The mean hourly wage for preschool teachers (occupational code 25-2011) of $17.57 is used for teachers and assistant teachers that will complete the teaching staff survey.

A.13. Costs

With OMB approval, the study team will offer each participating center a $150 honorarium for assistance and continued support of the study provided by the primary site leader and/or the on-site coordinator (if appointed by the leader). After the recruitment call, the primary site leader or an on-site coordinator from each center will participate in two interviews (engagement interview and SSLP interview) and complete the teaching staff roster, prior to any surveys released for data collection. They will also be asked to coordinate data collection activities by distributing survey invitation packets that include the pre-pay incentives to each eligible member of their teaching staff. Finally, they will assist with follow-up for staff completion of the center manager and teaching staff surveys.

A.14. Estimated annualized costs to the federal government

The total and annual cost for the data collection activities under this current request will be $2,114,000 and $1,057,000, respectively. This includes direct and indirect costs of data collection. Table A.4 displays the costs by the different work categories.

Table A.4. Estimated costs by category

Cost category	Estimated costs
Sampling and recruitment	$398,000
Data collection	$1,121,000
Data analysis and reporting	$473,000
Data archiving	$122,000
Total costs over the request period	$2,114,000
Annual costs	$1,057,000

A.15. Reasons for changes in burden

This is a new information collection request.

A.16. Timeline

Table A.5 lays out the timeline for the ExCELS descriptive study. Recruitment and data collection activities are expected to occur from February 2022 through May 2022. Data analysis will follow and findings are expected to be released in early 2023 (see B.7, Data Use for information on planned products).

Table A.5. ExCELS descriptive study timeline

Activity	Timinga
Data collection
Center recruitment	March 2022–June 2022
Data collection	April 2022–July 2022
Analysis
Data processing and analysis for data tables	Summer/Fall 2022
Data processing and analysis for final report	Summer/Fall 2022
Reporting
Data tables	January 2023
Descriptive study report	January 2023
Descriptive study briefs	Spring 2023
Restricted-use data file	Spring 2023

^a After obtaining OMB approval.

A.17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendices

Appendix A. ExCELS Theory of Change
Appendix B. Center Recruitment Materials
Appendix C. Interview Confirmation Emails
Appendix D. Center Manager Survey Respondent Materials

Appendix E. Teaching Staff Survey Respondent Materials

Instruments

Instrument 1. Center Recruitment Call Script
Instrument 2. Umbrella Organization Recruitment Approval Call Script
Instrument 3. Engagement Interview Guide
Instrument 4. Staffing Structure and Leadership Positions (SSLP) Interview Guide
Instrument 5. Teaching Staff Roster
Instrument 6. Center Manager Survey
Instrument 7. Teaching Staff Survey

1 Dhuey, Elizabeth, and Justin Smith. “How Important Are School Principals in the Production of Student Achievement?” Canadian Journal of Economics, vol. 47, no. 2, 2014, pp. 634-663.

2 Douglass, Anne L. Leading for Change in Early Care and Education: Cultivating Leadership from Within. New York: Teachers College Press, 2017.

3 Douglass, Anne L. “Redefining Leadership: Lessons from an Early Education Leadership Development Initiative.” Early Childhood Education Journal, vol. 46, no. 4, 2018, pp. 387-396.

4 Malone, L., S. Albanese, C. Jones, Y. Xue, and G. Kirby. “Compendium of Existing Measures for Understanding Leadership in Early Care and Education.” OPRE Report. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, forthcoming.

5 Kirby, G., A. Douglass, J. Lyskawa, C. Jones, and L. Malone. “Understanding Leadership in Early Care and Education: A Literature Review.” OPRE Report 2021-02. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021.

6 Please see B.3.1 design of data collection instruments, development of data collection instruments for details on the pretest.

7 Singer, Eleanor and Ye, Conge, 2013. The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science. Volume 645, Issue 1, January 2013, Pages 112-141

8 Lafauve, K., K. Rowan, K. Koepp, and G. Lawrence. “Effect of Incentives on Reducing Response Bias in a Web Survey of Parents.” Presented at the American Association of Public Opinion Research Annual Conference, Denver, CO, May 16–19, 2018.

9 Singer, Eleanor and Ye, Conge, 2013. The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science. Volume 645, Issue 1, January 2013, Pages 112-141

10 Andrew Mercer, Andrew Caporaso, David Cantor, and Reanne Townsend. “How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly, Volume 79, Issue 1, Spring 2015, pp 105–129.

11 Robbins, Michael, and Jennifer Hawes-Dawson. 2020. “The Effect of Incentives and Mode of Contact on the Recruitment of Teachers into Survey Panels.” Survey Practice 13 (1). https://doi.org/10.29115/SP-2020-0013.

12 U.S. Bureau of Labor Statistics. “Occupational Employment and Wages.” Washington, DC: U.S. Bureau of Labor Statistics, March 2021.