Supporting Statement A - Formative for Research GenIC - Child Care ECB

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Child Care Research and Evaluation Capacity Building Center Needs Assessment

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part A

March 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Meryl Barofsky and Alysia Blandon

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for ACF Research (0970-0356).

• Description of Request:

The Child Care Research and Evaluation Capacity Building Center (Center) will assess the research and evaluation capacity-building needs of State, Territory, and Tribal Child Care and Development Fund (CCDF) Lead Agencies through a web-based survey and virtual focus groups. Based on the information gathered, the Center will provide universal capacity-building resources and activities to support all CCDF Lead Agencies in conducting, understanding, consuming, and using research and evaluation for decision making (this includes facilitating agency use of state CCDF administrative data). We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: The information collected by the survey and focus groups will be used to inform the content and intensity of research and evaluation capacity-building supports in a currently active project. In order to stay on track with our project timeline we are targeting early 2021 for OMB approval.

A1. Necessity for Collection

The Administration for Children and Families Office of Planning, Research, and Evaluation (OPRE) would like to provide more targeted research and evaluation capacity building support to help the Child Care and Development Fund (CCDF) Lead Agencies increase their research and evaluation capacities. To do this, more comprehensive and systematic information is needed about the current research and evaluation capacities of the CCDF Lead Agencies to determine the most appropriate strategies to help CCDF Lead Agencies build their research and evaluation capacities.

A2. Purpose

Purpose and Use

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for research and evaluation (0970-0356):

• inform the development of ACF research

• maintain a research agenda that is rigorous and relevant

• ensure that research products are as current as possible

Information gathered on the survey will (1) describe the research and evaluation capacities of CCDF State, Territory, and Tribal Lead Agencies; and (2) contribute to OPRE’s understanding of their strengths and challenges related to conducting, partnering to obtain, and using research and evaluation. Focus groups with selected CCDF Lead agency staff will be done to further explore the challenges they have doing and using research and talk about potential solutions to those challenges.

The information we collect will be used to develop resources and supports (such as briefs and webinars) targeted to CCDF Lead Agencies about using research findings in decision making and conducting and/or partnering to do research and evaluation. These materials will be shared with CCDF Lead Agency staff and the general public through the Child Care Research and Evaluation Capacity Building Center (Child Care ECB Center), OPRE, and the Office of Child Care (OCC).

Data gathered will also be used to identify Lead Agencies that have multiple constraints and may benefit from intensive research and evaluation capacity-building supports. Findings are meant to inform ACF activities and may be incorporated into documents or presentations that are made public, but will not identify any specific state, territory, or tribe. OCC may also use the findings as they continue to guide supports for CCDF Lead Agencies through the national technical assistance network. Finally, the findings may be used to inform future ACF research on research and evaluation capacity building for ACF funded programs.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

The guiding research question is:

• What are the research and evaluation capacities of the CCDF State, Territory, and Tribal Lead Agencies?

Study Design

The study will use a mixed-methods design. The study includes a survey of the CCDF Lead Agencies and focus groups with selected CCDF Lead Agency administrators and staff responsible for CCDF Lead Agency research (when agencies have such staff). The study will include a census of state and territory administrators for the survey and a sample for the focus groups. The study will include a sample of tribal administrators for the survey and focus groups.

This design is appropriate for gathering data to assess the research and evaluation needs of CCDF Lead Agencies and inform development of resources to support CCDF Lead Agencies in improving research and evaluation capacities.

• The survey census of state and territory agencies is needed so we can develop resources for universal capacity-building, and the strategies that would be most appropriate for intensive capacity building for a small number of agencies with lower research and evaluation capacities. There are 56 CCDF state and territory lead agencies.

• We will select a sample of Tribal Lead Agencies to complete the survey to meet the goals of the study in an efficient manner without overburdening Tribal Agencies. There are 257 CCDF Tribal grantees. Tribal Lead Agencies are subject to some of the same requirements as State and Territory Lead Agencies, but also have different implementation milestones and reporting requirements. It is not feasible to do a census of tribal grantees, given how many there are. Finally, the follow up focus groups will help explore more deeply the associations among the constructs, individual perceptions about strengths and challenges, and possible strategies for building capacities.

For specific details about methods and design, and the collection of data, see Supporting Statement B.

The results of the study will only be generalizable to State and Territory CCDF Lead Agencies. The results of the tribal study will not be generalizable across the whole population of tribal CCDF Lead Agencies. However, we will select a stratified sample of tribes to get a diverse range of capacities and needs.

One study limitation is that we are relying on self-assessments of agency staff’s needs, strengths and weaknesses. This approach can be challenging because people may have poor insight into their strengths and weaknesses (Dunning, Heath, and Suls 2004; Mabe and West 1982). To help address this limitation, we have designed the instruments to follow best practices in eliciting accurate self-reports, including selecting or designing measures that ask for concrete examples of past performance to support opinions about past research activities and providing benchmarks and other comparative methods when assessing technical skills. Another limitation is that because the survey is designed to capture a broad range of research and evaluation capacities, we are gathering less detail about agencies’ needs related to more advanced research or analytic methods. Limitations will be clearly noted in any shared information resulting from this study.

Exhibit A1. Study Design Overview

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Survey of CCDF State, Territory, and Tribal Lead Agencies	Survey	Respondents: State, Territory, and Tribal agency staff Content: Agency structure and role in child care system; Agency’s use of existing research and data; Agency’s research capacity and barriers to conducting research; Agency’s interest in capacity building activities Purpose: Assess research and evaluation capacity and needs	Mode: Web with paper option to mail or scan Duration: 25 minutes
Focus Groups of Selected CCDF State, Territory, and Tribal Lead Agencies	Focus group protocol	Respondents: State, Territory, and Tribal Agency staff Content: Agency-level perspectives on strengths, barriers, and lessons learned in using and doing research; examples of research they have found helpful in their work; their priorities about questions they would like to answer; ways that staff are supported to learn more about doing and using research; examples of how their agency has used research and data to inform decision making; and the mode and focus of resources that could help them overcome barriers Purpose: Gain a more nuanced understanding of information gathered through the survey.	Mode: Virtual focus group Duration: 90 minutes for each focus group; each invited individual will participate in only one focus group

Other Data Sources and Uses of Information

We are not relying on any existing information for this project and the surveys and focus groups are the only information collection. We are not collecting any additional administrative data and there were no prior data collections for this project.

A3. Use of Information Technology to Reduce Burden

Survey

We will program and field the survey through Confirmit, a web-based survey software platform that enables the user to efficiently build and launch Section 508-compliant surveys while minimizing respondent burden. Confirmit includes many features that minimize burden. It allows surveys to be programmed with skip logic to avoid irrelevant questions. Information from previous responses are filled into subsequent question stems to reduce the need to recall prior responses. Responses are saved each time participants move forwards or backwards in a questionnaire, allowing participants to complete the questionnaire in stages without losing information. Multiple stakeholders in an agency can be sent the same survey, so different people can respond to questions that they can easily answer. Participants can print out a copy of their responses if they wish to keep them for internal records or use them to discuss internal capacity building efforts.

Focus Groups

Focus groups will be held in a virtual environment to facilitate participation of agency staff from States, Territories, and Tribes without the need for either the researchers or participants to travel. We will use a platform that allows individuals to see each other through video cameras, allows the moderator to type notes that appear as if on a flip chart in a room, and that enables easy recording of responses while maintaining a secure environment.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

Through our work, we have not identified other current or planned efforts to collect systematic, detailed information on what CCDF Lead Agencies need to better use and do research. None of the study instruments ask for information that can be obtained from alternative data sources (including administrative data). The design of the survey and focus group instruments ensures minimal duplication of data collected across instruments.

A5. Impact on Small Businesses

No small organizations are affected by the information collection.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on November 3, 2020, Volume 85, Number 213, page 69627, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

We consulted with external experts about how to sample tribal agencies in a way that will allow us to obtain a comprehensive view of the research and evaluation capacity building needs of tribal agencies. We also shared sections of the survey instrument and recruitment protocols with external experts to ensure that we used appropriate and relevant language.

A9. Tokens of Appreciation

No tokens of appreciation will be provided to participants.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

To ensure the correct identity of respondents, the survey and focus groups will collect the names of respondents and their State, Territory, or Tribal Agencies. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Focus groups will be recorded with the permission of respondents but only for the purposes of ensuring that complete notes can be taken. The notes and recording will not be shared beyond the research team. Entity-level identities (e.g. the State, Territories, and Tribes) with which the data are associated will be maintained throughout the study process and provided to ACF. We will keep the entity-level identities in order to provide support to selected agencies based on the need indicated in their survey responses. Although we will not provide respondent-level identifiers (e.g. names or initials) to ACF, anyone who knows the state, territory, or tribe will also know who participated in the survey because there is typically only one or two people in each Agency whose roles are appropriate for response. This means we cannot assure privacy. Findings may be shared publicly, but publicly shared results will not include state, territory, or tribal identifiers. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. The study has been approved by the Urban Institute Institutional Review Board.

Data Security and Monitoring

The study team has developed a data security and monitoring plan that assesses all protections of respondents’ personally identifiable information. Urban and Mathematica will ensure that all its employees and consultants who perform work under this contract are trained on data privacy issues and comply with the requirements. All project staff, including Urban Institute and subcontractors from Mathematica will sign the Urban Institute’s staff confidentiality pledge agreeing to follow the Urban Institute’s guidelines for data security.

As specified in OPRE’s contract, Urban and Mathematica will use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. Urban and Mathematica will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. Urban and Mathematica will (1) ensure that this standard is incorporated into the company’s property management and control system; and (2) establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, Urban and Mathematica must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive or personally identifiable information to ensure secure storage and limits on access.

A11. Sensitive Information ¹

We do not anticipate collection of any sensitive information.

A12. Burden

Explanation of Burden Estimates

Exhibit A2 summarizes the estimated reporting burden and costs for each of the study tools included in this information collection request. The estimates include time for respondents to review instructions, search data sources, complete and review their responses, and transmit information. Figures are estimated as follows:

1. Needs Assessment Survey (Instrument 1). The study team expects to reach out to 71 CCDF Lead Agencies for this study (56 State and Territories and 15 Tribes). We anticipate the survey will take about 25 minutes to complete. This burden estimate is based on a pretest conducted with four former CCDF agency leads. The pretest respondents provided responses to the survey in an electronic hardcopy version of the instrument and then participated in a debrief call with the research team. The research team used the feedback from pretest respondents to revise and edit the survey content to ensure it would take 25 minutes to complete.

2. Focus group protocol (Instrument 2). The team will gather data from up to 20 State, Territory and Tribal CCDF Lead Agencies for the virtual focus groups. We expect more than one participant from each agency and a total of approximately 45 participants. We will conduct five 90-minute long virtual focus groups, with each group having about 9 participants. One focus group will consist of staff from Tribal Agencies and the other four groups will include a mix of State and Territory staff. This burden estimate is based on 45 participants and the 90-minute length of the focus group.

Estimated Annualized Cost to Respondents

All respondents are members of state, local or tribal government. Estimates for these respondents are based on the Department of Labor (DOL) May 2019 estimates for the mean hourly wage of Social and Community Service Managers: $35.05 https://www.bls.gov/oes/current/oes_nat.htm#21-0000

Exhibit A2 Total burden requested under this information collection

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total /Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Needs Assessment Survey	71	1	.42	29.6	$35.05	$1,037.48
Focus group protocol	45	1	1.5	67.5	$35.05	$2,365.88
Total	116	1	N/A	97	$35.05	$3,403.36

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The cost for the data collection activities under this current request is $372,692. It includes direct and indirect costs for developing plans, designing data collection instruments and putting instruments through OMB clearance; the field work of recruiting participants and collecting and processing all the data; and costs associated with analyzing the data and writing the draft and final needs assessment reports. The cost estimate includes costs for both Urban Institute and its subcontractor, Mathematica.

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$147,500
Field Work	$152,100
Publications/Dissemination	$73,092
Total costs over the request period	$372,692

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for ACF research (0970-0356).

A16. Timeline

Exhibit A3 shows the timeline for the study. The final report, expected in November 2021, will present findings based on the survey and focus groups.

Exhibit A3. Study timeline

Tasks	Date
Survey recruitment and data collection	April 2021 – August 2021
Focus group recruitment and data collection	August 2021 – September 2021
Data analysis	September – October 2021
Report	December 2021

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Attachment A: Recruitment Materials

• A1. Advance email for State and Territory CCDF Lead Agencies

• A2. Study Flyer: Seeking Your Help to Assess CCDF Lead Agency Research and Evaluation Capacity Building Needs

• A3. Office of Child Care Letter of Support

• A4. Recruitment call script for Tribal CCDF Lead Agencies

• A5. Advance email for Tribal CCDF Lead Agencies

• A6. Survey invitation email

• A7. Survey reminder email

• A8. Survey phone follow-up scripts

• A9. Focus group invitation email

• A10. Focus group follow-up email

• A11. Focus group follow-up phone script

• A12. Focus group registration email

• A13. Focus group reminder email

Instrument 1: Needs Assessment Survey

Instrument 2: Focus Group protocol

References

Banghart, Patti, C. King, E. Bedrick, A. Hirilall, and S. Daily. 2019. State Priorities for Child Care and Development Block Grant Funding Increase: 2019 National Overview. Child Trends

Bourgeois, I. and J. B. Cousins. 2013. “Understanding Dimensions of Organizational Evaluation Capacity,” American Journal of Evaluation 34 (3): 299–319.

Brennan, Sue E., J. McKenzie, T. Turner, S. Redman, S. Makkar, A. Williamson, A. Haynes, and S. Green. 2017. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research. Health Research Policy and Systems, 15(1), 1-19. (also referenced as SEER)

Connors-Tadros, L. and N. DeCrecchio. 2019. The Views of State Early Childhood Education Agency Staff on Their Work and Their Vision for Young Children: Informing a Legacy for Young Children by 2030. New Brunswick, NJ: Center on Enhancing Early Learning

Derrick-Mills, T. M. 2012. “How Do Performance Data Inform Design and Management of Child Care Development Fund (CCDF) Programs in the U.S. States?” PhD Dissertation, the George Washington University. Ann Arbor, MI: UMI/ProQuest.

The Early Childhood Data Collaborative. 2014. 2013 State of States’ Early Childhood Data Systems. The Early Childhood Data Collaborative.

James Bell Associates. 2018. How Can Child Welfare Organizational Capacity be Measured? Evaluation Brief. Washington, DC: Department of Health and Human Services, Administration for Children, Youth and Families, Children’s Bureau.

King, Carlise, V. Perkins, C. Nugent and E. Jordan. 2018. “2018 State of State Early Childhood Data Systems.” The Early Childhood Data Collaborative.

Maxwell, Kelly, S. Moodle, C. King, V. Lin, and A. Blasberg. 2015. Child Care Administrators' Use of Administrative Data to Address Program and Policy Questions. Internal Use Only OPRE Report: CCADAC and CCEEPRA.

Palinkas, Lawrence A., A. Garcia, G. Aarons, M. Finno-Velasquez, I. Holloway, T. Mackie, L. Leslie, and P. Chamberlain. 2016. Measuring Use of Research Evidence: The Structured Interview for Evidence Use. Res Soc Work Pract, 26(5), 550-564. (also referenced as SIEU)

Penuel, W.R., D.C. Briggs, K.L. Davidson, C. Herlihy, D. Sherer, H.C. Hill, C.C. Farrell, & A-R Allen. 2016. Findings from a national survey of research use among school and district leaders (Technical Report No. 1). Boulder, CO: National Center for Research in Policy and Practice.

Pennsylvania Office of Child Development and Early Learning. 2018. Using Data to Inform Decision-Making: Staff Perspectives and Experiences (Instrument). (also referenced as PA)

Rohacek, M. 2017. Research and Evaluation Capacity: Self-Assessment Tool and Discussion Guide for CCDF Lead Agencies. OPRE Report #2017-63. Washington, DC: US Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.

Yanovitzky, Itzhak and C. Blitz. 2017. The Capacity-Opportunity-Motivation (COM) Model of Data-Informed Decision-Making in Education. Proceedings of EDULEARN17 Conference 3rd-5th July 2017, Barcelona, Spain. (also referenced as COM)

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

12