CONTENTS
A JUSTIFICATION 1
A1. Necessity for the Data Collection 1
A2. Purpose of the Survey and Data Collection Procedures 2
A3. Improved Information Technology to Reduce Burden 4
A4. Efforts to Identify Duplication 4
A5. Involvement of Small Organizations 4
A6. Consequences of Less Frequent Data Collection 4
A7. Special Circumstances 4
A8. Federal Register Notice and Consultation 4
A9. Gifts of Appreciation for Respondents 4
A10. Privacy of Respondents 5
A11. Sensitive Questions 5
A12. Estimation of Information Collection Burden 5
A13. Cost Burden to Respondents or Record Keepers 5
A14. Estimate of Cost to the Federal Government 5
A15. Change in Burden 5
A16. Plan and Time Schedule for Information Collection, Tabulation, and Publication 5
A17. Reasons Not to Display OMB Expiration Date 5
A18. Exceptions to Certification for Paperwork Reduction Act Submissions 5
A. JUSTIFICATION
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (DHHS) seeks approval for pilot data collection activities to support the Child Care and Early Education Quality Features, Thresholds and Dosage and Child Outcomes (Q-DOT) Pilot Study. ACF requests permission to (1) enroll child care centers and participants (teachers and center directors) into the pilot study and (2) pilot test two self-administered questionnaires (SAQs—Teacher SAQ and Center Director SAQ). We also plan to conduct classroom observations to analyze the relations between indicators of state Quality Rating and Improvement System (QRIS) quality ratings and observed process quality in classrooms. We also will evaluate the feasibility of recruiting and conducting the data collection among centers with varying auspices.
A1. Necessity for the Data Collection
The purpose of the Q-DOT Pilot Study is to learn more about the associations among QRIS ratings, quality-related features, and measures of observed quality for early care and education settings. The data collection is designed to explore the feasibility of obtaining QRIS ratings for programs, recruiting QRIS-rated programs for observation, and examining the relations between reported QRIS ratings and observable classroom process quality.
Q-DOT is a design project funded by ACF to examine associations between the quality of early care and education settings and child outcomes. The goal of the Q-DOT overall project is to assess whether thresholds of quality or dosage need to be met, or particular aspects of quality need to be present, before linkages to child outcomes are apparent. To date, project activities have encompassed a literature review and synthesis, secondary analyses of comprehensive data sets to determine key quality factors, and development of a conceptual framework to guide preparation of a research design. The proposed research design (to be carried out through a future procurement) calls for identification of center-based preschool programs across a range of auspices that vary on their level of initial process (interaction) quality. An evidence-based intervention designed to improve the quality of interactions between teachers and children would be selected for implementation. Random assignment to receive the intervention or serve as a control group would be carried out within two blocks: centers classified as higher quality and those classified as lower quality on the basis of process quality observations.
The design report proposed an approach to selection of sites that begins with identification of center-level QRIS ratings within a state or jurisdiction. These ratings then would be validated using direct process quality observations in a sample of classrooms. This approach would benefit from a pilot study to determine feasibility and effectiveness. Information from this pilot would assist ACF in the design of a future comprehensive study.
There are no legal or administrative requirements that necessitate the data collection activities.
A2. Purpose of the Survey and Data Collection Procedures
The Q-DOT Pilot Study seeks to evaluate whether the use of center-level QRIS ratings can accurately predict average levels of classroom process quality; in other words, whether the QRIS ratings can be used as a discriminating indicator of high vs. low classroom quality. First, the pilot study intends to explore whether QRIS ratings within a state or region can be sorted into discrete categories of high and low quality; and second, whether an observational process quality measure such as the Classroom Assessment Scoring System (CLASS) can be used to validate these ratings.
This pilot is designed to address several main questions:
How feasible is it to identify a geographic location (e.g., a state or region), obtain QRIS ratings for programs, and then recruit programs for observation? This will help us to determine the necessary characteristics and potential challenges for recruitment in the overall study.
What is the association between QRIS ratings and classroom process quality? In the overall study, we have proposed to use QRIS as a first step toward finding centers with classrooms that are high and low in process quality. Testing this process in a pilot will help us to determine whether this will be a feasible method of identification and recruitment.
What is the variation in classroom process quality within a center? This will inform the degree to which we think we can easily assign centers into the high or low process quality group and may influence decisions about the level of random assignment (center vs. classroom).
How do the answers to these questions vary by auspice? This will help address whether we should have concerns about any clustering of quality and auspice (for example, Head Start, community-based, and state pre-kindergarten programs).
Are centers willing to participate in the implementation of an intervention and research study? A short set of questions asked of center directors would help to identify how difficult it may be to recruit programs to participate in the larger proposed study, including mounting a quality intervention and tracking process quality and child outcomes.
The basic steps to conducting the pilot data collection efforts are to (1) select the sample and enroll programs into the study, (2) train observers who already work for Mathematica Policy Research (the contractor), (3) conduct data collection, (4) process the data, and (5) debrief the observers who carried out data collection.
Select sample and enroll programs. Once a list of programs representing high and low quality, as determined by the QRIS scores (distributed across community child care programs, Head Start programs, and state-funded pre-kindergartens), is compiled, we will send the directors a letter inviting them to participate in the pilot study and explaining the minimal burden involved (Attachment A1). Mathematica survey staff will follow up the letters with telephone calls, responding to questions directors may have, and facilitate the identification of up to three classrooms per program for the pilot study observations (Attachment A2).
Train observers. We will select and train four Mathematica survey staff on the CLASS1. A Mathematica staff member, trained as a CLASS-certified trainer, will conduct the training using the online protocol designed by the publisher. Observers will complete the online certification program required by the publisher. Observers also will be tested for interrater reliability in the field.
Collect and process the data. We will mail program/center directors and teachers an SAQ (Attachments A2 and A3) with a postage paid return envelope and conduct classroom observations over a four-week period. The observer will prompt teachers and program/center directors to return their SAQs if they have not done so by the time of the scheduled observation. Consistent with previous FACES data collections, we expect an 85 percent response rate from teachers and program/center directors. All completed observations and SAQs will be logged into the study database and the data entered and double-verified. We also will perform quality assurance visits for 10 percent of the observations. This will help in evaluating whether observers scores have strayed from reliability in their use of the observation measure.
Analysis. We will collect two types of data: (1) SAQs from center/program directors and teachers regarding the context of implementing a quality improvement initiative and the characteristics of the center (such as number of classes, group size, ratio, teacher qualifications, curriculum); and (2) direct observations of classroom quality.
Analyses will focus on several questions, including the following:
The correlation between QRIS ratings and CLASS scores.
Variability in CLASS scores between classrooms within a center.
Exploring different methods of analyzing CLASS scores for centers: for example, averaging all CLASS scores for a given center, or using only those CLASS scores within a particular range of one another.
Exploring the role of auspice in the overall distribution of scores and the relations among scores (for example, across auspices).
Descriptively examining teacher- and center-level characteristics, and possibly examining their correlation with CLASS scores or QRIS ratings.
A3. Improved Information Technology to Reduce Burden
The surveys for the center directors and teachers will be self-administered paper questionnaires, each of which will require approximately 15 minutes of the respondents’ time. The questionnaire packets will contain a business reply envelope. It would not be cost effective to use information technologies for the survey instrument for this one-time set of pilot activities.
A4. Efforts to Identify Duplication
We have selected items for the Q-DOT teacher and center director questionnaires that do not duplicate the information collected through other current and ongoing studies funded by federal agencies.
A5. Involvement of Small Organizations
Some programs or centers recruited into the project may be in this category. In order to reduce burden on small programs and centers, the SAQs for center director and teacher may be completed at a time convenient for the respondents, and the observations will be scheduled at times that are convenient for the programs.
A6. Consequences of Less Frequent Data Collection
Not applicable. This is a one-time data collection.
A7. Special Circumstances
There are no special circumstances for the proposed data collection efforts.
A8. Federal Register Notice and Consultation
The first Federal Register notice for ACF’s generic clearance for information gathering was published in the Federal Register, Volume 76, page 34078 on June 10, 2011. The agency did not receive any comments in response to the Federal Register notice for the generic clearance. The second Federal Register notice was published in the Federal Register, Volume 76, page 53682 on August 29, 2011.
The research team for the Q-DOT Pilot Study includes Dr. Margaret Burchinal from the University of North Carolina-Chapel Hill and Dr. Martha Zaslow from Child Trends and the Society for Research in Child Development. Drs. Burchinal and Zaslow have helped to identify research questions for the pilot study and reviewed items included in the SAQs.
A9. Gifts of Appreciation for Respondents
With OMB approval, we will offer a $100 post-pay check to centers for taking part in the pilot classroom observations and a gift valued at $20.00 to classroom teachers for completing the brief teacher questionnaire. These amounts were determined based on the estimated burden to participants and are consistent with those offered in prior studies using similar methodologies and data collection instruments. For example, the 2009 Head Start Family and Child Experiences Survey (FACES 2009) offered programs $250 for participation in the full study and an average of $50-$70 for completing the web-based teacher survey and teacher child report forms.
A10. Privacy of Respondents
The study will comply with government regulations for securing and protecting paper records, field notes, or other documents that contain sensitive or personally identifiable information. The study will not include personal identifiers on the brief questionnaires or field notes prepared during the classroom observations. We will assign a unique identification number to staff to facilitate linking information across data sources. All respondents will be assured that their information will be kept private to the extent permitted by law.
In the field, classroom observers will collect paper field notes on their classroom observation instrument. Interviewers will be instructed to keep these materials on their person or in locked storage at all times. Interviewers will hand carry all paper data (including field notes and completed questionnaires) back to Mathematica’s offices when they have finished field work. None of these data documents will include names or personal identifiers.
Once at Mathematica, we will scan paper field notes and store them in the project’s secure study-specific electronic folder. We will store the paper copies in a secure location, such as a locked file cabinet or locked drawer, when not in use.
We will maintain a data log, saved on the project’s secure study-specific electronic folder, to record the source, handling, receipt, location, and disposition of files that contain private data. Following the end of the project, when no longer required, we will destroy hardcopy materials and other physical media using a cross-cut shredder.
A11. Sensitive Questions
The center/program director SAQ includes items about salary that some directors may consider sensitive in nature. The SAQ instructions state that they may skip any questions they prefer not to answer.
The teacher SAQ includes items about salary and depressive symptoms (Center for Epidemiologic Studies Depression Scale [CES-D]; Radloff 1977). Some teachers may consider these questions to be somewhat sensitive in nature; they also will be instructed that they may skip any questions they prefer not to answer. Given established associations between such characteristics of teachers and classroom quality, we believe it is important to include these items.
A12. Estimation of Information Collection Burden
Total Annual Burden
The estimated burden for programs’ staff participating in the Q-DOT Pilot Study is listed in Table A.1. The total annual burden for the pilot activity is expected to be 18.75 hours. We estimated the response times based on prior experience with similar data collection instruments and materials.
Following the program’s receipt of the Program/Center Director Invitation Letter, we will call the program director to more formally invite him or her to participate in the study. During this call, we will describe the study purpose, provide an overview of the study activities, and confirm the program’s interest in participating using the Director Information Call Script. These activities are estimated to take a combined 15 minutes to complete. In addition, we anticipate that the center director SAQ will take approximately 15 minutes to complete, for a total burden of 30 minutes per center director respondent. The teacher SAQ (for three teachers per center) will take approximately 15 minutes for each teacher to complete.
To compute the total estimated annual cost, we multiplied the total burden hours by the average hourly wage for staff teachers and directors. For teachers, we used $14.72 per hour, which is the average hourly wage reported by the Bureau of Labor Statistics, Current Employment Statistics Survey, 2012. For center/program directors, we used 2011 data on the median weekly salary for full-time employees with a degree higher than a bachelor’s degree ($33.65 per hour, assuming an average work week of 40 hours).
Table A.1. Total Questionnaire and Administrative Tasks Burden Requested Under this Information Collection
Instrument/Material |
Total |
Number of Responses per Respondent |
Average Burden Hours per Response |
Total Burden Hours |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Center Director Survey/Screener |
15 |
1 |
0.50 |
7.50 |
8 |
$33.65 |
$269.20 |
Teacher Survey |
45 |
1 |
0.25 |
11.25 |
12 |
$14.72 |
$176.64 |
Estimated Annual Burden Subtotal |
--- |
20 |
--- |
$445.84 |
|||
A13. Cost Burden to Respondents or Record Keepers
Not applicable. There are no additional costs to respondents; they spend only their time to participate in the study.
A14. Estimate of Cost to the Federal Government
The total cost to the federal government of identifying and recruiting 15 child care and early education programs, identifying classrooms, conducting observations, and receipting questionnaires under the terms of the Q-DOT Pilot Study contract is estimated to be $255,149, including direct and in direct costs and fees. All work will be completed in the 2013-2014 fiscal year.
The total cost to the federal government of analyzing the collected data, summarizing findings in response to the study’s research questions, and developing a final report is estimated to be $80,509, including direct and indirect costs and fees. All work will be completed in the 2013-2014 fiscal year.
The total estimated cost to the federal government is $355,658.
A15. Change in Burden
This is an additional request under the pre-testing generic clearance (0970-0355).
A16. Plan and Time Schedule for Information Collection, Tabulation, and Publication
There are no plans for publishing the data gathered from the Q-DOT Pilot Study. Findings from the analysis and tabulation of data will be shared only with ACF staff. All information reported will be limited to methodological findings.
The Q-DOT Pilot Study activities will take place over a six-month period, commencing upon OMB approval and ending in August 2014. Recruitment and data collection activities are slated to begin in February 2014. Analysis activities will occur between May and June 2014. Reporting activities will take place between June and July 2014, with the draft report completed by July 2014.
The Q-DOT Pilot Study will be reported as a chapter of the Q-DOT Final Design Report focusing on the methodological findings of the pilot study and will discuss the following:
The feasibility of using QRIS indicators as a measure of quality.
The relations between QRIS ratings and classroom process quality.
The variation of classroom process quality within a center.
How auspice relates to measures of quality.
Barriers and incentives to participating in an intervention and study, as perceived by programs.
Considerations needed before moving forward with large-scale data collection.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the OMB approval number and expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 Classroom Assessment Scoring System (CLASS), by Robert C. Pianta, Karen M. LaParo, Bridget K. Hamre. Copyright 2008 by Paul H. Brookes Publishing Co. used with permission of the publisher.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Q-DOT OMB Section A\Q-DOT - Section A_TO OMB_FINAL.docx |
| Subject | Q-DOT Pilot Study--OMB Information Collection Request New Collection |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |