Information Collection Request
Existing Collection in Use Without OMB Control Number
Sealant Efficiency Assessment for Locals and States (SEALS)
Supporting Statement: Part B
Program Official/Contact
Cassie Brailer, Public Health Advisor
National Center for Chronic Disease Prevention and Health Promotion, Division of Oral Health
Centers for Disease Control and Prevention
Atlanta, GA
770-488-7414
March 12, 2019
TABLE OF CONTENTS
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B1. Respondent Universe and Sampling Methods
B2. Procedures for the Collection of Information
B3. Methods to Maximize Response Rates and Deal with Nonresponse
B4. Test of Procedures or Methods to be Undertaken
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
ATTACHMENTS
1. The Public Health Service Act (42 U.S.C. 301; and 42 U.S.C 247b–14 Oral health promotion and disease prevention)
2a. Sample SEALS cost reports
2b. Sample state comparison report
2c. Screenshots of entry forms from SEALS User Manual
2d. Paper data collection form
2e. Screenshot of default cost values
2f. Request to participate for funded states and unfunded states/territories/tribes
3. Federal Registry Notice
4. Institutional Review Board determination
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B1. Respondent Universe and Sampling Methods
The universe for this data collection is school sealant programs (SSPs) in 50 states, Washington, D.C., and the seven US territories (heretofore referred to as “states”). An SSP is defined as a program that delivers dental sealants typically using portable dental equipment in elementary and middle public schools attended by a large number of students who lack access to dental care (https://www.thecommunityguide.org/sites/default/files/assets/Oral-Health-Caries-School-based-Sealants_0.pdf). There are no data on the size of the sampling universe, i.e., number of SSPs in the US or by state. Historically, states funded by CDC report both the number of schools in the state where 50% or more of students participate in the free/reduced meal program and the number of these schools that the state knows have an SSP. So, although the universe of SSPs is unknown, historic data collection by CDC suggests that data will be available for at least one-third of the universe of low-income schools.
Data for this project will be obtained through a convenience sample of local SSPs recruited by each state oral health department. Historically, about 76% of funded states were able to recruit SSPs to share similar SSP data during the 5-year CDC funding period. The average number of responding SSPs per funded state was 9. Based on non-funded states requesting CDC assistance in analyzing similar data provided by SSPs in their state, CDC anticipates that SSPs in 4 non-funded states will also participate.
Over the next 5 years, CDC will fund 20 state oral health programs for SSPs. Based on historic response rates, we expect 162 SSPs to respond (9 programs * (0.76*20 + 4) states). Data about each SSP will only be included in the analysis one time so that observations are independent (i.e., observations from the same SSP would be correlated over time) to ensure accurate standard errors.
B2. Procedures for the Collection of Information
CDC-funded state respondents are asked to provide information for at least one school year during the State Actions to Improve Oral Health Outcomes (funding award DP1810). SSPs will enter information into SEALS, a web-based tool that stores and analyzes SSP data. At the end of the school year when SSPs have entered all data, users can generates reports with various performance measures. The CDC cooperative agreement asks funded state oral health departments to recruit SSPs in their state to enter data into SEALS for one school year. Non-funded states that request to participate will also need to recruit SSPs in their state to participate.
CDC manages the SEALS web app and provides a helpdesk for users during the school year. At the beginning of the funding period CDC sets up accounts for each participating state. The state in turn sets up accounts for each participating SSP. Once all accounts are created at the beginning of the funding period, CDC provides a recorded SEALS training to state sealant coordinators and all participating local SSPs. A detailed user manual, created by CDC, is also provided.
SSPs enter data into three “forms” or screens. In the first two screens, data is entered one time at the beginning of the school year.
The first screen, Program Options (Attachment 2c) asks questions related to sealant delivery logistics (e.g., does program screen and seal children during same seating, number of operators per child).
The second screen, Cost Options (Attachment 2c), asks questions about SSP per unit resource costs such as dental hygiene costs per hour. SSPs can select a default cost for each resource unit or input their own value.
In the last screen, SSPs enter data at each school. This screen, Add Event, (Attachment 2c) asks for information on quantity of resource units used (e.g., hours at school for each worker category), services delivered, and risk for tooth decay among those children served.
Some SSPs may already enter information on child’s caries risk and services delivered into their billing software. To avoid duplicate data entry, SSPs that are already collecting this information can skip entering data into the 3rd screen and instead export a file with this information from their billing software. This file would then be sent to CDC who would upload the data into SEALS.
From these data, SEALS generates reports with performance measures related to program efficiency (Attachment 2a, and 2b), which can be downloaded by each SSP, by a state (for individual SSPs in state and also aggregated across state), and CDC. CDC can access reports for individual SSPs, SSP data aggregated across a given state, and data across all states.
CDC will use information from SEALS to develop feasible benchmarks (i.e., what is attainable after controlling for factors beyond an SSPs control) for the following measures: total resource cost-per-child sealed; total resource cost-per-tooth sealed; clinical labor time per child sealed; clinical labor cost-per-child sealed; and cost-per-averted cavity.
In order to identify feasible benchmarks, CDC will first use multivariate regression modeling to identify which factors beyond an SSP’s control are significantly associated with lower performance measure values. Five regressions will be run – each regression will have one of the performance measures as the dependent variable. Independent variables in each regression include:
State dental practice acts (these allow dental hygienists to prescribe and place sealants under the following conditions):
with no dentist supervision;
allows hygienists to place sealants without dentist supervision; or
expressly prohibits hygienists to prescribe or place sealants without a dentist supervision;
urbanicity (percentage of served schools that are in rural counties), and
program size.
Information about state dental practice acts and whether served school is in rural county is publicly available and information on number of children served is recorded in SEALS.
Cut-points for urbanicity (whether SSPs primarily serve schools in urban areas) and program size will be determined after examining their respective distributions. Each regression will have indicator variables for each state to capture state effects. Using an online power calculator (https://www.danielsoper.com/statcalc/calculator.aspx?id=1) and assuming 24 independent variables, we found that we would need 132 SSPs total to respond in order to detect a 20% effect sizes at beta (i.e., probability of false negative or rejecting an invalid null hypothesis) = 0.2.
After significant factors are identified, CDC will partition the data by these factors and then examine the distribution of each performance measure (mean, median, mode, and quintiles) to establish feasible benchmarks (benchmark interval would include median and values at 40 and 60 percentile). CDC has also convened a team of experts who will conduct additional analyses based on the distribution of the data using either parametric (stochastic production function) or non-parametric analyses (data envelopment analyses) to identify the set of the most efficient SSPs and to identify factors/practices (collected in SEALS or publicly available) contributing to efficiency.
B3. Methods to Maximize Response Rates and Deal with No Response
The first activity CDC will undertake, once approval for this information collection to continue is received, is to reach out to all states and territories to invite them to participate, and to explain the training opportunities and support system in place.
Because the sampling universe is unknown CDC will use a convenience sample. Based on the expected number of independent variables (including state effects) CDC will have sufficient power to detect factors that are significantly associated with the performance measures in the 5 regressions if 132 vs. the anticipated 162 SSPs respond. CDC will also test whether the individual state effects in the regressions are significant. If they are significant, then the findings of this analysis will only be applicable to the participating 18 states. It should be noted that at present there have been no multivariate analyses of factors associated with SSP cost and efficiency and thus this analysis will lay the foundation for future analyses.
CDC has worked with several experts and stakeholders to develop the most efficient data collection tools so that the information collection is consistent with their program goals. It should also be noted that a panel of 13 SSP experts participating in a sealant work group convened by the Children’s Dental Health Project, independently recommended that all SSPs collect the data elements that CDC is requesting. CDC has also worked with experts to provide default values for per unit resource costs when possible and to avoid duplicate data entry by allowing batch data entry of data exported from billing software and other SSP data systems.
B4. Test of Procedures or Methods to be Undertaken
All calculations used by SEALS have been peer-reviewed. The methodology to estimate 9-year averted cavities attributable to SSP sealants was developed by 3 PhD economists and an administrator of a large SSP that has existed since 2000. The Markov model was published in a peer-reviewed journal in 20142.
CDC also developed a methodology to obtain reasonable estimates of costs, resource use, and services delivered with minimal data collection in collaboration with SSP experts – three state sealant coordinators, four SSP administrators, and two PhD economists. CDC staff (1 public health dentist and 1 PhD economist) had 4 conference calls with the SSP experts to document sealant delivery logistics; to identify SSP cost components and the most appropriate units for their measurement (e.g., per operator); and optimal collection frequency (e.g., per day). For costs that were homogenous per unit across SSPs, CDC worked with the SSP experts to develop default values that SSPs could select in lieu of collecting their own program data. CDC used information obtained in these meetings to develop data collection logs, which were piloted at 2 different schools by each of the 4 SSP administrators. The data collection logs served as the template for the web-based data collection tool, SEALS. This cost estimation methodology was accepted for publication in a peer-reviewed journal in 2016 and was published in 20171.
CDC and a contractor, Northrup Grumman, developed a web app, SEALS, to collect, store, and analyze SSP data. This data collection instrument was piloted in six CDC-funded states in 2017-2018 (OMB # 0920-0739). From this pilot CDC has upgraded SEALS to better meet user needs. For example, batch data entry capacity was added based on feedback from users.
Finally, for the benchmarking analysis, of performance measures generated by SEALS, CDC convened a team of academics with expertise in economics and benchmarking. The initial study design described in B2 was developed by the expert academic panel. This team will also work on the analysis of the SSP data obtained during the next 5 years.
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Expert Consultants: Statistical Benchmarking
Susan Griffin, PhD, Health Economist, Division of Oral Health
Kari Jones, PhD, Quantitative Health Research [email protected] 770-598.4780.
Rene Villalobos, PhD, Associate Professor, Industrial Engineering Department, Arizona State University [email protected], 480-965-0437
Christina Scherrer, PhD, Professor Systems and Industrial Engineering, Kennesaw State University, [email protected], 470-578-5413
Paul M Griffin, PhD, St. Vincent Health Chair of Healthcare Engineering and Professor of Industrial Engineering, Purdue University, [email protected], 765-4967395
SEALS is programmed to analyze much of the data for the parameters sought. The CDC economist provides quality assurance on the data.
REFERENCES
Griffin SO, Jones K, Naavaal S, Arlotta D, Demopoulos C, & O’Connell J. (2018). Estimating the cost of school sealant programs with minimal data. Journal of Public Health Dentistry. 78(1):17-24.
Griffin SO, Jones K, & Crespin M. (2014). Calculating averted caries attributable to school-based sealant programs with a minimal dataset. Journal of Public Health Dentistry, 74(3):202-9.
Griffin SO, Naavaal S, Scherrer CR, Patel M, & Chattopadhay S. (2017). Evaluation of school-based dental sealant programs: A Community Guide systematic economic review. American Journal of Preventive Medicine, 52(3):407-415.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |