Contact:
Amy McMillen
National Center for Emerging and Zoonotic Infectious Diseases
Centers for Disease Control and Prevention
1600 Clifton Road, NE
Atlanta, Georgia 30333
Phone: (404) 639-1045
Email: [email protected]
1. Circumstances Making the Collection of Information Necessary
2. Purpose and Use of Information Collection
3. Use of Improved Information Technology and Burden Reduction
4. Efforts to Identify Duplication and Use of Similar Information
5. Impact on Small Businesses or Other Small Entities
6. Consequences of Collecting the Information Less Frequently
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
9. Explanation of Any Payment or Gift to Respondents
10. Protection of the Privacy and Confidentiality of Information Provided by Respondents
11. Institutional Review Board (IRB) and Justification for Sensitive Questions
12. Estimates of Annualized Burden Hours and Costs
13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
14. Annualized Cost to the Government
15. Explanation for Program Changes or Adjustments
16. Plans for Tabulation and Publication and Project Time Schedule
17. Reason(s) Display of OMB Expiration Date is Inappropriate
18. Exceptions to Certification for Paperwork Reduction Act Submissions
19. References
Exhibits
Exhibit 12.A Estimated Annualized Burden Hours
Exhibit 12.B Estimated Annualized Burden Costs
Exhibit 14.A Estimated Cost to the Government
Exhibit 16.A Project Time Schedule
Attachments
Attachment Designation |
Document Description |
||
A |
Electronic Platform: Quarterly Chronic Absenteeism Data Reporting Form |
||
B |
Demographic Data Collection Points |
||
C |
Pilot Site Baseline Survey |
||
D |
Webinar 1 Feedback Form |
||
E |
Question Guide for Face to Face Evaluation Interviews |
||
F |
School District Feedback Form |
||
G |
Privacy Impact Checklist |
||
H |
IRB Protocol Approval with Modifications
|
||
I |
Consent Form and Authorization for School District Representatives
|
||
J |
Consent Form and Authorization for School Nurses
|
||
|
|
|
Goal of the study: The
goal of this GEN-IC is to field test an approach for surveillance
of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) by
building on and improving ongoing active surveillance of chronic
conditions associated with absenteeism and school withdrawal among
US schoolchildren.
Intended use of the
resulting data: The data from the proposed GEN-IC will be used
1) to fine- tune a new electronic data collection tool and active
surveillance methods for school-level surveillance of chronic
conditions, including ME/CFS; 2) to enhance technical assistance
and training to support this surveillance activity; 3) to identify
considerations for scaling up the piloted approach.
Methods to be used to
collect data: The methods will include usability testing of a
new electronic surveillance data collection tool, field testing of
the surveillance tool after modification guided by usability
testing results, and online surveys and focus groups of users and
potential users of the tool. The subpopulation to be
studied: Respondents for the proposed data collection will be
from U.S. school districts in states where physicians experienced
in caring for patients with ME/CFS practice. School districts have
been chosen to include urban, suburban, and rural areas. The
respondents will be school nurses in the selected school districts,
who will test the new data collection tool and describe their needs
and experiences related to this activity; administrators in the
same school districts, who will describe the impact of the activity
on their district; and school data coordinators in all fifty
states, who will report on training and technical assistance needs
relevant to school surveillance of chronic health conditions,
including ME/CFS. How data will be analyzed:
The data from the surveillance tool and online surveys will be
analyzed using descriptive statistics; qualitative analysis methods
will be used to identify themes from focus group interviews.
JUSTIFICATION
The Centers for Disease Control and Prevention (CDC) requests approval for a new Gen-IC, “Formative Research to Support Active Surveillance of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Among Schoolchildren” under OMB Control No. 0920-1154.
This request is for a pilot project to field test and refine data collection tools and methods for determining the frequency of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) among other currently monitored chronic health conditions associated with school absenteeism and school withdrawal among US schoolchildren. Collection of information on better-known conditions that school nurses have already been tracking, in addition to ME/CFS, will allow assessment of the quality of this pilot surveillance approach.
Background
ME/CFS is a complex, chronic, debilitating disease that involves several body systems and is hypothesized to contribute to chronic school absenteeism.1 The condition is characterized by reduced ability to perform pre-illness activities that lasts for more than six months and is accompanied by profound fatigue, which is not improved by rest. ME/CFS affects children and adolescents as well as adults.2,3 Less is known about the condition in children and youth because less research has been focused on this age group. A few studies have estimated the number of children affected by ME/CFS, but these estimates vary, and the studies have limitations.4-6 Information on how many children are affected by ME/CFS and the impact of their condition on school attendance is needed to raise awareness of ME/CFS and to plan for children’s health care needs. Recognizing this need, in January 2017, the Chronic Fatigue Syndrome Advisory Committee (CFSAC) of the U.S. Department of Health and Human Services (HHS) recommended that HHS educate educators and school nurses on ME/CFS affecting children and adolescents.7
The National Association of School Nurses (NASN) has been promoting surveillance of health conditions affecting schoolchildren for years, but ME/CFS has not been among the conditions tracked. Furthermore, data collection is not currently standardized, and there is not an electronic data collection platform that allows for data entry at the school level. Consequently, data on chronic conditions, absenteeism, and health concerns is inadequate. By implementing an electronic platform for school-based surveillance of chronic conditions associated with absenteeism and school withdrawal that includes ME/CFS, the proposed pilot can help address both the CFSAC recommendation and the limitations of current surveillance of chronic health conditions among US schoolchildren. Trained school nurse respondents in a small number of pilot sites will test an approach that involves proactively identifying students affected by ME/CFS and other chronic health conditions and using the new electronic data collection tool to report aggregate, de-identified data to NASN. The active surveillance methods, data collection tool, and related training and technical assistance will be evaluated to identify considerations for national rollout of the piloted approach. Active national surveillance of ME/CFS in schools, coupled with education of school nurses about ME/CFS, could help improve estimates of the burden of ME/CFS in children and lead to the improvement of services to children suffering from this and other chronic health conditions.
The purpose of this pilot project is to field test and refine an approach for surveillance of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) by building on and improving ongoing active surveillance of chronic conditions associated with absenteeism and school withdrawal among US schoolchildren. Several states are already tracking data on students who are chronically absent. In many school districts, school nurses already monitor how many students have certain chronic conditions, like asthma, diabetes, and seizure disorders, but the data collection is not standardized at the school level. Adding ME/CFS to the list of chronic conditions that school nurses already track could provide a more accurate picture of how many schoolchildren have ME/CFS, and could, in the process, educate more school nursing professionals about ME/CFS. Active surveillance could potentially assist communities in better identifying students at risk of ME/CFS early, tracking trends, and proactively meeting the health needs of students and their families.
An environmental scan conducted by NASN found that having school electronic health records, school health data coordinators, and data training improved active surveillance outcomes.8 For this pilot project, CDC will implement an electronic data collection tool to allow school-level standardized data entry at the pilot sites by school nurses trained in this activity. If data collection can be standardized at the school level, data might be more easily rolled up to the state and national levels, providing greater consistency compared with current system, where data points are not consistently defined, collected, and reported in a standardized manner by states. Therefore, the results from this pilot will be used to identify considerations for scaling up the piloted approach.
Six school district pilot sites in four states (one in Massachusetts, one in Florida, two in Michigan, and two in Utah) will participate in this project. These specific states were chosen because they are known to be states where physicians experienced in caring for patients with ME/CFS practice. School districts were chosen to include urban, suburban, and rural areas. In each of the demonstration sites, one frontline school nurse will identify children who are chronically absent or have withdrawn from school, as well as children with ME/CFS and other chronic conditions, through school records, including school health records.
Three types of data will be collected on six forms for this pilot project: 1) data on health-related chronic absenteeism and school withdrawal (2 forms—Attachments A and B), 2) data on technical assistance and training needs (2 forms—Attachments C and D), and 3) data on feasibility and usability of the active surveillance process and electronic data collection tool (2 forms—Attachments E and F).
Data on Health-related Chronic Absenteeism and School Withdrawal
School nurse respondents will collect data on health-related school withdrawal and chronic absenteeism, which has been defined as missing more than 10% of a school year, or more than three days per month.17 This definition has been modified slightly because of operational changes related to the coronavirus pandemic. For this surveillance pilot project, chronic absenteeism is defined as missing 10% or more of virtual and/or traditional learning opportunities. The data from this component of the pilot project will be used to assess whether school nurses can provide accurate, useful information to track ME/CFS and other chronic health conditions associated with absenteeism and school withdrawal in schoolchildren, and to identify considerations for implementing this approach on a larger scale in the U.S. The surveillance information to be collected includes both data from the school health record and follow-up data from outreach to students who have ME/CFS symptoms. Collecting data about other chronic conditions, including, but not limited to asthma, diabetes, seizures, as well as ME/CFS, will allow assessment of the quality of the data from the pilot system, as more is known about these conditions, allowing benchmarking of surveillance estimates. Without standardized data entry at the school level--the approach that the proposed pilot project is intended to test--information on ME/CFS and other chronic conditions in schoolchildren will continue to be limited, and the patchwork approach to current school surveillance will continue to be a barrier to a coordinated system that provides national data.
School nurses will enter and submit data related to chronic absenteeism and school withdrawal surveillance in an online data collection tool (Attachment A). This form includes the following aggregate data elements from each site: number of students chronically absent or withdrawn for any reason and for health reasons, number of families contacted for students with chronic absenteeism, number of undiagnosed students with symptoms related to ME/CFS, number of students referred to a provider for possible ME/CFS, types of nursing interventions, and nursing diagnoses for students who are chronically absent. Additionally, the school nurses will note the time needed to complete parts of this data collection tool. In addition, school nurses will electronically enter the number of enrolled students in the school, school level demographics, and school nurse workforce data as listed in the “Demographic Data Collection Points” document (Attachment B). The data entered in the platform will be reported by the school nurses to NASN quarterly, and NASN will send a summary to CDC at the end of the school year. The quarters are defined as follows: Quarter 1 (beginning of school year to November 30), Quarter 2 (December 1 to February 28), and Quarter 3 (March 1 to end of school year).
With the escalation of the COVID-19 pandemic, the operation of school systems has been evolving. Educational modifications, including virtual learning, traditional learning, or hybrid models, are starting to be implemented across the nation as schools start in August and September 2020. Tracking chronic absenteeism and school withdrawal by taking physical attendance is still possible in schools that are operating as they did pre-COVID. However, other methods are necessary for tracking attendance in virtual or hybrid educational settings made necessary by the pandemic, such as monitoring school engagement and turning in assignments. The National Association of School Nurses (NASN), has been collaborating with Attendance Works to identify alternative methods of tracking chronic absenteeism and school withdrawal in the pilot schools. Guidance on identifying alternatives for monitoring attendance in distance learning has been posted on the Attendance Works site here: (https://www.attendanceworks.org/chronic-absence/addressing-chronic-absence/monitoring-attendance-in-distance-learning/).
These alternative methods for monitoring absenteeism and school withdrawal are designed to be flexible enough to accommodate COVID-related changes in school operations. As planned for this data collection, school nurses will follow up on children to identify and document chronic health problems that are interfering with school participation. This project is not designed to monitor student absences related to COVID-19, unless COVID-19 leads to chronic health consequences.
Data on Technical Assistance and Training Needs
Considering the variability between states in data collection infrastructure and practices and knowledge about ME/CFS, school nurses in the pilot sites will be asked to complete an online survey (Attachment C) on technical assistance and training needs before the pilot sites start data collection on health-related chronic absenteeism and school withdrawal. The questionnaire is designed to provide a baseline understanding of the ME/CFS knowledge and information technology status and needs of each site. This information will be used by NASN to provide continued nursing education and technical assistance for school nurses and leaders to improve critical skills related to data collection and use. Without this information, NASN would not be able to provide appropriate support for the pilot sites throughout the surveillance process.
The state data coordinator in each of the fifty states will be asked to complete a second online survey (Attachment D). This survey gathers feedback on the status of, and training and technical assistance needs relevant to the ongoing school surveillance of chronic health conditions, as well as the level of ME/CFS knowledge, awareness of activities related to ME/CFS and chronic absenteeism, and availability to participate in webinars. The data will be used to identify planning considerations for scaling up the piloted approach nationally, should the pilot be successful and national rollout be determined to be feasible. Without these data from all fifty states, NASN would be missing critical information needed for planning for training and technical assistance related to surveillance of chronic conditions, including ME/CFS, among U.S. schoolchildren, to support a future national rollout of the piloted approach.
Data on Feasibility and Usability of the Active Surveillance Process and Data Collection Tool
Data on the feasibility and usability of the surveillance process and data entry platform will be collected through three separate focus groups (each lasting about 1.5 hours) with the school nurses from the six pilot sites, conducted by a NASN facilitator. The interview guides for focus group interviews (Attachment E), are semi-structured, allowing for in-depth discussions. These questions are designed to elicit information about school nurses’ experience working with students and entering data into the platform, how well the surveillance data entry platform guided their practice, and suggestions for improving this tool. Information collected through these semi-structured interviews may be used to gain insight into the experience school nurses have with the active surveillance process and to inform changes that may be needed in tools and procedures. The potential changes, best practices, and lessons learned will be documented so that they may be used to inform a national roll-out of the piloted approach, if the pilot is successful and national scale-up is determined to be feasible.
School district administrators for each of the six pilot sites will be asked to complete the “School District Feedback Form” (Attachment F). This form is designed to evaluate the feasibility of this school nurse-led surveillance process from the perspective of the school district representative. The answers will be used to adjust and revise the process. The form includes inquiries about the impact on the chronic absenteeism surveillance on the school districts, any changes that are needed for tracking purposes, and general insights and feedback. Successful school-based health surveillance requires the engagement of multiple organizational levels. Without the data from this form on the impact on school districts, NASN will be missing key considerations related to scaling up the piloted approach.
Only the minimum amount of de-identified aggregate data about children who are chronically absent or withdrawn will be collected. An online electronic database platform will be used to facilitate entry of surveillance data by the school nurses, and online surveys will be used to assess the feasibility of the piloted approach and to collect information about training technical assistance needs. Provision of electronic data collection software, training, and technical assistance will help to reduce the burden of data collection on school nurse respondents involved in this pilot project. Automated edit checks will be built into the computer software programs as an additional quality control measure. This will eliminate the need for data cleaning associated with data entry and subsequent errors and will result in a reduction in the time to transfer the data itself. The use of an online survey may reduce the burden on respondents by improving comprehension and reducing the amount of time needed to complete the assessment.
We reviewed currently funded programs and did not identify potential areas of duplication. We are not aware of any department or agency that collects or maintains data on ME/CFS and other chronic conditions and school absenteeism. School nurses have collected information through NASN on other chronic diseases contributing to school absenteeism and school withdrawal, but not in a standardized way. They have not reported the data to the federal government, and ME/CFS has not previously been among the chronic diseases tracked.
This data collection will not involve small businesses. All six pilot site schools are public schools.
The frequency of data collection from school nurses in the six pilot districts is quarterly (for a total of five quarters) for data on health-related chronic absenteeism and school withdrawal, and a total of three times for focus groups on the feasibility and usability of the active surveillance process and data collection tool. Collecting this information less frequently would not allow evaluation, refinement, and re-evaluation of the data collection methods and tool. All other data collection will occur only one time.
This request fully complies with the regulation 5 CFR 1320.5.
A. The Federal Register notice was published for this collection on August 23, 2019, Vol. 84, No. 164, pp. 44308. No public comments were received.
B. The following ME/CFS experts have been consulted regarding this project:
Lucinda Bateman M.D.
Bateman Horn Center
24 South 1100 East, Suite 205
Salt Lake City, UT 84102
Email: [email protected]
Tel.: 801-359-7400
Nathan Holladay MD, PhD865 E 4800 S Ste 160
Murray, UT 84107
Email: nbholladay@gmail
Tel.: 385-251-6028
Ruth Walkotten DO, MA
Walkotten Wellness Center
2947 Russell Rd.
Muskegon, MI 49445
Tel.: (231) 733-1989
Edward Conley MD
G-3494 Beecher Rd Suite A
Flint, MI 48532
Email: [email protected]
Tel.: 810.230.8677
No major unresolved problems were identified.
The district school nurses will collect aggregated, de-identified data on the students’ health and reasons for absenteeism/school withdrawal and report the data to NASN. NASN will collect the school nurse respondent’s personal identifiers (first name, last name, school district) to allow for NASN’s potential future analysis by school district. Before sending the data to CDC, NASN will remove the personal and school district identifiers. NASN will be conducting focus group interviews with the six pilot district school nurses and will know the identity of the nurses who participate, and which comments/responses are associated with a particular school nurse. However, data from focus group interviews will be reported to CDC without identifiers.
Focus group interviews will be conducted three times during the pilot data collection, to gather information to guide local implementation of the pilot. The focus group conversations will be recorded by a notetaker. The notetaker may use a recording device as a backup for transcription purposes only. When the focus groups have been completed, the notes/transcriptions will be stored at NASN. The notes (or transcription) will be qualitatively analyzed for themes without the use of qualitative analysis software. Only authorized persons will have access to the notes, and they will not be transmitted to CDC.
School district administrators for each of the six pilot sites will be asked to complete the “School District Feedback Form” (Attachment F). NASN will be aware of the identity of the respondents. However, their responses will be shared with CDC only in aggregate form from all districts; no personally identifying information will be reported.
NASN will transmit summaries of data from this pilot project without identifiers to the CDC team by email.
A Privacy Impact Checklist is included as part of this submission (Attachment G).
Institutional Review Board (IRB)
NASN received IRB approval of the protocol with modifications on March 5, 2019, through Advarra IRB (Attachment H).
Justification for Sensitive Questions
No sensitive questions are to be included.
A. Estimated Annualized Burden Hours
The estimate of annualized burden hours for this sub collection is 228 hours; details are provided in exhibit 12.A. Six frontline school nurses are expected to assemble and enter data into the data collection tool five times and participate in three focus groups and one online survey. Assembling and entering data is estimated to take five hours each of the five times in addition to a demographic form submitted one time yearly that will take six hours to complete; the focus groups are expected to last for 1.5 hours each; and the online survey is estimated to take 10 minutes. Six school district representatives are expected to complete the School District Feedback Form once, which is estimated to take 15 minutes. Fifty state data coordinators are expected to complete a one-time webinar feedback survey, which is estimated to take 15 minutes.
Exhibit 12.A Estimate of Annualized Burden Hours
Type of Respondent |
Form Name |
No. of Respondents |
No. Responses per Respondent |
Avg. Burden per response (in hrs.) |
Total Burden (in hrs.) |
Frontline School Nurse |
Electronic Platform Quarterly Chronic Absenteeism Data Reporting Form |
6 |
5 |
5 |
150 |
Frontline School Nurse |
Demographic Data Collection Points |
6 |
1 |
6 |
36 |
Frontline School Nurse |
Pilot Site Baseline Survey |
6 |
1 |
10/60 |
1 |
State Data Coordinators |
Webinar 1 Feedback Form |
50 |
1 |
15/60 |
12.5 |
Frontline School Nurse |
Question Guide for Face to Face Evaluation Interviews |
6 |
3 |
1.5 |
27 |
School District Representative |
School District Feedback Form |
6 |
1 |
15/60 |
1.5 |
TOTAL |
|
|
|
|
228 |
B. Estimated Annualized Burden Costs
The annualized cost to respondents for the burden hours is estimated to be $7,879; details are provided in Exhibit 12.B.
The estimates of hourly wages were obtained from the Department of labor (Bureau of Labor Statistics Wage Data (https://www.bls.gov/ooh/healthcare/registered-nurses.htm). The median hourly wage for Registered Nurses is $34.48. Both Frontline School Nurses and State Data Coordinators correspond with this profile.
U.S. Bureau of Labor Statistics website states that Education Administrators for elementary and secondary schools have an annual median wage of $95,310. (https://www.bls.gov/oes/2018/may/oes119032.htm#(4)). An hourly wage is not provided on the website. Assuming a 40-hour work week, 52 weeks/year, the mean hourly wage is estimated as $45.82. School District Representatives correspond with this profile.
Exhibit 12.B Estimate of Annualized Burden Costs
Type of Respondent |
Form Name |
Total Burden Hours |
Hourly Wage Rate Dept of Labor |
Total Respondent Costs |
Frontline School Nurse |
Electronic Platform Quarterly Chronic Absenteeism Data Reporting Form |
150 |
$34.48/hr |
$5,172 |
Frontline School Nurse |
Demographic Data Collection Points |
36 |
$34.48/hr |
$1,241 |
Frontline School Nurse |
Pilot Site Baseline Survey |
1 |
$34.48/hr |
$35 |
State Data Coordinators |
Webinar 1 Feedback Form |
12.5 |
$34.48/hr |
$431 |
Frontline School Nurse |
Question Guide for Face to Face Evaluation Interviews |
27 |
$34.48/hr |
$931 |
School District Representative |
School District Feedback Form |
1.5 |
$45.82/hr |
$69 |
TOTAL |
|
|
|
$7,879 |
There are no costs to respondents other than their time to participate.
The annualized cost of this project is estimated to be $193,980. This includes one GS-13 Medical Officer’s time. The source for salary is fedsdatacenter.com. Contract costs comprise the remainder of the estimated annualized cost to the government.
Exhibit 14.A Estimate of Annualized Cost to Government per Activity
Expense Type |
Expense Explanation |
Estimated Annualized Cost |
Direct Costs to the Federal Government |
CVDB: Personnel Medical Officer-13 $102,195 – 1 officer at 10% effort
|
$10,220
|
Contract |
Contract funds to NASN |
$183,760 |
TOTAL |
|
$193,980 |
Not applicable – request is for a sub-collection under a generic approval.
All data collection will be completed by January 31, 2023, the expiration date for OMB No. 0920-1154. The following is a brief overview of the timeline for the proposed pilot project.
Exhibit 16.A Project Time Schedule
Activity |
Time Schedule |
Data/information collection and reporting |
0–16 months after OMB approval |
Data cleaning and analysis |
10–12 months after OMB approval |
Preparation of final report(s) / manuscript |
12-24 months after OMB approval |
Manuscript publication |
24-36 months after OMB approval |
The display of the OMB Expiration Date is not inappropriate.
There are no exceptions to the certification.
19. References
1. IOM (Institute of Medicine). 2015. Beyond myalgic encephalomyelitis / chronic fatigue syndrome: Redefining an illness. Washington, DC: The National Academies Press. Bauer L, Liu P, Schanzenback DW, and Shambaugh J.Reducing Chronic Absenteeism under the Every Student Succeeds Act. Brookings Institute. [Online] April 2018. [Cited: Feb 26, 2020.] https://www.brookings.edu/wp-content/uploads/2018/04/reducing_chronic_absenteeism_under_the_ every_student_succeeds_act2.pdf.
2. Jason L, Torres-Harding S, Njok M. The face of CFS in the US. CFIDS Chronicle 2006, 16-21. https://www.researchgate.net/publication/236995875_The_Face_of_CFS_in_the_US.
3. Jason L, Richman JA, Rademaker AW, et al. A community-based study of chronic fatigue syndrome. Archives of Internal Medicine 1999, 159(18):2129-2137.
4. Chalder T, Goodman R, Wessely S, Hotopf M, Meltzer H. Epidemiology of chronic fatigue syndrome and self-reported myalgic encephalomyelitis in 5-15 year olds: cross sectional study. BMJ 2003, 327:654–5.
5. Bell K, Cookfair D, Bell DS, Reese P, Cooper L. Risk factors associated with chronic fatigue syndrome in a cluster of pediatric cases. Reviews of Infectious Diseases 1991;13(Suppl 1):S32-8.
6. Geraghty KJ, Adeniji C. The importance of accurate diagnosis of ME/CFS in children and adolescents: A commentary. Front. Pediatr. 2019, 6:435.
7. Chronic Fatigue Syndrome Advisory Committee. Transcript for January 13, 2017 CFSAC meeting. [Online] Jan 13, 2017. [Cited: March 3, 2020.] https://wayback.archive-it.org/org-745/20181018061119/https://www.hhs.gov/sites/default/files/january-13-2017-transcript.pdf.
8. National Association of School Nurses. Environmental Scan: Current School-Based Active Surveillance Efforts. 2018.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Samuel, Lee (CDC/OID/NCEZID) |
File Modified | 0000-00-00 |
File Created | 2021-01-13 |