You Snooze, You Win: the PhysioNet/Computing in Cardiology Challenge 2018

If you have any questions or comments regarding this challenge,
please post it directly in our Community
Discussion Forum. This will increase transparency (benefiting all
the competitors) and ensure that all the challenge organizers see your
question.

Introduction:

At the end of last year, American scientists Jeffrey Hall, Michael
Rosbash and Michael Young received a Nobel Prize in Physiology “for
their discoveries of molecular mechanisms controlling the circadian
rhythm"— the mechanism that regulates sleep (Osborn, 2017). The
precise reasons why humans sleep (and even how much sleep we need)
remains a topic of scientific inquiry. Contemporary theorists indicate
that sleep may be responsible for learning and/or the clearing of
neural waste products (Ogilvie and Patel, 2017).

While the precise reasons why we sleep are not perfectly understood,
there is consensus on the importance of sleep for our overall health,
and well-being. Inadequate sleep is associated with a wide range of
negative outcomes including: impaired memory and learning, obesity,
irritability, cardiovascular dysfunction, hypotension, diminished
immune function (Harvard Medical School, 2006), depression (Nutt et
al, 2008), and quality of life (Lee, 2009). Further studies even
suggest causal links between quality of sleep, and important
outcomes including mental health.

It follows that improving the quality of sleep could be used to
improve a range of societal health outcomes, more generally. Of
course, the treatment of sleep disorders is necessarily preceded by
the diagnosis of sleep disorders. Traditionally, such diagnoses are
developed in sleep laboratory settings, where polysomnography, audio,
and videography of sleeping subject may be carefully inspected by
sleep experts to identify potential sleep disorders.

One of the more well-studied sleep disorders is Obstructive Sleep Apnea
Hypopnea Syndrome (or simply, apnea). Apneas are characterized by a
complete collapse of the airway, leading to awakening, and consequent
disturbances of sleep. While apneas are arguably the best understood
of sleep disturbances, they are not the only cause of
disturbance. Sleep arousals can also be spontaneous, result from teeth
grinding, partial airway obstructions, or even snoring. In this year's
PhysioNet Challenge we will use a variety of physiological signals,
collected during polysomnographic sleep studies, to detect these
other sources of arousal (non-apnea) during sleep.

Challenge Data

Data for this challenge were contributed by the Massachusetts General
Hospital’s (MGH) Computational Clinical Neurophysiology Laboratory
(CCNL), and the Clinical Data Animation Laboratory (CDAC). The dataset
includes 1,985 subjects which were monitored at an MGH sleep
laboratory for the diagnosis of sleep disorders. The data were
partitioned into balanced training (n = 994), and test sets (n = 989).

The sleep stages of the subjects were annotated by clinical staff at
the MGH according to the American Academy of Sleep Medicine (AASM)
manual for the scoring of sleep. More specifically, the following six
sleep stages were annotated in 30 second contiguous
intervals: wakefulness, stage 1, stage 2, stage 3, rapid eye
movement (REM), and undefined.

The subjects had a variety of physiological signals recorded as they
slept through the night including: electroencephalography (EEG),
electrooculography (EOG), electromyography (EMG), electrocardiology
(EKG), and oxygen saturation (SaO2). Excluding SaO2, all signals were
sampled to 200 Hz and were measured in microvolts. For analytic
convenience, SaO2 was resampled to 200 Hz, and is measured as a
percentage.

Objective of the Challenge

The goal of the challenge is use information from the available
signals to correctly classify target arousal regions. For the purpose
of the Challenge, target arousals are defined as regions where
either of the following conditions were met:

From 2 seconds before a RERA arousal begins, up to 10 seconds after it ends or,

From 2 seconds before a non-RERA, non-apnea arousal begins, up to 2 seconds after it ends.

Please note that regions falling within 10 seconds before or after a
subject wakes up, has an apnea arousal, or a hypopnea
arousal will not be scored for the Challenge.

The Challenge data
repository contains two directories (training
and test) which are each approximately 135 GB in
size. Each directory contains one subdirectory per subject
(e.g. training/tr03-0005). Each subdirectory contains
signal, header, and arousal files; for example:

tr03-0005.mat: a Matlab V4 file containing the
signal data.

tr03-0005.hea: record header file - a text file
which describes the format of the signal data.

Submitting your Entry

Participants should use the provided signal and arousal data to
develop a model that classifies test-set subjects. More specifically,
for each subject in /test, participants must generate two outputs:

A .csv file containing a list of arousal onset and offset times
in samples (200 Hz), such as the following:

arousal_onset,arousal_offset
101,110
1034,1134
21344,21788

A .vec text file that describes the probability of arousal at each sample, such as:

0.01
0.00
0.02
0.05

The names of the generated annotation files
should match the name of the test subject. For instance,
test/te09-0094.mat should have corresponding files
named annotations/te09-0094.csv
and annotations/te09-0094.vec.

Scoring

Your final algorithm will only be graded for its binary
classification performance on target arousal and non-arousal regions
(designated by +1 and 0
in teNN-NNNN-arousals.mat).

Sample Submission

In the coming days, we will release a baseline algorithm, and
accompanying scripts for the challenge. Please check for updates.

Rules and Deadlines

Entrants may have an overall total of up to three submitted entries
over both the unofficial and official phases of the competition
(see Table 2). Following submission, entrants
will receive an email confirming their submission and reporting how
well their arousal annotations match those of the held-out test set.

All deadlines occur at noon GMT (UTC) on the dates mentioned
below. If you do not know the difference between GMT and your local
time, find out what it is before the deadline!

Table 2: Rules and deadlines.

Start at noon GMT on

Entry limit

End at noon GMT on

Unofficial Phase

15 February

1

9 April

[Hiatus]

9 April

0

15 April

Official Phase

16 April

2

1 September

All official entries must be received no later than noon
GMT on Friday, 1 September 2018. In the interest of fairness
to all participants, late entries will not be accepted or
scored. Entries that cannot be scored (because of missing components,
improper formatting, or excessive run time) are not counted against
the entry limits.

To be eligible for the open-source award, you must do all of the following:

Submit at least one open-source entry that can be scored before the Phase I deadline (noon GMT on Sunday, 9 April 2018).

Submit at least one entry during the second phase (between noon GMT on Sunday, 16 April 2018 and noon GMT on Friday, 1 September 2018). Only your final entry will count for ranking.

Entering an Abstract to CinC: Submit an acceptable abstract (about 299 words) on your work on the Challenge to Computing in Cardiology no later than 15 April 2018. Include the overall score for your Phase I entry in your abstract. Please select “PhysioNet/CinC Challenge” as the topic of your abstract, so it can be identified easily by the abstract review committee. You will be notified if your abstract has been accepted by email from CinC during the first week in June.

Submit a full (4-page) paper on your work on the Challenge to CinC
no later than the deadline of conference paper submission.

Attend CinC 2018 (23-26 September 2018) in
Maastricht and present your work there.

Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after CinC 2018 has taken place, so the competitors are able to discuss the results in a single forum. We expect a special issue from the journal Physiological Measurement to follow the conference and encourage all entrants (and those who missed the opportunity to compete or attend CinC 2018) to submit extended analysis and articles to that issue, taking into account the publications and discussions at CinC 2018.

Abstract Submission

Don't forget you must also submit an abstract to Computing in Cardiology before the imminent deadline on the 15th of April and attend the conference in September where we will announce the winner. See cinc.org. Note that your methods and score, and therefore your abstract, will most likely change by the end of summer. That is acceptable and expected. If you don't submit now though, you won't reserve your place to discuss your methods at the conference. Please note that abstracts should include your current methods and score. We encourage you to include cross validation stats on the training data too to show the reviewers you know what you are doing. In many ways toys is far more important than your test set score we give you at this stage, so if you have a poor challenge score but a great cross validated score then you are well on your way!

Incoherent or information poor abstracts are unlikely to be accepted as they indicate low quality approaches and an inability to communicate ideas. A well thought out abstract indicates a high likelihood of a good presentation and high quality scientific approach. Do not include a description of the competition in the abstract (it's very clear what the competition is about.) Focus on your methods and results.

Please make sure you select the PhysioNet Challenge category in the abstract submission. If you don't, it may get reviewed outside of the challenge track and get rejected.

Please give your abstract and paper a title distinct from that of the Challenge itself (“You Snooze, You Win: The PhysioNet/Computing in Cardiology Challenge 2018.”)

After the Challenge

As is customary, we hope to run a special issue in Physiological Measurement with a closing date of 31 January 2019. We will therefore encourage competitors (and non-competitors) to submit updates and further reworks based on the Challenge after the award ceremony at the Computing in Cardiology Conference in Maastricht in September.

Obtaining complimentary MATLAB licenses

The MathWorks has kindly decided to sponsor Physionet’s 2018 Challenge providing licenses. The MathWorks is offering to all teams that wish to use MATLAB, complimentary licenses. User can apply for a license and learn more about MATLAB support through The Mathworks’ PhysioNet Challenge link. If you have questions or need technical support, please contact The MathWorks at academicsupport@mathworks.com.

Community Discussion Forum

Note: Please check the FAQ below before posting on the Forum.

To display the Community Forum, please enable JavaScript in your browser.