Description

Task specific mailing list

All discussions take place on the MIREX "EvalFest" list. If you have an question or comment, simply include the task name in the subject heading.

Data

A collection of 100 clips of recorded pop music (vocals plus music) are used to evaluate the singing voice separation algorithms (these are the hidden parts of the iKala dataset). If your algorithm is a supervised one, you are welcome to use the public part of the iKala dataset for training.

Evaluation

For evaluation we use Vincent et al.'s (2012) Source to Distortion Ratio (SDR), Source to Interferences Ratio (SIR), and Sources to Artifacts Ratio (SAR), as implemented by bss_eval_sources.m in BSS Eval Version 3.0. Everything will be normalized to enable a fairer evaluation. More specifically, their function will be invoked as follows:

The final scores will be determined by the mean over all 100 clips (note that GSIR and GSAR are not normalized):

,

,

.

In addition, sd, min, max and median will also be reported.

Submission format

Participants are required to submit an entry that takes in an input filename (full native pathname ending in *.wav) and an output directory as arguments. The entries must send their voice-separated outputs to *-voice.wav and *-music.wav under the output directory. For example:

Packaging submissions

Be sure to follow the MIREX 2014 Submission Instructions. For example, under Very Important Things to Note, Clause 6 states that if you plan to submit more than one algorithm or algorithm variant to a given task, EACH algorithm or variant needs its own complete submission to be made including the README and binary bundle upload. Each package will be given its own unique identifier. Tell us in the README the priority of a given algorithm in case we have to limit a task to only one or two algorithms/variants per submitter/team. [Note: our current limit is two entries per team.]

All submissions should include a README file including the following the information:

Command line calling format for all executables and an example formatted set of commands

Number of threads/cores used or whether this should be specified on the command line

Expected memory footprint

Expected runtime

Approximately how much scratch disk space will the submission need to store any feature/cache files?