I did some experiments with sparse coding methods in order to remove noise from CW audio signals.

There is a signal processing algorithm called Matching Pursuit that decomposes any signal into linear expansion of waveforms that are selected from a redundant dictionary of functions. A matching pursuit isolates the signal structures that are coherent with respect to given dictionary. I spent some time trying to find a good set of dictionary "atoms" aka wavelets that would work well with noisy CW signals.

As far as I know you can expand a function in a series of orthagonal functions usually sine waves, but not necessarily, Some times a set of Bessel functions , or Walsh functions suits better.

Especially Walsh functions may be interesting in CW decoding

The mathematics behind the Matching Pursuit algorithm is explained in Mallat's paper: blanche.polytechnique.fr/~mallat/papiers/MallatPursuit93.pdf. As far as I understand the algorithm works also with a smaller dictionary of time-frequency atoms (like Gabor functions) but getting small enough error term becomes increasingly difficult. One of the parameters you can assign for MPTK toolkit matching pursuit decomposition software is the target signal-to-noise ratio. It may take either much longer time or you don't reach your target SNR without having some redundant functions in the dictionary of time-frequency atoms.

Walsh functions might be interesting to try out and see if they could improve the results. Perhaps some sort of combination of Walsh functions and Morlet wavelets might also work. I did some experiments earlier with modified Morlet wavelets http://ag1le.blogspot.com/2012/06/morse-code-detection-using-modified.html and for very low SNR cases they seemed to improve ability to extract CW signals. Matching Pursuit algorithm provides a great framework to find all CW signals in the baseband.

The nice thing about this MPTK toolkit is that it is written in C++ and is very fast. Right now it works with WAV files but it may be possible to tweak it so that it could be integrated with FLDIGI or something similar. Being able to automatically extract all (even low SNR) CW signals from baseband would be a nice feature.

If you want to confirm the results yourself I have posted links where you can download and compile the software, provided example WAV file and even what dictionary I was using to produce the results above. I think this might be a useful tool for folks who are interested in latest noise reduction techniques and posted this to find out if anybody else in this ham radio community has been playing with MPTK toolkit. The dictionary "atoms" I used were clearly not optimized very well for CW.

With ever increasing computing power and more capable software defined radios the good old CW mode might also benefit of these fairly new signal processing methods, right?

1: Sparse code consisting of three symbols: "Dit (including empty following dit-space)", "Dash (including empty following dit-space)", and "intercharacter space" (actually one dit short because taken up by previous dit or dah). This is how, for example, some old reperferators stored morse code onto paper tape. It may have been called "Wheatstone's Method" used for dense efficient coding onto transatlantic cables.

2: Sparse code consisting of 26 letters, 10 numbers, and a couple punctuation and common prosigns.

I have done some of my own testing where I take wide-bandwidth audio recordings of 160M contest CW, and convolute them with "well known callsigns". In this case the "alphabet" of well-known callsigns are the most well-known 160M regulars. This is able to pull callsigns well out of the noise better than the human ear, but the computational power to do this (especially at varying WPM's and varying center frequencies) is nowhere near real-time.

I have also done the above with contest RTTY (and a different "alphabet" of well-known RTTY callsigns).

I think CW skimmer does something similar, but with a "broader" and "better" choice of callsigns and an algorithm that deals with varying center frequencies and varying WPM's. CW Skimmer obviously has very good callsign knowledge, and is working way above the "dit" and "dah" and "character" level, probably working much more like a human decodes iffy CW via probabalistic decision tree and pattern recognition rather than my brute force attempts. I observe it can also back up the decision tree and correct a previous decoding when it is obvious there's a better choice given by later data. Very impressive!

I have done some of my own testing where I take wide-bandwidth audio recordings of 160M contest CW, and convolute them with "well known callsigns". In this case the "alphabet" of well-known callsigns are the most well-known 160M regulars. This is able to pull callsigns well out of the noise better than the human ear, but the computational power to do this (especially at varying WPM's and varying center frequencies) is nowhere near real-time.

I have also done the above with contest RTTY (and a different "alphabet" of well-known RTTY callsigns).

1: Sparse code consisting of three symbols: "Dit (including empty following dit-space)", "Dash (including empty following dit-space)", and "intercharacter space" (actually one dit short because taken up by previous dit or dah).

2: Sparse code consisting of 26 letters, 10 numbers, and a couple punctuation and common prosigns.

Thanks for sharing this idea. To express this "Dit" function in a formula, something like this:

Based on the suggestion above I used new atoms in the dictionary and the results sound much better than in the original experiment. I also tested the run time of this new algorithm - you can process WAV file in almost real time with a ThinkPad X301 laptop.

Anybody else willing to give this a test spin? The software is open source and available for Windows, Linux and Mac platforms.

Looks like significant noise reduction but it also took out some weak signals.

Try to download the audio files and listen the difference. I think it is possible to improve this method even further. I spent some 3 hours experimenting with different atoms and parameters and I am pretty sure that we can do much better.Compared to traditional band pass filtering this method finds and extracts (almost) all CW signals in the base band while reducing noise. Pretty cool, eh?

Copyright 2000-2017 eHam.net, LLC
eHam.net is a community web site for amateur (ham) radio operators around the world.
Contact the site with comments or questions.
WEBMASTER@EHAM.NETSite Privacy Statement