Lecture 11: Shannon vs. Hamming

In today’s lecture, we redid the last part of Shannon’s proof for the capacity theorem for . We then looked at the questions that were left open by Shannon’s proof and compared them to the corresponding questions in Hamming’s world. We then did a quick comparison of the salient points of Shannon’s and Hamming’s theory. We then started to look at some bounds on the rate of a code (as a function of the relative distance). At the end of the class we proved the Singleton bound. Next class, we will look a particular family of codes called Reed-Solomon codes that meet the Singleton bound.

In the lecture, I mentioned the following paper: Error-free coding by Elias. This paper gives a very hands down construction of a code with rate that can be used for reliable communication over for some . You might also want to look at the lecture notes from Venkat Guruswami‘s coding theory course that talks about this paper. I encourage you to read the paper: the construction is very nice and we have covered most of the bakcground needed for the result. I also encourage you to read the introduction of the paper: it’s from an era where introductions were written in a different manner than today 🙂