Abstract

Information theory originated in a series of papers by Shannon in 1948, where he sketched an important theoretical framework for characterizing information sources and communication channels. Subsequently, information theory has primarily studied theoretical problems based on known source and channel properties. Although Shannon’s theory was in spirit applicable to a wider class of problems, information theory has paid less attention to analysis problems, where an observed signal is accessible to the researcher but the underlying true mechanism is unknown, e.g., statistical estimation and hypothesis testing problems. This paper reviews the properties of informationentropy and common techniques for estimating and interpreting it. The performance and pitfalls of common entropy estimators are illustrated with examples. Specifically, the naive technique of substituting relative frequencies or empirical distribution into the definition of entropy is shown to produce inaccurate estimates. An alternative technique for analyzing observed data, the sliding window match length estimator, is relatively simple to implement, can be intuitively understood, and is mathematically well supported.