Then the three weighings give the ternary expansion of the index of the odd coin.

If the expansion is the same as the expansion in the matrix, it indicates that the coin is heavier. If the expansion is of the opposite sign, the coin is lighter.

Why does this scheme work? It is a single error correcting Hamming code for the ternary alphabet discussed in Section 8. Here are some details. First note a few properties of the matrix above that was used for the scheme.

All the columns are distinct and no two columns add to 0,0,0. Also if any coin 14 Entropy, Relative Entropy and Mutual Information is heavier, it will produce the sequence of weighings that matches its column in the matrix.

If it is lighter, it produces the negative of its column as a sequence of weighings. Combining all these facts, we can see that any single odd coin will produce a unique sequence of weighings, and that the coin can be determined from the sequence. One of the questions that many of you had whether the bound derived in part a was actually achievable. For example, can one distinguish 13 coins in 3 weighings?

No, not with a scheme like the one above.

Elements of Information Theory 2nd Edition PDF

Yes, under the assumptions under which the bound was derived. The bound did not prohibit the division of coins into halves, neither did it disallow the existence of another coin known to be normal.

Under both these conditions, it is possible to nd the odd coin of 13 coins in 3 weighings. You could try modifying the above scheme to these cases.

Drawing with and without replacement. An urn contains r red, w white, and b black balls.

Information theory

Which has higher entropy, drawing k 2 balls from the urn with replacement or without replacement? Set it up and show why.

There is both a hard way and a relatively simple way to do this. Solution: Drawing with and without replacement. Intuitively, it is clear that if the balls are drawn with replacement, the number of possible choices for the i -th ball is larger, and therefore the conditional entropy is larger.

But computing the conditional distributions is slightly involved. It is easier to compute the unconditional entropy.

Institutionen för systemteknik (ISY)

With replacement. In this case the conditional distribution of each draw is the same for every draw. Thus r with prob. Thus the unconditional entropy H X i is still the same as with replacement. The conditional entropy H X i Xi1 ,. A metric.

Elements of Information Theory

By problem 2. Turing 's information unit, the ban , was used in the Ultra project, breaking the German Enigma machine code and hastening the end of World War II in Europe. Shannon himself defined an important concept now called the unicity distance.

Based on the redundancy of the plaintext , it attempts to give a minimum amount of ciphertext necessary to ensure unique decipherability.

Information theory leads us to believe it is much more difficult to keep secrets than it might first appear. A brute force attack can break systems based on asymmetric key algorithms or on most commonly used methods of symmetric key algorithms sometimes called secret key algorithms , such as block ciphers.

The security of all such methods currently comes from the assumption that no known attack can break them in a practical amount of time. Information theoretic security refers to methods such as the one-time pad that are not vulnerable to such brute force attacks.

In such cases, the positive conditional mutual information between the plaintext and ciphertext conditioned on the key can ensure proper transmission, while the unconditional mutual information between the plaintext and ciphertext remains zero, resulting in absolutely secure communications.

In other words, an eavesdropper would not be able to improve his or her guess of the plaintext by gaining knowledge of the ciphertext but not of the key. However, as in any other cryptographic system, care must be used to correctly apply even information-theoretically secure methods; the Venona project was able to crack the one-time pads of the Soviet Union due to their improper reuse of key material.

Pseudorandom number generation[ edit ] Pseudorandom number generators are widely available in computer language libraries and application programs. They are, almost universally, unsuited to cryptographic use as they do not evade the deterministic nature of modern computer equipment and software.