Some Observations

• The best sequence of yes-or-no questions can be constructed via a Huffman tree.
• For this example, we can summarize the results as follows:
  

  Track	Probability	Answers (yes=1, no=0)	Number of bits	$-\log_2$ (Probability)
  1	$\frac{1}{8}$	000	3	3
  2	$\frac{1}{8}$	001	3	3
  3	$\frac{1}{8}$	010	3	3
  4	$\frac{1}{8}$	011	3	3
  5	$\frac{1}{16}$	1000	4	4
  6	$\frac{1}{16}$	1001	4	4
  7	$\frac{1}{16}$	1010	4	4
  8	$\frac{1}{16}$	1011	4	4
  9	$\frac{1}{16}$	1100	4	4
  10	$\frac{1}{16}$	1101	4	4
  11	$\frac{1}{16}$	1110	4	4
  12	$\frac{1}{16}$	1111	4	4
  	Sum: 1		Average: 3.5	Average: 3.5

• On average 3.5 questions are required to find out the track.
• Notice that $-\log_2$ (Probability) predicts the number of bits accurately.
• We can say, by finding out that the train leaves on track 3 you have gained 3 bits of information.
• On the other hand, finding the train leave on track 5 provides 4 bits of information.
• More information is conveyed when an unlikely event occurs.
• The average of the second column from the right is the avearge number of questions required.
• The average of the rightmost column in the table above is called the entropy.
• Remember that the averages are computed as a weighted sum with weights given by the probailities.
• I.e., the entropy is computed as:
  
  $$H = \mbox{Probability(1)} \cdot (-\log_2 \mbox{Probability(1)... ...mbox{Probability(12)} \cdot (-\log_2 \mbox{Probability(12)}).$$
  
  

• The entropy is always smaller than (or at most equal to) the average number of questions (bits) to determine the answer.