Information content

In information theory, information content, self-information, or surprisal of a random variable or signal is the amount of information gained when it is sampled. Formally, information content is a random variable defined for any event in probability theory regardless of whether a random variable is being measured or not.

Information content is expressed in a unit of information, as explained below. The expected value of self-information is information theoretic entropy, the average amount of information an observer would expect to gain about a system when sampling the random variable.[1]


Given a random variable   with probability mass function  , the self-information of measuring   as outcome   is defined as  [2]

Broadly given an event   with probability  , information content is defined analogously:


In general, the base of the logarithmic chosen does not matter for most information-theoretic properties; however, different units of information are assigned based on popular choices of base.

If the logarithmic base is 2, the unit is named the Shannon but "bit" is also used. If the base of the logarithm is the natural logarithm (logarithm to base Euler's number e ≈ 2.7182818284), the unit is called the nat, short for "natural". If the logarithm is to base 10, the units are called hartleys or decimal digits.

The Shannon entropy of the random variable   above is defined as


by definition equal to the expected information content of measurement of  .[3]:11[4]:19-20


Antitonicity for probabilityEdit

For a given probability space, measurement of rarer events will yield more information content than more common values. Thus, self-information is antitonic in probability for events under observation.

  • Intuitively, more information is gained from observing an unexpected event—it is "surprising".
    • For example, if there is a one-in-a-million chance of Alice winning the lottery, her friend Bob will gain significantly more information from learning that she won than that she lost on a given day. (See also: Lottery mathematics.)
  • This establishes an implicit relationship between the self-information of a random variable and its variance.

Additivity of independent eventsEdit

The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics, and sigma additivity in particular in measure and probability theory. Consider two independent random variables   with probability mass functions   and   respectively. The joint probability mass function is


because   and   are independent. The information content of the outcome   is

See § Two independent, identically distributed dice below for an example.

The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.


This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined by Myron Tribus in his 1961 book Thermostatics and Thermodynamics.[5][6]

When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization.

Self-information is an example of a proper scoring rule.[clarification needed]


Fair coin tossEdit

Consider the Bernoulli trial of tossing a fair coin  . The probabilities of the events of the coin landing as heads   and tails   (see fair coin and obverse and reverse) are one half each,  . Upon measuring the variable as heads, the associated information gain is

so the information gain of a fair coin landing as heads is 1 shannon.[2] Likewise, the information gain of measuring   tails is

Fair dice rollEdit

Suppose we have a fair six-sided die. The value of a dice roll is a discrete uniform random variable   with probability mass function

The probability of rolling a 4 is  , as for any other valid roll. The information content of rolling a 4 is thus
of information.

Two independent, identically distributed diceEdit

Suppose we have two independent, identically distributed random variables   each corresponding to an independent fair 6-sided dice roll. The joint distribution of   and   is


The information content of the random variate   is

just as

as explained in § Additivity of independent events.

Information from frequency of rollsEdit

If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables


for  , then   and the counts have the multinomial distribution


To verify this, the 6 outcomes   correspond to the event   and a total probability of 1/6. These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other   combinations correspond to one die rolling one number and the other die rolling a different number, each having probability 1/18. Indeed,  , as required.

Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events   and  for  . For example,  and  .

The information contents are

Let   be the event that both dice rolled the same value and   be the event that the dice differed. Then   and  . The information contents of the events are


Information from sum of dieEdit

The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable   has probability mass function  , where   represents the discrete convolution. The outcome   has probability  . Therefore, the information asserted is


General discrete uniform distributionEdit

Generalizing the § Fair dice roll example above, consider a general discrete uniform random variable (DURV)   For convenience, define  . The p.m.f. is

In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable.[2] The information gain of any observation  is

Special case: constant random variableEdit

If   above,   degenerates to a constant random variable with probability distribution deterministically given by   and probability measure the Dirac measure  . The only value   can take is deterministically  , so the information content of any measurement of   is

In general, there is no information gained from measuring a known value.[2]

Categorical distributionEdit

Generalizing all of the above cases, consider a categorical discrete random variable with support   and p.m.f. given by


For the purposes of information theory, the values   do not even have to be numbers at all; they can just be mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure  . Without loss of generality, we can assume the categorical distribution is supported on the set  ; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.

The information of the outcome   is given


From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.

Relationship to entropyEdit

The entropy is the expected value of the information content of the discrete random variable, with expectation taken over the discrete values it takes. Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies  , where   is the mutual information of   with itself.[7]


By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero.

For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin, “Weather forecast for tonight: dark. Continued dark overnight, with widely scattered light by morning.” Assuming one does not reside near the Earth's poles or polar circles, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.

When the content of a message is known a priori with certainty, with probability of 1, there is no actual information conveyed in the message. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.

Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event,  , depends only on the probability of that event.


for some function   to be determined below. If  , then  . If  , then  .

Further, by definition, the measure of self-information is nonnegative and additive. If a message informing of event   is the intersection of two independent events   and  , then the information of event   occurring is that of the compound message of both independent events   and   occurring. The quantity of information of compound message   would be expected to equal the sum of the amounts of information of the individual component messages   and   respectively:


Because of the independence of events   and  , the probability of event   is


However, applying function   results in


The class of function   having the property such that


is the logarithm function of any base. The only operational difference between logarithms of different bases is that of different scaling constants.


Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that  .

Taking into account these properties, the self-information   associated with outcome   with probability   is defined as:


The smaller the probability of event  , the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of   is bits. This is the most common practice. When using the natural logarithm of base  , the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.

As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 bits (probability 1/16), and the information content associated with getting a result other than the one specified would be ~0.09 bits (probability 15/16). See below for detailed examples.

See alsoEdit


  1. ^ Jones, D.S., Elementary Information Theory, Vol., Clarendon Press, Oxford pp 11-15 1979
  2. ^ a b c d McMahon, David M. (2008). Quantum Computing Explained. Hoboken, NJ: Wiley-Interscience. ISBN 9780470181386. OCLC 608622533.
  3. ^ Borda, Monica (2011). Fundamentals in Information Theory and Coding. Springer. ISBN 978-3-642-20346-6.
  4. ^ Han, Te Sun & Kobayashi, Kingo (2002). Mathematics of Information and Coding. American Mathematical Society. ISBN 978-0-8218-4256-0.CS1 maint: uses authors parameter (link)
  5. ^ R. B. Bernstein and R. D. Levine (1972) "Entropy and Chemical Change. I. Characterization of Product (and Reactant) Energy Distributions in Reactive Molecular Collisions: Information and Entropy Deficiency", The Journal of Chemical Physics 57, 434-449 link.
  6. ^ Myron Tribus (1961) Thermodynamics and Thermostatics: An Introduction to Energy, Information and States of Matter, with Engineering Applications (D. Van Nostrand, 24 West 40 Street, New York 18, New York, U.S.A) Tribus, Myron (1961), pp. 64-66 borrow.
  7. ^ Thomas M. Cover, Joy A. Thomas; Elements of Information Theory; p. 20; 1991.

Further readingEdit

External linksEdit