| Mutual Information |
Article Index for Mutual |
Website Links For Mutual |
Information AboutMutual Information |
| CATEGORIES ABOUT MUTUAL INFORMATION | |
| information theory | |
|
Intuitively, mutual information measures the information about ''X'' that is shared by ''Y''. If ''X'' and ''Y'' are independent, then ''X'' contains no information about ''Y'' and vice versa, so their mutual information is zero. If ''X'' and ''Y'' are identical then all information conveyed by ''X'' is shared with ''Y'': knowing ''X'' reveals nothing new about ''Y'' and vice versa, therefore the mutual information is the same as the information conveyed by ''X'' (or ''Y'') alone, namely the Entropy of ''X''. In a specific sense (see below), mutual information quantifies the distance between the Joint Distribution of ''X'' and ''Y'' and the product of their Marginal Distribution s. Formally, the mutual information of two discrete random variables ''X'' and ''Y'' can be defined as: : where ''p''(''x'',''y'') is the Joint Probability Distribution Function of ''X'' and ''Y'', and ''p''(''x'') and ''p''(''y'') are the marginal probability distribution functions of ''X'' and ''Y'' respectively. In the Continuous case, we replace summation by a definite Double Integral : : where ''p''(''x'',''y'') is now the joint probability ''density'' function of ''X'' and ''Y'', and ''p''(''x'') and ''p''(''y'') are the marginal probability density functions of ''X'' and ''Y'' respectively. Mutual information is a measure of independence in the following sense: ''I''(''X''; ''Y'') = 0 Iff ''X'' and ''Y'' are independent random variables. This is easy to see in one direction: if ''X'' and ''Y'' are independent, then ''p''(''x'',''y'') = ''p''(''x'') × ''p''(''y''), and therefore: : Moreover, mutual information is nonnegative (i.e. ''I''(''X'';''Y'') ≥ 0; see below) and Symmetric (i.e. ''I''(''X'';''Y'') = ''I''(''Y'';''X'')). Several generalizations of mutual information to more than two random variables have been proposed, but a widely agreed on definition has not yet emerged. RELATION TO OTHER QUANTITIES Mutual information can be equivalently expressed as | ||
| ::<math> | H(Y) - H(YX) \,</math> |
|
| Where ''H''(''X'') And ''H''(''Y'') Are | "http://wwwinformationdelightinfo/encyclopedia/entry/information_entropy" class="copylinks">Entropies , ''H''(''X''''Y'') and ''H''(''Y''''X'') are Conditional Entropies , and ''H''(''Y'',''X'') is the Joint Entropy of ''X'' and ''Y'' Since ''H''(''X'') &ge ''H''(''X''''Y''), this characterization is consistent with the nonnegativity property stated above |
|
| Note That ''H''(''X''''X'') | 0 and therefore ''H''(''X'') = ''I''(''X''''X'') This is the reason why Entropy is often called self-information Thus ''I''(''X''''X'') &ge ''I''(''X''''Y''), and one can formulate the basic principle that a variable contains more information about itself than any other variable can provide |
|
|