| Differential Entropy |
Article Index for Differential |
Shopping Entropy |
Website Links For Differential |
Information AboutDifferential Entropy |
| CATEGORIES ABOUT DIFFERENTIAL ENTROPY | |
| entropy and information | |
| information theory | |
| statistical randomness | |
| randomness | |
|
DEFINITION Let ''X'' be a random variable with a Probability Density Function ''f'' whose Support is a set . The ''differential entropy'' or is defined as : As with its discrete analog, the Unit s of differential entropy depend on the base of the Logarithm , which is usually ''2'' (i.e., the units are Bit s). See Logarithmic Units for logarithms taken in different bases. Related concepts such as Joint and Conditional differential entropy are defined in a similar fashion. One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than ''1''. For example, Uniform (''0'',''1/2'') has differential entropy . Note that the continuous Mutual Information has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of ''partitions'' of ''X'' and ''Y'' as these partitions become finer and finer. Thus it is invariant under quite general transformations of ''X'' and ''Y'', and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values. EXAMPLE: EXPONENTIAL DISTRIBUTION Let ''X'' be an Exponentially Distributed random variable with parameter , that is, with probability density function : Its differential entropy is then Here, was used rather than to make it explicit that the logarithm was taken to base ''e'', to simplify the calculation. SEE ALSO EXTERNAL LINKS |
|
|