Entropy In Thermodynamics And Information Theory Article Index for
Entropy 		Shopping
Entropy 		Website Links For
Thermodynamics 		 

Information About

Entropy In Thermodynamics And Information Theory
  CATEGORIES ABOUT ENTROPY IN THERMODYNAMICS AND INFORMATION THEORY
   
thermodynamic entropy
   
philosophy of thermal and statistical physics
   
APPAREL
BABY
BEAUTY
BOOKS
CAR TOYS
CELL PHONES
DVD'S
ELECTRONICS
GOURMET FOOD
GROCERIES
HEALTH & PERSONAL
HOME & GARDEN
JEWELRY
MUSIC
MUSIC INSTRUMENTS
OFFICE PRODUCTS
SOFTWARE
SPORTING GOODS
TOOLS & HARDWARE
TOYS
VIDEO GAMES
SHOPPING HOME
MORE SHOPPING...



There are close parallels between the mathematical expressions for the thermodynamic Entropy , usually denoted by ''S'', of a physical system in the Statistical Thermodynamics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s ; and the Information-theoretic Entropy , usually expressed as ''H'', of Claude Shannon and Ralph Hartley developed in the 1940s . Shannon, although not initially aware of this similarity, commented in it upon publicizing information theory in '' A Mathematical Theory Of Communication ''.

This article explores what links there are between the two concepts, and how far they can be regarded as connected.


EQUIVALENCE OF FORM OF THE DEFINING EXPRESSIONS


Discrete case


The defining expression for Entropy in the theory of Statistical Mechanics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s , is of the form:
: S = - k \sum_i p_i \log p_i,\,
where p_i is the probability of the microstate ''i'' taken from an equilibrium ensemble.

The defining expression for Entropy in the theory of Information established by Claude E. Shannon in 1948 is of the form:
: H = - \sum_i p_i \log p_i,\,
where p_i is the probability of the message m_i taken from the message space ''M''.

Mathematically ''H'' may also be seen as an Average Information , taken over the message space, because when a certain message occurs with probability ''p''''i'', the information
-log(''p''''i'') will be obtained.

If all the microstates are equiprobable (a Microcanonical Ensemble ), the statistical thermodynamic entropy reduces to the form on Boltzmann's tombstone,
: S = k \log W \,
where ''W'' is the number of microstates.

If all the messages are equiprobable, the information entropy reduces to the Hartley Entropy
  Where <math>M</math> Is The "http://wwwinformationdelightinfo/information/entry/cardinality" class="copylinks">Cardinality of the message space ''M''