| Entropy In Thermodynamics And Information Theory |
Article Index for Entropy |
Shopping Entropy |
Website Links For Thermodynamics |
Information AboutEntropy In Thermodynamics And Information Theory |
| CATEGORIES ABOUT ENTROPY IN THERMODYNAMICS AND INFORMATION THEORY | |
| thermodynamic entropy | |
| philosophy of thermal and statistical physics | |
|
INTRODUCTION "Gain in entropy always means loss of information, and nothing more" There are close links between the Information-theoretic Entropy of Shannon and Hartley , usually expressed as ''H'', and the Thermodynamic Entropy of Clausius and Carnot , usually denoted by ''S'', of a physical system — in particular between the Shannon entropy and the Statistical Interpretation of thermodynamic entropy, established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s . A quantity defined by the entropy formula for was first introduced by Boltzmann in 1872 , in the context of his H-theorem . Boltzmann's definition, based on frequency distribution for a single particle in a gas of like particles, was subsequently reworked by Gibbs into a general formula for the statistical-mechanical entropy (or "mixedupness"), based on the probability distribution ''pi'' for a complete microstate ''i'' of the total system: : The relation between Gibbs's statistical mechanical definition of entropy and Clausius's classical thermodynamical definition is explored further in the article: '' Thermodynamic Entropy ''. It is evident that : where the Shannon entropy ''H'' is measured in Nats , and the constant of proportionality ''kB'' is Boltzmann's Constant . Boltzmann's constant appears here due to the conventional definition of the units of temperature. Beyond that it has no particular fundamental ''physical'' significance in the definition of statistical mechanical entropy here. In fact, in the view of ''). Equilibrium statistical mechanics gives the prescription that the probability distribution which should be assigned for the unknown Microstate of a thermodynamic system is that which has maximum Shannon entropy, given that it must also satisfy the macroscopic description of the system. But this is just an application of a quite general rule in information theory, if one wishes to a Maximally Uninformative Distribution . The thermodynamic entropy, measuring the phase-space spread of this equilibrium distribution, is just this maximum Shannon entropy, multiplied by Boltzmann's Constant for historical reasons. A neat physical implication was established by Szilard in 1929 , in a refinement of the famous Maxwell's Demon thought-experiment. Consider Maxwell's set-up, but with only a single gas particle in a box. If the supernatural demon knows which half of the box the particle is in, it can close a shutter between the two halves of the box, close a piston unopposed into the empty half of the box, and then extract joules of useful work if the shutter is opened again, and particle isothermally expands back to its original equilibrium occupied volume. In just the right circumstances therefore, the possession of a single bit of Shannon information (a single bit of Negentropy in Brillouin's term) really does correspond to a reduction in physical entropy, which theoretically can indeed be parlayed into useful physical work. A corollary is that in storing one bit of previously unstored information in a system, one inevitably potentially reduces the system's entropy by J K-1. This is only thermodynamically possible if the storage process releases at least joules of energy into the system's surroundings. Rolf Landauer ( 1961 ) showed that the couterpart of this process can occur as well: an array of ordered bits of memory can become "thermalized," or populated with random data, and in the process cool off its surroundings. N bits would then increase the entropy of the system by as they thermalize. Heat generation is one of the banes of computer hardware design; so , which in turn proved essential for research into Quantum Computers . The relation between information entropy and thermodynamic entropy has become common currency in physics. Thus Stephen Hawking often speaks of the thermodynamic entropy of Black Hole s in terms of their information content; and it is not surprising that computers must obey the same physical laws that steam engines do, even though they are radically different devices. But it should also be remembered that Gibbs's statistical mechanical entropy is only one application of information theory to physical systems, relevant when the particular 'message' not yet communicated is the underlying microstate of the physical system. Other physical 'messages' will have their own information entropies. For example, the information rate of a macroscopic physical system obeying stochastic or chaotic behavior can be equal to the information rate of an equivalent Markov process. This entropy is quite likely negligibly tiny and practically quite irrelevant as a contribution to the overall thermodynamic entropy. But if this is the message of interest, then it is the thermodynamic entropy which is irrelevant, and this Shannon information which is everything. EQUIVALENCE OF FORM OF DEFINING EQUATIONS Discrete case The defining equation for Entropy in the theory of Statistical Mechanics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s , is of the form: : where is the probability of the microstate ''i'' taken from an equilibrium ensemble; which reduces for the special case of the microcanonical ensemble to : where ''W'' is the number of microstates, given the Fundamental Postulate that all the microstates are equiprobable. The defining equation for Entropy in the theory of Information established by Claude E. Shannon in 1948 is of the form: : where is the probability of the message taken from the message space ''M''. This also reduces to | ||
| Where <math>M</math> Is The | "http://wwwinformationdelightinfo/encyclopedia/entry/cardinality" class="copylinks">Cardinality of the message space ''M'', under the assumption that all the messages are equiprobable |
|
|