Fisher Information

DEFINITION

Fisher information is the amount of Information that an observable Random Variable ''X'' carries about an unobservable Parameter θ upon which the Probability Distribution of ''X'' depends. Since the Expectation of the Score is zero, the Variance is also the second Moment of the score, and the Fisher information can be written

:

\mathcal{I}(	heta)
=
\mathrm{E}
\left[
 \left[
  rac{\partial}{\partial	heta} \ln f(X;	heta)
 
ight]^2

ight],

where ''f'' is the Probability Density Function of random variable ''X'' and, consequently,

0 \leq \mathcal{I}(	heta) < \infty

.
The Fisher information is thus the expectation of the square of the score. A random variable carrying high Fisher information implies that the absolute value of the score is frequently high. (Remember that the expectation of the score is zero.)

Note that the information as defined above is not a function of a particular observation, as the random variable ''X'' has been averaged out. The concept of information is useful when comparing two methods of observing some random process.

If the following regularity condition is met:

:

\int rac{rac{\partial^2}{\partial 	heta^2}f(X ; 	heta )}{f(X ; 	heta)^2} \, d	heta  = 0,

then the Fisher information may also be written as:

:

\mathcal{I}(	heta) = - \mathrm{E} \left[ rac{\partial^2}{\partial	heta^2} \ln f(X;	heta) 
ight].

Thus Fisher information is the expectation of the second Derivative of the Log of ''f'' with respect to θ.
Information may thus be seen to be a measure of the "sharpness" of the Support Curve near the Maximum Likelihood Estimate of θ. A "blunt" support curve (one with a shallow maximum) would have low expected second derivative, and thus low information; while a sharp one would have a high expected second derivative and thus high information.

Information is additive, in that the information yielded by two Independent experiments is the sum of the information from each experiment separately:

:

\mathcal{I}_{X,Y}(	heta) = \mathcal{I}_X(	heta) + \mathcal{I}_Y(	heta).

This result follows from the elementary fact that if random variables are independent, the variance of their sum is the sum of their variances.
Hence the information in a random sample of size ''n'' is ''n'' times that in a sample of size 1 (if observations are independent).

The information provided by a Sufficient Statistic is same as that of the sample ''X''. This may be seen by using Fisher's Factorization Criterion for a sufficient statistic. If ''T''(''X'') is sufficient for θ, then

:

f(X;	heta) = g(T(X), 	heta) h(X)

for some functions ''g'' and ''h''. See Sufficient Statistic for a more detailed explanation. The equality of information then follows from the following fact:

:

rac{\partial}{\partial	heta} \ln \left[f(X   ;	heta)
ight]
= rac{\partial}{\partial	heta} \ln \left[g(T(X);	heta)
ight]

which follows from the definition of Fisher information, and the independence of ''h''(''X'') from θ. More generally, if ''T'' = ''t''(''X'') is a Statistic , then

:

\mathcal{I}_T(	heta)
\leq
\mathcal{I}_X(	heta)

with equality If And Only If ''T'' is a Sufficient Statistic .

The Cramér-Rao Inequality states that the reciprocal of the Fisher information is an asymptotic lower bound on the variance of any Maximum Likelihood Estimator of θ.

Single parameter Bernoulli experiment

A Bernoulli Trial is a random variable with two possible outcomes, "success" and "failure", with "success" having a probability of θ. The outcome can be thought of as determined by a coin toss, with the probability of obtaining a "head" being θ and the probability of obtaining a "tail" being 1 - θ.

The Fisher information contained in ''n'' independent Bernoulli Trial s may be calculated as follows. In the following, ''A'' represents the number of successes, ''B'' the number of failures, and ''n'' = ''A'' + ''B'' is the total number of trials.

:

\mathcal{I}(	heta)
=
-\mathrm{E}
\left[
 rac{\partial^2}{\partial	heta^2} \ln(f(A;	heta))

ight]  (1)

=
-\mathrm{E}
\left[
 rac{\partial^2}{\partial	heta^2} \ln
 \left[
  	heta^A(1-	heta)^Brac{(A+B)!}{A!B!}
 
ight]

ight]  (2)

=
-\mathrm{E}
\left[
 rac{\partial^2}{\partial	heta^2} 
 \left[
  A \ln (	heta) + B \ln(1-	heta)
 
ight]

ight]  (3)

=
-\mathrm{E}
\left[
 rac{\partial}{\partial	heta}
 \left[
  rac{A}{	heta} - rac{B}{1-	heta}
 
ight]

ight]

(on differentiating ln ''x'', see Logarithm ) (4)

::

=
+\mathrm{E}
\left[
 rac{A}{	heta^2} + rac{B}{(1-	heta)^2}

ight]  (5)

=
rac{n	heta}{	heta^2} + rac{n(1-	heta)}{(1-	heta)^2}

(as the expected value of ''A'' = ''n''θ, etc.) (6)

::

= rac{n}{	heta(1-	heta)}  (7)

(1) defines Fisher information.
(2) invokes the fact that the information in a Sufficient Statistic is the same as that of the sample itself.
(3) expands the Log term and drops a constant.
(4) and (5) differentiate with respect to θ.
(6) replaces ''A'' and ''B'' with their expectations. (7) is algebra.

The end result, namely,
:

\mathcal{I}(	heta) = rac{n}{	heta(1-	heta)},

is the reciprocal of the Variance of the mean number of successes in ''n'' Bernoulli Trial s, as expected (see last sentence of the preceding section).

MATRIX FORM

When there are ''N'' parameters, so that θ is a ''N''x1 vector

heta = \begin{bmatrix}
 	heta_{1}, 	heta_{2}, \cdots , 	heta_{N} \end{bmatrix},

, then the Fisher information takes the form of an ''N''x''N'' matrix, the Fisher information matrix (FIM), with typical element:

:

{\left(\mathcal{I} \left(	heta 
ight) 
ight)}_{i, j}
=
\mathrm{E}
\left[
 rac{\partial}{\partial	heta_i} \ln f(X;	heta)
 rac{\partial}{\partial	heta_j} \ln f(X;	heta)

ight].

The FIM is a ''N''x''N'' Positive Definite Symmetric Matrix , defining a Metric on the ''N''-dimensional parameter space. Exploring this topic requires Differential Geometry .

Multivariate normal distribution

The FIM for a ''N''-variate Multivariate Normal Distribution has a special form. Let

\mu(	heta) = \begin{bmatrix}
 \mu_{1}(	heta), \mu_{2}(	heta), \cdots , \mu_{N}(	heta) \end{bmatrix},

and let

\Sigma(	heta)

be the Covariance Matrix of

\mu(	heta)

. Then the typical element

\mathcal{I}_{m,n}

, 0≤''m'',''n''<''N'', of the FIM for

X \sim N(\mu(	heta), \Sigma(	heta))

is:

:

\mathcal{I}_{m,n}
=
rac{\partial \mu}{\partial 	heta_m}
\Sigma^{-1}
rac{\partial \mu^	op}{\partial 	heta_n}
+
rac{1}{2}
\mathrm{tr}
\left(
 \Sigma^{-1}
 rac{\partial \Sigma}{\partial 	heta_m}
 \Sigma^{-1}
 rac{\partial \Sigma}{\partial 	heta_n}

ight),

where

(..)^	op

denotes the Transpose of a Vector ,

\mathrm{tr}(..)

denotes the Trace of a Square Matrix , and:

= \begin{bmatrix} rac{\partial \mu_1}{\partial heta_m} & rac{\partial \mu_2}{\partial heta_m} & \cdots & rac{\partial \mu_N}{\partial heta_m} & \end{bmatrix};

= \begin{bmatrix} rac{\partial \Sigma_{1,1}}{\partial heta_m} & rac{\partial \Sigma_{1,2}}{\partial heta_m} & \cdots & rac{\partial \Sigma_{1,N}}{\partial heta_m} \ \ rac{\partial \Sigma_{2,1}}{\partial heta_m} & rac{\partial \Sigma_{2,2}}{\partial heta_m} & \cdots & rac{\partial \Sigma_{2,N}}{\partial heta_m} \ \ dots & dots & \ddots & dots \ \ rac{\partial \Sigma_{N,1}}{\partial heta_m} & rac{\partial \Sigma_{N,2}}{\partial heta_m} & \cdots & rac{\partial \Sigma_{N,N}}{\partial heta_m} \end{bmatrix}.

FISHER INFORMATION AND SCIENTIFIC THEORY

Fisher information is a powerful new method for deriving laws governing many aspects of nature and human society. Frieden (2004) sets out in detail how Fisher information can ground a great deal of contemporary Physical Theory , including Newtonian Mechanics , Virial Theorem , Statistical Mechanics , Thermodynamics , Maxwell's Equations , Lorentz Transformation , General Relativity , EPR Experiment , Schrodinger Equation , Klein-Gordon Equation , Dirac Equation , Rarita-Schwinger Equation , and the fundamental Physical Constants . Frieden and coauthors have also used EPI to derive some established principles and new laws of Biology , the Biophysics of Cancer growth, Chemistry , and Economics .

Frieden begins with the obvious: any information channel is imperfect. The amount of Fisher information lost while observing a physical effect is called the Physical Information . To extremize (usually minimizing via the Calculus Of Variations ) the physical information through variation of the system probability amplitudes is called the principle of Extreme Physical Information , EPI. The result is Differential Equation s and Probability Density Function s describing the physics of the source effect.

SEE ALSO

Cramér-Rao Inequality

B. Roy Frieden

Other measures employed in Information Theory :

Self-information

Kullback-Leibler Divergence

Shannon Entropy

BOOKS

B. Roy Frieden , 2004. ''Science from Fisher Information: A Unification'', 2nd ed. Cambridge University Press.ISBN 0-52-100911-1.

EXTERNAL LINKS

B. Roy Frieden , " Fisher Information, a New Paradigm for Science: Introduction, Uncertainty principles, Wave equations, Ideas of Escher, Kant, Plato and Wheeler. " This essay is continually revised in the light of ongoing research.

	APPAREL
	BABY
	BEAUTY
	BOOKS
	CAR TOYS
	CELL PHONES
	DVD'S
	ELECTRONICS
	GOURMET FOOD
	GROCERIES
	HEALTH & PERSONAL
	HOME & GARDEN
	JEWELRY
	MUSIC
	MUSIC INSTRUMENTS
	OFFICE PRODUCTS
	SOFTWARE
	SPORTING GOODS
	TOOLS & HARDWARE
	TOYS
	VIDEO GAMES
	SHOPPING HOME

	MORE SHOPPING...

	CATEGORIES ABOUT FISHER INFORMATION

	estimation theory

	information theory

Information About

Fisher Information