The is a fundamental result in the field of Information Theory , in particular Telecommunication s and Signal Processing . The theorem is commonly called the '''Shannon sampling theorem''', and is also known as '''Nyquist–Shannon–Kotelnikov''', '''Whittaker–Shannon–Kotelnikov''', '''Whittaker–Nyquist–Kotelnikov–Shannon''', '''WKS''', etc., sampling theorem, as well as the '''Cardinal Theorem of Interpolation Theory'''. It is often referred to as simply '''''the sampling theorem'''''. See the historical background section below.
Sampling is the process of converting a signal (for example, a function of continuous time or space) into a numeric sequence (a function of discrete time or space).
The theorem states that
The theorem also leads to a formula for the reconstruction. The assumptions necessary to prove the theorem form a mathematical model that is only an idealized approximation, at best, to any realistic situation. The conclusion, that perfect reconstruction is possible, is mathematically correct for the model, but only an approximation for actual signals and actual sampling techniques.
A signal or function is bandlimited if it contains no energy at frequencies higher than some bandlimit or Bandwidth . A signal that is bandlimited is constrained in terms of how rapidly it changes in time, and therefore how much detail it can convey, in between discrete instants of time. The sampling theorem means that the uniformly spaced discrete samples are a complete representation of the signal if this bandwidth is less than half the sampling rate.
To formalize these concepts, let represent a Continuous-time signal and be the Continuous Fourier Transform of that signal (which exists if is Square-integrable ):
:
The signal is bandlimited to a one-sided baseband bandwidth if:
|
and the samples of
are denoted by
:
(
Integers )
The sampling theorem leads to a procedure for reconstructing the original
from the samples
, and states sufficient conditions for such a reconstruction to be exact.
From a
Signal Processing perspective, the theorem describes two processes; a
Sampling process, in which a
Continuous Time signal is converted to a
Discrete Time signal, and a reconstruction process, in which the continuous signal is recovered from the discrete signal.
The continuous signal varies over ''time'' (or ''space'' as in a digitized
Image or another independent variable in some other application) and the sampling process is done by simply measuring the continuous signal's value every ''T'' units of time (or space), which is called the ''sampling interval''. In practice, for signals that are a function of time, the sampling interval is typically quite small, on the order of milliseconds or microseconds or less. This results in a sequence of numbers, called ''samples'', which is to represent the original signal. Each sample is associated to the specific point in time where it was measured. The reciprocal of the sampling interval, 1/''T'' is the
Sampling Frequency , ''f''
''s'', and measured in samples per unit time. If ''T'' is expressed in
Second s then ''f''
''s'' is expressed in
Hz .
The reconstruction process is an
Interpolation process that mathematically defines a continuous-time signal, ''x''(''t''), from the discrete samples ''x''
{Link without Title} and at times in between the sample instants, ''nT''.
: sin(πx) / (πx)... showing the central peak at ''x''= 0, and zero-crossings at the other integer values of ''x''.]]
- Each sample is multiplied by the Sinc Function scaled so that the zero-crossings of the sinc function occur at the sampling instants and that the sinc-function's central point is shifted to the time of that sample, ''nT''. All of these shifted and scaled functions are then added together to recover the original signal. The scaled and time-shifted sinc-functions are Continuous making the sum of these also continuous. This means that the result of this operation is indeed a continuous signal. This procedure is represented by the Whittaker–Shannon Interpolation Formula .
- The signal obtained from this reconstruction process will have no frequencies higher than one-half the sampling frequency. This reconstructed signal will match the original signal if the original signal contains no frequencies equal to or above half the sampling frequency; that is, if the sampling frequency exceeds twice the highest frequency in the original signal. This condition is called the ''Nyquist criterion'' or sometimes the ''Raabe condition.''
Note that if the original signal contains a frequency component exactly equal to one-half the sampling rate, this condition is not satisfied, and the resulting reconstructed signal may have a component at that frequency but the amplitude and phase of that component will not, in general, match the original component.
This reconstruction or interpolation using sinc functions is not the only interpolation scheme, and indeed, is practically impossible because it requires summing an infinite number of terms. However, it is the interpolation method that exactly reconstructs ''any'' given bandlimited ''x''(''t'') with ''any'' bandlimit ''B''<1/(2''T''); any other method that does so is formally equivalent to it.
A few consequences can be drawn from the theorem:
- If it is known that the signal which we sample has a certain highest frequency ''B'', the theorem gives us a lower bound on the sampling frequency to assure perfect reconstruction. This lower bound to the sampling frequency, 2''B'', is called the Nyquist Rate .
- If instead the sampling frequency is known, the theorem gives us an upper bound for frequency components, ''B''<''f''''s''/2, of the signal to allow for perfect reconstruction. This upper bound is the Nyquist Frequency , denoted ''f''''N''.
- Both of these cases imply that the signal to be sampled must be Bandlimited ; that is, any component of this signal which has a frequency above a certain bound should be zero, or at least sufficiently close to zero to allow us to neglect its influence on the resulting reconstruction. In the first case, the condition of bandlimitation of the sampled signal can be accomplished by assuming a model of the signal which can be analysed in terms of the frequency components it contains; for example, sounds that are made by a speaking human normally contain very small frequency components at or above 10 kHz and it is then sufficient to sample such an audio signal with a sampling frequency of at least 20 kHz. For the second case, we have to assure that the sampled signal is bandlimited such that frequency components at or above half of the sampling frequency can be neglected. This is usually accomplished by means of a suitable low-pass filter; for example, if it is desired to sample speech waveforms at 8 kHz, the signals should first be lowpass filtered to below 4 kHz.
- In practice, neither of the two statements of the sampling theorem described above can be completely satisfied, and neither can the reconstruction formula be precisely implemented. The reconstruction process that involves scaled and delayed Sinc Function s can be described as ''ideal''. It cannot be realized in practice since it implies that each sample contributes to the reconstructed signal at almost all time points, requiring summing an infinite number of terms. Instead, some type of approximation of the sinc functions, finite in length, has to be used. The error that corresponds to the sinc-function approximation is referred to as ''interpolation error''. Practical Digital-to-analog Converter s produce neither scaled and delayed sinc functions nor ideal Impulses (that if ideally low-pass filtered would yield the original signal), but a sequence of scaled and delayed Rectangular Pulses . This practical Piecewise-constant output can be modeled as a Zero-order Hold filter driven by the sequence of scaled and delayed dirac impulses referred to in the mathematical basis section below. A shaping filter is sometimes used after the DAC with zero-order hold to make a better overall approximation.
- Furthermore, in practice, a sampled signal that is "time-limited", or finite length, can never be fully bandlimited. This means that even if an ideal reconstruction could be made, the reconstructed signal would not be exactly the original signal. The error that corresponds to the failure of bandlimitation is referred to as ''aliasing''.
- The sampling theorem does not say what happens when the conditions and procedures are not exactly met, but its proof suggests an analytical framework in which the non-ideality can be studied. A designer of a system that deals with sampling and reconstruction processes needs a thorough understanding of the signal to be sampled, in particular its frequency content, the sampling frequency, how the signal is reconstructed in terms of interpolation, and the requirement for the total reconstruction error, including aliasing and interpolation error. These properties and parameters may need to be carefully tuned in order to obtain a useful system.
If the sampling condition is not satisfied, then frequencies will overlap; that is, frequencies above half the sampling rate will be reconstructed as, and appear as, frequencies below half the sampling rate. The resulting distortion is called
Aliasing ; the reconstructed signal is said to be an alias of the original signal, in the sense that it has the same set of sample values.
For a sinusoidal component of exactly half the sampling frequency, the component will in general alias to another sinusoid of the same frequency, but with a different phase and amplitude.
To prevent or reduce aliasing, two things can be done:
# Increase the sampling rate, to above twice some or all of the frequencies that are aliasing.
# Introduce an
Anti-aliasing Filter or make the anti-aliasing filter more stringent.
The anti-aliasing filter is to restrict the bandwidth of the signal to satisfy the condition for proper sampling. Such a restriction works in theory, but is not precisely satisfiable in reality, because realizable filters will always allow some ''leakage'' of high frequencies. However, the leakage energy can be made small enough so that the aliasing effects are negligible.
]]
The sampling theorem is usually formulated for functions of a single variable. Consequently, the theorem is directly applicable to time-dependent signals and is normally formulated in that context. However, the sampling theorem can be extended in a straightforward way to functions of arbitrarily many variables. Grayscale images, for example, are often represented as two-dimensional arrays (or matrices) of real numbers representing the relative intensities of
Pixel s (picture elements) located at the intersections of row and column sample locations. As a result, images require two independent variables, or indices, to specify each pixel uniquely — one for the row, and one for the column.
Color images typically consist of a composite of three separate grayscale images, one to represent each of the three primary colors — red, green, and blue, or ''RGB'' for short. Other colorspaces using 3-vectors for colors include HSV, LAB, XYZ, etc. Some colorspaces such as cyan, magenta, yellow, and black (CMYK) may represent color by four dimensions. All of these are treated as vector-valued functions over a two-dimensional sampled domain.
Similar to one-dimensional discrete-time signals, images can also suffer from aliasing if the sampling resolution, or pixel density, is inadequate. For example, a digital photograph of a striped shirt with high frequencies (in other words, the distance between the stripes is small), can cause aliasing of the shirt when it is sampled by the camera's
Image Sensor . The aliasing appears as a
Moiré Pattern . The "solution" to higher sampling in the spatial domain for this case would be to move closer to the shirt or use a higher resolution sensor.
Another example is shown to the right in the brick patterns.
The top image shows the effects when the sampling theorem's condition is not satisfied.
When software rescales an image (the same process that creates the thumbnail shown in the lower image) it, in effect, runs the image through a low-pass filter first and then
Downsamples the image to result in a smaller image that does not exhibit the
Moiré Pattern .
The top image is what happens when the image is downsampled without low-pass filtering: aliasing results.
The top image was created by zooming out in
GIMP and then taking a
Screenshot of it. The likely reason that this causes a banding problem is that the zooming feature simply downsamples without low-pass filtering (probably for performance reasons) since the zoomed image is for on-screen display instead of printing or saving.
The application of the sampling theorem to images should not be made without care. For example, the sampling process in any standard image sensor (CCD or CMOS camera) is relatively far from the ideal sampling which would measure the image intensity at a single point. Instead these devices have a relatively large sensor area at each sample point in order to obtain sufficient amount of light. Also, it is not obvious that the analog image intensity function which is sampled by the sensor device is bandlimited. It should be noted, however, that the non-ideal sampling is itself a type of low-pass filter, although far from one that ideally removes high frequency components. Despite images having these problems in relation to the sampling theorem, the theorem can be used to describe the basics of down and up sampling of images.
When a signal is
Downsampled , the sampling theorem can be invoked via the artifice of resampling a hypothetical continuous-time reconstruction. The Nyquist criterion must still be satisfied with respect to the new lower sampling frequency in order to avoid aliasing. To meet the requirements of the theorem, the signal must usually pass through a
Low-pass Filter of appropriate cutoff frequency as part of the downsampling operation. This low-pass filter, which prevents aliasing, is called an
Anti-aliasing Filter .
The is defined as twice the bandwidth of the
Continuous-time signal. It should be noted that the sampling frequency must be strictly ''greater'' than the Nyquist rate of the signal to achieve unambiguous representation of the signal. This constraint is equivalent to requiring that the system's '''
Nyquist Frequency ''' (also known as '''''critical frequency''''', and equal to half the sample rate) be strictly greater than the bandwidth of the signal. If the signal contains a frequency component at precisely the Nyquist frequency then the corresponding component of the sample values cannot have sufficient information to reconstruct the Nyquist-frequency component in the continuous-time signal because of phase ambiguity. In such a case, there would be an infinite number of possible and ''different'' sinusoids (of varying amplitude and phase) of the Nyquist-frequency component that are represented by the discrete samples.
As an example, consider this family of signals at the critical frequency
:
Where the samples
:
are in every case just alternating –1 and +1, for any phase θ. There is no way to determine either the amplitude or the phase of the continuous-time sinusoid ''x''(''t'') that ''x''
{Link without Title} was sampled from. This ambiguity is the reason for the ''strict'' inequality of the sampling theorem's condition.
The Nyquist–Shannon sampling theorem states that, given a
Bandlimited Continuous-time signal ''x''(''t'') that is uniformly sampled at a sufficient rate, even if all of the information in the signal between samples is discarded, there remains sufficient information in the samples that the original continuous-time signal can be mathematically reconstructed perfectly from only those discrete samples. To prove this, a different function is first constructed, conceptually, from the whole original signal, but preserving information from just the sample instants
:
:Δ
''T''(''t'') is the sampling operator called the
Dirac Comb and, being periodic with period ''T'', can be formally expressed as a
Fourier Series
| |
T \cdot rac{1}{T}\sum_{k=-\infty}^{\infty} e^{i 2 \pi k t/T} \ </math>
|
| |
\sum_{k=-\infty}^{\infty} e^{i 2 \pi k f_s t} \ </math>
|
|
| |
\mathcal{F} \left \{ \sum_{k=-\infty}^{\infty} x(t) \cdot e^{i 2 \pi k f_s t}
ight \} \ </math>
|
| |
\sum_{k=-\infty}^{\infty} \mathcal{F} \left \{ x(t) \cdot e^{i 2 \pi k f_s t}
ight \} \ </math>
|
| |
\sum_{k=-\infty}^{\infty} X(f - k f_s) \ </math>
|
|
| |
0 for all ''f'' > ''B''), and consider what condition precludes overlapping of the adjacent images ''X''(''f''-''kf<sub>s</sub>'') ''':'''
|
| |
\begin{cases}1 & f \le B \ 0 & f \ge f_s - B \end{cases}</math>
|
| |
\mathrm{rect} \left( rac{f}{f_s}
ight) = \begin{cases}1 & f < rac{f_s}{2} \ 0 & f > rac{f_s}{2} \end{cases}</math>
|
|
:where ''n'' is any positive or negative integer, we obtain
::
:On the left are values of
at the sampling points. The integral on the right will be recognized as essentially the ''n''th coefficient in a Fourier-series expansion of the function
, taking the interval –''W'' to ''W'' as a fundamental period. This means that the values of the samples
determine the Fourier coefficients in the series expansion of
. Thus they determine
, since
is zero for frequencies greater than ''W'', and for lower frequencies
is determined if its Fourier coefficients are determined. But
determines the original function
completely, since a function is determined if its spectrum is known. Therefore the original samples determine the function
completely.
Shannon's proof of the theorem is complete at that point, but he goes on to discuss reconstruction via sinc functions, what we now call the
Whittaker–Shannon Interpolation Formula as discussed above. He does not derive or prove the properties of the
Sinc Function , but these would have been familiar to engineers reading his works at the time, since the Fourier pair relationship between
Rect and
Sinc was well known. Quoting Shannon:
:Let
be the ''n''th sample. Then the function
is represented by:
::
As in the other proof, the existence of the Fourier transform of the original signal is assumed, so the proof does not say whether the sampling theorem extends to bandlimited stationary random processes.
For sampling a non-
Baseband Signal , the conditions to avoid information loss and to allow perfect reconstruction can be generalized in terms of conditions on the frequency interval of nonzero spectrum. See ''
Sampling (signal Processing) '' for more details and examples.
A bandpass condition is that
for all nonnegative
outside the open band of frequencies
::
for some nonnegative integer
. This formulation includes the normal baseband condition as the case ''N''=0.
The corresponding interpolation function is the impulse response of a bandpass filter with cutoffs at the upper and lower edges of the specified band, which is the difference between a pair of lowpass impulse responses:
::
.
Other generalizations, for example to signals occupying multiple non-contiguous bands, are possible as well. Even the most generalized form of the sampling theorem does not have a provably true converse. That is, one cannot conclude that information is necessarily lost just because the conditions of the sampling theorem are not satisfied; from an engineering perspective, however, it is generally safe to assume that if the sampling theorem is not satisfied then information will most likely be lost.
The was implied by the work of
Harry Nyquist in 1928 ("Certain topics in telegraph transmission theory"), in which he showed that up to 2B independent pulse samples could be sent through a system of bandwidth B; but he did not explicitly consider the problem of sampling and reconstruction of continuous signals. About the same time,
Karl Küpfmüller showed a similar result
K. Küpfmüller , "Über die Dynamik der selbsttätigen Verstärkungsregler", ''Elektrische Nachrichtentechnik'', vol. 5, no. 11, pp. 459-467, 1928. (German)
K. Küpfmüller,
On the dynamics of automatic gain controllers , ''Elektrische Nachrichtentechnik'', vol. 5, no. 11, pp. 459-467. (English translation) , and discussed the sinc-function impulse response of a band-limiting filter, via its integral, the step response ''Integralsinus''; this bandlimiting and reconstruction filter that is so central to the sampling theorem is sometimes referred to as a ''Küpfmüller filter'' (but seldom so in English).
The sampling theorem, essentially a
Dual of Nyquist's result, was proved by
Claude E. Shannon in 1949 ("Communication in the presence of noise").
V. A. Kotelnikov published similar results in 1933 ("On the transmission capacity of the 'ether' and of cables in electrical communications", translation from the Russian), as did the mathematician
E. T. Whittaker in 1915 ("Expansions of the Interpolation-Theory", "Theorie der Kardinalfunktionen"), J. M. Whittaker in 1935 ("Interpolatory function theory"), and
Gabor in 1946 ("Theory of communication").
Others who have independently discovered or played roles in the development of the sampling theorem have been discussed in several historical articles, for example by JerriAbdul J. Jerri,
The Shannon Sampling Theorem—Its Various Extensions and Applications: A Tutorial Review , ''Proceedings of the IEEE'', 65:1565–1595, Nov. 1977. See also [http://ieeexplore.ieee.org/search/wrapper.jsp?arnumber=1455576 Correction to "The Shannon sampling theorem—Its various extensions and applications: A tutorial review"], Proceedings of the IEEE, 67:695, April 1979 and by Lüke.Hans Dieter Lüke, , ''IEEE Communications Magazine,'' pp.106–108, April 1999. For example, Lüke points out that H. Raabe, an assistant to Küpfmüller, proved the theorem in his
1939 Ph.D. dissertation; the term ''Raabe condition'' came to be associated with the criterion for unambiguous representation (sampling rate greater than twice the bandwidth).
MeijeringErik Meijering, , ''Proc. IEEE,'' 90, 2002. mentions several other discoverers and names in a paragraph and pair of footnotes:
:As pointed out by Higgins
the sampling theorem should really be considered in two parts, as done above: the first stating the fact that a bandlimited function is completely determined by its samples, the second describing how to reconstruct the function using its samples. Both parts of the sampling theorem were given in a somewhat different form by J. M. Whittaker [350, 351, 353 and before him also by Ogura
242 . They were probably not aware of the fact that the first part of the theorem had been stated as early as 1897 by Borel
As we have seen, Borel also used around that time what became known as the cardinal series. However, he appears not to have made the link [135 . In later years it became known that the sampling theorem had been presented before Shannon to the Russian communication community by Kotel'nikov
In more implicit, verbal form, it had also been described in the German literature by Raabe [257 . Several authors
205 have mentioned that Someya
introduced the theorem in the Japanese literature parallel to Shannon. In the English literature, Weston [347 introduced it independently of Shannon around the same time.
28
:
27 Several authors, following Black
have claimed that this first part of the sampling theorem was stated even earlier by Cauchy, in a paper [41 published in 1841. However, the paper of Cauchy does not contain such a statement, as has been pointed out by Higgins [135].
:
28 As a consequence of the discovery of the several independent introductions of the sampling theorem, people started to refer to the theorem by including the names of the aforementioned authors, resulting in such catchphrases as “the Whittaker-Kotel’nikov-Shannon (WKS) sampling theorem"
or even "the Whittaker-Kotel'nikov-Raabe-Shannon-Someya sampling theorem" [33 . To avoid confusion, perhaps the best thing to do is to refer to it as the sampling theorem, "rather than trying to find a title that does justice to all claimants" [136].
Exactly how, when, or why Nyquist had his name attached to the sampling theorem remains obscure. The first known use of the term ''Nyquist sampling theorem'' is in a 1965 book.Richard A. Roberts and Ben F. Barton, ''Theory of Signal Detectability: Composite Deferred Decision Theory'', 1965. It had been called the ''Shannon Sampling Theorem'' as early as 1954,Truman S. Gray, ''Applied Electronics: A First Course in Electronics, Electron Tubes, and Associated Circuits'', 1954. but also just ''the sampling theorem'' by several other books in the early 1950s.
In 1958, Blackman and TukeyR. B. Blackman and J. W. Tukey, ''The Measurement of Power Spectra : From the Point of View of Communications Engineering'', New York: Dover, 1958. cited Nyquist's 1928 paper as a reference for ''the sampling theorem of information theory'', even though that paper does not treat sampling and reconstruction of continuous signals as others did. Their glossary of terms includes these entries:
: Nyquist's result that equi-spaced data, with two or more points per cycle of highest frequency, allows reconstruction of band-limited functions. (See ''Cardinal theorem.'')
: A precise statement of the conditions under which values given at a doubly infinite set of equally spaced points can be interpolated to yield a continuous band-limited function with the aid of the function
::
Exactly what result of Nyquist they are referring to remains mysterious.
When Shannon stated and proved the sampling theorem in his 1949 paper, according to Meijering "he referred to the critical sampling interval T = 1/2W as the ''Nyquist interval'' corresponding to the band W, in recognition of Nyquist’s discovery of the fundamental importance of this interval in connection with telegraphy." This explains Nyquist's name on the critical interval, but not on the theorem.
Similarly, Nyquist's name was attached to ''
Nyquist Rate '' in 1953 by
Harold S. Black :Harold S. Black, ''Modulation Theory,'' 1953
:"If the essential frequency range is limited to ''B'' cycles per second, 2''B'' was given by Nyquist as the maximum number of code elements per second that could be unambiguously resolved, assuming the peak interference is less half a quantum step. This rate is generally referred to as and 1/2''B'' has been termed a ''Nyquist interval''." (bold added for emphasis; italics as in the original)
According to the
OED , this may be the origin of the term ''Nyquist rate''. In Black's usage, it is not a sampling rate, but a signaling rate.
- E. T. Whittaker , "On the Functions Which are Represented by the Expansions of the Interpolation Theory", Proc. Royal Soc. Edinburgh, Sec. A, vol.35, pp.181-194, 1915
- H. Nyquist , "Certain topics in telegraph transmission theory", Trans. AIEE, vol. 47, pp. 617-644, Apr. 1928 Reprint as classic paper in: ''Proc. IEEE, Vol. 90, No. 2, Feb 2002 .
- Karl Küpfmüller , "Utjämningsförlopp inom Telegraf- och Telefontekniken", ("Transients in telegraph and telephone engineering"), Teknisk Tidskrift, no. 9 pp.153-160 and 10 pp.178-182, 1931. [http://runeberg.org/tektid/1931e/0182.html
- V. A. Kotelnikov , "On the carrying capacity of the ether and wire in telecommunications", Material for the First All-Union Conference on Questions of Communication, Izd. Red. Upr. Svyazi RKKA, Moscow, 1933 (Russian). (english translation, PDF)
- J. M. Whittaker , ''Interpolatory Function Theory'', Cambridge Univ. Press, Cambridge, England, 1935.
- C. E. Shannon , "Communication in the presence of noise", Proc. Institute of Radio Engineers, vol. 37, no.1, pp. 10-21, Jan. 1949. Reprint as classic paper in: ''Proc. IEEE'', Vol. 86, No. 2, (Feb 1998)
- J. R. Higgins: ''Five short stories about the cardinal series'', Bulletin of the AMS 12(1985)
- Michael Unser: '' Sampling-50 Years after Shannon '', Proc. IEEE, vol. 88, no. 4, pp. 569-587, April 2000
