Weighted Least Squares Article Index for
Weighted 		Shopping
Least 		Website Links For
Squares 		 

Information About

Weighted Least Squares
  CATEGORIES ABOUT WEIGHTED LEAST SQUARES
   
regression analysis
   
APPAREL
BABY
BEAUTY
BOOKS
CAR TOYS
CELL PHONES
DVD'S
ELECTRONICS
GOURMET FOOD
GROCERIES
HEALTH & PERSONAL
HOME & GARDEN
JEWELRY
MUSIC
MUSIC INSTRUMENTS
OFFICE PRODUCTS
SOFTWARE
SPORTING GOODS
TOOLS & HARDWARE
TOYS
VIDEO GAMES
SHOPPING HOME
MORE SHOPPING...



: S = \sum_{i=1}^n (y_i - f(x_i))^2.

However, instead of weighting all points equally, they are weighted such that points with a greater weight contribute more to the fit:
: S = \sum_{i=1}^n w_i(y_i - f(x_i))^2.

Often, ''w''i is given as the inverse of the Variance , giving points with a lower variance a greater statistical weight:

: w_i = 1/\sigma_i^2.

In a Linear Regression context,

: f(\mathbf x_i) = \mathbf x_i^{\mathrm T} \boldsymbol\beta.

Denote

  • = w_i^{1/2} y_i, \quad \mathbf x_i^--- = w_i^{1/2} \mathbf x_i,


then minimizing the Weighted Least Squares

: S = \sum_{i=1}^n w_i(y_i - \mathbf x_i^{\mathrm T} \boldsymbol\beta)^2,

is the same as minimizing the Ordinary Least Squares

  • - \mathbf x_i^{---\mathrm T}\boldsymbol \beta)^2.


In Matrix notation, the weighted least squares estimator of β is

  • \mathrm T}\mathbf X^---)^{-1}\mathbf X^{---\mathrm T}\mathbf Y^--- = (\mathbf X^{\mathrm T} \mathbf W\mathbf X)^{-1}\mathbf X^{\mathrm T}\mathbf W\mathbf Y, \quad \mathbf W=\operatorname{diag}\{w_1, \dots, w_n\}.



LINEAR ALGEBRAIC DERIVATION

Say we are trying to solve for an overdetermined system like

\begin{bmatrix}
1 & x_{11} & x_{12} & ... \
1 & x_{21} & x_{22} & ... \
1 & x_{31} & x_{32} & ... \
1 & ... & ... & ... \end{bmatrix}
\begin{bmatrix} a_0 \ a_1 \ a_2 \ ... \end{bmatrix} =
\begin{bmatrix} y_1 \ y_2 \ y_3 \ ... \end{bmatrix}

Which we can denote as

X \bold{a} = \bold{y}.

We wish to solve for a.
The least squares solution to this problem will be

\bold{a} = (X^T X)^{-1} X^T \bold{y}


Weighted Case

Now, if we want a weighted least-squares solution, we can simply
multiply each row of X, and each row of y by an appropriate weight, w.


  • \ w_2 --- \ w_3 --- \ ... \end{matrix}

    • \begin{bmatrix}

1 & x_{11} & x_{12} & ... \
1 & x_{21} & x_{22} & ... \
1 & x_{31} & x_{32} & ... \
1 & ... & ... & ... \end{bmatrix}
\begin{bmatrix} a_0 \ a_1 \ a_2 \ ... \end{bmatrix} =
  • \ w_2 --- \ w_3 --- \ ... \end{matrix}

    • \begin{bmatrix} y_1 \ y_2 \ y_3 \ ... \end{bmatrix}


Which we can refer to as:

X_w \bold{a} = \bold{y}_w

Which is the same form as the regular least-squares case, but with weighted versions
of the X matrix and y vector.
The same solution used for the basic least-squares case now yields a weighted solution.