Explained sum of squares

The explained sum of squares (ESS) is the sum of the squares of the deviations of the predicted values from the mean value of a response variable, in a standard regression model — for example, y_i = a + b₁x_1i + b₂x_2i + ... + ε_i, where y_i is the i ^th observation of the response variable, x_ji is the i ^th observation of the j ^th explanatory variable, a and b_j are coefficients, i indexes the observations from 1 to n, and ε_i is the i ^th value of the error term. In general, the greater the ESS, the better the estimated model performs.

If${\displaystyle {\hat {a}}}$  and ${\displaystyle {\hat {b}}_{i}}$  are the estimated coefficients, then

${\displaystyle {\hat {y}}_{i}={\hat {a}}+{\hat {b}}_{1}x_{1i}+{\hat {b}}_{2}x_{2i}+\cdots \,}$ 

is the i ^th predicted value of the response variable. The ESS is then:

${\displaystyle {\text{ESS}}=\sum _{i=1}^{n}\left({\hat {y}}_{i}-{\bar {y}}\right)^{2}.}$ 

where ${\displaystyle {\hat {y}}_{i}}$  is the value estimated by the regression line .^[1]

In some cases (see below): total sum of squares (TSS) = explained sum of squares (ESS) + residual sum of squares (RSS).

The following equality, stating that the total sum of squares (TSS) equals the residual sum of squares (=SSE : the sum of squared errors of prediction) plus the explained sum of squares (SSR :the sum of squares due to regression or explained sum of squares), is generally true in simple linear regression:

${\displaystyle \sum _{i=1}^{n}\left(y_{i}-{\bar {y}}\right)^{2}=\sum _{i=1}^{n}\left(y_{i}-{\hat {y}}_{i}\right)^{2}+\sum _{i=1}^{n}\left({\hat {y}}_{i}-{\bar {y}}\right)^{2}.}$ 

${\displaystyle {\begin{aligned}(y_{i}-{\bar {y}})=(y_{i}-{\hat {y}}_{i})+({\hat {y}}_{i}-{\bar {y}}).\end{aligned}}}$ 

Square both sides and sum over all i:

${\displaystyle \sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}=\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2}+\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}2({\hat {y}}_{i}-{\bar {y}})(y_{i}-{\hat {y}}_{i}).}$ 

Here is how the last term above is zero from simple linear regression^[2]

${\displaystyle {\hat {y_{i}}}={\hat {a}}+{\hat {b}}x_{i}}$ 

${\displaystyle {\bar {y}}={\hat {a}}+{\hat {b}}{\bar {x}}}$ 

${\displaystyle {\hat {b}}={\frac {\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})}{\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}}}}$ 

So,

${\displaystyle {\hat {y_{i}}}-{\bar {y}}={\hat {b}}(x_{i}-{\bar {x}})}$ 

${\displaystyle y_{i}-{\hat {y}}_{i}=(y_{i}-{\bar {y}})-({\hat {y}}_{i}-{\bar {y}})=(y_{i}-{\bar {y}})-{\hat {b}}(x_{i}-{\bar {x}})}$ 

Therefore,

${\displaystyle {\begin{aligned}&\sum _{i=1}^{n}2({\hat {y}}_{i}-{\bar {y}})(y_{i}-{\hat {y}}_{i})=2{\hat {b}}\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\hat {y}}_{i})\\[4pt]={}&2{\hat {b}}\sum _{i=1}^{n}(x_{i}-{\bar {x}})((y_{i}-{\bar {y}})-{\hat {b}}(x_{i}-{\bar {x}}))\\[4pt]={}&2{\hat {b}}\left(\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}})-\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2}{\frac {\sum _{j=1}^{n}(x_{j}-{\bar {x}})(y_{j}-{\bar {y}})}{\sum _{j=1}^{n}(x_{j}-{\bar {x}})^{2}}}\right)\\[4pt]={}&2{\hat {b}}(0)=0\end{aligned}}}$ 

The general regression model with n observations and k explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is

${\displaystyle y=X\beta +e}$ 

where y is an n × 1 vector of dependent variable observations, each column of the n × k matrix X is a vector of observations on one of the k explanators, ${\displaystyle \beta }$  is a k × 1 vector of true coefficients, and e is an n × 1 vector of the true underlying errors. The ordinary least squares estimator for ${\displaystyle \beta }$ is

${\displaystyle {\hat {\beta }}=(X^{T}X)^{-1}X^{T}y.}$ 

The residual vector ${\displaystyle {\hat {e}}}$ is${\displaystyle y-X{\hat {\beta }}=y-X(X^{T}X)^{-1}X^{T}y}$ , so the residual sum of squares ${\displaystyle {\hat {e}}^{T}{\hat {e}}}$  is, after simplification,

${\displaystyle RSS=y^{T}y-y^{T}X(X^{T}X)^{-1}X^{T}y.}$ 

Denote as ${\displaystyle {\bar {y}}}$  the constant vector all of whose elements are the sample mean ${\displaystyle y_{m}}$  of the dependent variable values in the vector y. Then the total sum of squares is

${\displaystyle TSS=(y-{\bar {y}})^{T}(y-{\bar {y}})=y^{T}y-2y^{T}{\bar {y}}+{\bar {y}}^{T}{\bar {y}}.}$ 

The explained sum of squares, defined as the sum of squared deviations of the predicted values from the observed mean of y, is

${\displaystyle ESS=({\hat {y}}-{\bar {y}})^{T}({\hat {y}}-{\bar {y}})={\hat {y}}^{T}{\hat {y}}-2{\hat {y}}^{T}{\bar {y}}+{\bar {y}}^{T}{\bar {y}}.}$ 

Using ${\displaystyle {\hat {y}}=X{\hat {\beta }}}$  in this, and simplifying to obtain ${\displaystyle {\hat {y}}^{T}{\hat {y}}=y^{T}X(X^{T}X)^{-1}X^{T}y}$ , gives the result that TSS = ESS + RSS if and only if ${\displaystyle y^{T}{\bar {y}}={\hat {y}}^{T}{\bar {y}}}$ . The left side of this is ${\displaystyle y_{m}}$  times the sum of the elements of y, and the right side is ${\displaystyle y_{m}}$  times the sum of the elements of ${\displaystyle {\hat {y}}}$ , so the condition is that the sum of the elements of y equals the sum of the elements of ${\displaystyle {\hat {y}}}$ , or equivalently that the sum of the prediction errors (residuals) ${\displaystyle y_{i}-{\hat {y}}_{i}}$  is zero. This can be seen to be true by noting the well-known OLS property that the k × 1 vector ${\displaystyle X^{T}{\hat {e}}=X^{T}[I-X(X^{T}X)^{-1}X^{T}]y=0}$ : since the first column of X is a vector of ones, the first element of this vector ${\displaystyle X^{T}{\hat {e}}}$  is the sum of the residuals and is equal to zero. This proves that the condition holds for the result that TSS = ESS + RSS.

In linear algebra terms, we have ${\displaystyle RSS=\|y-{\hat {y}}\|^{2}}$ , ${\displaystyle TSS=\|y-{\bar {y}}\|^{2}}$ , ${\displaystyle ESS=\|{\hat {y}}-{\bar {y}}\|^{2}}$ . The proof can be simplified by noting that ${\displaystyle {\hat {y}}^{T}{\hat {y}}={\hat {y}}^{T}y}$ . The proof is as follows:

${\displaystyle {\hat {y}}^{T}{\hat {y}}=y^{T}X(X^{T}X)^{-1}X^{T}X(X^{T}X)^{-1}X^{T}y=y^{T}X(X^{T}X)^{-1}X^{T}y={\hat {y}}^{T}y,}$ 

Thus,

${\displaystyle {\begin{aligned}TSS&=\|y-{\bar {y}}\|^{2}=\|y-{\hat {y}}+{\hat {y}}-{\bar {y}}\|^{2}\\&=\|y-{\hat {y}}\|^{2}+\|{\hat {y}}-{\bar {y}}\|^{2}+2\langle y-{\hat {y}},{\hat {y}}-{\bar {y}}\rangle \\&=RSS+ESS+2y^{T}{\hat {y}}-2{\hat {y}}^{T}{\hat {y}}-2y^{T}{\bar {y}}+2{\hat {y}}^{T}{\bar {y}}\\&=RSS+ESS-2y^{T}{\bar {y}}+2{\hat {y}}^{T}{\bar {y}}\end{aligned}}}$ 

which again gives the result that TSS = ESS + RSS, since ${\displaystyle (y-{\hat {y}})^{T}{\bar {y}}=0}$ .

• Sum of squares (statistics)
• Lack-of-fit sum of squares
• Fraction of variance unexplained

• S. E. Maxwell and H. D. Delaney (1990), "Designing experiments and analyzing data: A model comparison perspective". Wadsworth. pp. 289–290.
• G. A. Milliken and D. E. Johnson (1984), "Analysis of messy data", Vol. I: Designed experiments. Van Nostrand Reinhold. pp. 146–151.
• B. G. Tabachnick and L. S. Fidell (2007), "Experimental design using ANOVA". Duxbury. p. 220.
• B. G. Tabachnick and L. S. Fidell (2007), "Using multivariate statistics", 5th ed. Pearson Education. pp. 217–218.