Deriving covariance and variances of coefficients <slope and intercept> of Simple Linear Regression <Var(beta), Cov(beta0, beta1)>

数学

Suppose a simple linear regression model:

Y_i = \beta _0 + \beta _ 1 X_i + \varepsilon _ i

This post will explain how to obtain the following formulae:

\displaystyle Var ( \hat{\beta _ 0} ) = \sigma ^ 2 \left ( \frac{1}{n} + \frac{\bar{x}_{n}^{2}}{s^{2}_{x}} \right )

\displaystyle Var( \hat{\beta _ 1}) = \frac{\sigma ^2}{s^{2}_{x}}

\displaystyle Cov( \hat{\beta _ 0}, \hat{\beta _ 1}) = -\frac{\bar{x}_n \sigma^{2}}{s^{2}_{x}}

① \displaystyle Var( \hat{\beta _ 1}).

\begin{align} \displaystyle Var( \hat{\beta _ 1}) &= Var \left ( \frac{ \sum ^{n}_{i=1} (X_i - \bar{X} ) Y_i }{s^{2}_{x}} \right ) \\ &= \frac{1}{s^{4}_{x}} Var ( \sum ^{n}_{i = 1} (X_i - \bar{X} ) (\beta _{0} + \beta _{1} X_{i} + \varepsilon _{i}))\\ &= \frac{1}{s^{4}_{x}} Var ( \sum ^{n}_{i = 1} (X_i - \bar{X} ) \varepsilon _{i})\\ &= \frac{1}{s^{4}_{x}} \sum ^{n}_{i = 1} Var ( (X_i - \bar{X} ) \varepsilon _{i})\\ &= \frac{1}{s^{4}_{x}} \sum ^{n}_{i = 1} (X_i - \bar{X} ) Var (\varepsilon _{i})\\ &= \frac{1}{s^{4}_{x}} \sum ^{n}_{i = 1} (X_i - \bar{X} ) σ^{2}\\ &= \frac{ σ^{2}}{s^{4}_{x}} \sum ^{n}_{i = 1} (X_i - \bar{X} )\\ &= \frac{ σ^{2}}{s^{4}_{x}} s^{2}_{x}\\ &= \frac{ σ^{2}}{s^{2}_{x}} \end{align}

  • Line 1→2: Y_i = \beta _0 + \beta _ 1 X_i + \varepsilon _ i
  • Line 2→3: We consider \beta _0 + \beta _ 1 X_i as constant, and Var(A+constant) = Var(A)
  • Line 3→4: We assume homoscedasticity

② \displaystyle Var ( \hat{\beta _ 0} )

\begin{align} \displaystyle Var ( \hat{\beta _ 0} ) &= Var(\bar{Y} - \beta_{1} \bar{X} )\\ &= Var( \bar{Y} ) + \bar{X}^{2} Var(β_{1} ) - 2 \bar{X} C o v ( \bar{Y}, \hat{β} _{1} )\\ &= A + B + C \end{align} We consider each term of left hand side (= A, B, C) one by one.

A

\begin{align} \displaystyle Var( \bar{Y} ) &= Var( \frac{1}{n} \sum ^ {n} _{i=1} Y_{i} )\\ &= \frac{1}{n^{2}} n Var(Y_{i} )\\ &= \frac{σ^{2}}{n} \\ \end{align}

B

\begin{align} \displaystyle \bar{X}^{2} Var(β_{1} ) = \bar{X}^{2} \frac{ σ^{2}}{s^{2}_{x}} \end{align}

↑From previous result

C

\begin{align} \displaystyle C o v ( \bar{Y}, \hat{\beta}_{1} ) &= C o v ( \frac{1}{n} \sum_{i=1}^{n} Y_{i}, \frac{\sum_{j=1}^{n} x_{j}-\bar{x} Y_{j}}{s_{x}^{2}} ) \\ &= \frac{1}{n} \frac{1}{s_{x}^{2}} C o v ( \sum_{i=1}^{n} Y_{i}, \sum_{j=1}^{n} (X_{j}-\bar{X} ) Y_{j} \\ &=\frac{1}{n s_{x}^{2}} \sum_{i=1}^{n} ( X_{i}-\bar{X} ) \sum_{j=1}^{n} C o v ( Y_{i}, Y_{j} ) \\ &=\frac{1}{n s_{x}^{2}} \sum_{i=1}^{n} ( X_{i}-\bar{X} ) σ^{2} \\ &=\frac{σ^{2}}{n s_{x}^{2}} \sum_{i=1}^{n} ( X_{i}-\bar{X} ) \\ &=0 \end{align}

A + B + C

Therefore,

\begin{align} \displaystyle V a r ( \hat{β }_ {0} ) &= \frac{σ^{2}}{n} + \bar{X}^{2} \frac{ σ^{2}}{s^{2}_{x}} + 0 \\ &= σ^{2} ( \frac{1}{n} + \frac{ \bar{X} ^{2} }{s^{2}_{x} } ) \end{align}

③ \displaystyle Cov( \hat{\beta _ 0}, \hat{\beta _ 1})

\begin{align} C o v ( β_{1}, \hat{β}_{1} ) &=\mathbb{E} ( ( \hat{\beta}-\bar{\hat{β}}_{0} ) ( \hat{β}_{1}-\bar{\hat{β}}_{1} ) ) \\ &=\mathbb{E} ( ( \hat{β}_{0} - β_{0} ) ( \hat{β}_{1}-β_{1} ) ) \\ &=\mathbb{E} ( ( \sum_{i=1}^{n} ( \frac{1}{n} - \frac{\bar{X}(X_{i} - \bar{X} ) } {\hat{s}^{2}_{x} }) ε_{i} \sum ^ {n} _{i=1} ( \frac{X_{i}-\bar{X}}{s_{x}^{2}} ) ε_{i} ) ) \\ &=\mathbb{E} ( \varepsilon_{i}^{2} ) \sum_{i=1}^{n} ( - ( \frac{1}{n} - \frac{ \bar{X} ( X_{i}- \bar{X} ) }{s_{x}^{2}} ) \frac{X_{i}-\bar{X}}{s_{x}^{2}} ) \\ &=σ^{2} ( - \frac{\bar{X}}{ ( s_{x}^{2} ) ^{2}} \sum_{i=1}^{n} ( X_{i}-\bar{X} ) ^{2} ) \\ &=σ^{2} ( - \frac{\bar{X}}{ ( s_{x}^{2} ) ^{2}} s_{x}^{2} ) \\ &=-\frac{σ^{2} \bar{X}}{s_{x}^{2}} \end{align}