Variance and Standard Error of OLS Estimators and Gauss-Markov Theorem

In the previous article, Assumptions of Classical Linear Regression Model (CLRM), we discussed the assumptions of CLRM. In this article, we discuss the variance and standard error of OLS estimators and the Gauss-Markov Theorem.

It has been shown in the post Simple Linear Regression Model that the OLS estimates are a function of sample values of X and Y. But since the data are likely to change from sample to sample (X is fixed in repeated sampling), the estimates will also change from sample to sample. How much will our sample estimates change if we repeat the hypothetical process of repeated sampling? In other words, how accurate or precise are our sample estimates?

This accuracy and precision of OLS estimates can be measured by their standard error. Given the Gaussian assumptions as discussed in the post Assumptions of Classical Linear Regression Model (CLRM), the variance and standard error of OLS estimates can be obtained as follows:

Table of Contents

Variance and Standard Error of OLS Estimators

Variance of $\hat{\beta}_1$

$Var(\hat{\beta}_1)=\frac{\sigma^2}{\sum x_i^2}$

Standard error of $\hat{\beta}_1$

$se(\hat{\beta}_1)=\frac{\sigma}{\sqrt{\sum x_i^2}}$

Variance of $\hat{\beta}_0$

$Var(\hat{\beta}_0)=\frac{\sum X_i^2}{n\sum x_i^2}\sigma^2$

Standard error of $\hat{\beta}_0$

$se(\hat{\beta}_0)=\sqrt{\frac{\sum X_i^2}{n\sum x_i^2}}\sigma$

where var = variance and se = standard error, and where σ² is the constant or homoscedastic variance of u_i,which is unknown since the population is unknown; therefore, we estimate it using $\hat{\sigma}^2$ , which itself can be found using the following formula:

$\hat{\sigma}^2=\frac{\sum \hat{u}_i^2}{n-2}$

Where $\hat{\sigma}^2$ is the OLS estimator of the true but unknown $\sigma^2$ , and where the expression n − 2 is known as the number of degrees of freedom (df), $\sum \hat{u}_i^2$ is the residual sum of squares (RSS).

Once $\sum \hat{u}_i^2$ is known, $\hat{\sigma}^2$ can be easily computed. $\sum \hat{u}_i^2$ itself can be computed either from $\sum (Y_i - \hat{Y}_i)^2$ or from the following expression:

$\sum \hat{u}_i^2 = \sum y_i^2 - \hat{\beta}_1^2 \sum x_i^2$

$\sum \hat{u}_i^{2} = \sum y_i^{2} - \frac{\left(\sum x_i y_i\right)^{2}}{\sum x_i^{2}}$

Note that the positive square root of $\hat{\sigma}^2$ is known as the standard error of estimate (SEE) or standard error of regression (SER) given as:

The variance of $\hat{\beta}_1$ is directly proportional to $\sigma}^2$ but inversely proportional to $\sum x_i^2$ . That is, given σ², the larger the variation in the X values, the smaller the variance of $\hat{\beta}_1$ and, hence, the greater the precision with which β₂ can be estimated. Also, given $\sum x_i^2$ , the larger the variance of σ², the larger the variance of $\hat{\beta}_1$ . As n increases, the precision with which β₂ can be estimated also increases.
The variance of $\hat{\beta}_0$ is directly proportional to σ² and $\sum X_i^2$ but inversely proportional to $\sum x_i^2$ and the sample size n.
Since $\hat{\beta}_0$ and $\hat{\beta}_1$ are estimators, they will not only vary from sample to sample but in a given sample they are likely to be dependent on each other, this dependence being measured by the covariance between them, given as

$cov(\hat{\beta}_0,\hat{\beta}_1)=-\bar{X}\,var(\hat{\beta}_1)$

$=-\bar{X}\frac{\sigma^2}{\sum x_i^2}$

Since var ( $\hat{\beta}_1$ ) is always positive. The nature of the covariance between $\hat{\beta}_0$ and $\hat{\beta}_1$ depends on the sign of $-\bar{X}$ . If $\bar{X}$ is positive, then as the formula shows, the covariance will be negative.

Thus, if the slope coefficient β₁ is overestimated (i.e., the slope is too steep), the intercept coefficient β₀ will be underestimated (i.e., the intercept will be too small).

Gauss-Markov Theorem (Finite Sample Properties of OLS Estimators)

Given the assumptions of the classical linear regression model (CLRM), the OLS estimators have the minimum variance among the class of all linear unbiased estimators; that is, they are the best linear unbiased estimators (BLUE).

The best linear unbiased property of OLS estimators is explained below.

$\hat{\beta}_0$ and $\hat{\beta}_1$ are Linear estimators; that is, they are linear functions of the random variable Y.
They are Unbiased, that is, $E(\hat{\beta}_0)=\beta_0,\; E(\hat{\beta}_1)=\beta_1$ . Therefore, in repeated applications, on average $\hat{\beta}_0$ and $\hat{\beta}_1$ will converge with their true values $\beta_0$ and $\beta_1$ , respectively.
They are Best, that is they have minimum variance in the class of all linear unbiased estimators.

Sampling Distribution of OLS Estimators

The sampling distribution of an estimator is simply a probability or frequency distribution of the estimator, that is, a distribution of the set of values of the estimator obtained from all possible samples of the same size from a given population. Sampling distributions are used to draw inferences about the values of the population parameters on the basis of the values of the estimators calculated from one or more samples.

The sampling distribution of the OLS estimator $\hat{\beta}_1$ is the distribution of the values taken by $\hat{\beta}_1$ in repeated sampling experiments. Suppose that we take 100 random samples from a given population of the same size ‘n’, its probability distribution is shown in Figure 1(a).

If we assume that the $\hat{\beta}_1$ is distributed symmetrically, as shown, the mean or expected value of $\hat{\beta}_1$ is equal to $\beta_1$ . This is the unbiasedness property of $\hat{\beta}_1$ . Thus, $\hat{\beta}_1$ is the unbiased estimator of $\beta_1$ . An estimator is said to be unbiased if its expected or mean value is equal to its true parameter, that is $E(\hat{\beta})=\beta$ .

In Figure 1(b), we show the sampling distribution of $\beta_1^*$ . It is an alternative estimator of $\beta_1$ obtained by using another (i.e., other than OLS) method. Assume that $\beta_1^*$ like $\hat{\beta}_1$ is unbiased and linear. Which estimator should we rely on?

To answer this question, we have combined both parts of Figure 1 in Figure 1(c). You can observe that $\beta_1^*$ is more widely spread around the mean value than the distribution of $\hat{\beta}_1$ . In other words, the variance of $\beta_1^*$ is larger than the variance of $\hat{\beta}_1$ , that is, $Var(\hat{\beta}_1) < Var(\beta_1^*)$ .

Thus, we can say that though both $\hat{\beta}_1$ and $\beta_1^*$ are linear and unbiased estimators of $\beta_1$ , $\hat{\beta}_1$ is more efficient than $\beta_1^*$ , since the former has smaller variance. Therefore, $\hat{\beta}_1$ is the best estimator of $\beta_1$ . An estimator is said to be best if it has minimum variance among all linear unbiased estimators.

Some Concepts

Expected Value of OLS Estimator

The expected value of the OLS estimator is the mean value of the estimator in its sampling distribution.

Standard Error

Standard error is the standard deviation of the sampling distribution of the estimator.

Linear Estimator

An estimator is said to be linear if it can be expressed as a linear function of the dependent variable, Y

Unbiased Estimator

An estimator is said to be unbiased if its expected or mean value is equal to its true parameter, that is $E(\hat{\beta})=\beta$ .

Best Estimator

An estimator is said to be best if it has minimum variance among all linear unbiased estimators.

Efficient Estimator

An estimator is said to be efficient if it has minimum variance among all unbased estimators.

Finite Sample Properties of OLS Estimators

The BLUE (Best Linear Unbiased Estimator) property of OLS is a finite sample property of OLS estimator. BLUE property of the OLS estimator holds as long as the Gauss-Markov assumptions hold.

Asymptotic Properties of OLS Estimators

Asymptotic properties are the properties that hold only if the sample size is very large (technically, infinite). These properties are consistency, asymptotic unbiasedness, asymptotic normality, and asymptotic efficiency.

Share this article

Muhammad Minhaj Akhtar

Muhammad Minhaj Akhtar is a Lecturer in Economics at Government Graduate College Jauharabad, Pakistan. He holds an M.Phil. in Economics from Quaid-i-Azam University, Islamabad, and an MSc in Economics from the University of Sargodha, where he earned a Silver Medal. His academic passion lies in Econometrics, with a strong focus on applying empirical methods to real-world economic issues. Through MinhajMetrixHub, he shares learning resources, research guidance, and practical econometric insights for students and researchers.

Read Posts

Variance and Standard Error of OLS Estimators and Gauss-Markov Theorem

Variance and Standard Error of OLS Estimators

Features of Variance and Standard Error of OLS Estimators

Gauss-Markov Theorem (Finite Sample Properties of OLS Estimators)

Sampling Distribution of OLS Estimators

Share this article

Muhammad Minhaj Akhtar

Leave a Reply Cancel reply

CATEGORIES

TAGS

Recent Posts

Sen Capability Approach

Assumptions of Classical Linear Regression Model (CLRM)

Regression Through Origin

Education and Economic Development

A Numerical Example of Multiple Linear Regression by Hand