Simple Linear Regression Model

Table of Contents

Simple Linear Regression Model

Simple Linear Regression Model is used to estimate the relationship between one dependent and one independent variable. It is also called bivariate or two variable regression model. For example, regression of consumption on disposable income, regression of sales revenue on advertisement expenses, regression of log of wages on years of education. In previous lectures, we discussed the concepts of population and sample regression function.

Least Square Principle

From previous blogs, we found out the primary objective was to estimate PRF.

$Y_i = \beta_0 + \beta_1 X_i + u_i$ … 1

on the basis of the SRF

${ Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{u}_i$ … 2

We cannot estimate the PRF accurately because of sampling fluctuations. Therefore, given a sample, we approximate the true relationship by estimating the SRF. But how is the SRF itself estimated from the sample? This is the main focus of this post.

We want to derive SRF such that it is as close as possible to the actual or observed values of Y. It can be done by choosing SRF so that the difference between the actual Y and the estimated Y is as small as possible. We have various criterions to minimize this distance. These are:

Minimize the sum of errors i.e. $\sum e_i$
Minimise the absolute sum of errors i.e. $\sum |e_i|$
Minimise the weighted sum of absolute errors i.e. $\sum w_i |e_i|$

Minimize the sum of residuals i.e. $\sum e_i$

Minimizing the sum of residuals is not a good criterion because of two reasons. Firstly, sum of residuals are zero because positive and negative errors cancel out each other or too small even though errors are widely spread around the regression line. If we minimise sum of residuals this would be misleading. Secondly, each error gets equal weight in the sum $(u_1 + u_2 + u_3 + u_4)$ though large or small. In other words, large and small errors have equal importance and are treated as same.

ei	Case A	Case B	Case C	Case D
e1	500	0	0	\|14\|
e2	-900	10	0	\|-14\|
e3	-500	0	50	14\|
e4	900	0	0	\|-14\|
Sum of ei	0	10	50	56

In table 1 case B is more desirable than case A, although it has non-zero sum, but error is small but in case A although sum of residuals is zero, but the errors are large.

Minimise the absolute sum of residuals i.e. $\sum |e_i|$

The second option we have is to take the absolute sum of residuals so that the sum of residuals cannot be zero. There is another problem in this method, that it ignores outliers. Suppose that in case C we have one outlier while other errors are zero. If we fit the line, the fit might not be as good because it ignores the outlier. In case D we have relatively large errors in absolute terms, but we might like case D because it considers all the errors value, but still, it has a problem of equal weights.

Minimise the weighted sum of absolute residuals i.e. $\sum w_i |e_i|$

The third option we have is to minimize the sum of squared residuals to give more importance to large errors and less importance to small errors and then minimize the errors. Let wi =|ei|

$\sum w_i |e_i|$

$\sum |e_i| \, |e_i|$

$\sum (|e_i|)^2$

$\sum e_i^2$

This is the basis of the Ordinary Least Squares Method. So, minimising the squared residuals has several advantages over either minimising the sum of residuals or minimising the sum of absolute residuals. Firstly, the sum of squared residuals is not always zero and, secondly, it gives more importance to large errors and little importance to small errors. A further justification for the least-squares method lies in the fact that the estimators obtained by it have some very desirable statistical properties, as we shall see later.

Least square principle is the mathematical procedure that uses the data to position a line with the objective of minimizing sum of the squared vertical distance between the actual Y values and the predicted values of Y.

In other words, the least-squares method chooses the $\hat{\beta}_0$ and $\hat{\beta}_1$ in such a manner that in a given set of data $\sum e_i^2$ is as small as possible.

Derivation of OLS

Consider a two-variable population regression model

$Y_i = \beta_0 + \beta_1 X_i + u_i$

which we estimate using SRF

$\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i$

Residuals are given as:

$\hat{u}_i = Y_i - \hat{Y}_i$

$\hat{u}_i = Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i$

Squaring the residuals

$SSR = \sum_{i=1}^{n} \hat{u}_i^2$

$SSR = \sum_{i=1}^{n} (Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i)^2$ … (1)

Take partial derivative of equation 1 w.r.t $\hat{\beta}_0$ and set it equal to zero

$\frac{\partial SSR}{\partial \hat{\beta}_0} = -2 \sum_{i=1}^{n} (Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i) = 0$

$\sum_{i=1}^{n} Y_i = n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^{n} X_i$ … (2)

Now take partial derivative of equation 1 w.r.t $\hat{\beta}_1$ and set it equal to zero

$\frac{\partial SSR}{\partial \hat{\beta}_1} = -2 \sum_{i=1}^{n} X_i (Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i) = 0$

$\sum_{i=1}^{n} X_i Y_i = \hat{\beta}_0 \sum_{i=1}^{n} X_i + \hat{\beta}_1 \sum_{i=1}^{n} X_i^2$ … (3)

Solving equations (2) and (3) simultaneously, we get formulas of $\hat{\beta}_0$ and $\hat{\beta}_1$

$\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^{n} (X_i - \bar{X})^2}$

$\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}$

Numerical Properties of OLS Estimators

Numerical properties are those that hold as a result of the use of ordinary least squares, regardless of how the data were generated. These properties are:

The OLS estimators are expressed solely in terms of the observable (i.e., sample) quantities (i.e., X and Y). Thus, we have given the data of X and Y we can easily estimate the values of $\hat{\beta}_0$ and $\hat{\beta}_1$ .
OLS estimators are point estimator, it means they provide only a single value of population parameter. Interval estimators are those which provide a range of values for population parameter.
Once the OLS estimates are obtained, we can easily draw sample regression line. This sample line has the following five properties.

1. It passes through the sample means of Y and X. It can be seen from the following equation.

$\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}$

2. The mean value of the estimated $\hat{Y}_i$ is equal to the mean value of the actual Y. That is.

$\bar{\hat{Y}} = \bar{Y}$

3. The mean value of the residuals $Y = \hat{Y}_i$ is zero.

$E(\hat{u}_i) = 0$

4. The residuals $Y = \hat{Y}_i$ are uncorrelated with the predicted Y.

$\operatorname{Cov}(\hat{Y}_i, \hat{u}_i) = 0$

5. The residuals $Y = \hat{Y}_i$ are uncorrelated with X.

$\operatorname{Cov}(X_i, \hat{u}_i) = 0$

Another formula for the slope coefficient can be written as:

$\hat{\beta}_1 = r_{xy} \cdot \left( \frac{s_y}{s_x} \right)$

where rxy is the sample correlation between xi and yi and sx, sy denote the sample standard deviations.

Statistical Properties of OLS Estimators

The statistical properties of OLS estimators are those that hold only under certain assumptions about the way the data were generated. Statistical properties of OLS estimators are of two categories:

Finite or small sample properties (properties that hold regardless of the sample size under certain assumptions known as Gauss-Markov assumptions): These properties include unbiasedness, minimum variance and efficiency.
Asymptotic or large sample properties (properties that hold only if the sample size is very large (technically, infinite): These properties include consistency, asymptotic unbiasedness, asymptotic efficiency. asymptotic normality.

Some Concepts

Conditional Mean

The conditional mean in regression analysis is the expected value of the dependent variable Y given a specific value of the independent variable (X), denote as E (Y| Xi) It is expressed as the function of value of X, Xi written as E (Y| Xi) = f(Xi)

Stochastic Error Term

Stochastic error term of error term is PRF shows the difference between actual or observed value of Y and conditional mean value of Y. That is, $u_i = Y_i - E(Y \X_i)$ .

In other words, it represents the vertical distance between each value of Y and its conditional mean. The error term can be positive or negative. It is positive when Y>E(Y|Xi). It is negative when Y<E(Y|Xi). It is zero when Y=E(Y|Xi).

Thus, Y fluctuates around E(Y|Xi). Why? It is because there are lot of unobserved factors that affect Y but cannot be included in our model. These are omitted factors and error term captures their collective effect.

Error vs Residual

Error (µ) is the error of specification. It can be controlled if we choose a good model or we able to extract as much information from µ. Residual (e) is the error of estimation. It is inevitable. In OLS our task is to minimise the estimation error.

Interpretation of Simple Linear Regression Estimates

After estimating simple linear regression coefficients, it is important to interpret its results. Remember that in SLRM a dependent variable is expressed as a linear function of only one independent variable. The slope coefficient $\hat{\beta}_1$ in SLRM measures the change in Y for a one-unit change in X. In other words, the slope coefficient tells us if X increases by one unit, then Y will increase or decrease by how much (depending on the sign of the coefficient).

Or the slope $\hat{\beta}_1$ means that, for each one-unit increase in X, the mean value of Y, i.e., $\hat{Y}$ , changes by the magnitude of $\hat{\beta}_1$ units. The interpretation of the slope coefficient depends on the units of the dependent and independent variables in which they are measured. Consider a couple of examples:

Suppose Y = per-house cost in USD, X= house size in square feet. $\hat{\beta}_1$ =75, it means for each one square ft. increase in house size, its cost increases by USD 75.
Consider another example: Y = quantity consumed in kg, X=price in PKR. $\hat{\beta}_1$ =2.716, it means that for each PKR increase in price, quantity consumed will fall by 2.716 kg.

Now consider the interpretation of the intercept, also known as the constant, $\hat{\beta}_0$ . It is the mean value of the dependent variable when X=0. If $\hat{\beta}_0$ =25, it means that the model predicts an expected mean quantity consumed of 25 kg when the price is zero (although this makes no economic sense, but truly we interpret the intercept in this way)

Suggestions for further reading:

Share this article

About the author

Muhammad Minhaj Akhtar

Muhammad Minhaj Akhtar is a Lecturer in Economics at Government Graduate College Jauharabad, Pakistan. He holds an M.Phil. in Economics from Quaid-i-Azam University, Islamabad, and an MSc in Economics from the University of Sargodha, where he earned a Silver Medal. His academic passion lies in Econometrics, with a strong focus on applying empirical methods to real-world economic issues. Through MinhajMetrixHub, he shares learning resources, research guidance, and practical econometric insights for students and researchers.