Solving Ordinary Least Squares (OLS) Regression Using Matrix Algebra

2019-01-30

Tags:StatisticsR

In psychology, we typically learn how to calculate OLS regression by calculating each coefficient seperately. However, I recently learned how to calculate this using matrix algebra. Here is a brief tutorial on how to perform this using R.

The Salaries dataset is from the carData package, which shows the salary of professors in the US during the academic year of 2008 and 2009. Let’s say we are interested in determining if professors who have had their Ph.D. degree for longer are more likely to also have higher salaries.

Solve Using Matrix Algebra

Design Matrix

The design matrix is just a dataset of the all the predictors, which includes the intercept set at 1 and yrs.since.phd.

After multiplication, the matrix provides the total number of participants (\(n\) = 397; really, the sum of the intercept), sum of yrs.since.phd (\(\Sigma(yrs.since.phd)\) = 0), and sum of squared yrs.since.phd (\(\Sigma (yrs.since.phd^2)\) = 65765.64). Respectively, \(\Sigma (years.since.phd)\) and \(\Sigma (yrs.since.phd^2)\) are sum of error (\(\Sigma(yrs.since.phd-M_{yrs.since.phd})\)) and sum of squared error (\(\Sigma(yrs.since.phd-M_{yrs.since.phd})^2\)) because we first centered the yrs.since.phd variable.

Coefficients

To obtain the coefficients, we can multiply these last two matrices (\(b = (X'X)^{-1}X'Y\)).

coef <- x_prime_x_inverse %*% x_prime_y
coef

## [,1]
## intercept 113706.4584
## yrs.since.phd 985.3421

Standard Error

To calculate the standard error, we multiply the inverse matrix of \(X'X\) by the mean squared error (MSE) of the model and take the square root of its diagonal matrix (\(\sqrt{diag((X'X)^{-1} * MSE)}\)).

First, we need to calculate the \(MSE\) of the model. Calculating \(MSE\) of the model is still the same, \(MSE = \frac{\Sigma(Y-\hat{Y})^{2}}{n-p} = \frac{\Sigma(e^2)}{df}\) where \(Y\) is the DV, \(\hat{Y}\) is the predicted DV, \(n\) is the total number of participants (or data points), and \(p\) is the total number of variables in the design matrix (or predictors, which includes the intercept).

To obtain the predicted values (\(\hat{Y}\)), we can also use matrix algebra by multiplying the design matrix with the coefficients (\(\hat{Y} = Xb\)).