I have read a lot of advantages of regularisation methods (e.g. ridge, lasso), but in what cases might multiple regression with Ordinary Least Squares be more desirable? For example is there a downside to using ridge regression for n >> p problems (where n is the number of samples and p is the number of features), or when there is little multicollinearity between predictor variables? If not then I'm wondering what OLS is good for (except in cases where there's only one predictor variable of course).

$\begingroup$Could you give me a (concrete) example of a situation where one wants unbiased coefficients and doesn't care about statistical errors? Thanks!$\endgroup$
– user795305Oct 20 '17 at 13:43

1

$\begingroup$One aspect of multicollinearity is that small changes in the data set can make big changes in the parameter estimates. If you care about the coeffiicents at all then you have to care about collinearity. The only time when you do NOT care about it, as far as I know, is when your only aim is prediction. And the number of predictors isn't relevant in any way that I can see. Even with only 2 predictors, there could be collinearity.$\endgroup$
– Peter Flom♦Oct 20 '17 at 22:00

$\begingroup$Can you confirm: when you say the parameters in ridge regression are biased, do you mean they have been shrunk? In ridge regression the parameters are shrunk but their relative importances (e.g. b1=0.02 is double as important as b2=0.01) are still at least given?$\endgroup$
– Oliver AngelilOct 20 '17 at 19:41

1

$\begingroup$I don't think they are necessarily shrunk - they might be increased. And I don't think their relative importance is necessarily the same.$\endgroup$
– Peter Flom♦Oct 20 '17 at 21:58

$\begingroup$Ok. In my case I'm not interested in making predictions. I'm just interested in identifying the relative importance of each one of my X variables to explain the y variable. There is some multicollinearity between the X variables (VIF between ~3 and ~7). I think ridge would be the way to go?$\endgroup$
– Oliver AngelilOct 21 '17 at 0:03

3

$\begingroup$I'm not sure there is a good answer to this. Ridge results in bias and, unless the bias is uniform in a particular way, the ratio of two coefficients could be affected a lot. When variables are colinear, the question of how important each is doesn't really make sense. You could look at each in a bivarite regression with the Y variable, that might be the best answer. But then you are not controlling for other variables.$\endgroup$
– Peter Flom♦Oct 21 '17 at 12:39