Commercial manufactures such as Kodak, Agfa, etc. have long had access to manufacturing methods and chemicals that are not available to the average person. This allowed them to achieve higher film speeds in their emulsions. You are right that in the past films speeds were set more conservatively. This was done to prevent underexposure and insure that the user always got an image. Remember that light meters were not common or very accurate at this time. This practice also allowed products such as Acufine and Diafine to claim speed increases that weren't actually true.

A rock pile ceases to be a rock pile the moment a single man contemplates it, bearing within him the image of a cathedral.