The gold rush in the mid 1800s motivated tens of thousands of people to head out to California to seek their fortunes. It was also the source of one of today's better known business aphorisms. As some savvy business people figured out, selling "picks and shovels" to all of the aspiring miners was a pretty good business itself and certainly a more reliably profitable one.

With this understanding, it is at least somewhat surprising how little attention is paid to the tools of investing, i.e., investment research. Interestingly, a number of changes have swept over the research industry in the last twenty years that have made it more accessible to a broader base. While many developments have worked against investors over that period of time, changes in research are creating opportunities to improve investment outcomes for those investors willing and able to adapt to the new landscape.

It makes sense to begin the discussion of investment research with the big Wall Street firms for a couple of reasons. One is that arguably they still represent the largest source of research. A second is that the environment for Wall Street research has changed a lot and that has broader implications for investors. Throughout this discussion we will mainly be referring to the research output of analysts who exemplify the "picks and shovels" of the knowledge economy.

Wall Street analysts, or sell-side analysts, as they are referred to in the industry, normally know their companies and industries well and produce detailed financial models. While in many industries the generic term of "analyst" often refers to a lowly administrative employee, in investment research an analyst is central to the effort and has extensive education, skills, and contacts. Many senior analysts have become celebrities of sorts.

After the crash in technology stocks in 2000 the investment information provided by the sell-side started changing. Regulation was a key driver as both Regulation FD (full disclosure) and Sarbanes Oxley (SOX) were passed shortly after the crash. Both sought to correct problems that had emerged in the tech boom. Reg FD mandated that "all publicly traded companies must disclose material information to all investors at the same time" in order to deconstruct what had become a playground of "privileged access" to information. The SOX Act "mandated strict reforms to improve financial disclosures from corporations" with the intent of preventing accounting fraud.

At the same time, the Securities and Exchange Commission (SEC) ratcheted up enforcement actions. Special attention was paid to analysts who said one thing in published research reports but another thing in private meetings. Efforts were also made to reduce or eliminate the conflicts of interest inherent between research departments and highly profitable investment banking operations at the big investment banks.

The combination of these factors had an interesting effect. On one hand, it was true that these efforts did level the playing field and that the magnitude of abuse declined significantly. On the other hand, less information got out which diminished the value of the research to those who had access to it. Partly, companies became much more deliberate and constrained about what they released to the public. Partly, a lot of talent left the industry.

With less money, less glamour, and more career risk, it's not surprising that many of the most talented analysts left for greener pastures. Some simply retired. Others moved on to other things. Mary Meeker, for example, who at one time was arguably the best analyst on internet stocks, is now a partner at venture capital firm Kleiner Perkins. Anthony Noto, who was also a star internet analyst, is now the CFO of Twitter. Many others who were established enough sought to avoid the hassles of the large banks altogether and went off on their own.

As consumers of sell-side research ourselves, we started noticing the effects of this migration fairly quickly. Over time, there were fewer and fewer exceptional sell-side analysts who really knew their stuff. While there are still exceptional analysts plying the trade, in aggregate, the quality of industry knowledge and the number of quality investment ideas seemed to diminish.

Another wave of change occurred in the research world after the financial crisis in 2008, but for different reasons. One of the biggest problems that emerged this time was the inherent web of conflicts of interest on Wall Street often enabled, or even encouraged, financial firms to actually work against the interests of their clients. Trust in the investment banks and even the financial industry as a whole plunged to record low levels as investors saw up close just how little interest and/or ability many financial intermediaries had in their well-being. Widespread disappointment motivated a sweeping search for independent, and therefore less conflicted, research and advisory services that could do a better job of serving client interests.

As it turns out, research analysts had their own reasons for going independent besides the emerging demand from investors. A key motivation was similar to that of investors - simply to escape organizations that had flawed corporate cultures and/or structural conflicts of interest. This resulted in creating something of a diaspora of disenchanted analysts. Our perception is that many analysts are competitive and want to do a great job in their area of expertise. However, many of them also want their work to serve a socially useful purpose. After the financial crisis, it became much easier to accomplish these goals as independent analysts.

One aspect of useful research that can be easier to provide on an independent basis is that of communicating important risks. Too often it is not fraudulent misstatements by analysts that hurt investors, but what the analysts don't say that can be a material risk. Worse, such risks are often well known to active market participants but do not make it into the news articles for broader public consumption. As a result, the opportunity to more fully explore the thorny, but intellectually stimulating, investment issues has served as another motivation for analysts to take their skills and expertise out of the large financial firms and to provide them on an independent basis.

While an increasing supply of independent analysts has improved access to good research, so too has technology vastly improved access to financial information. It takes only seconds to download a Federal Reserve report or a company's 10-K statement to a computer or tablet device from which it can be reviewed and highlighted. Further, the SEC's requirement for XBRL tagging has also vastly improved access to financial data and facilitated financial modeling.

In addition, technology has also vastly improved the types and number of tools available for research. Data from all kinds of sources is being harnessed to provide new insights and to provide them faster. Satellite images are being used in a variety of ways, one of which is to identify idle oil tankers that are effectively being used as alternative oil storage. Several outfits are experimenting with using social media as an input for credit analysis. Further, simple off-the-shelf apps also can provide extremely useful and affordable utilities that can improve research efforts. For information junkies, it really is like being a kid in a candy shop.

These changes in the "picks and shovels" of investment research are also having an effect on the underlying business of investment management. Institutional Investor recently published an article [here] in which the authors believe that: "Asset management shows the typical earmarks of an industry ripe for disruption: unhappy customers and extremely proﬁtable incumbents." The claim that this condition is exacerbated by "a new set of client demands resulting from global social changes" is well corroborated by our observations and experience.

The authors get right to the heart of the matter: "In our industry the problem is exacerbated by the presence of so many conﬂicted intermediaries. To avoid career risk, an individual allocator is often motivated to assign assets to the most popular fund or type of investment. If an allocator hires a known player, underperformance will rarely prompt questions about that person's judgment. The resulting herd mentality penalizes new managers, potentially stiﬂes innovation and generates suboptimal returns."

This creates an interesting paradox. Despite the presence of thousands of mutual funds, thousands of money managers, and thousands of hedge funds, the industry isn't really that competitive in an important sense. Many elements of the "new set of client demands" have been frustrated by the fact that, "Historically, the asset management business has been slow to change. The stakes are high, the sector is closely regulated, and there are few early adopters of innovation." In business terms, the market is underserved.

Change is beginning to happen, however, and one of the more visible areas is with pension funds. Pensions and Investments reported [here] that, "An unexpected alignment of factors has made now a very good time for pension funds to insource some or more of their investment portfolios. The combination of a low-return environment for the foreseeable future, far closer ... scrutiny of money manager fees and a need to better control investments is fueling interest in bringing assets in-house for the first time for some plans and expanding internal management for experienced investment departments."

Not surprisingly, the report concludes, "The insourcing trend is definitely up," and attributes cost savings as a primary cause because it directly leads to higher investment returns.

As the landscape for investment research has changed, so too have the opportunities for investors. In particular, the hegemony of Wall Street research is cracking in many respects. As a wide variety of surprisingly insightful research services are becoming available at prices that are even affordable for a wide swath of individual, these investors are gaining new access to the "picks and shovels" of investing. No longer is useful data and good research exclusively the purview of institutional investors.

This greater access to high quality research and data provides important benefits. For one, greater access to research empowers investors to become better consumers of existing investment services. At very least, having a better background about an investment or a decision facilitates asking better questions and more thoroughly vetting the providers that are used.

Further, investors with the interest and ability can take a page out of the pension playbook and seriously evaluate which activities might be brought "in-house". It doesn't take that much to oversee a standard allocation of passive funds, for example, and the savings over external management can be significant. Such an approach also allows for leveraging providers who do have greater expertise in certain areas. The main point is that it doesn't have to be all-or-nothing. There is certainly an opportunity to stop demonizing the entire industry and to start leveraging the independent and affordable sources that are emerging.

Of course lower research costs also create the potential for significant change in the investment management industry itself. Lower costs reduce the scale necessary to found an investment business which in turn would allow smaller providers to flourish. Further, as data and access become commoditized, the dimensions of differentiation shift from size, access, and brand recognition to expertise, analysis and lack of organizational conflict. These attributes may well be the hallmarks of the best research in the future.

Finally, there will always be investors who don't want to have anything to do with their investments and prefer to completely outsource the entire effort. In light of expectations for low returns, this approach is likely to be extremely costly for all but the very wealthiest. Don't fear though; there will continue to be many providers all too happy to accommodate these investors.