Nick Holeman, a certified financial planner at Betterment, agrees. Holeman explains that an index fund can be thought of as a benchmark for a certain type of investment. In this case, the S&P 500 follows the largest 500 companies in the United States.

"But there are so many other indexes out there," he tells CNBC Make It. "There's smaller companies in the United States, there's companies in Europe and Asia and Australia, and there's bond indexes."

In other words, although investing in the S&P 500 is more diversified than owning stock in a single company, it's still only reaching a small fraction of the thousands of potential assets available worldwide. While it's okay to invest in it, that index alone won't provide you with a very diversified portfolio.

"You're not exposed to any other asset classes," Roberge says. "If you look at the historical returns of the S&P 500 over time, the average is great, but average doesn't mean much when you're sitting in 2008 and it's a 48 percent drop. That's not going to diversify your assets appropriately over the long term."

Index funds in general are still considered a low-cost, low-risk way to get into the market, but investing successfully is more complicated than choosing a single fund. "You can't just pick one index and think that all of your work is satisfied," Holeman says. "There's smaller companies in the United States, there's companies in Europe and Asia and Australia and there's bond indexes."

To make sure you're properly diversified, look into resources like target-date funds and robo-advisors, which automatically create a diversified portfolio of both stocks and bonds for you. You can also use a traditional advisor, although most first-time investors usually don't have enough wealth yet for the fees to be worth it.

If you choose to hand-pick a set of funds yourself, make sure that you have multiple indexes included in your portfolio, ranging from U.S. stocks to international stocks, Roberge says.

"That kind of diversification is really important," he adds. "Sometimes, Europe might be doing well and the U.S. is not. We can't just rely on our home country bias, which most people have."

It's also worth noting that investing in just stocks carries a certain amount of risk, so make sure to balance out your portfolio accordingly, depending on the level of risk you're seeking to take on. "In general, bonds are a great way to lower your risk," Holeman says. "So if maybe you're a conservative investor or an older investor, you don't want 100 percent of your money in stocks, and let alone 100 percent of your money in U.S. stocks."

And remember that it's always a good idea to consult a professional you trust, such as a certified financial planner, before investing. A CFP is a great resource if you have questions or want individualized guidance, especially if you're new to the game.