"Part of REITs' modest underperformance this year has been a function of the ongoing dialogue about rising interest rates," said Mizuho REITs analyst Richard Anderson. "We've have also seen some fairly quick moves in the 10-year Treasury and that has coincided with REITs' underperformance as people who are owning the REITs for their dividend yield seek a better option."

The FTSE Nareit All Equity REITs Index, which is made up of 167 REITs, has a price return so far this year of about 6.7 percent, while its total return, which includes dividends, is about 10.1 percent, according to data from the National Association of Real Estate Investment Trusts.

That lags the Standard & Poor's 500 index, which is up about 14.6 percent, or roughly 17.2 percent including dividends.

REITs tend to appeal to investors seeking high yields because the companies' tax structure requires them to pay out most of their income, which usually comes from rent and lease payments, as distributions to shareholders. But REITs also require a lot of money, and often debt, to operate.

So when interest rates rise, it drives the cost of borrowing up, which can cut into profits and lead to smaller dividend payouts.

The Federal Reserve has raised interest rates twice this year and is expected to do so again next month.

REITs are also vulnerable to an increase in long-term bond yields. That's because as bonds start paying out more, they can become more attractive alternatives to REITs, which are comparatively riskier investments.

The yield on the 10-year Treasury note declined to 2.32 percent on Wednesday. It started the year at around 2.45 percent and declined gradually through the first half of the year to a low of about 2.04 percent in early September. It has been moving mostly moving higher since then.

Still, Anderson notes that REITs have often bounced back strong after a period of rising interest rates. Heading into 2018, he recommends investors take a targeted, rather than broad, approach to investing in REITs.

"There are certainly reasons to own the space, and as you selectively look within sectors and be selective at the stock level, there are definitely ways to make money," he said.

The outlook for commercial real estate remains largely positive, with a slowdown in construction expected to help keep rents rising modestly next year amid mostly stable vacancy rates, according to a forecast by the National Association of Realtors.

The store closures and other woes that have weighed on mall owners are expected to continue next year. This year, the sector has a negative total return of 8.4 percent, according to NAREIT data.

Even so, the prospect of a buyout has the potential to drive shares higher. This week, mall owner GGP surged after Brookfield Property Partners offered to buy the rest of the company it doesn't already own for $14 billion.

Other types of REITs are having a much better year, including companies that own warehouses catering to Amazon.com and other retailers pushing to capture a bigger slice of the e-commerce market. Industrial REITs are up about 24.7 percent this year.

REITs that lease space on towers to wireless companies are up the most. Known as infrastructure REITs, their total return is 38.4 percent.

The question going into next year with REITs like these is: Will demand for their offerings be enough to justify jumping into these stocks at their high-flying prices?

Anderson says residential real estate REITs may be a safer bet.

Also known as the multifamily REITs, the sector is up about 11 percent, according to NAREIT data.

These REITs have been held back somewhat by a surge in new apartment construction in recent years, which limited rent growth. The impact from that rapid supply growth is expected to lessen in 2019 and 2020, Anderson said.

"That's one area in the REITs space where we can actually see owning a sector at a reasonable price with a reasonable, nice inflection point ahead," he said.