TOKYO—The Bank of Japan’s aggressive purchasing of stock funds has helped Japanese shares climb to multiyear highs in recent months. But some within the central bank are growing uncomfortable about the fast-paced rally and the bank’s own role in fueling it.

Since Gov. Haruhiko Kuroda took office in March 2013 and introduced monetary easing of what he called a “different dimension,” the central bank has sharply increased its buying of baskets of stocks known as exchange-traded funds. By directly underpinning the market,...