Turning to the labor market, Yellen said it is "approaching full strength," and that unemployment should fall to near 5 percent by the end of the year.

Benchmark 10-year Treasury note yields, which move inversely to prices, were last at 2.21 percent after hitting a session high of 2.23 percent shortly after the release of her prepared remarks. Three- and seven-year notes also remained higher, but pulled back from their highs as the speech was underway.

The Labor Department said earlier its consumer price index gained 0.1 percent last month after increasing 0.2 percent in March. In the 12 months through April, the CPI fell 0.2 percent, the largest decline since October 2009, after dipping 0.1 percent in March.

CPI is important since it is expected that inflation will continue to be low, while the Fed has been looking for signs it will start to pick up.

The U.S. bond market will close at 2 p.m. ET on Friday and stay shut on Monday for the Memorial Day holiday.

More than $42 billion worth of investment-grade corporate bonds have been sold so far this week, putting May on pace for a record month for issuance according to Thomson Reuters data.

While a $13 billion auction of 10-year Treasury Inflation-Protected Securities (TIPS) on Thursday was met with decent demand.