Palmisano, who became IBM's CEO in 2002 and board chairman in 2003, will keep his spot as chairman and Rometty will join the board.

"Ginni Rometty has successfully led several of IBM's most important businesses over the past decade -- from the formation of IBM Global Business Services to the build-out of our Growth Markets Unit," Palmisano said in a statement. "But she is more than a superb operational executive. With every leadership role, she has strengthened our ability to integrate IBM's capabilities for our clients. She has spurred us to keep pace with the needs and aspirations of our clients by deepening our expertise and industry knowledge."

Rometty will direct IBM as it looks to grow its cloud computing and analytics businesses, as well as the commercialization of technology behind its Jeopardy-winning super-computer Watson.

Palmisano described Rometty, 54, as the "ideal CEO to lead IBM into its second century."

Rometty joined IBM in 1981 as a systems engineer and holds a computer science and electrical engineering degree from Northwestern University, according to her company bio.

In her current role, Rometty has been responsible for the "revenue, profit and client satisfaction in the 170 global markets in which IBM does business" and has also overseen "IBM's global strategy, marketing and communications functions."

"There is no greater privilege in business than to be asked to lead IBM, especially at this moment," Rometty said in a statement. "Sam had the courage to transform the company based on his belief that computing technology, our industry, even world economies would shift in historic ways. All of that has come to pass. Today, IBM's strategies and business model are correct."

The soon-to-be-CEO also praised Palmisano by saying he "taught us, above all, that we must never stop reinventing IBM."

And reinvent Palmisano did, pushing IBM out of the personal computer business, which was sold to Lenovo, and ending the company's production of printers and hard drives.

Rometty will be the first woman to serve as IBM's CEO and, as pointed out by Jon Fortt of CNBC on Twitter, she will join Ursula Burns at Xerox and Whitman at HP as females running massive tech firms.