Technically, it comes from mathematics when you factor an expression into an equivalence--the factors are cleaner ways of expressing the same statement.

Actually, that is the history of factoring. One might say this is a well factored design, which is different from a design that is refactored. The former is initial while the latter is continual.

The reason we talk about refactoring is that if we have a factored formula in math, and we *add* something to it, the new formula isn't factored and we have to factor it again... refactor it, if you will. Likewise, a well-factored program design will probably need refactoring after a bunch of features are added to it.

RalphJohnson was the first person to define refactoring and use it in
print, but he probably picked it up from Smalltalk programmers at Tektronix.
He and Bill Opdyke wrote a paper about refactoring in 1990. The emphasis
was on the importance of refactoring in the design of frameworks. Bill
Opdyke wrote a PhD thesis on refactoring (ftp://st.cs.uiuc.edu/pub/papers/refactoring/opdyke-thesis.ps.Z
). John Brant and Don Roberts built
the RefactoringBrowser for Smalltalk, and Don wrote a PhD thesis on it in 1999.

Refactoring is not an XP-specific term at all, but has been rightfully adopted into XP like so many other best-practices in the field of software engineering and management.
Before refactoring, the Object-Oriented community used to use terms like incremental refinement <yawn>.
The important thing to realize about refactoring is that it has a long and dignified history -- initially for Object-Oriented Frameworks -- in the O-O community. And, finally, while refactoring doesn't seem to be an actual word, I still prefer it over having to repeatedly say incremental refinement.

A long and dignified history indeed. I remember the 1980 PointyHairedBoss called it ApplePolishing?. It was considered superfluous to the project, just the tidiness fetish of a few techies. Refactoring was even acknowledged in the WaterFall systems of the day - Post Implementation Review - but that was one phase never performed. At least now it is out of the closet. RefactorMercilessly and the benefits will outweigh your expectations.

Factoring has been a key technique used in the ForthLanguage from its inception in the 70's. The absence of named local variables, minimal syntax, and use of a parameter stack made refactoring equivalent to textual substitution. As with Lisp, Forth's interactivity encouraged factoring into small words which could be immediately tested. At the time of Forth's development, the object of refactoring was as much to save space (threaded code can be extremely compact) as it was to zero in on a good design. The book ThinkingForth details a number of factoring patterns that would be familiar to the XP crowd.