What exactly happens when competent cells like DH5ɑ are heatshocked with DNA present? How does the DNA get inside the cells?

Specifically, why are all the steps necessary? What if you heatshock right after adding DNA? What if you don't put on ice after heatshock? What does the calcium chloride do? What actually happens when cells are "competent"? What governs transformation efficiency (besides obvious things like amount of DNA or cells)?

To make it clear what I'm talking about, I use a protocol like the following:

Spread 300 ul of the culture on LB-agar plates with appropriate selection.

I usually get thousands of colonies from this (in fact, often 1:10 or 1:100 dilution is necessary so I can actually get isolated colonies). Even if I skip the outgrowth, I still get hundreds.

I don't have my competent cell protocol, but basically I use that one rubidium chloride that everyone uses: DH5ɑ cells are washed with some buffers, suspended in a solution with CaCl2 and RbCl, then frozen in liquid nitrogen. I never actually measured, but usually the concentration of cells in the frozen aliquots is about 10 times as many as you would get from an overnight liquid culture (they are spun down and resuspended in a small volume).

I think it is still true that no-one knows "exactly" what happens. There is a piece of received wisdom that by growing cells at 37C then shocking them at 42C, the membrane becomes more fluid (true) and this helps DNA to cross this barrier. If anyone writes an answer with evidence for this I for one will be very interested.
–
Alan BoydJun 11 '14 at 10:42

@AlanBoyd I have heard many scientists also claim that nobody knows. I have always found it bizarre. How can such a simple, popular technique not be well understood? Are the experimental techniques to study it not available? Or does nobody bother because they think even if they figure it out, nobody will cite it or care?
–
SuperbestJun 11 '14 at 20:58

I also learned/was told the same mechanism is responsible for E. coli transformation as written by Alan Boyd. This paper (Large-volume transformation with high-throughput efficiency chemically competent cells. Focus 20:2 (1998)) seems to get referenced a lot about the procedure; however, I do not know if the study offers any mechanistic explanation of heat-shock based E. coli transformation as I cannot seem to be able to access/find it.
–
BezJun 11 '14 at 23:56

Can't be sure about this but natural competence is affected by environmental stress.. May be it is the same case with chemically induced competence. See this post.
–
WYSIWYG♦Jun 12 '14 at 9:28

... heat-pulse (0 degrees C42 degrees C) step of the standard transformation procedure had lowered considerably outer membrane fluidity of cells. The decrease in fluidity was caused by release of lipids from cell surface to extra-cellular medium. A subsequent cold-shock (42 degrees C0 degrees C) to the cells raised the fluidity further to its original value and this was caused by release of membrane proteins to extra-cellular medium.