Transmission electron microcopy (TEM) in both the parallel illumination and scanning probe mode (i.e. STEM) has revealed morphological and structural transformations in self-assembled II-VI, III-V and group IV semiconductor quantum dots (QDs). These transformations result in the standard compressively strained QDs being converted over time into lower energy structurally transformed or compositionally modulated QDs. We believe that it is the release of the excess Gibbs free energy that drives these structural transformations, proceeding by either atomic ordering [2], spinodal decomposition, or phase transformations [1]. A simple estimate of the excess Gibbs free energy yields a value between 0.1 and 1.5 eV per atom in a compressively strained QD, a large driving force for structural re-organization. Furthermore, as in compressively strained quantum dots there is typically a hydrostatic pressure between 1 and 10 GPa on the quantum dot [1], structural re-organization is also driven towards orientation relationships that minimize the lattice mismatch strain. Experimental observations indicating the specific crystallographic relationships between the quantum dots and the surrounding matrix will be presented for these systems. The implications for the structural stability and therefore lifetime of devices involving QDs will also be discussed.