Depends on which mil spec you are looking at.
Usually the packaging is much higher quality
for a start: ceramic packaging or a hermetic
metal can, some shielding for rad hardened
devices etc, stronger casing for higher
G-forces ratings.

Wider temp ranges mean thermal expansion will
be more of an issue which is also a package
design issue. Noise of plastic encapsulated
devices can also be higher than metal can
units from surface contamination of the active
device element. So for a start I'd say the
packaging would be a major consideration of
mil-spec components.