I get the same thing when I go from Vent to Defrost. It tends to continue to blow out the same vents.
My truck is an 81 GMC k2500 4x4....My guess is something is stuck in the vents somewhere (debris) and not creating a seal.