Thats the principle behind an "atmospheric water generator", it basically pulls the moisture out of the air via condensation to produce water. The biggest difference is that these generators have TONS of filtering.

Dehumidifier water would most likely be drinkable if filtered through a Katadyn or similar camping filter. Dehumidifiers tend to suck in airborne crap, so whatever dust etc is floating around will be in the water that it produces.