When starting a new colony, my colonists keep dying of hypothermia before I manage to accomplish any real milestones. Is this normal?

What I usually do is fill the empty space with workshop tiles, add a door, place a workshop table. Add an energy source outside (portable solar seems to be the fastest to place) and put an atmo generator inside (usually in the hallway next to the airlock.

About the time when the workshop table has been built and either the atmo or the energy source is up, the first colonist dies.

With most of the recent builds, an atmosphere generator in a couple of rooms is usually enough to keep heat levels high enough to avoid colonist dying.
It might be you're waiting too long to build your atmosphere generator(s) or not building enough of them.

What I would advise is as soon as your workshop table is complete and you're building an energy source... also build 1 atmosphere generator in your workshop and another in your storage area.
That's a solid start, since atmosphere generators also generate a little bit of heat. Not much, just enough to avoid early hypothermia.

... and to avoid you're next problem. As soon as at least one of those atmosphere generators is complete, also build a solar still outside to collect water.

You don't need to build an atmosphere generator in every room, but I would recommend "most rooms" as a general rule.

Edit:Oh, one other quick thought: When you start a new game, there's a generic room attached to the base's airlock. Quickest way to get started is to convert that area into your workshop rather than using other spaces.

There are also heaters you can build which attach to walls. They are a lower build priority than atmosphere generators, but throw out a lot more heat. You can also switch them off once you're happy with the temperature.
If you haven't already noticed them, just scroll left and right (usually Q and E) when hovering over the walls while in object mode. It will always start with doors being the default object, but heaters (and screens, and circuit breakers and other stuff) are all there too.

Personally, dependant upon how Simon is tweaking stuff, I've found that I can generally heat a base with atmosphere generators alone - but I wouldn't rely on that to always be true.

Finally, again another "you may or may not have already noticed" tip. If you hover your mouse over the walls of all the named room, you'll see a bit of tooltip details about that room. Including the air quality and the temperature.

Simon has been travelling and working recently. Combining a holiday with time away to focus on some of the core features of the game. Allegedly, that includes a lot of the game balancing issues people were having recently (like colonists not prioritising things that would ultimately keep them alive). So you may notice that in version 0.58 colonists don't die as easily to dehydration or hypothermia... but chances are, they'll die to something new like crop diseases or something. For example, I believe 0.58 will include water dispensers for each room, so colonists don't have to walk across half the base just to get a much needed drink of water. It also rebalances how colonists react to cold rooms, with colonists choosing quicker to go and stand in warmer places when they are cold.

Thanks. What's made the difference is placing an Atmo Gen in every room that I build, instead of relying on heat spreading around the base as I thought would work.

I'm now able to construct bases where the colonists don't die of hypothermia. Instead, malnutrition seems to be the next hurdle. Oh well, your skill should've been a bit more useful than throat singing, then I would've made more effort to save you