I've heard that 2 layers are good. Used to make crocheted ones from yarn, and just the 2 layers of yarn seemed good enough. With fabric, however, you want to make sure to give enough distance between you and the heat.

I have seen raw-edged trivet/pot holders created with many layers of fabric, perhaps some denim thrown in (we country people really do like our denim )
one suggestion on the board was to use a couple of towels.
The potholders I remember seeing were stitched about 1/2" around all the sides and then a simple x through the middle ... or, I would probably do the x first, and then anchor things around the edges. No binding, no turning ... and, my mother washes things a LOT. These simple potholder/trivets have held up many years