Public transport in Taiwan is pretty good. Lot's of buses, and the more expensive but super fast high speed rail from Taibei in the north to Kao Hsiung in the south, almost one end of Taiwan to the other in less than 2 hrs.

Don't know much about where to say, because I always stay in temples / monasteries when in the area. But, there should be plenty of back-packer type places, hostels, etc. Huseng will probably be clearer about this.

Come and visit us at Fo Guang University if you're in the area, we're only 1+hr away!