I like the natural look so I don't tend to buy any of those resin or ceramic decorations. Since they're made and sold exclusively for aquarium use, they're probably going to be pretty expensive no matter where you look, but like most aquarium products they're probably cheaper online.

As for the natural stuff, any type of driftwood will work in a fish tank but they have different characteristics. Most driftwood you find on the beach won't sink, so you'd need to waterlog it. Same goes for any sort of dried branch or log that's on dry land. Stuff that's submerged will probably already sink. Of course, any driftwood you find should be boiled at least once and scrubbed thoroughly. However, stuff you find outside might not last very long in your tank as it will decay. The more dense driftwoods sold for use in aquariums (Malaysian, mopani, etc) will usually sink on their own or are screwed to slate bases. They will also last a very long time (decades) in a fish tank without rotting. For these reasons, I usually dish out the extra money for these types of wood. They also look really nice.

Rocks: many fish stores will sell rocks for aquarium use. They tend to be expensive. You can use any inert shale or slate from a landscaping place and build your own caves and hiding places for a fraction of the cost. If you can find it on your own property, that's fine too. Just be sure the rocks are inert (try putting a drop from the #1 nitrate test bottle from your API liquid test kit) and see if it fizzes. If not, it's inert and safe. Just give it a good scrubbing and pour some boiling water over the rocks to clean them. DO NOT BOIL THE ROCKS! Trapped moisture in the rocks can boil, causing them to explode.