talking on the radio is only part of it ... for me the main fun is building your own equipment and understanding how it all works so that you can talk to people all over the world (conditions allowing )
ive had my amateur radio license for around 15yrs ..

there have been times when ive had to sell my kit ...but ive always gone back

Extremely Talkative

I haven't used CB in ages, but I remember being some 8 years old, getting my first piece of gear:
One of dad's buddies worked as a lorry driver for a food wholesale-company. I remember that we used to pay DM 20,- for the radio plus microphone (that would be EUR 10,- today). The antenna was bought at a specialised store for DM 80,- / EUR 40,-

Maybe you should get in touch with a company doing logistics of whatever sort. Maybe you can score some spare parts or outdated CB radios there.

Quite Talkative

Now I don't want to go into the whole HAM thing, but I would like to just have a dabble and see if it's something I could get into, so what I am looking for is a budget friendly, entry level, idiot proof set up that I can just have a play with. Then If I find I like it, I can move on to something a bit more in depth and expensive.
So my quiestion is can anyone recommend kit that I can use for the above.

How's about you try and get hold of a MIDLAND porta Pac,small 4 watt portable cb.Then get a fibreglass fishing pole and a t2lt antenna (think that's it's name) just coax cable.Take it with you when your out and see who you can get.Look for Jester bushcraft on you tube he's got one.