Thinking about moving away!

What will happen if I move away as I am a single mum and father does not see my son much and no maintenance is paid. Will he still have the right to see him and he says he will stop me from moving! please help!