When is it appropriate to take access/guardianship AWAY from a Father?

I will try to keep emotion out of my post but feel free to express any heart ache you may be feeling on this issue as it is a very contentious one.
Basically, we seperated and were getting on fine. We agreed shared access and guardianship. The Judge didn't do anything formal about the access he said as we were amenable to see how it goes, any difficulties to return to court.
There was some unpleasantness afterward but we got sorted out. The baby was spending mon and wed afternoons with his dad, having great fun. If we were in town, I would send a text and get invited over with the baby for coffee and a chat. So most weeks, he would see him probably 5 days out of 7. And I like that.
Then the day arrived 10 weeks ago when I picked up the baby and was told he (Dad) would be away for a couple of weeks from the next day. I thought it was funny he hadn't mentioned it, inconvenient too as I had to notify the childminder and, well, inconsiderate. Fumed a bit, but no harm done really. Except the two weeks came and went and there was no word. A month went by, still nothing. Christmas came and went, still no word.
Until two weeks ago when he told me he was moving house, did I want the baby's stuff? And then he sent me a long email saying that as he can never be a 'proper' Dad, he shouldn't see the baby at all.
At this point, I'm thinking 'fair enough, you are clearly unstable perhaps you staying away would not be a bad idea' - given that he had gone so long without contact when I had never ever obstructed him from seeing that lovely little boy. If he had mentioned his insecurities BEFORE going missing I would have tried to talk him down. But he didn't and now here we are.
Should I ask to be granted sole custody legally or what do people feel is the correct way to handle this?

I wouldn't specifically go about removing everything YET. Just see what happens. This is a fairly new situation for you so don't make any permanent decisions just yet.

To be honest his justification sounds like a pretty feeble cop out to me. Maybe he's just not bothered. Maybe he has a new girlfriend, and the baby is cramping his style. Maybe he's enjoying the single life, and the baby is cramping his style. Maybe he's depressed. Etc etc.

Can you sit and talk to him. Ask him questions. Tell him what you think, and what you are thinking about doing. And see what his reaction to it is.

As things seem to have been fairly informal between you up to this, I don't see any reason YET, to make the removal of rights formal.

His family are in England. I told him I am happy to keep in touch with the baby's paternal grandmother, half-sister etc. I'm hugely in favour of children having contact with both parents. But not if it is on an ad-hoc, unreliable basis.

Things have turned nasty - he is swinging from questioning paternity to texting me asking can he tell people the decision for him to stay away was mutual as he is getting asked 'awkward questions' (!)

I am thinking about the passport situation and that is one of the reasons I need to clarify things permanently.

That's really unfair on you and the baby and must be so hard for you. There is obviously something up with him, I wouldnt go down the legal route yet, its a lot of time and money and it really is his loss at this stage hopefully he will come around.

He's being asked awkward questions !!! he may keep being asked awkward questions if he is willing to abandon his child... definitely don't play into that he needs to take responsibility for the decision he made, sounds like he doesn't want to do that, that's why he's using cop outs like "the child's better off without him" and "it might not even be his"

Get him to put it in writing to you that currently he does not want access, even email will do.
When issues arise which should be decisions made by both of you as per guardianship, write/email to him about them and see if he wants input, if he replies that he does not then you have it all to show to the court if needs be.

Once you have it from him via email/writing that he doesnt want access then I would suggest writing to his parents saying that while he does not wish contact you are happy to keep in touch with them if they wish.

I think it sounds like he has someone new in his life and does not want them knowing he has a child.

Would you consider mediation services? These may help him see his role as a father clearer. His turnabout sounds really strange I would guess at depression or a third party intervening. What is the story with maintainance, does he think not seeing the child will let himoff the hook?

Dont say it was a mutual decision if it wasn't.
Dont see how removing access would do any favours for the kid.
You can look forward to having an unhappy kid, with anger towards you, if you start things like these.
You've every right to have issues with the father's behaviour. But it's something that has to be separate to your son's access to his father.

I have it in email and multiple texts that he doesn't want to have any further contact with the baby for the baby's 'own good'. He says how heartbreaking it is and he hopes that he will come to know another man 'like a father' who will guide him on the right paths.... it sounds like he has lost the plot to be honest.
What if he changes his mind? This has happened before - after we went to court and agreed shared access. He took him for one weekend, then went weird and said he didn't believe he was the father. I though he was joking. 5 weeks went by and no contact. Then an apology in which he said the only reason he wanted to now get a paternity test was to vindicate me as he felt awful for bad mouthing me to the men in the pub.
I think there is a bad precedent and I'm glad the baby is too young to miss him properly. But let me stress - they really did get on well together when he had the afternoon access up until ten weeks ago.
Again, thanks for the input.

Would you consider mediation services? These may help him see his role as a father clearer. His turnabout sounds really strange I would guess at depression or a third party intervening. What is the story with maintainance, does he think not seeing the child will let himoff the hook?

Have asked before about mediation - he didn't see the point. I, however, am glad of the counselling!
As for maintenance, not a shilling has been offered and I hate the thought of asking

If you are married, guardianship cannot be either given or taken away. I'm not sure if you were married or not but the term "separated" normally applied to married parents.

It can be taken away in extreme circumstances.

This however is not one of them. OP, you can't make him have anything to do with the child, though Iam not suggesting you are trying to. It must be very sad and disappointing for you that he doesn't. If he has said this to you on numerous occasions, then there is little you can do, except prepare yourself for a life of single parenting. I suggest you ask him would be be willing to sign away his guardianship so you can get on with things. At this point, I would advise him that you will be taking this very seriously, and will not allow him to dip in and out of the childs life whenever he feels like it.

Children need consistency, a proper arrangement of dual parenting. If this cannot be, the child may grow up to resent him for his Daddy one minute, disappearing act the next.

You're going to have to try hard to set your emotions aside on this one, and do the best for your child, and for yourself.