I just wanted to point out that although you can get a passport without his permission you do need his permission to take your child out of the country because he also has parental rights and responsibilities. However, if you have a residence order then you can take the child out for up to 28 days without his permission. Have a look at this:

They is nothing from the courts to say how has how many days etc when we split ex hubby had him fri-Monday.but he moved to Cheshire in 2014 and only seen his son four times since then,my new partner lives with me now and we have gave him lots of chancers but he's messed up again etc sending my fella abusive voice msg and saying he's gonna smack his teeth in.ive had nothing but bother from them both since 2014.