Previous skills insufficient

Hi guys!
So today I decided to zak for phantom blow on my dual blader. After getting it on the second try, I used the mastery book and poof there it was. I had 3 sp saved and spent all of the required sp in my other jobs.
Whenever I'd try to apply sp into phantom blow(or any 4th job skill) I'd get the message "Previous skills insufficient" nothing about 2nd or 3rd job, just those 3 words. I used the sp reset given to me during chaos and nothing else.
Help?