How exactly are these two types different from one another? To be more specific, what is the difference between analyzing "protocol activity" and examining "network traffic"?

My current understanding is that a NIPS may dig deeper into packets (e.g. possibly examine multiple OSI layer contents) and check if any of the parameters are invalid/prohibited etc. On the other hand, a NBA may concentrate on parameters such as packets per second, number of connections per host etc. Is this understanding correct?

Signature based IPS don't work for today's threats. Go for Network Anomaly and Behavior Detection to catch unknown threat within your network. Just look at the last major incidents in the industry and ask yourself: Would had an IPS stopped that threat? In almost every case the answer will be NO. If I had to choose one out of IPS and NBAD and would always go for NBAD.