I've setup a custom property on the node level that tells which email group should notified incase of an alert. This seems to work fine on basic alerts like Node down but when trying to use it on an advanced alert for APM (Application is critical) the variable is not being replaced.

Is this due to the fact that it's an advanced alert or a custom property on the node? How could I due this with APM alerts?

I made that mistake a few times too because originally the alert manager did have (still does?) a variety of general properties in just ${PropertyName} format, e.g. ${NodeID}. I think it was the move from 9.5.1 to 10.0 that introduced the ability of having any Property usable (which is a god send as we had some awful looking actions that had to use ${SQL..} queries to access the data).