I think you would have more success posting this on an E39 M5 technical forum. Most of us here don't have too much knowledge of the E39 M5 and without seeing the car it would be very difficult to troubleshoot this for you. It is very possible that someone on an E39 M5 forum could be aware of an issue(s) that could cause these symptoms and will offer you more help.

To me it seems like you may have an intake leak of some sort which is allowing un-metered air into the front 4 cylinders. Check gaskets, vacuum hoses, etc. that could be the source of an air leak. Just to note, bad coils usually would not cause the affected cylinders to burn lean, on the contrary, in the majority of cases it would cause those cylinders to burn rich.