You could limit the ram usage through the settings from "Working Set" - qmc/system/setup/performance. But in general a qv server shouldn't run in a (real) shared environment - fix ranges of cpu + ram will be needed.

This has been a requested feature in the pass, but in has not be included into Qlik. I am sure you can search the web and find a utility to do what you are looking to do. I remember on called KillProcess, that will kill tasks based on the settings you setup.