Does anybody know a work around or maybe a solution to solve this problem?

Search in the archive of fop-user mail list, there were many recipes there how
to avoid out of memory using fop, such as: get rid of (if you can) "page x of
y" stuff, don't use one big page sequence, etc etc. And upgrade fop to 0.20.3.