grid environment not set up automatically on the WN (!), e.g. due to bad NFS mount

...

or it got killed before it finished:

it ran into the wall-clock or CPU time limit (should be the most common cause!)

the job got pre-empted by the batch system

NFS server overloaded/stuck (when a standard job manager is used)

WN ran out of memory and killed "random" processes

WN crashed

...

Note: the "lcg" job managers need globus-url-copy to work from WN to CE,
otherwise a job cannot even start. The command can fail in particular when
the CE does not have the correct contents in /etc/grid-security/vomsdir,
e.g. an outdated VOMS server host certificate or bad contents in some of the
vo/*.lsc files. The occurrence of such a problem may prevent certain
new jobs from being submitted to the CE, while previously submitted jobs may
experience fatal errors when they actually start on the WN.

Jobs may be submitted successfully to the CE, but fail to start on the WN
when the gatekeeper and the gridftpd on the CE do not map the user proxy
to the same local account (this should never happen on an LCG-CE).
Check this as follows for an LCG-CE or OSG-CE: