Re: How to track batch jobs in spark ?

Thanks for all responses.

1) I am not using YARN. I am using Spark Standalone.

2) yes I want to be able to kill the whole Application.

3) I want to be able to monitor the status of the Application which is running a batch query and expected to run for an hour or so, therefore, I am looking for some mechanism where I can monitor the progress like a percentage or something.

3) I want to be able to monitor the status of the Application which is running a batch query and expected to run for an hour or so, therefore, I am looking for some mechanism where I can monitor the progress like a percentage or something.

Re: How to track batch jobs in spark ?

Hi Kant,

why would you want to kill a batch job at all, it leads to half written data in to the disk, and sometimes other issues. The general practice is to have exception handling code.

In case you are running into scenarios where the code is just consuming too much resources and you are running the spark job locally, I would prefer killing the entire spark application from command line using unix kill command or stop-all.sh.

Otherwise, as Jayesh mentioned killing from the application also makes sense.

3) I want to be able to monitor the status of the Application which is running a batch query and expected to run for an hour or so, therefore, I am looking for some mechanism where I can monitor the progress like a percentage or something.