this has been changed recently [1] in order to properly support
failover in job cluster mode.

A workaround for you would be to add an application identifier to the
checkpoint path of each application, resulting in S3 paths like
application-XXXX/00...00/chk-64.

Is that a feasible solution?

As a side note: It was considered to keep the job ID fixed, but make
it configurable (e.g. by providing a --job-id argument) which would
also help to avoid this situation, but I'm not aware of any concrete
plans to move forward with that approach.

On Sun, Nov 4, 2018 at 3:39 AM Hao Sun <hasun@xxxxxxxxxxx> wrote:
>
> I am wondering if I can customize job_id for job cluster mode. Currently it is always 00000000000000000000000000000000. I am running multiple job clusters and sharing s3, it means checkpoints will be shared by different jobs as well e.g. 00000000000000000000000000000000/chk-64, how can I avoid this?
>
> Thanks