hadoop-mapreduce-issues mailing list archives

[jira] [Comment Edited] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered

Date

Sat, 05 Oct 2013 22:29:42 GMT

[ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787380#comment-13787380
]
Vinod Kumar Vavilapalli edited comment on MAPREDUCE-5547 at 10/5/13 10:28 PM:
------------------------------------------------------------------------------
We finally sat down and reasoned about all things in MR App that are broken because of various
race conditions during RM restart. While fixing that is a larger effort, looking at this specific
problem, I think we should stick to the invariant that if RM sees the app as failed, we should
make sure clients also see the same.
In such cases where the AM has successfully finished the 'job', but failed to unregister or
failed to write history file(we ran into this also), the client will still the job as running
till the last attempt. And if the last attempt also fails with the same reason, it sees the
job as failed. In corner cases, we will lose work but that's better than clients struggling
with successful jobs with no history files or a failure information on RM.
That said, to really fix this issue, we should change the order of things in the AM - unregister
should be the first thing that should happen. Previously we moved JobHistory flush/close to
be before unregister as we didn't have the AM grace period as we have now. Given that we now
have the AM grace period, we can do the following
- Flush history events & close the current history file
- unregister and loop till RM acknowledges (YARN-540)
- If unregister fails, don't do anything any more - irrespective of whether this is the last
retry or not. This is done at MAPREDUCE-5562.
- Otherwise, if this is not the last retry, then
-- let the client loop (safeTermination flag) (MAPREDUCE-5505)
-- Don't copy the history file (TODO, this JIRA)
-- Don't send the job-end notification (MAPREDUCE-5538)
-- Don't delete the staging directory (Done)
-- Exit
- Otherwise, this is the last retry (usual code path, all DONE)
-- copy the history file to intermediate done directory
-- send the job notification URL and
-- let the client know job succeeded/failed/killed
-- remove the staging directory.
-- Exit
There are a couple more things that need fixing in separate JIRAs
- AM reports final tracking URL as part of finishApplicationRequest. RM should set this URL
for clients to see ONLY after the state operations are done
- There is a duration during which AM unregistered successfully with RM, but hasn't yet copied
the history file to the intermediate done directory and so JHS cannot serve this job to the
client. We should change JHS to throw a ComeBackToMeLaterException and client retrying for
the availability of the hist file till a timeout.
Few more things that need more thought
- _SUCCESS files and their creation time given the above design
- The commit start and end files and their cleanup
was (Author: vinodkv):
We finally sat down and reasoned about all things in MR App that are broken because of various
race conditions during RM restart. While fixing that is a larger effort, looking at this specific
problem, I think we should stick to the invariant that if RM sees the app as failed, we should
make sure clients also see the same.
In such cases where the AM has successfully finished the 'job', but failed to unregister or
failed to write history file(we ran into this also), the client will still the job as running
till the last attempt. And if the last attempt also fails with the same reason, it sees the
job as failed. In corner cases, we will lose work but that's better than clients struggling
with successful jobs with no history files or a failure information on RM.
That said, to really fix this issue, we should change the order of things in the AM - unregister
should be the first thing that should happen. Previously we moved JobHistory flush/close to
be before unregister as we didn't have the AM grace period as we have now. Given that we now
have the AM grace period, we can do the following
- Flush history events & close the current history file
- unregister
- If unregister fails, don't do anything any more - irrespective of whether this is the last
retry or not. This is done at MAPREDUCE-5562.
- Otherwise, if this is not the last retry, then
-- let the client loop (safeTermination flag)
-- Don't copy the history file
-- Don't send the job-end notification
-- Don't delete the staging directory
-- Exit
- Otherwise, this is the last retry
-- copy the history file to intermediate done directory
-- send the job notification URL and
-- let the client know job succeeded/failed/killed
-- remove the staging directory.
> Job history should not be flushed to JHS until AM gets unregistered
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-5547
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Zhijie Shen
> Assignee: Zhijie Shen
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)