Activity

There should be some way to pass a DAG like structure. Look at the DAG node as a configuration. So each node will represent a conf file. Users can generate conf files, arrange then in a DAG and pass it to the JobControl. User should specify just the input folder(s) and the output folder(s) and JC should generate/manage in-between output folders. The users can provide hints if the intermediate job outputs are desired or not. Also some level of fault-tolerance feature should be provided so that in case of some failures the meta job should complete.

Amar Kamat
added a comment - 03/Apr/08 12:29 There should be some way to pass a DAG like structure. Look at the DAG node as a configuration. So each node will represent a conf file. Users can generate conf files, arrange then in a DAG and pass it to the JobControl . User should specify just the input folder(s) and the output folder(s) and JC should generate/manage in-between output folders. The users can provide hints if the intermediate job outputs are desired or not. Also some level of fault-tolerance feature should be provided so that in case of some failures the meta job should complete.

Arun C Murthy
added a comment - 08/Apr/08 09:08 More features:
Ability to submit and monitor more than one job in a parallel-manner (independent sub-graphs in a DAG)
Ability to have custom notifications at various states
Features to re-submit jobs on failures
Ability to 'cron' jobs
Allow users to track 'progress' of the DAG
Allow users to submit templatized configs (same set of jobs run daily)
Heh, fancy UI

Amar Kamat
added a comment - 14/Apr/08 19:46
Facility to re-execute a subgraph. This will fasten the process of incorporating dynamic config changes since only the related nodes/jobs will be re-run with the changed environment.

Allen Wittenauer
added a comment - 17/Jul/14 20:19 I'm going to close as fixed for a variety of reasons:
some of these features are now native to Hadoop
some of these features are now part of Oozie, Azkaban, etc
some of these features are part of Tez