It can be null when doing sampler job and order by job. we can solve it by forcing getLoadFunc() instantiate LoadFunc.

Note in the design of LoadFunc, setLocation is not guaranteed to be called in the frontend. But seems now some LoadFunc depends on it. Shall we make the statement that setLocation should be called in the frontend?

Daniel Dai
added a comment - 24/Apr/12 23:11 Yes, we does not call setLocation in some cases. Specifically in JobControlCompiler:
LoadFunc lf = ld.getLoadFunc();
if (lf != null ) {
lf.setLocation(ld.getLFile().getFileName(), nwJob);
}
It can be null when doing sampler job and order by job. we can solve it by forcing getLoadFunc() instantiate LoadFunc.
Note in the design of LoadFunc, setLocation is not guaranteed to be called in the frontend. But seems now some LoadFunc depends on it. Shall we make the statement that setLocation should be called in the frontend?

It would make sense to have something that guarantees to be get called on the FE so that there's a way to pass information to the inputformat as well as to the MR job. Not unless there's already something existing that does this?

Francis Liu
added a comment - 25/Apr/12 21:51 It would make sense to have something that guarantees to be get called on the FE so that there's a way to pass information to the inputformat as well as to the MR job. Not unless there's already something existing that does this?

In the short term, it seems fair to ensure setLocation is called on the frontend...in the (ideally near) future, though, we need to work on working some lifecycle into LoadFunc. The fact that there is a convention of stuffing a bunch of logic in setLocation is pretty confusing and suboptimal.

Jonathan Coveney
added a comment - 26/Apr/12 05:11 In the short term, it seems fair to ensure setLocation is called on the frontend...in the (ideally near) future, though, we need to work on working some lifecycle into LoadFunc. The fact that there is a convention of stuffing a bunch of logic in setLocation is pretty confusing and suboptimal.

In the short term, it seems fair to ensure setLocation is called on the frontend...in the (ideally near) future, though, we need to work on working some lifecycle into LoadFunc. The fact that there is a convention of stuffing a bunch of logic in setLocation is pretty confusing and suboptimal.

That would be great. StoreFunc needs to be taken care of as well. I haven't used the other UDF interfaces.

Francis Liu
added a comment - 27/Apr/12 00:37 In the short term, it seems fair to ensure setLocation is called on the frontend...in the (ideally near) future, though, we need to work on working some lifecycle into LoadFunc. The fact that there is a convention of stuffing a bunch of logic in setLocation is pretty confusing and suboptimal.
That would be great. StoreFunc needs to be taken care of as well. I haven't used the other UDF interfaces.