Pinned topicCustom literal as runtime parameter?

Got a situation where it would be extremely convenient to pass in a custom literal as a runtime parameter so as to avoid having to recompile the code to reuse a job for various purposes.

As a simple example, think of a TopN job where I get the top N tuples from the data stream passed through, and some times I want the highest 25 tuples of data through and sometimes the lowest 25 tuples. So I'd want to pass "ascending" or "descending" in to be used in the Sort operator.

Re: Custom literal as runtime parameter?

‏2013-02-01T15:36:33Z

This is the accepted answer.
This is the accepted answer.

Bruce,

The short answer is no, you can't do that. There are two reasons. First, assuming you could specify the value at submission time, the Sort's param value for order is used to generate order specific code at compile-time. The second is, at this time, you can't use a value passed in, at least where you would need it, to specify order. I think you would have to code something like:

param order : (SortedOrder)getSubmissionTimeValue(...);

That will not compile today, although I think you could argue that it should.

Bruce,

The short answer is no, you can't do that. There are two reasons. First, assuming you could specify the value at submission time, the Sort's param value for order is used to generate order specific code at compile-time. The second is, at this time, you can't use a value passed in, at least where you would need it, to specify order. I think you would have to code something like:
<pre class="jive-pre">
param order : (SortedOrder)getSubmissionTimeValue(...);
</pre>
That will not compile today, although I think you could argue that it should.

Re: Custom literal as runtime parameter?

The short answer is no, you can't do that. There are two reasons. First, assuming you could specify the value at submission time, the Sort's param value for order is used to generate order specific code at compile-time. The second is, at this time, you can't use a value passed in, at least where you would need it, to specify order. I think you would have to code something like:
<pre class="jive-pre">
param order : (SortedOrder)getSubmissionTimeValue(...);
</pre>
That will not compile today, although I think you could argue that it should.

Thanks for the confirmation. So if I'm building my own operators, I should avoid custom literals if I want to make it easier for the users to switch at run time and just pass in strings. And yes, I'd argue it should compile. Probably work if the literal was exposed as an enum, but that's not really feasible in general.

Re: Custom literal as runtime parameter?

Thanks for the confirmation. So if I'm building my own operators, I should avoid custom literals if I want to make it easier for the users to switch at run time and just pass in strings. And yes, I'd argue it should compile. Probably work if the literal was exposed as an enum, but that's not really feasible in general.

The literal actually is exposed as an enum, but the enum type (for the cast) isn't visible in that clause. Using a literal is the desirable way to do it. I'll track the desire to make the enum type visible in that scope.

Re: Custom literal as runtime parameter?

The literal actually is exposed as an enum, but the enum type (for the cast) isn't visible in that clause. Using a literal is the desirable way to do it. I'll track the desire to make the enum type visible in that scope.

Thanks. That would help. Of course that also means that for it to work the operator's author must rely on runtime logic instead of compile-time, so would have to be able to distinguish between runtime-changeable parameters and compile time only. We have that problem already, though - which raises the question of how to distinguish now? If I have a parameter I rely on in the perl code in a generic C++ operator, meaning it must be compile-time only, how do I indicate that in the operator model?

Re: Custom literal as runtime parameter?

Thanks. That would help. Of course that also means that for it to work the operator's author must rely on runtime logic instead of compile-time, so would have to be able to distinguish between runtime-changeable parameters and compile time only. We have that problem already, though - which raises the question of how to distinguish now? If I have a parameter I rely on in the perl code in a generic C++ operator, meaning it must be compile-time only, how do I indicate that in the operator model?

Note the SPL expression and the equivalent cpp expression. If the code generator is correctly written then submission time values can be used.
The operator model does not seem to be able to definitively express this characteristic. Perhaps it should. It would be good if this was specified in words in the toolkit reference.

This is doable today, generally speaking. Consider the following:
<pre class="jive-pre">
stream<int32 i> Beat = Beacon()
{ param iterations : (int32)getSubmissionTimeValue(
"count"); output Beat : i = (int32) IterationCount();
}
</pre>
Here we are passing a submission time value to a param. When the Beacon is generated, the instance model contains transformed code that will use a literal that is supplied at submission time:
<pre class="jive-pre">
<cppExpression>::SPL::spl_cast<SPL::int32, SPL::rstring >::cast(lit$0)</cppExpression> <splExpression>(int32)(getSubmissionTimeValue(
"count"))</splExpression>
</pre>
Note the SPL expression and the equivalent cpp expression. If the code generator is correctly written then submission time values can be used.
The operator model does not seem to be able to definitively express this characteristic. Perhaps it should. It would be good if this was specified in words in the toolkit reference.

Re: Custom literal as runtime parameter?

This is doable today, generally speaking. Consider the following:
<pre class="jive-pre">
stream<int32 i> Beat = Beacon()
{ param iterations : (int32)getSubmissionTimeValue(
"count"); output Beat : i = (int32) IterationCount();
}
</pre>
Here we are passing a submission time value to a param. When the Beacon is generated, the instance model contains transformed code that will use a literal that is supplied at submission time:
<pre class="jive-pre">
<cppExpression>::SPL::spl_cast<SPL::int32, SPL::rstring >::cast(lit$0)</cppExpression> <splExpression>(int32)(getSubmissionTimeValue(
"count"))</splExpression>
</pre>
Note the SPL expression and the equivalent cpp expression. If the code generator is correctly written then submission time values can be used.
The operator model does not seem to be able to definitively express this characteristic. Perhaps it should. It would be good if this was specified in words in the toolkit reference.

Thanks for the details. I knew it was possible. What I'm wondering about is if I'm using an input parameter (like Sort apparently does) at compile-time to determine generated code, how to mark that in the operator model to prevent submission time parameters being used.

Re: Custom literal as runtime parameter?

Thanks for the details. I knew it was possible. What I'm wondering about is if I'm using an input parameter (like Sort apparently does) at compile-time to determine generated code, how to mark that in the operator model to prevent submission time parameters being used.

In this particular case I would expect the operator's code generator to complain at compile time if it did not get a constant. I think that the compile doesn't get far enough, due to the fact that the SortedOrder CustomLiteral type is not recognized, to get to the point where the code generator could complain. It fails before the code generator is invoked.

I suspect that the Sort could be changed so that order could be changed at runtime without any significant performance degradation. Then, if that cast would work, the order could be specified at submit time. Unfortunately, I don't expect to be able to change that in the near future.