Implementing custom StreamAlgorithm

Implementing custom StreamAlgorithm

Hi all,

I wish to implement a custom StreamAlgorithm whose parse() method can
potentially return false if the incoming request has not been received
completely into the input buffer. (For the curious, I'm trying to write a
custom protocol that handles the Typed Parameter Language
(tpl.sourceforge.net)).

However, when my StreamAlgorithm's parse() method returns false, the request
is not processed. I tried inspecting the ContentLengthAlgorithm's parse()
method, but found that this does not seem to work either. I create a simple
SelectorThread for the ContentLengthAlgorithm and POST a request that's
greater than the default buffer size of 8192 bytes. The request is never
processed.

Re: Implementing custom StreamAlgorithm

Eric Heaton wrote:
> I wish to implement a custom StreamAlgorithm whose parse() method can
> potentially return false if the incoming request has not been received
> completely into the input buffer. (For the curious, I'm trying to
> write a custom protocol that handles the Typed Parameter Language
> (tpl.sourceforge.net)).
>
> However, when my StreamAlgorithm's parse() method returns false, the
> request is not processed. I tried inspecting the
> ContentLengthAlgorithm's parse() method, but found that this does not
> seem to work either. I create a simple SelectorThread for the
> ContentLengthAlgorithm and POST a request that's greater than the
> default buffer size of 8192 bytes. The request is never processed.

Eric --

I hope to post my design work up on my blog soon
(http://blogs.sun.com/djhagberg) but I decided to cut out the
StreamAlgorithm pattern in my application (it seemed like it was more
tightly coupled to http-type protocols) and went with the lower-level
ProtocolFilter API. I have the SSLReadFilter in front of this to decode
SSL into cleartext bytes...

My Message's have a fixed-length MessageHeader followed by a
variable-length byte[] content.

There are a few methods here that deal with partial message reads:

- prepareParseBuffer prepends any remaining bytes from the last read
onto the current ByteBuffer.

I can't say this code has passed all regression/unit tests yet but it
might help.

Note it also depends on an SSLSession to act as a Map for partial reads.
If you don't have an SSLSession, you may have something else attached
to or associated with your SelectionKey that could be used to stow
remaining stuff...

Re: Implementing custom StreamAlgorithm

> Hi all,
>
> I wish to implement a custom StreamAlgorithm whose parse() method can
> potentially return false if the incoming request has not been received
> completely into the input buffer. (For the curious, I'm trying to write
> a custom protocol that handles the Typed Parameter Language
> (tpl.sourceforge.net)).
>
> However, when my StreamAlgorithm's parse() method returns false, the
> request is not processed. I tried inspecting the
> ContentLengthAlgorithm's parse() method, but found that this does not
> seem to work either. I create a simple SelectorThread for the
> ContentLengthAlgorithm and POST a request that's greater than the
> default buffer size of 8192 bytes. The request is never processed.

Hum...looks like a bug I've introduced in 1.5, as I've just tried with
1.0 and it worked.

No you are absolutely right. The current http sub module has a bug that
prevent the StreamAlgorithm to work as expected. Now why do you need to
use the http submodule for your project? Would it be easier to use
directly the framework classes? This way you don't have the http overhead.

I will fix the StreamAlgorithm so it can work with http like it was in 1.5.

Re: Implementing custom StreamAlgorithm

> Eric Heaton wrote:
> > I wish to implement a custom StreamAlgorithm whose parse() method can
> > potentially return false if the incoming request has not been received
> > completely into the input buffer. (For the curious, I'm trying to
> > write a custom protocol that handles the Typed Parameter Language
> > (tpl.sourceforge.net)).
> >
> > However, when my StreamAlgorithm's parse() method returns false, the
> > request is not processed. I tried inspecting the
> > ContentLengthAlgorithm's parse() method, but found that this does not
> > seem to work either. I create a simple SelectorThread for the
> > ContentLengthAlgorithm and POST a request that's greater than the
> > default buffer size of 8192 bytes. The request is never processed.
>
> Eric --
>
> I hope to post my design work up on my blog soon
> (http://blogs.sun.com/djhagberg) but I decided to cut out the
> StreamAlgorithm pattern in my application (it seemed like it was more
> tightly coupled to http-type protocols) and went with the lower-level
> ProtocolFilter API. I have the SSLReadFilter in front of this to decode
> SSL into cleartext bytes...

Good :-) The http module should really be used, if possible, only to
improve/support the http protocol (that was the main reason why we have
dropped active development on Grizzly 1.0).

This looks really good. Would you agree, once you think it is stable
enough, to give it to the community? I think people will like if made
available such filter with the default distribution. Let me know and I
can try to work on a clear text version of your filter (by removing the
SSLSession and add a Grizzly's Session object that can be used when more
bytes are required).

Re: Implementing custom StreamAlgorithm

> Jeanfrancois wrote:
>
> No you are absolutely right. The current http sub module has a bug that
> prevent the StreamAlgorithm to work as expected. Now why do you need to
> use the http submodule for your project? Would it be easier to use
> directly the framework classes? This way you don't have the http overhead.
>

I've only been looking at Grizzly for a couple of days, and had completely
missed the possibility of directly using a Controller to implement my custom
protocol, instead of subclassing SelectorThread. I'll change my approach
(with the excellent ProtocolFilter headstart that D.J. posted) and not worry
about the 'http' module.

Re: Implementing custom StreamAlgorithm

This is exactly the response I needed! I'll start with the Controller from
the 'grizzly' framework and implement a custom ProtocolFiler as you describe
below. I don't need SSL support, so for partial reads, I think I'll instead
try a custom Pipeline that creates a custom WorkerThreadImpl that contains
the partially read message header/body.

Re: Implementing custom StreamAlgorithm

Hi Eric,

Eric Heaton wrote:
> D.J.,
>
> This is exactly the response I needed! I'll start with the Controller
> from the 'grizzly' framework and implement a custom ProtocolFiler as you
> describe below. I don't need SSL support, so for partial reads, I think
> I'll instead try a custom Pipeline that creates a custom
> WorkerThreadImpl that contains the partially read message header/body.

Let us know the result and if you think your Filter can be donated to
the community, feel free to send it :-)

Re: Implementing custom StreamAlgorithm

Jeanfrancois,

As it turns out, I decided to follow D.J.'s suggestion and instead store my
partially-read message as an attachment on the SelectionKey. This didn't
work initially, because the standard cleartext ReadFilter that I was using
earlier in the ProtocolChain was doing a key.attach(null) first thing in its
execute() method. So, I replaced this ReadFilter with another version of a
"read" filter that didn't remove the SelectionKey attachment, and things are
working just peachy now.

I don't know if my ProtocolFilter is ready for prime-time yet, since I've
got all of my protocol-specific parsing code inside the MesasgeFilter. Other
than that, it looks very much like D.J.'s implementation (thanks again,
D.J.!). The ideal thing might be to extract all of the protocol-specific
parsing into a MessageParser interface that could be injected into the
MessageFilter... I'll try and think about this after my deadline.

I wonder if the "correct" thing for me to be doing is to store state across
calls to execute() in the SelectionKey? My initial idea to use a custom
WorkerThreadImpl that held on to the state wouldn't work, right, because the
WorkerThreads are not assigned to a specific SelectionKey (or are they?).

Re: Implementing custom StreamAlgorithm

Hi Eric

Eric Heaton wrote:

> Jeanfrancois,
>
> As it turns out, I decided to follow D.J.'s suggestion and instead store
> my partially-read message as an attachment on the SelectionKey. This
> didn't work initially, because the standard cleartext ReadFilter that I
> was using earlier in the ProtocolChain was doing a key.attach(null)
> first thing in its execute() method. So, I replaced this ReadFilter with
> another version of a "read" filter that didn't remove the SelectionKey
> attachment, and things are working just peachy now.
>
> I don't know if my ProtocolFilter is ready for prime-time yet, since
> I've got all of my protocol-specific parsing code inside the
> MesasgeFilter. Other than that, it looks very much like D.J.'s
> implementation (thanks again, D.J.!). The ideal thing might be to
> extract all of the protocol-specific parsing into a MessageParser
> interface that could be injected into the MessageFilter... I'll try and
> think about this after my deadline.
>
> I wonder if the "correct" thing for me to be doing is to store state
> across calls to execute() in the SelectionKey?

Yes it is, but make sure the object attached are removed when the key is
canceled, to avoid potential memory leak:

My initial idea to use a
> custom WorkerThreadImpl that held on to the state wouldn't work, right,
> because the WorkerThreads are not assigned to a specific SelectionKey
> (or are they?).

Yes that's a good idea. I will propose a generic way today to handle
partial read and attachment to SelectionKey. If all goes well I should
have it be the end of the day so you should be able to try it. The
SailFin project is asking for weeks about that feature :-)