[nanomsg] Re: accessing control IDs

From: Garrett D'Amore <garrett@xxxxxxxxxx>

To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>

Date: Tue, 6 May 2014 21:40:20 -0700

That would indeed be a new pattern. I see lots of challenges with it.
Particularly session resumption and preventing runaway memory consumption due
to left over state At some level this feels like solving TCP at the
application level. At which point my first thought is "why?"
Sent from my iPhone
> On May 6, 2014, at 9:00 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx> wrote:
>
> How about some kind of API-compatible req/rep clone, say "sreq/srep" (s for
> state) that implements session functionality?
>
> Sent from my iPad
>
>> On May 6, 2014, at 10:50 PM, Achille Roussel <achille.roussel@xxxxxxxxx>
>> wrote:
>>
>> I think REQ/REP being stateless is a great thing, it protects the user from
>> misusing the tool and solves a single problem. Now it is true that in its
>> current state nanomsg is missing some sort of stateful protocol, but if
>> you're trying to use REQ/REP when you need to maintain states between
>> requests coming from a remote node then you're probably not using the right
>> tool to solve your problem.
>>
>> HTTP is stateless but it is showing its limits for today's web, and is being
>> slowly replaced by SPDY which is a stateful protocol... Maybe nanomsg should
>> get some sort of hybrid protocol or implement SPDY itself (which is a
>> stateful req/rep with some sort of pub/sub on top of it).
>>
>> Achille Roussel
>> +1 415 490 6339
>>
>>> On May 6, 2014, at 8:13 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Nanomsg is stateless from the point of view of the app. Adding session
>>>> state would be inappropriate.
>>>
>>> I have seen this stated several times in this thread, but I have not seen
>>> any design justification for this position. Such a position should be
>>> defensible.
>>>
>>> I can construct a rationale for why there should be a REQ-REP-*like*
>>> protocol that is stateless. That would clearly be useful. However, in my
>>> view it is an error to give it the name REQ/REP, because this is
>>> confusingly similar to the ZeroMQ protocol with the same name. It would be
>>> tolerable if nanomsg REQ/REP was merely API-incompatible and
>>> wire-incompatible with the protocol from ZeroMQ; that is to be expected.
>>> However when the difference is not just in wireformat or argument order but
>>> is actually conceptually orthogonal to the way ZeroMQ REQ/REP is used
>>> (which is stateful) in a way that is hard to see up front it is time to
>>> invent a new name for a new idea. The connection between the two protocols
>>> is so flimsy that sharing a name is actively misleading.
>>>
>>> That aside, you are taking a much stronger position than merely one
>>> protocol. You are saying “nobody should implement a stateful protocol in
>>> nanomsg”. Well, why not?
>>>
>>> Clearly many applications have stateful networking problems. How should
>>> they solve these problems?
>>>
>>> Should they each implement state atop nanomsg in their own incompatible
>>> way? This is duplication of effort, and probably leads to buggy code
>>> Should somebody consolidate those stateful implementations into a library
>>> that sits atop nanomsg? Now language bindings have to be written for that
>>> library in addition to nanomsg. This is duplication of effort, and
>>> probably leads to buggy code
>>> Should these applications not use nanomsg at all, and instead use ZeroMQ?
>>> But there are many reasons to prefer nanomsg including use of C, BSD socket
>>> API, context-free implementation, pluggable transports, routing priorities,
>>> zero-copy, etc. etc. This option basically tells applications “because you
>>> have stateful networking, you can enjoy none of these benefits”
>>> Should these applications fork nanomsg and hack in a stateful alternative?
>>> This is duplication of effort, and probably leads to buggy code
>>>
>>> I think what you imagine is that if you make stateful programs difficult,
>>> people will rethink their stateful problems into stateless ones. Sometimes
>>> that is true. But other times, people will rethink their stateful programs
>>> and decide being stateful is the right decision for their problem. There
>>> should be some vision to address this case other than “deal with it”.
>>>
>>> It is one thing to declare stateful problems out of scope for a protocol.
>>> It is a different thing to declare them out of scope for a networking
>>> library. People have stateful problems; and this doesn’t disappear merely
>>> because stateless protocols are well-behaved or more scalable.
>>>
>>> I don’t know exactly what the right solution looks like. Maybe it’s
>>> creating protocols that are deliberately stateful. Maybe it’s introducing
>>> some kind of pluggable session layer. But I am absolutely certain that
>>> throwing up our hands and declaring anything stateful as out of scope would
>>> move nanomsg out of the “zeromq done right” search space and turn it into a
>>> very specialized tool for very specialized problems.
>>>
>>>> On May 6, 2014, at 8:23 PM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
>>>>
>>>> Nanomsg is stateless from the point of view of the app. Adding session
>>>> state would be inappropriate. The pipe is almost certainly insufficient
>>>> by itself and recognize that it is subject to change as a result of
>>>> connects or disconnects in the underlying transport.
>>>>
>>>> The information I would like to make available would allow for actual
>>>> strong authentication on those transports that can support it, but
>>>> admittedly it's only on receipt and it ties to a specific message so at
>>>> the abstract level is still stateless.
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On May 6, 2014, at 4:20 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>> I don’t think putting it in the body is the right solution, or at least
>>>>> not right for every case.
>>>>>
>>>>> For one thing it requires allocating storage for this is in the messages.
>>>>> It is an interesting question how much storage is required but the naive
>>>>> implementation would be a 128-bit GUID. For a 4-byte messages this
>>>>> bloats the network traffic significantly. Particularly when there is a
>>>>> simple solution with zero overhead—pull the pipe key from nanomsg.
>>>>>
>>>>> For a another thing, ZeroMQ clearly and unambiguously supports this model
>>>>> (via ROUTER). So anyone porting ZeroMQ code is in for a rough time
>>>>> implementing their own scheme atop nanomsg on all the codebases to get
>>>>> the same behavior they had before. There are of course legitimate
>>>>> reasons to be incompatible with ZeroMQ (such as achieving compatibility
>>>>> instead with BSD sockets) but I think this case is much more harmful than
>>>>> helpful and does not isolate itself to one particular system or codebase.
>>>>>
>>>>>> let’s think about cases where the client sending requests loses
>>>>>> connection, then reconnects with a different address, or connects from
>>>>>> multiple endpoints (mobile device, desktop computer, …), if remote
>>>>>> endpoint information is provided by nanomsg all these endpoints will
>>>>>> appear as different entities, whereas in your application logic they
>>>>>> should be considered the same.
>>>>>
>>>>>
>>>>> You should probably not rely on transport-layer guarantees to
>>>>> authenticate a user. However *given* transport-layer information, it
>>>>> *becomes possible to implement* many authentication schemes.
>>>>>
>>>>> For example (and this is the problem that motivated this discussion)
>>>>> suppose I have some decision oracle which can determine with complete
>>>>> certainty whether a particular user intended to send a packet. Then
>>>>>
>>>>>> if (!oracle_user_sent_message(user,message)) {
>>>>>> end_session();
>>>>>> }
>>>>>
>>>>> Now if we know (or can guess) that John sent the message via
>>>>> transport-layer information, the solution is straightforward. But trying
>>>>> all the possible users is impractical for a slow oracle. So the
>>>>> transport-layer information can comprise part of an application-level
>>>>> authentication scheme to identify which user the oracle should be asked
>>>>> about. Even for sockets that do not have 1:1 fanout to users, the fanout
>>>>> may divide the users into enough buckets that asking the oracle about
>>>>> each member in the bucket becomes practical.
>>>>>
>>>>> Now of course we could prepend some session ID to the message rather than
>>>>> rely on transport-level data, and bloat the message accordingly. However
>>>>> anybody who gets ahold of the session ID could spoof messages with that
>>>>> session ID from anywhere on the network. Now these would be rejected by
>>>>> our perfect oracle, but not before ending the user’s session, comprising
>>>>> a DDoS attack against the legitimate user. Alternatively, relying on the
>>>>> TCP information significantly increases the difficulty of the attack,
>>>>> requires a TCP MITM technique or some other advanced persistent threat
>>>>> capability to execute. This is a major security and reliability
>>>>> advantage to relying on TCP data in my situation.
>>>>>
>>>>> Drew
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On May 6, 2014, at 12:13 PM, Achille Roussel <achille.roussel@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>> Could you put the state information in the body of your message instead
>>>>>> of attempting to get it from nanomsg. HTTP is also stateless but
>>>>>> websites maintain more or less state using cookies, session ids or
>>>>>> access tokens… maybe you can implement this at the application logic
>>>>>> level.
>>>>>>
>>>>>> I think it’s a sain design to have your transport protocol separated
>>>>>> from your application logic, let’s think about cases where the client
>>>>>> sending requests loses connection, then reconnects with a different
>>>>>> address, or connects from multiple endpoints (mobile device, desktop
>>>>>> computer, …), if remote endpoint information is provided by nanomsg all
>>>>>> these endpoints will appear as different entities, whereas in your
>>>>>> application logic they should be considered the same.
>>>>>>
>>>>>>> On May 6, 2014, at 1:23 AM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> There are many cases that require state full networking
>>>>>>>>
>>>>>>> I’m in such a case. The open question at this point is how to achieve
>>>>>>> it.
>>>>>>>
>>>>>>>
>>>>>>>> On May 5, 2014, at 10:52 PM, Apostolis Xekoukoulotakis
>>>>>>>> <xekoukou@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>> Req rep were designed by default to be stateless, that is why finding
>>>>>>>> the address of the message has been hidden on purpose.
>>>>>>>>
>>>>>>>> There are many cases that require state full networking but state full
>>>>>>>> is more difficult because it requires that you implement an update
>>>>>>>> mechanism on the routing information.
>>>>>>>>
>>>>>>>>> On May 6, 2014 5:02 AM, "Drew Crawford" <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>> I have dug a little deeper into this. it appears that in global.c
>>>>>>>>> [1] msg_controllen is never set. I’m not sure if that’s expected.
>>>>>>>>>
>>>>>>>>> The attached patch sets controllen based on the size of the chunk.
>>>>>>>>> Whether right or wrong, this seems to produce the behavior expected
>>>>>>>>> by zerotacg and Achille, e.g., control bytes are emitted in the RAW
>>>>>>>>> case. The 8 bytes are
>>>>>>>>>
>>>>>>>>>> d0,4e,c0,00,c1,7f,00,00,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Three of which (bytes[3],bytes[4],and bytes[5]) seem to change from
>>>>>>>>> run-to-run. This is mildly surprising, because the RFC documents the
>>>>>>>>> control ID at being 32 bits, so one would expect four bytes to change
>>>>>>>>> from one execution to the next. I’m also unable to account for the
>>>>>>>>> presence of the remaining bytes. Something may be wrong with my
>>>>>>>>> patch, or with my understanding of the codebase or RFC.
>>>>>>>>>
>>>>>>>>> This is an interesting line of inquiry, but since a solution along
>>>>>>>>> this line has the limitation of requiring me to implement my own
>>>>>>>>> end-to-end behaviors on top of a raw socket, I’m wondering if it
>>>>>>>>> would be desirable to introduce an API for this purpose
>>>>>>>>>
>>>>>>>>>> /* Returns an integer that uniquely identifies the immediate sender
>>>>>>>>>> of the most-recently-received message. Returns an error if no
>>>>>>>>>> messages have ever been received on the socket */
>>>>>>>>>
>>>>>>>>>> int nn_sender(int socket);
>>>>>>>>>
>>>>>>>>> Such API could work equally well for raw sockets as full sockets,
>>>>>>>>> could be implemented for different socket topologies, and does not
>>>>>>>>> introduce an application-layer dependency on parsing the control
>>>>>>>>> header format.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://github.com/nanomsg/nanomsg/blob/master/src/core/global.c#L817
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On May 5, 2014, at 4:38 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I thought about that, however, msg_controllen still returns -1 when
>>>>>>>>>> using raw sockets, suggesting there are no control information
>>>>>>>>>> available, as the sample below illustrates. Maybe something is
>>>>>>>>>> wrong with the code sample?
>>>>>>>>>>
>>>>>>>>>> Another problem is that use of raw sockets would require me to roll
>>>>>>>>>> my own end-to-end behavior which may be undesirable.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> int client = nn_socket(AF_SP,NN_REQ);
>>>>>>>>>>> int server = nn_socket(AF_SP_RAW,NN_REP);
>>>>>>>>>>> nn_connect(client,"inproc://test");
>>>>>>>>>>> nn_bind(server,"inproc://test");
>>>>>>>>>>> nn_send(client,"A",1,0);
>>>>>>>>>>>
>>>>>>>>>>> int rc;
>>>>>>>>>>> void *body;
>>>>>>>>>>> void *control;
>>>>>>>>>>> struct nn_iovec iov;
>>>>>>>>>>> struct nn_msghdr hdr;
>>>>>>>>>>>
>>>>>>>>>>> iov.iov_base = &body;
>>>>>>>>>>> iov.iov_len = NN_MSG;
>>>>>>>>>>> memset (&hdr, 0, sizeof (hdr));
>>>>>>>>>>> hdr.msg_iov = &iov;
>>>>>>>>>>> hdr.msg_iovlen = 1;
>>>>>>>>>>> hdr.msg_control = &control;
>>>>>>>>>>> hdr.msg_controllen = NN_MSG;
>>>>>>>>>>> rc = nn_recvmsg (server, &hdr, 0);
>>>>>>>>>>> print_array(body,rc,"body”); //contains only A
>>>>>>>>>>>
>>>>>>>>>>> printf("msg_iovlen %d\n",hdr.msg_iovlen); // 1
>>>>>>>>>>> printf("msg_controllen %d\n",hdr.msg_controllen); // -1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 5, 2014, at 4:32 PM, Achille Roussel
>>>>>>>>>>> <achille.roussel@xxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>> You have to use AF_SP_RAW sockets to get access to these info in
>>>>>>>>>>> the control header when receiving a message with nn_recvmsg.
>>>>>>>>>>>
>>>>>>>>>>>> On May 5, 2014, at 2:27 PM, Drew Crawford
>>>>>>>>>>>> <drew@xxxxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a REP socket. I’m trying to identify the channel (sender
>>>>>>>>>>>> or forwarder) on which some message has arrived to the socket. A
>>>>>>>>>>>> transport-layer understanding of the sender is not required; any
>>>>>>>>>>>> identifying value, such as an integer, is sufficient. Consulting
>>>>>>>>>>>> the REQREP spec suggests that the topmost “channel ID”, one of
>>>>>>>>>>>> the records in the “backtrace”, is the identifier I’m looking for.
>>>>>>>>>>>>
>>>>>>>>>>>> Clearly this identifier is not exposed over the nn_recv interface.
>>>>>>>>>>>> I had some hopes that it would be accessible in the nn_recvmsg
>>>>>>>>>>>> interface, possibly as control information, but it seems not to be
>>>>>>>>>>>> the case:
>>>>>>>>>>>>
>>>>>>>>>>>>> int client = nn_socket(AF_SP,NN_REQ);
>>>>>>>>>>>>> int server = nn_socket(AF_SP,NN_REP);
>>>>>>>>>>>>> nn_connect(client,"inproc://test");
>>>>>>>>>>>>> nn_bind(server,"inproc://test");
>>>>>>>>>>>>> nn_send(client,"A",1,0);
>>>>>>>>>>>>>
>>>>>>>>>>>>> int rc;
>>>>>>>>>>>>> void *body;
>>>>>>>>>>>>> void *control;
>>>>>>>>>>>>> struct nn_iovec iov;
>>>>>>>>>>>>> struct nn_msghdr hdr;
>>>>>>>>>>>>>
>>>>>>>>>>>>> iov.iov_base = &body;
>>>>>>>>>>>>> iov.iov_len = NN_MSG;
>>>>>>>>>>>>> memset (&hdr, 0, sizeof (hdr));
>>>>>>>>>>>>> hdr.msg_iov = &iov;
>>>>>>>>>>>>> hdr.msg_iovlen = 1;
>>>>>>>>>>>>> hdr.msg_control = &control;
>>>>>>>>>>>>> hdr.msg_controllen = NN_MSG;
>>>>>>>>>>>>> rc = nn_recvmsg (server, &hdr, 0);
>>>>>>>>>>>>> print_array(body,rc,"body”); //contains only A
>>>>>>>>>>>>>
>>>>>>>>>>>>> printf("msg_iovlen %d\n",hdr.msg_iovlen); //1
>>>>>>>>>>>>> printf("msg_controllen %d\n",hdr.msg_controllen); //-1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have consulted a previous mailing thread on this topic which
>>>>>>>>>>>> suggests channel IDs are manipulated in rep.c. Indeed, the
>>>>>>>>>>>> information I’m looking for seems to be moved around between
>>>>>>>>>>>> nn_sockbase, nn_msg, nn_rep, and similar structures. However I
>>>>>>>>>>>> cannot work out a sane way to get those structures from
>>>>>>>>>>>> application code.
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions on identifying the sender of a remote message?
>>>>>>>>>>>>
>>>>>>>>>>>> Drew
>>>