XMPP

Formerly known as Jabber, XMPP is the 'Extensible Messaging and Presence Protocol.' This page discusses how PSYC relates to XMPP and how psyced implements it. But you probably came here because we describe its fundamental architectural problems. Documentation on how to use Jabber with psyced and PSYC is kept at http://www.psyced.org/psyc-jabber.en.html

psyced and the XMPP

The psyced implements XMPP according to its RFCs for interservergatewaying and client access. Please note that XMPP client access isn't as stable as IRC's. Here's a list of XMPP extension protocols that we support:

XEP-0012: Last Activity

XEP-0030: Service Discovery

XEP-0045: Multi-User Chat

XEP-0054: vcard-temp

XEP-0077: In-Band Registration

XEP-0091: Legacy Delayed Delivery

XEP-0092: Software Version

XEP-0107: User Mood

XEP-0154: User Profile

XEP-0178: Best Practices for Use of SASL EXTERNAL with Certificates (up to revision 1.0)

Technical Problems with XMPP

Wrong use of XML

It's not a question of religion – it's not true that one format is like another – there are clear and logical technical reasons why XML and specifically the XMPP dialect of XML are unsuitable for instant messagingapplications. These deficiencies can be compensated by using more processing power and network bandwidth, but still they could be avoided (look at syntax for an example):

Major disadvantage: Performance inefficiency

Many people are currently turning away from XML and turning to JSON for efficiency gains, but if you look at the libpsycbenchmarks comparing XML, JSON and PSYC you will find out that they both aren't very performant formats unless you really make use of their respective strengths. Check those benchmark results out, they're really quite enlightening. PSYC turns out several factors faster than both, whatever input you feed it.

Major disadvantage: Binary Data Transfer

Just like HTTP, PSYC has the ability to say, the following n bytes are binary data - transfer them without trying to interpret them. This may sound totally unspectacular, but several protocols cannot do that. It makes File Transfers or embedding of a photograph or cryptographic key as simple as a fingersnap. XML instead has no simple solution for binary transfers. You either have to escape all XML control characters, or encode the data using something like base64 that needlessly consumes computation power and bandwidth, or you have to encode each message as a separate XML document, and put a binary capable framing protocol around it. XMPP doesn't do that, either.

It also means that it can't just deliver XML data. Ironically, any non-XML protocol is much better suited for delivery of XML than XMPP – because XMPP itself looks like XML so it can't handle anything similar to it. Just like HTTP, PSYC can simply transmit XML as is while XMPP has to encode it to avoid collision with its own syntax. The result will also be quite hard to read, should anyone need to do so. Either escaped or base64-encoded.

Alternatively, if you twist the XML data in a way that the XML document becomes itself part of the XMPP protocol, then it works. But that's not simple, not transparent, not spontaneous at all: you have to define an XMPP protocol extension with a custom namespace for that. You can't say you're delivering an XML document at the snap of a finger as you would when using HTTP.

XMPP extensions like SOAP over XMPP or ATOM over XMPP re-implement the foreign XML format into XMPP stanzas. This means that whenever you want to receive those packets, you have to parse them, then render the tree structure into a regular XML document, then pass it on to the software library which implements SOAP, ATOM (or whatever).

On the other side when you want to publish a document you received via SOAP or ATOM you have to parse it, then re-render it into XMPP. This is waste of processing power! The only way to avoid this is either to not use libraries and implement everything yourself, or to not use XMPP. Guess what PSYC does: It knows there is an XML thing in its message body. It can pass it on without even looking at it.

Major problem with XMPP: Missing framing

Being able to provide the length of a packet isn't an advantage only for binary packets. It generally allows to wait for the complete packet before starting to work on it, which allows for more efficient low-level read operations. In the case of PSYC nodes providing routing services only, it is even better as parsing of the packet can stop immediately after the routing header of the packet. Everything else is just read into a buffer at once - very processing-friendly.

In XML you have to parse the complete tree before you can find out if something has reached completion, so you have to try parsing every little chunk of data coming from the network, because it might just be the last one you've been waiting for - this makes processing more expensive. XML parser implementations have gotten pretty smart at guessing the number of closed braces to figure out when it is worthwhile to start parsing, still it is an unnecessarily complicated heuristic approach. Several protocols solve this by wrapping a non-XML length-prefixed framing protocol around XML. In the case of XMPP this is however not being done:

"You're correct in your assertion that framed data would mean clean binary transfers, but that isn't a goal of XMPP - anyone shipping binary data directly through XMPP is simply doing something inefficient. If you do want to ship large binary objects, it's more efficient to send them either via email, or send a URL via XMPP, and ship the data by a more suitable protocol."

And we would prefer to transfer any data across our PSYC links once we
have them up and ready for use, not be limited to XML packets, especially
since an HTTP file transfer cannot be multicast.

Flash and some other applications introduced null bytes into the XML stream to frame packets. This doesn't help optimizing read operations, but at least gives you a chance to postpone parsing. XMPP could adopt that.

XMPP isn't even proper XML

Some say, in the light of today's processing power, it doesn't matter if XMPP uses an inefficient syntax, and yet performance optimizations were important enough for XMPP not to fully embrace the XML standard.

One advantage of XML is the existence of ready to use parsers, and by now, many of them handle the XMPP dialect pretty well. In fact XMPP developers prefer if you don't roll your own. In order to employ such an XML parser, Jabber developers need to tweak certain things before and after parsing:

XMPP does not allow encodings other than UTF-8, but UTF-16 is a requirement for XML compliance.

XMPP forbids XML comments, but they are a requirement for XML compliance.

Federation vs Scalability

From a protocol and architectural point of view, Jabber/XMPP generates a lot more traffic than it necessarily should and has not performed well when users had many friends or entered huge chatrooms in the past. This may be compensated to a certain degree by processing power and network bandwidth. Unfortunately we have no solid numbers to prove this.

Here are some statistics that Matthias Wimmer, the friendly systems administrator of amessage, has provided us (Thank you!):

"I just checked the log of the three biggest domains (amessage.be, amessage.de, and amessage.info) where for the last 9,211,845 stanzas that have been delivered to these three domains (either from remote or locale JIDs) it's the following:"

Then, he investigated a bit further, by looking at how many presence stanzas are actually identical copies being retransmitted over and over, and found out that:

"6200 out of 10374 presence stanzas have been dupes (59.76 %)."

That means that 42.66% of packets sent across the amessage servers were redundant. These numbers are from 2006. Would be cool if there were newer stats than these, but no-one has provided any. Maybe the situation is worse now.

Why is XMPP sending the same information again and again? Presence, like groupchat, is information that needs to be distributed to many recipients, so XMPP has to do it somehow. Currently, XMPP simply generates a copy of each message to each and every recipient and even resubmits the same information onto the same TCP connection for each recipient.

At least, when both sides of the XMPP communication support XEP 0033, the list of recipients can be delivered in an SMTP-like fashion, but that is only a slight improvement, since the Cc: lists create overhead, too.

True bandwidth benefits appear only if

subscription state is not retransmitted with every communication,

the burden of transmission is not going from the source to every single recipient, but using a distribution tree instead.

When you have hundreds of friends on your roster and distributed on several remote servers, that's when XMPP servers start connecting each of the recipient servers and send a presence update to each person one by one.

The same is done for pubsub distribution. Let's say a news source has a million subscribers. Each time it has news, the publishing server needs to address all of these people one by one. In PSYC, by contrast, the server will contact a set of servers, which will forward the news to another set of servers each, until all the recipients receive the news. Since each of these servers knows who of their users have subscribed to this distribution context, the list of recipients does not need to be retransmitted with the message.

It's a protocol design issue that cannot easily be circumvented by server design. Some XEP drafts have been written to address the problem, but they didn't find enough acceptance in the XMPP community. If you're interested:

The SIMPLE working group has published an Internet Draft on Optimizing Federated Presence with View Sharing. Their approach is similar to what we proposed for XMPP, but they have an explicit negotation of distribution context while we intended to couple that with presence subscriptions. See presence for more details.

Ironically, even if Jabber was using IRC as a kind of multicast overlay network to distribute its presence, it would probably perform better in the whole. fippo wrote up a little proposal: Jabber Relay Chat. IRC has its weaknesses, but some things can be learnt from it.

XMPP is generally claimed to be scalable, but its scalability is best achieved within a single domain, in a closed cloud silo, that uses its own decentralization strategies, like the Erlang language provides, not trying to use S2S XMPP internally. The open federation network of XMPP instead would benefit greatly from more elaborate routing strategies like stanza repeaters and multicasting.

By not providing those, XMPP cements the likelihood that people will just stick to large popular servers as XMPP works best when your contacts are on the same server as you.

Still, a distribution network based on established TCP lines should generally be superior to similar efforts using HTTP POST requests (see also XMPP To The Rescue), so if you are still doing things using HTTP, you may of course experience an improvement switching to XMPP. Then again, in the light of REST-ful APIs, web servers have improved a lot on handling small notifications in recent years. HTTP/2.0 is probably going to be several times more efficient than XMPP. Especially if you deploy a multicast solution over HTTP like FETHR has tentatively done.

When asked about multicast, Ralph Meijer, member of the XSF Council, stated in a conversation at the FSW2011 that you cannot entrust subscriber nodes to correctly forward your message to further subscribers. They might censor or fake messages. FETHR approaches this problem by applying digital signatures and using hash chains. It's not such an attractive option for XMPP however, since all this cryptographic data cannot be transmitted in native binary form – inefficient base64 encoding is needed instead, but it would probably still be better than giving up on efficient distribution altogether.

Interestingly however, since 2009 Ralph Meijer's XEP-0253 exists, which provides for multicast distribution at least for pubsub. In combination with XEP-0163: Personal Eventing Protocol this could actually turn into a scalable replacement for traditional <presence/>.

What if PSYC and XMPP were to team up?

It is being said((r)) many large XMPP deployments like Google Talk, Wave or Facebook Chat use custom tree-based distribution protocols within the cluster, while only interfacing to outer XMPP federation using the XMPP protocols.

At least when Google Talk started, its first experiments weren't using XMPP. Also Facebook Chat's first incarnation was no XMPP. Both implementations are custom and didn't start out from an XMPP codebase. Since it is in both companies' interest not to tell that they might be using more efficient inter-server communications, we will probably not be told. In any case their existence is a proof, that you can do XMPP in a scalable way if you solve internal communications efficiently somehow. Even ejabberd uses its own XML-free internal protocol between processes, the one provided by Erlang, so it can't be a bad idea.

It would be good for the federation, decentralized communications and open source in general, if such strengths weren't limited to cloud-based commercial offerings (here's an example for that – don't try this without a cloud at hand!), but rather generally available and evenly integrated into the XMPP experience. The ejabberd solution is freely available, but it only works in homogenous network clusters.

In PSYC as Jabber S2S we have laid out a plan on how to use PSYC as an interserver federation protocol for XMPP applications. This would combine the respective strengths of the two technologies. psyced already implements several parts of that, but it sure would be fruitful to extend solid XMPP server technologies to also process PSYC as a more effective network serialisation format and distribution system for the same content (see below for benchmarks comparing PSYC and XMPP syntax).

We are however not putting further effort into this, as we have come to the conclusion that federation is the wrong architecture for private communications. We still do multicast, but we no longer expect our users to trust a server (see secushare).

Having to Guess the Meaning of a Packet

The PSYC protocol provides you with a method (or message code, message type) for every protocol message, which immediately makes it clear what the meaning (semantics) of the packet is, whereas an XMPP server, even psyced, has to closely look at what a client or server sends to figure out what it wants. This means parsing the complete XML packet, no matter who it is intended for, then inspecting a lot of individual aspects of each packet. For example,

is a _status_friend_present in PSYC terminology.
While in PSYC we can quickly glance at the method name, then decide what other parts of the packet we need, in XMPP we have to look at all the attributes and children to figure out what semantic meaning the packet has, before we can start acting upon it.

The lack of a method name as a unified speedway to packet interpretation makes programming XMPP applications harder than it needs to be. Oh, since XMPP developers reading this paragraph mentioned that it's bad if a client needs to know all methods – no, just because this page is criticizing XMPP doesn't mean you get a complete introduction to PSYC. Clients make use of method inheritance and only need to understand three or four basic method families to already provide for all basic functionality, so PSYC achieves the same with a lot less code complexity.

MUC the "Multi-User Conference"

MUCs are organized in the most natural way you would expect a chatroom to be organized: It is running somewhere on a server, all users register with it, all communication flows through the server of the chatroom. You could say XMPP MUC implements the most straightforward way to implement a chatroom, similarely to most webchats.

In situations of either highly distributed topology or large audiences, this approach has the same scalability issues we have seen above with presence and pubsub. Other technologies have more elaborate strategies.

IRC hosts its channels on each and every server of its network, multicasts the messages to each other in a close to optimal way, then each server takes care of letting its users take part. IRC has hosted moderated chatrooms with thousands of participants in the past. The drawback is, that no single authority is in charge of conference control of an IRC channel, so a single rogue server can cause havoc.

PSYC uses a combination of both approaches, it has a central authority organizing a chatroom, yet uses multicast distribution to even out the load on the network. As early as 2001 PSYC events have been run hosting ten thousands of users in a single conference. Not all of them were permitted to speak freely, but all of them were able to write up submissions for the editorial team to choose from, in realtime.

Some MUC implementations use custom overlay networks to achieve a similar result as IRC channels. The drawback is, it is not a generic solution all XMPP servers can use to distribute the load - it only works for a set of servers run under the same administration (cloud-friendly thinking again).

It would be nice to see these kind of improvements in a generic way, so that all servers in the XMPP federation can take part (see the Smart Chat proposal above for example). This would address those frequently discussed scalability issues of MUC. And even better, if such a multicast improvement was generic enough to also encompass presence and pubsub. But, as we found out, this is not going to happen.

Distributed MUC

Since the MUC strategy turns out to have several drawbacks and we found that this distributed MUC proposal submitted to council has several fundamental shortcomings we submitted our own version, which is mainly a straightforward mapping of the PSYC group join to XMPP. This was accepted by the council for publication on 2010-03-29 (see minutes).

Feedback was meagre so we decided not to update the specification. Some feedback given on obscure mailing lists and slides here and here.

Are we afraid of trust issues in a distributed scenario? If a server X is trusted to bring in new users and servers it could just generate fake user and fake server names. The PSYC approach to this depends on the use case. If the use case is an anonymous broadcast, then PSYC doesn't want to know any names of users and servers behind your node. If the use case instead is such, that each node must be authenticated and authorized, then each node must make its admission request to a master node before getting assigned to a node in the tree. You may also have multiple masters, as in the case of "FMUC," but that is a mere question of conference control and no reason to provide less distribution capability.

So with all of these developments there is no convincing reason why XMPP shouldn't provide scalable distributed MUCs by default and add any authentication paranoia later. Of course this may harm some business models of some companies.

/msg doesn't care for OTR

A minor privacy concern with MUCs is the /msg command which sends private messages through the MUC, not directly to the recipient. This is frequently not obvious to the people using this function. The MUC could therefore listen into private conversations if suitably modified. Also, if people usually do end-to-end encryption, they may fail to notice that while an IRC client would also encrypt /msg, a Jabber client won't, thus transmitting over the network in the clear.

Potential identity confusion

Another minor concern is about identity. XMPP resources are case sensitive. Due to that, there can be 3 persons in a MUC with the identities foo@bar/AbC, foo@bar/aBC and foo@bar/ABc. This could cause confusion, as they may appear as the same person to other participants.

Supporting MUCs from PSYC

PSYC will however not be able to apply multicast optimizations retroactively: If several PSYC users join the same MUC, each of them will receive its own copy of each message. For a cleaner set-up it's good to use PSYC chatrooms, and invite XMPP users in. In this constellation, all PSYC and IRC users will benefit from interserver multicast routing. Only XMPP users will be handed out redundant copies of messages. This is how it works for Jabber users:

XMPP users entering PSYC places

If you want to enter psyc://example.org/@someplace, you need to join an XMPP JID like *someplace@example.org. The best way is to query the official addresses of the room by issuing the /statuscommand in it.

This applies to both people using psyced as a Jabber server directly and regular Jabber users in the XMPP federation network, given the PSYC server runs interserver XMPP, too. As explained before, entering via XMPP defeats multicast routing and similar PSYC functions, but provides you with the extras of a native XMPP server in exchange.

Profiles in Jabber

Jabber comes with two different profile formats. One (XEP 0154 aka User Profile), which is very elaborate, but rarely implemented and another one that is very limited, but available everywhere (XEP 0054 aka vcard-temp).

The latter is a mapping from the vCard format defined in RFC 2426
into XML. The way it keeps phone numbers is a bit impractical:

<TEL><VOICE/><HOME/><NUMBER>303-555-1212</NUMBER></TEL>

VOICE, HOME and NUMBER are adjacent children of TEL, defeating XPath-like XML access strategies. Even parsing this XML with a line mode parser doesn't work as the DTD doesn't enforce this order of children. HOME may come before VOICE and VOICE after NUMBER.

You can read about the PSYC way to encode this sort of data on the Profile page – it, of course, supports keyword inheritance, so you can extend any field with new semantics as you require.

Minor protocol optimizations

XMPP uses uptime as a technical value (JEP-0012). That's impractical, as the value gets old while it travels across the network. Using boottime is much better: You can keep it for as long as you know the other side is up (typically while the TCP line is up) and calculate the uptime from it by a simple time() - boottime subtraction. Needless to say, PSYC uses boottime. For end user messages it provides both values.

JIDs aren't flexible enough

Because the world is round and not flat, I mean because there are other chat systems out there beyond Jabber, the user@host syntax, also known as Jabber Identity, is bound to create problems.

It could have all been so easy if the JID was simply an opaquestring, where only the user's server needed to understand what's in it. Why does a client need to know what an icq:1234 is, or an INFITD0E at DM0TUI1S - as long as both the user and his server know what it means, that should be fine. Okay, forget the second example (which is a BITNET address, by the way). If JIDs in XMPP client-server were opaque, then we could have asked our users to drop the uniform into the add buddy wizard's address field, and poof connection is made. Instead people often have seperate username and hostname input fields, and the majority of chatsystems out there aren't compatible with it.

Therefore Jabber has to mingle all kinds of gateway addresses into the user@host syntax. This may work for icq:NUMBER as you can turn it around into NUMBER@icq, and that's kind of logical to the average user too. But once you have an address like msn:user@example.net, how are you going to go about it? 'msn:user' is not a legal user name in Jabber, and even if it was, it is just nothing the end user would guess - to put the protocol scheme into the username field. Oh, XMPP uses UUCP syntax for that: user%example.net@msn.transport.example.org.

All of this is just theory anyway, as according to Jabber philosophy your home server is not supposed to handle any gateways itself (psyced is the exception, as usual). You get friends with a transport service instead, then you encode all your friends as being at home in that transport - This means you have to shift all your friends to new addresses if that transport service goes out of business or you start doubting its disinterest in your private affairs. These are issues that are being worked upon, however. We haven't heard of any improvements in this area in the past >5 years.

The fact, that the protocol scheme is not supposed to be transmitted in an XMPP JID has a strength and a weakness:

CON: Jabber servers have no easy way to figure out if XMPP is the only possible protocol to reach a certain person.

PRO: All Jabber clients and servers have a scheme-free form of address of all people in their data structures. Thus one could replace the XMPP protocol at the backbone of Jabber and use PSYC (or whatever else) instead. So, if a sufficient number of Jabber installations also provided PSYC protocol, the network would slowly and transparently shift to a possibly more efficient interserver protocol. This idea is discussed in 'PSYC as Jabber S2S'. But this is only applicable to a structurally similar protocol such as PSYC in federation mode. So we did:

Users of Jabber clients connected to a psyced can communicate using user@host JIDs as usual, yet psyced will figure out which protocol is best to use to reach each recipient.

The Odyssey of Dialback

Concerning dialback, one of the reasons why it was preferred over an IP address check is jabberd was not able to bind() the socket to a particular IP at the time. And why does it require two TCP connections? Because in Java it is nicer to have a reader and a writer thread, and having them use the same socket is impractical (as mentioned here). At times there were even more than two TCP links between two XMPP servers, because of multiple hostnames. PSYC uses just a single so-called circuit in all of these circumstances. There have been many attempts to deal with this and close the book on dialback, but it kept coming back as the only properly functioning method. Fifteen years later it looks like XMPP has finally caught up and solved this.

Other Techie Details

Full JID as a stumbling block

XMPP distinguishes between a 'bare JID' which points to your identity as in joe@example.org and a 'full JID' such as joe@example.org/coolChat. The 'resource' appended at the end distinguishes the client connection which is supposed to receive certain messages from your identity and potential other clients of yours.

It all kind of fits together somehow, although it is a bit strange, that a remote chatroom (a MUC) should know which client you are chatting on, rather having your own server know about that. It would be interesting, if your client was entering the MUC directly using its own IP address (using something like a UNL in PSYC-speak) – that would allow for high bandwidth applications if the data no longer needs to flow throw your server.

It would also be practical if any application or chatroom could send you things without having to keep track of the client apps you are currently using, just by sending to your 'bare JID.' Unfortunately, several servers will reject such a 'stateless' mode of operation.

All in all it's not dramatic, but a bit strange how XMPP introduces an in-between concept, that combines the disadvantages of going through a server and not having a person's full peer address with the disadvantages of having to keep track what resource a user is currently using networkwide. If you think it is good to hide your peer address for privacy or security (which is a question of trust architecture really, but in many cases that's a valid point), then you shouldn't have to let everyone keep track of your current client resource. The server should figure out where packets need to go. It knows which chatrooms a client entered, after all.

But that breaks with XMPP philosophy. XMPP servers are intentionally agnostic and don't look at packets they deliver on behalf of clients. So, end of story, you can't always send messages to the 'bare JID.' Sometimes they will be considered illegal and return an error. It's up to you to consider this an achievement or an unnecessary complication of things.

Vocabulary

Broadcast stands for a single transmission operation to reach all recipients on a medium in a single go - like on a radio. Once you transmit, everyone can listen - there is no extra effort involved in getting the content to each recipient. In the XMPP world the word however actually means a quite inefficient fan-out of copies of a message to each member of a collection of recipients, which, in terms of bandwidth consumption, results in pretty much the opposite of what is expected from a broadcast.

Also the word multicast is used in a confusing way to actually mean what the XCASTRFC describes as multi-unicast on page 14. The naming of XEP 0033 itself has been corrected, but the contents still frequently mentions multicast with an incorrect semantic connotation while XEP-0253 implements multicast but makes no mention of it. It uses the word chaining instead. Twenty years of earlier research work do not matter, the world deserves new vocabulary.

Inter-server (S2S) encryption

Quite often Jabber is being advertised as being more secure because of its interserverencryption using TLS. Some even propose to remove dialback from the XMPP standard, because Jabber is secure now. fippo's 2007 survey however shows that only 4% of publicly accessible XMPP servers actually provide proper valid certification, everything else is either untrustworthy or unencrypted.

With all the recent efforts to improve that, we expect the number to have risen consistently – still the conversation data is unencrypted within server memory, and your talking partner's server might be participating in the PRISM program, so you better employ some OTR. PSYC in the meantime has given up on federation and TLS along with it and is now moving on to more secure strategies based on public-key addressing.

How to protect who is talking to whom?

From a privacy standpoint the fact that, even if you are employing OTR for end-to-end encryption, servers always see who is talking to whom, is a serious problem – especially in the light of the PRISM program and the largest XMPP deployments on earth being affected by that.

PSYC can deliver encrypted PSYC packets in an onion routing style in its native binary form, creating not much overhead besides the fact of doing onion routing. The secushare project is working on this. Doing so in XMPP however would mean to shuffle around big blobs of base64-encoded text. It can be done, but it sure is ugly. First of all you need to get your friends off of the PRISM servers.

pointless having a big standard noone wants to implement. have a working implementation first, have the standard come later.

Unfortunately these points have not been published in the paper document
but rather presented in voice at the FSW2011 (see the video for the W3C
Codebase session on June 4th).
They're in the OneSocialWeb-for-FSW slides however!

Questions and Answers

Why don't you help XMPP getting better instead of spending so much time documenting its weaknesses?

By writing a Jabber server and gateway as a part of psyced we are just documenting the problems we keep encountering. This page is a by-product of that. At the same time we are helping Jabber. Our first XEP (XMPP Extension Protocol, formerly known as Jabber Enhancement Proposal aka JEP) on Dialback Key Generation and Validation has been accepted as number 0185. Also XEP 0190 on Best Practice for Closing Idle Streams has been adopted.

See above for our Smart Unicast XEPs, which have been rejected in a council meeting on 2006-06-14. The hard data had been given to them before. See in the Federation vs Scalability paragraph above. Also the test framework existed at the time. psyced implements this experimental extension (just #define XMPPERIMENTAL).

Here we are in 2013, the historic chance to fix XMPP has been missed – by now there are devices implementing design mistakes in hardware – new technology will be so radically different, that it will slowly lure people away from XMPP's cloud architecture into the darknets.

So you are saying that XMPP is bad?

<fippo> No, otherwise we would not support it. We think it's an ugly protocol and many of the design decision are wrong, but since it enables us to communicate with more people than without it, we deal with it. After all, you do, too.

We did however experience situations that suggest that the XMPP Extension Protocols aren't always developed in an open fashion, contrary to the statement that the XSF be a "completely open and transparent standards development organization." See for example how the Jingle protocol was developed). Nor that the "developer-friendly standards process [...] places a premium on the values that built the Internet in the first place: rough consensus and running code." See this thread on standards-jig for that.

<lynX> Actually yes, I would say XMPP is bad as it has impeded better solutions from finding momentum and public interest. Everyone is so keen on having a standard, it doesn't matter as much how many bugs it has. Doing chat with XML as a wire syntax sounded like a grand idea in the year of the big XML hype, but it puts up a hard barrier on how far XMPP can go.

Why do messages to XMPP users take so long to echo on the screen sometimes?

First of all, feedback that a message has successfully reached its destination,
so-called echoes, is a PSYC feature. The Jabber gateway generates an echo as
soon as it successfully delivered a message to the recipient Jabber server,
which isn't exactly the same thing, but an acceptable approximation.

Jabber echoes should therefore actually appear quicker than PSYC echoes.
They sometimes however don't, because most Jabber servers shutdown
inactive network links after a short period. They are considered a cause
for load rather than a chance for quick delivery - which is an architectural
problem. By consequence after that short period a new connection has to be
established between the servers. Using the XMPP protocol this means
opening two TCP connections involving two TCP handshake processes plus the
XMPP dialback handshakes. This totals in a typical delay of several
seconds. It takes less if authentication based on TLS certificates is used instead of so-called dial-back.

This still is no cause for concern. Only when no echoes appear at all, you should presume that your message has not successfully reached the destination, as is the case with PSYC messages, too.
Normally you will receive a detailed error notice as soon as your server gives
up on delivery of your message, but this may take a while as psyced keeps a queue of outgoing messages and tries to recover the communication link a couple of times before returning the queue to its respective senders.

By reading your lines I get the impression these Jabber people never get anything right

No no that is certainly not the case. The competence is there, in abundance, but for some reason it doesn't always end in the technology itself. Maybe it is simply because the technology started on several wrong feet, and you
just can't keep tradition and compatibility at the same time with radical repairs. Also the new way of doing things has grown roots, and all the
XML structures and acknowledgement stanzas may look and feel beautiful to those who like them.
After so many years using the technology this way, tradition prevales
over the disadvantage count. Only an by-stander still has the capacity to
see all the problems in their full ugliness.

On the other hand, so many aspects of technology are falling apart as we look at them, especially in the light of 2013's revelations. Some folks broke the Internet. The task at hand is to fix it, rewrite a GNU Internet stack, and XMPP is among the technologies with a completely wrong architecture to be helpful. We have to restart from scratch anyhow.