Binary Data is XMPP's Achilles Heel
June 10, 2008
A few months back, our users started to complain that clocks were skipping a lot and that chat messages were getting delayed. After a little bit of digging we realized the root cause: avatars. A few random users had avatars that were quite large (relative to normal xmpp stanzas). Because a TCP connection can only be retrieving one XMPP stanza at a time, all other incoming stanzas were blocked until the avatar was fully downloaded. This is bad news for applications that users expect to be very low latency like a game.
An Analogy To Multitasking
XMPP is like cooperative multitasking. Each stanza must wait until the previous stanzas have been received before it can be sent. There is no way to preempt a stanza to achieve a fairer usage of the connection. This means that large stanzas do not play nicely and cause other stanzas to hang. The only way to achieve balance is for all the stanzas to limit themselves to some small size and not consume the whole connection for long periods of time.
If all the stanzas don’t cooperate to achieve a low latency for stanza handling then you get a poor experience for users. This is similar to that of OS 9 or Windows 3.x when one application didn’t cede control as often as it should. This problem becomes a lot worse as the connection speed declines. Our broadband users were able to notice this with avatars under 20kB in size.
A Proposed Solution
The only real solution without retooling the entire XMPP spec is to enforce a small maximum stanza size. I suggest 4kB, which is about 1 second for a dialup user. If any stanza would be bigger than 4kB, then it needs to be chunked and sent out as a group of stanzas, each of which is 4kB or less.
The in-band bytestreams specification (XEP 47) already exists to do exactly this kind of thing. Unfortunately it is used mostly for file transfers and not for more common tasks like avatars and large messages. The community needs to address this by building in-band bytestream support into places where large data may occur. It should be part of the avatar specifications (XEP 84 and the historical XEP 153). There should be a XEP documenting how clients would send large messages to each other. New specifications like the data element spec (XEP 231) need to be more clear about their use cases and the consequences of abuse.
A Workaround For Now
Until we can address some of the specification problems and client and server support issues, we will have to work around the large stanza issues. Thankfully, large stanzas in common use are mostly limited to avatars (roster queries are not usually large unless you are Peter Saint-Andre and they happen once at the very start of a connection where high latency is not a big issue). For these the best workaround is to request them out of band.
Large stanzas and XMPP don’t combine well, and this fact seems not well enough known in the XMPP community. I think the primary reason for this is that most XMPP clients today are only scratching the surface of what is possible with the platform. As more advanced functionality is used, more and more people will discover this issue as we did.
Let’s address this issue now before it becomes a bigger issue by:
- Limiting stanza size to some small value (4kB)
- Sending large bits of data in chunks with in-band bytestreams (XEP 47)
- Retooling XEPs like User Avatar and Pubsub to support the use of in-band bytestreams
- Making it obvious in new XEPs (like XEP 231) that they need to be used with care and cooperate with small stanzas
- Upgrading clients and servers to support in-band bytestreams for large stanzas
- Working around the problem while we figure out canonical solutions so that our users have the best experience possible
Let’s make sure that XMPP continues to improve, since it will take over the world.