Comments on: Why is Twitter using XMPP/Jabber protocols?

By: mgkimsal

mgkimsal — Wed, 21 May 2008 14:23:24 +0000

OK guys – you’re probably correct. Not sure where I inferred that XMPP was used internally – it’s probably only for external interactions.

By: George Hotelling

George Hotelling — Wed, 21 May 2008 14:20:23 +0000

I don’t think they’re using XMPP on the backend, they have a front-end bot at twitter@twitter.com that can send and receive updates once you link your account to your Jabber ID.

I get twitter notifications via IM from that bot, I’m pretty sure they don’t use XMPP as a messaging protocol between their servers.

By: Vidar Hokstad

Vidar Hokstad — Wed, 21 May 2008 14:00:28 +0000

IF they were running XMPP in the backend, scaling it would be trivial (just partition the userbase over many domains with a hashing function and make that handled by separate servers), so I very much doubt they do depend on it for their core infrastructure. More likely they just gateway to it for external use.

In fact, there are no good reasons for Twitter to go down as much as they do – their scalability is trivially solved by partitioning (even if it leads to more data duplication ultimately, because you’d need to duplicate “tweets” on each partition where there’s a follower for it to be effective). It’s just not a hard problems to scale the moment you’ve accepted you need to partition the data.

Besides, your network costs with polling are larger (the polling requests needs to be received too) and the cost of keeping the list of listeners is trivial. It scales close to linearly with replication/chaining, and you’re likely to run into CPU and network limits long before you run into RAM limits over the number of connections on a properly tuned system.