We restarted our cluster to deal with the session-close performance problem, but just encountered another issue: when there's a large number of existing users on other nodes, new nodes can't join the cluster since the initial cluster cache sync packet is too large for the XML parser, trips the DOS protections and then causes the cluster connection to be closed. Effectively this means that if a node goes down (as this one did) it won't rejoin the cluster but will still accept c2s connections, leading to a "split brain" where anyone on that node is isolated from the rest of the cluster.

Here's the log message in question, I won't paste in the full packet since it's far too large:

2013-06-02 17:50:33 ClusterConnectionManager.xmppStreamOpened() INFO: Stream opened: {id=c7c9b815-9957-4f74-8ff6-ef538b49c5e3, to=8.gs.ea.com, xmlns:stream=http://etherx.jabber.org/streams, from=2.gs.ea.com, xmlns=tigase:cluster}

2013-06-02 17:50:33 XMPPIOService.processSocketData() INFO: null, type: connect, Socket: nullSocket[addr=5.gs.ea.com/10.82.107.125,port=5277,localport=38639], Incorrect XML data: ="http://jabber.org/protocol/caps" node="http://ea.com/xmpp" hash="sha-1"/></presence {packet continues for several hundred lines} , stopping connection: null, exception:

tigase.xmpp.XMPPParserException: Too many elements for staza, possible DOS attack.

    at tigase.xmpp.XMPPDomBuilderHandler.newElement(XMPPDomBuilderHandler.java:379)

    at tigase.xmpp.XMPPDomBuilderHandler.startElement(XMPPDomBuilderHandler.java:347)

    at tigase.xml.SimpleParser.parse(SimpleParser.java:230)

    at tigase.xmpp.XMPPIOService.processSocketData(XMPPIOService.java:456)

    at tigase.net.IOService.call(IOService.java:235)

    at tigase.net.IOService.call(IOService.java:87)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)