tigase.xmpp.XMPPParserException: Too many elements for staza, possible DoS attack.Current service class tigase.xmpp.XMPPIOService limit of elements: 10000000 (#371)

kellogs . opened 1 decade ago

Hi,

in a cluster setup where there are two quite powerful machines (16 CPUs each, 48 and respectively 72 GB RAM) I was surprised to see this warning happening. 10 million elements limit seems not enough (Lowered it from 100 million where the 48GB server couldn't keep up and mayhem broke loose). Would it not be better not to glue all the stanzas together in those mega-stanzas that the clusters exchange between them, but instead to place a limit on the maximum number of stanzas that can be glued together before sending through the cluster socket ? I think it would be a great RAM saver and a great plus for overall tigase server health.

Thank you!

Activities

kellogs . commented 1 decade ago

LE: this happens during a tsung load test that would target both servers and fire a maximum of 360 req/s (40 auth get + 40 auth set + 40 presence available + 160 custom iq's + 40 messages + 40 presence unavailable) / second.
Artur Hefczyc commented 1 decade ago

I am not 100% sure, as I have too little information but as far as I remember this problem has been fixed in version 5.2.1 and later. For sure in version 5.3.0. It could be also a misconfiguration of the cluster which causes an incorrect behavior.

To be certain we would need some example/sample of the data with so many subelements for the XML element.

kellogs . commented 1 decade ago

Here are two cases:

2014-09-23 01:07:27.601 [pool-11-thread-29]  XMPPIOService.processSocketData()  INFO:   null, type: connect, Socket: nullSocket[addr=server25.domain2.com/192.168.101.25,port=5277,localport=56270], jid: null, Incorrect XML data: type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391886@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921776" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391903@domain1.com"/></data></cluster><cluster id="cl-31921814" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391903@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921833" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391886@domain1.com"/></data></cluster><cluster id="cl-31921822" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391897@domain1.com"/></data></cluster><cluster id="cl-31921830" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391930@domain1.com"/></data></cluster><cluster id="cl-31921837" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391914@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921827" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391905@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921839" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391914@domain1.com"/></data></cluster><cluster id="cl-31921832" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391912@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921825" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391901@domain1.com"/></data></cluster><cluster id="cl-31921849" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391899@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921852" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391927@domain1.com"/></data></cluster><cluster id="cl-31921856" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391934@domain1.com"/></data></cluster><cluster id="cl-31921846" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391918@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921864" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+391911@domain1.com" xmlns="jabber:client" type="probe" to="+391910@domain1.com"/></data></cluster><cluster id="cl-31921853" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391907@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921867" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391929@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921869" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"/><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391932@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921870" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391925@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921871" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="299186" to="+391936@domain1.com" from="+391911@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-31921873" to="sess-man@server26.domain2.com" from=", stopping connection: null, exception: 
tigase.xmpp.XMPPParserException: Too many elements for staza, possible DoS attack.Current service class tigase.xmpp.XMPPIOService limit of elements: 7000000


2014-09-23 01:21:45.833 [pool-11-thread-5]  XMPPIOService.processSocketData()  INFO:    null, type: connect, Socket: nullSocket[addr=server25.domain2.com/192.168.101.25,port=5277,localport=56330], jid: null, Incorrect XML data: e.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471071" to="+39111105@domain1.com" from="+39111099@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207884" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471033" to="+39111105@domain1.com" from="+39111087@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207881" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471114" to="+39111095@domain1.com" from="+39111105@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207885" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471114" to="+39111097@domain1.com" from="+39111105@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207889" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence from="+39111105@domain1.com" xmlns="jabber:client" type="probe" to="+39111101@domain1.com"/></data></cluster><cluster id="cl-34207890" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471114" to="+39111101@domain1.com" from="+39111105@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207891" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471114" to="+39111099@domain1.com" from="+39111105@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207893" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471096" to="+39111105@domain1.com" from="+39111101@domain1.com/tsung" xmlns="jabber:client"/></data></cluster><cluster id="cl-34207899" to="sess-man@server26.domain2.com" from="sess-man@server25.domain2.com" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server25.domain2.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server25.domain2.com</par></method-call><first-node>sess-man@server25.domain2.com</first-node></control><data><presence id="471114" to="+39111114@domain1.com" from="+39111105@domain1.com/tsung" xmlns="jabber, stopping connection: null, exception: 
tigase.xmpp.XMPPParserException: Too many elements for staza, possible DoS attack.Current service class tigase.xmpp.XMPPIOService limit of elements: 7000000

Always followed by:

2014-09-23 01:52:34.465 [ConnectionOpenThread]  ClusterConnectionManager.serviceStarted()  INFO: cluster connection opened: 192.168.101.25, type: connect, id=192.168.101.26_56622_192.168.101.25_5277
2014-09-23 01:52:34.465 [ConnectionOpenThread]  ClusterConnectionManager.serviceStarted()  INFO: cid: null, sending: <stream:stream xmlns='tigase:cluster' xmlns:stream='http://etherx.jabber.org/streams' from='server26.domain2.com' to='server25.domain2.com'>
2014-09-23 01:52:34.467 [pool-11-thread-57]  ClusterConnectionManager.xmppStreamOpened()  INFO: Stream opened: {id=7b70b149-d497-4dcb-8065-f312f9538fae, to=server26.domain2.com, xmlns:stream=http://etherx.jabber.org/streams, from=server25.domain2.com, xmlns=tigase:cluster}

Artur Hefczyc commented 1 decade ago

This is strange. Above data should not trigger this error. Could you try to reproduce the problem on our latest dev: 5.3.0? If it still happen we will investigate.
kellogs . commented 1 decade ago

built tigase-server 800c2460 from master branch and it almost fine when elements limit is at 2 million (had one DoS) but not so fine when at 700k where there were a bit more:

2014-09-23 22:53:31.579 [pool-8-thread-8] XMPPIOService.processSocketData() INFO: null, type: connect, Socket: nullSocket[addr=server25.domain2.com/192.168.101.25,port=5277,localport=33896], jid: null, Incorrect XML data: sess-man@server25.domain2.com +39111364@domain1.comtsungc2s@server25.domain2.com/192.168.101.25_1443_192.168.101.34_62708e76b6205-d044-4479-82e9-845d2b8a71182004sess-man@server25.domain2.com, stopping connection: null, exception:

tigase.xmpp.XMPPParserException: Too many elements for staza, possible DoS attack.Current service class tigase.xmpp.XMPPIOService limit of elements: 700000

2014-09-23 22:57:40.542 [pool-8-thread-17] XMPPIOService.processSocketData() INFO: null, type: accept, Socket: nullSocket[addr=/192.168.101.26,port=33822,localport=5277], jid: null, Incorrect XML data: sess-man@server26.domain2.com sess-man@server26.domain2.com sess-man@server26.domain2.com https://server26/blah/blahblah/+39140815@domain1.com//profile/IMG491-051956.jpg?temp_url_sig=c0cddc9109f93b6be7a034ee8694fb69b1054e22&temp_url_expires=2357564260 https://server26/blah/blahblah/+39140815@domain1.com//profile/IMG491-051956.jpg?temp_url_sig=fc671e68d12670db5ffd5292ac2855ea57bc5899&temp_url_expires=2357564260, stopping connection: null, exception:

tigase.xmpp.XMPPParserException: Too many elements for staza, possible DoS attack.Current service class tigase.xmpp.XMPPIOService limit of elements: 700000

Oh, and no rosters for these runs; initial setup involved some dynamic rosters in place.
Artur Hefczyc commented 1 decade ago

To me, it looks like some problem with installation, configuration mistake or some custom code causing kind of a loop which bounces packet back and forth between cluster nodes.
kellogs . commented 1 decade ago

Hmm, tried out the initial tigase-server.jar we were testing with but this time without that custom component, just presence and message stanzas. And no more DoS. What the custom component causing DoS did is receiving some IQs, asynchronously processing them and then sending back the results from the processing threads (non-tigase threads) via a call to tigase.server.AbstractMessageReceiver.addOutPacket(Packet); Perhaps there is some different way of returning a result were the processing takes place asynchronously ?

Thank you!
Artur Hefczyc commented 1 decade ago

I think, what you do is correct in principle. Most likely the problem is with incorrect addressing in either Packet object or stanza element.
Artur Hefczyc commented 1 decade ago

Not a bug in our code.
Login to comment

Type	Bug
Priority	Normal
Assignee	Artur Hefczyc
RedmineID	2295

Issue Votes (0)

Watchers (0)

Reference

tigase/_server/server-core#371