After an initial investigation, the issue was narrowed down to the size of direct buffers and their pool in IOUtil.

While default buffer size was taken from the operating system via JVM (java.lang.Math.max(socket.getReceiveBufferSize(), socket.getSendBufferSize()), 8MB per each buffer), this value for each connection was overwritten in Tigase XMPP Server connection managers to use only 2KB or 64KB.

This led to the situation in which Tigase XMPP Server was allocating 8MB buffer (would be w 2MB for typical deployment) for each thread processing data for each connection manager. In this case, I would end up having 16 threads, 4 connection managers, and 8MB multiplied. If all connections and threads would be used then 512MB of direct memory would be required.

On the other hand, the maximum size of the buffer allowed by Tigase XMPP Server is only 64K (S2S connections, other connections have lower values). Keeping other values the same, I would end up with only 4MB of direct memory.

I think that we can remove this direct buffers pool and use it directly. With this change, we would not keep large and mostly unused buffers and we would allocate smaller buffers when required. I do not expect any performance issues, while we would reduce memory usage which would be very good for smaller installations.

I've applied a fix and it looks like memory requirements are now lowered. In my case, memory usage dropped from around 1.2GB to 0.9GB of RAM.

Looks good.