Projects tigase _server tigase-pubsub Issues #18
Properly handling high traffic in PubSub without OOM (#18)
Closed
Artur Hefczyc opened 1 decade ago

The basic question here is how to properly handle a notification to let's say 10M or 100M subscribers? Or generally how to handle notifications to a high traffic PubSub installation without a OOM error? Let's say we have 100M PubSub nodes with 100 subscribers each but publishing to nodes is quite frequent.

The main goal is to avoid OOM. It is acceptable to drop some packets in order to prevent queue overload as it may happen the generated traffic exceeds physical capabilities of the HW the PubSub is running on.

I was experimenting with a code which monitors memory usage and sleeps until it is low enough before generating more packets. Something similar could be implemented in PubSub I guess: Look at createNodes method

Andrzej Wójcik (Tigase) commented 1 decade ago

I added similar handling on low memory as suggested, but I applied lower delay and lower low memory level.

I also changed mechanism of generation of PubSub notifications so that component will send them as they are created instead of adding them to a queue and sending them later in one batch.

Maybe we should consider other improvement to mechanism of handling low memory case, ie. we should try to generate notification only n times, and if for each time we get info about low memory then maybe we should drop generation of this notification? I would suggest adding this as it would protect us against case in which under very heavy traffic server will be using almost all memory which will lead to permanently blocking generation of notifications by PubSub Component which could block whole component. But then we may never generate notifications on very high memory usage.

Artur Hefczyc commented 1 decade ago

I added similar handling on low memory as suggested, but I applied lower delay and lower low memory level.

I also changed mechanism of generation of PubSub notifications so that component will send them as they are created instead of adding them to a queue and sending them later in one batch.

Thank you, good work.

Maybe we should consider other improvement to mechanism of handling low memory case, ie. we should try to generate notification only n times, and if for each time we get info about low memory then maybe we should drop generation of this notification? I would suggest adding this as it would protect us against case in which under very heavy traffic server will be using almost all memory which will lead to permanently blocking generation of notifications by PubSub Component which could block whole component. But then we may never generate notifications on very high memory usage.

This is a good point but I do not think we have to really worry about this. If the service is permanently running at the memory limits the installation is kind of broken anyway. This is more a deployment issue rather then implementation.

Referenced from commit 11 months ago
#18
git-svn-id: file:///home/svn/repos/tigase-pubsub/trunk@165 ea752f3b-7626-48e1-ae42-c030457b2254
Bartosz Małkowski committed 2 decades ago
Referenced from commit 11 months ago
#18 #19 #12 #13 #15 #21 #22
git-svn-id: file:///home/svn/repos/tigase-pubsub/trunk@43 ea752f3b-7626-48e1-ae42-c030457b2254
Bartosz Małkowski committed 2 decades ago
Referenced from commit 11 months ago
#18 works better, but not fully...
git-svn-id: file:///home/svn/repos/tigase-pubsub/trunk@36 ea752f3b-7626-48e1-ae42-c030457b2254
Bartosz Małkowski committed 2 decades ago
issue 1 of 1
Type
Task
Priority
Normal
Assignee
RedmineID
1694
Spent time
4h
Issue Votes (0)
Watchers (0)
Reference
tigase/_server/tigase-pubsub#18
Please wait...
Page is in error, reload to recover