Projects tigase _server server-core Issues #1210
Make Kernel initialisation parallel (#1210)
wojciech.kapcia@tigase.net opened 4 years ago

Currently Tigase XMPP Server requires dozens of seconds to start up (15s and up, depending on machine specification). It would be great if Kernel beans initialisation would be don in parallel instead of serial.

(With this implemented we could probably do away with parallel data source initialisation)

Andrzej Wójcik (Tigase) commented 3 years ago

I've looked at this and made a few adjustments to internal methods which will now not convert arrays into lists and then filter those lists but will use streams for processing. That should slightly improve performance.

As for parallel initialization, that is very difficult as we currently are building "beans" on-demand, which works fine even with circular dependencies. However, when I tried to convert this to be executed concurrently it resulted either in deadlocks or in multiple instances of the same bean being instantiated - both results are not acceptable.

Taking into account how difficult and time-consuming this may be, I wonder if we still need this as improvements for MySQL initialization (optional flag) was done and startup now is around 7s.

wojciech.kapcia@tigase.net commented 3 years ago

Out of curiosity I compared original tigase (regular configuration, almost "default"; using local postgres instance):

== 2021-09-09 14:58:26.046 Server finished starting up in (21s) and (if there wasn't any error) is ready to use

And your changes:

== 2021-09-09 15:05:12.022 Server finished starting up in (22s) and (if there wasn't any error) is ready to use
  • not that much of an improvement
  • YMMV but it's kinda far from "7s"

Though, considering it may be non trivial and there are other more important/better task (like getting rid of groovy, which should also help with startup) I think this could be made low priority.

Andrzej Wójcik (Tigase) commented 3 years ago

With useCallableMysqlWorkaround = true and MySQL I've Tigase running in 7s.

== 2021-09-09 16:07:06.347 Server finished starting up in (8s) and (if there wasn't any error) is ready to use
Andrzej Wójcik (Tigase) commented 3 years ago

You were running it with PostgreSQL, so maybe that is the same issue as with MySQL without useCallableMysqlWorkaround - SQL driver is slowing Tigase down.

wojciech.kapcia@tigase.net commented 3 years ago

MySQL with useCallableMysqlWorkaround = false

[2021-09-10 12:54:51:426] [WARNING ] [                  main ] DSLBeanConfigurator.dumpConfiguration(): Dumping full server configuration to: etc/config-dump.properties
== 2021-09-10 12:54:51.882 Server finished starting up in (62s) and (if there wasn't any error) is ready to use

MySQL with useCallableMysqlWorkaround = true:

[2021-09-10 13:02:54:023] [WARNING ] [                  main ] DSLBeanConfigurator.dumpConfiguration(): Dumping full server configuration to: etc/config-dump.properties
== 2021-09-10 13:02:54.494 Server finished starting up in (21s) and (if there wasn't any error) is ready to use

So, for me, the baseline is ~20s. Yes, I know I have older machine but it still rocks ~4500 points in cpubenchmark... which is quite similar to lot's of machines now being sold thus we could assume that lot's of people trying Tigase would get same result.

Our tigase.org installation (with postgres) also starts with similar time:

[2021-09-07 18:28:46:875] [WARNING ] [                  main ] DSLBeanConfigurator.dumpConfiguration(): Dumping full server configuration to: etc/config-dump.properties
== 2021-09-07 18:28:48.181 Server finished starting up in (18s) and (if there wasn't any error) is ready to use

I made a simple experiment and removed majority of dependencies (virtually left only with our own binaries; save for removing tigase-http and http-api-jetty, tigase-auditlog, tigase-push, tigase-extras*) and the startup time went down to 5s.

Then I removed only groovy + http-api (and tigase-extras as it references groovy as well) and the result was:

[2021-09-10 13:17:18:290] [WARNING ] [                  main ] DSLBeanConfigurator.dumpConfiguration(): Dumping full server configuration to: etc/config-dump.properties
== 2021-09-10 13:17:18.839 Server finished starting up in (6s) and (if there wasn't any error) is ready to use

Thus, I would conclude that the issue is not with the kernel itself but rather by groovy (and maybe http-api using it extensively). As I said in the previous message: "getting rid of groovy, which should also help with startup" -- seems to be on point here.

Andrzej Wójcik (Tigase) commented 3 years ago

Yes, you are correct. Groovy not only is slow to load (code using groovy usually loads groovy language libraries) but also takes time being scanned for annotations.

wojciech.kapcia@tigase.net commented 3 years ago

One more "tidbid": normal Tigase distribution takes after startup ~110M from heap and ~100M of metaspace. Without groovy it goes down to ~35M and ~60M... (metaspace would probably go even lower with even less external dependencies as they total to ~40M)

wojciech.kapcia@tigase.net batch edited 7 months ago
Name Previous Value Current Value
Iterations
empty
tigase-server-9.0.0
issue 1 of 1
Type
New Feature
Priority
Minor
Assignee
Version
tigase-server-9.0.0
Spent time
45m
Subsystem
Kernel
Iterations
Issue Votes (0)
Watchers (2)
Reference
tigase/_server/server-core#1210
Please wait...
Page is in error, reload to recover