Trim "Processed packets thread" statistics (#709)

Closed

Activities

wojciech.kapcia@tigase.net opened 8 years ago

Due Date
2016-09-01

While looking at the statistics (and sizes of them) I've noticed lot's of similar entries: @cl-comp/Processed packets thread…@. This is especially noticeable with higher number of CPU/cores. With only 4 of them we get roughly 2x16 entries for majority of components, but 2x32 entries for cl-comp, and 2x64 for sess-man (see below). With 16 cores this grows even more - uses more memory, size on disk and actually makes it more difficult to look at the statistics.

I think that turning it into array (like @cl-comp/Last day disconnects@) would be a good idea:

cl-comp/Last day disconnects    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

In addition, the biggest advantage of this information is to know whether the load is spread evenly across the threads, so we could add additional statistic with information about the distribution and/or if there are any outliers.

Thoughts?

cl-comp/Processed packets thread: in_0-cl-comp  235809
cl-comp/Processed packets thread: in_1-cl-comp  32364
cl-comp/Processed packets thread: in_2-cl-comp  33437
cl-comp/Processed packets thread: in_3-cl-comp  33144
cl-comp/Processed packets thread: in_4-cl-comp  36534
cl-comp/Processed packets thread: in_5-cl-comp  31350
cl-comp/Processed packets thread: in_6-cl-comp  29767
cl-comp/Processed packets thread: in_7-cl-comp  30533
cl-comp/Processed packets thread: in_8-cl-comp  29350
cl-comp/Processed packets thread: in_9-cl-comp  32395
cl-comp/Processed packets thread: in_10-cl-comp 32271
cl-comp/Processed packets thread: in_11-cl-comp 35581
cl-comp/Processed packets thread: in_12-cl-comp 32870
cl-comp/Processed packets thread: in_13-cl-comp 34462
cl-comp/Processed packets thread: in_14-cl-comp 49423
cl-comp/Processed packets thread: in_15-cl-comp 34374
cl-comp/Processed packets thread: in_16-cl-comp 44389
cl-comp/Processed packets thread: in_17-cl-comp 42973
cl-comp/Processed packets thread: in_18-cl-comp 50507
cl-comp/Processed packets thread: in_19-cl-comp 43489
cl-comp/Processed packets thread: in_20-cl-comp 45282
cl-comp/Processed packets thread: in_21-cl-comp 46531
cl-comp/Processed packets thread: in_22-cl-comp 46720
cl-comp/Processed packets thread: in_23-cl-comp 46180
cl-comp/Processed packets thread: in_24-cl-comp 43065
cl-comp/Processed packets thread: in_25-cl-comp 47136
cl-comp/Processed packets thread: in_26-cl-comp 43368
cl-comp/Processed packets thread: in_27-cl-comp 40855
cl-comp/Processed packets thread: in_28-cl-comp 38304
cl-comp/Processed packets thread: in_29-cl-comp 37258
cl-comp/Processed packets thread: in_30-cl-comp 36933
cl-comp/Processed packets thread: in_31-cl-comp 34420
cl-comp/Processed packets thread: out_0-cl-comp 240205
cl-comp/Processed packets thread: out_1-cl-comp 31831
cl-comp/Processed packets thread: out_2-cl-comp 34506
cl-comp/Processed packets thread: out_3-cl-comp 32562
cl-comp/Processed packets thread: out_4-cl-comp 35489
cl-comp/Processed packets thread: out_5-cl-comp 29036
cl-comp/Processed packets thread: out_6-cl-comp 26575
cl-comp/Processed packets thread: out_7-cl-comp 25369
cl-comp/Processed packets thread: out_8-cl-comp 26140
cl-comp/Processed packets thread: out_9-cl-comp 27647
cl-comp/Processed packets thread: out_10-cl-comp        26724
cl-comp/Processed packets thread: out_11-cl-comp        27452
cl-comp/Processed packets thread: out_12-cl-comp        23183
cl-comp/Processed packets thread: out_13-cl-comp        21398
cl-comp/Processed packets thread: out_14-cl-comp        35675
cl-comp/Processed packets thread: out_15-cl-comp        22915
cl-comp/Processed packets thread: out_16-cl-comp        30951
cl-comp/Processed packets thread: out_17-cl-comp        29240
cl-comp/Processed packets thread: out_18-cl-comp        34888
cl-comp/Processed packets thread: out_19-cl-comp        26656
cl-comp/Processed packets thread: out_20-cl-comp        29296
cl-comp/Processed packets thread: out_21-cl-comp        32229
cl-comp/Processed packets thread: out_22-cl-comp        30679
cl-comp/Processed packets thread: out_23-cl-comp        31927
cl-comp/Processed packets thread: out_24-cl-comp        28184
cl-comp/Processed packets thread: out_25-cl-comp        34138
cl-comp/Processed packets thread: out_26-cl-comp        34280
cl-comp/Processed packets thread: out_27-cl-comp        35319
cl-comp/Processed packets thread: out_28-cl-comp        34173
cl-comp/Processed packets thread: out_29-cl-comp        32387
cl-comp/Processed packets thread: out_30-cl-comp        30950
cl-comp/Processed packets thread: out_31-cl-comp        32969
…
sess-man/Processed packets thread: in_63-sess-man       27844

Artur Hefczyc commented 8 years ago

Good idea, especially to changing it into an array of values.

wojciech.kapcia@tigase.net commented 8 years ago

Given that recently I was working with statistics code and this was rather small change I've decided to resolve this one as well and include it in origin/release (i.e. 7.1.0).

Processed packets thread is now presented as an array and for threads IN/OUT we also calculate standard deviation (and list outliers as thread name:value), the result looks like this:

message-router/Processed packets thread IN=[2, 5, 0, 0, 1, 0, 1, 0, 1, 0, 2570, 0, 5019, 0, 2508, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 4, 4, 1, 1]
message-router/Processed packets thread OUT=[12]
message-router/Processed packets thread (outliers) IN=mean: 316.0, deviation: 1044.1141131361073, outliers: [in_10-message-router:2570, in_12-message-router:5019, in_14-message-router:2508]
message-router/Processed packets thread (outliers) OUT=mean: 12.0, deviation: 0.0
…
c2s/Processed packets thread IN=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2539, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
c2s/Processed packets thread OUT=[3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
c2s/Processed packets thread (outliers) IN=mean: 79.0, deviation: 441.76743740343744, outliers: [in_10-c2s:2539]
c2s/Processed packets thread (outliers) OUT=mean: 1.0, deviation: 5.75, outliers: [out_10-c2s:33]
…
pubsub/Processed packets thread IN=[246, 231, 210, 237, 223, 227, 224, 251, 267, 233, 240, 200, 219, 250, 243, 233, 243, 218, 209, 235, 249, 236, 233, 228, 236, 247, 209, 249, 259, 250, 251, 223]
pubsub/Processed packets thread OUT=[239, 233, 230, 245, 220, 201, 217, 241, 260, 225, 234, 211, 244, 249, 225, 249, 251, 218, 216, 261, 235, 222, 227, 239, 237, 242, 202, 265, 243, 257, 234, 237]
pubsub/Processed packets thread (outliers) IN=mean: 234.0, deviation: 15.496975511369952, outliers: [in_8-pubsub:267, in_11-pubsub:200]
pubsub/Processed packets thread (outliers) OUT=mean: 234.0, deviation: 16.00488206766923, outliers: [out_5-pubsub:201]
…
sess-man/Processed packets thread IN=[1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 33, 0, 2501, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0]
sess-man/Processed packets thread OUT=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2537, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]
sess-man/Processed packets thread (outliers) IN=mean: 19.0, deviation: 220.17743200087514, outliers: [in_108-sess-man:2501]
sess-man/Processed packets thread (outliers) OUT=mean: 20.0, deviation: 223.34862743030234, outliers: [out_106-sess-man:2537]

Btw. I've noticed that eventbus is single-threaded:

eventbus/Processed packets thread IN=[1]
eventbus/Processed packets thread OUT=[1]
eventbus/Processed packets thread (outliers) IN=mean: 1.0, deviation: 0.0
eventbus/Processed packets thread (outliers) OUT=mean: 1.0, deviation: 0.0

Daniel Wisnewski commented 8 years ago

%kobit please assign to me if works/looks OK, I will update documentation.

Artur Hefczyc commented 8 years ago

Daniel, yes, this looks good to me. Please update docs.

Daniel Wisnewski commented 8 years ago

Documentation updated, closing issue.

Type	New Feature
Priority	Normal
Assignee	Daniel Wisnewski
RedmineID	4504
Version	tigase-server-7.1.0
Spent time	13h 45m

#722 Closed

Eventbus is single-threaded

Issue Votes (0)

Watchers (0)

Reference

tigase/_server/server-core#709