@kobit Should I consider adding some grafana-loki integration on k8s (helm charts) level? Maybe we could add additional pod which would upload logs from our installations to loki server?

Yes, I have been using loki in my tests and experiments and it seem quite good.

CloudWatch is indeed very expensive.

I also use Prometheus for monitoring with Grafana to display metrics.

In my side project, I'm using Prometheus and Grafana for monitoring. Maybe it would be good to plan to add some metrics (Tigase specific) to Tigase, so we could monitor Tigase with Grafana & Prometheus?

Maybe it would be good to plan to add some metrics (Tigase specific) to Tigase, so we could monitor Tigase with Grafana & Prometheus?

@andrzej.wojcik #server-1034 :-)

Definitely yes.

Having a custom logger with tagging for loki can be useful but with docker/k8s we can easily get console log as a source for scrapping, see #k8s-7

What about the case where we have to manage 100s or 1,000s systems?

Having a custom logger with tagging for loki can be useful but with docker/k8s we can easily get console log as a source for scrapping, see #k8s-7

Two "buts":

1. not everyone uses k8s
2. console log doesn't contain all logs (though, scraping logs from tigase.log.x is probably a possibility as well) but more importantly - it has quite awful performance.

IMHO dedicated handler could still be useful.