Monitor Sztab JVM vital signs (SZ-143)

rk@tigase.net opened 4 weeks ago

Problem

During performance investigations, I have had to use raw curl calls inside the Docker container to check heap, GC, and endpoint metrics. Need a basic operational view before stress testing large repo imports.

Approach

Spring Boot Admin — zero custom code, surfaces existing Actuator endpoints in a usable UI.

Tasks

Add spring-boot-admin-starter-client to pom.xml Wire up SBA server container in compose override Configure client registration in application-docker.yml Verify heap, GC, and HTTP request rate metrics are visible in the dashboard

Activities

rk@tigase.net commented 4 weeks ago

rksuma@Ramakrishnans-MacBook-Pro sztab % git checkout -b feature/SZ-143-performance-dashboard
Switched to a new branch 'feature/SZ-143-performance-dashboard'
rksuma@Ramakrishnans-MacBook-Pro sztab %

rk@tigase.net changed state to 'In Progress' 4 weeks ago
Previous Value Current Value
Open

In Progress

Previous Value	Current Value
Open	In Progress

rk@tigase.net commented 4 weeks ago

Metrics exposed by Spring Boot Actuator

rksuma@Ramakrishnans-MacBook-Pro sztab % docker exec sztab-sztab-backend-1 curl http://localhost:8181/actuator
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1907    0  1907    0     0  51550      0 --:--:-- --:--:-- --:--:-- 52972
{"_links":{"self":{"href":"http://localhost:8181/actuator","templated":false},"beans":{"href":"http://localhost:8181/actuator/beans","templated":false},"caches-cache":{"href":"http://localhost:8181/actuator/caches/{cache}","templated":true},"caches":{"href":"http://localhost:8181/actuator/caches","templated":false},"health":{"href":"http://localhost:8181/actuator/health","templated":false},"health-path":{"href":"http://localhost:8181/actuator/health/{*path}","templated":true},"info":{"href":"http://localhost:8181/actuator/info","templated":false},"conditions":{"href":"http://localhost:8181/actuator/conditions","templated":false},"configprops":{"href":"http://localhost:8181/actuator/configprops","templated":false},"configprops-prefix":{"href":"http://localhost:8181/actuator/configprops/{prefix}","templated":true},"env":{"href":"http://localhost:8181/actuator/env","templated":false},"env-toMatch":{"href":"http://localhost:8181/actuator/env/{toMatch}","templated":true},"flyway":{"href":"http://localhost:8181/actuator/flyway","templated":false},"loggers":{"href":"http://localhost:8181/actuator/loggers","templated":false},"loggers-name":{"href":"http://localhost:8181/actuator/loggers/{name}","templated":true},"heapdump":{"href":"http://localhost:8181/actuator/heapdump","templated":false},"threaddump":{"href":"http://localhost:8181/actuator/threaddump","templated":false},"metrics-requiredMetricName":{"href":"http://localhost:8181/actuator/metrics/{requiredMetricName}","templated":true},"metrics":{"href":"http://localhost:8181/actuator/metrics","templated":false},"sbom":{"href":"http://localhost:8181/actuator/sbom","templated":false},"sbom-id":{"href":"http://localhost:8181/actuator/sbom/{id}","templated":true},"scheduledtasks":{"href":"http://localhost:8181/actuator/scheduledtasks","templated":false},"mappings":{"href":"http://localhost:8181/actuator/mappings","templated":false}}}%                                                                        rksuma@Ramakrishnans-MacBook-Pro sztab %

rk@tigase.net changed title 3 weeks ago

Previous Value	Current Value
Implement Performance Dashboard for operational monitoring	Monitor Sztab JVM vital signs

rk@tigase.net commented 3 weeks ago
I brought up Spring Boot Actuator and it's more of a developer view and less a performance monitor. Please see screens below. I will hence add Prometheus screens as an increment to see metrics under load.

Addendum — Prometheus scraping layer

Adding Prometheus as a time-series backend on top of Spring Boot Admin. SBA shows current state; Prometheus records history and enables rate queries under load — specifically the HTTP request rate and heap pressure curves needed to characterize Sztab behavior during large repo imports.

deploy/docker/prometheus.yml — scrape config targeting /actuator/prometheus docker-compose.override.yml — Prometheus sidecar on port 9091

Query of interest: rate(http_server_requests_seconds_count[1m]) — shows live request rate rising under stress.
rk@tigase.net commented 3 weeks ago
Added Prometheus screen - below are two example software monitoring using Prometheus graphs:

Metric under question The rate of incoming HTTP requests to the Sztab REST layer, broken down by endpoint, method, and result. This tells us how hard the Spring Boot layer is working.

What user action cuases this stress User actions that triggers REST calls (browsing issues, loading projects, running issue query, creating PRs). Under normal use the rate is low and steady. Under load — multiple users, CI pipelines, or a bulk import — the rate climbs and we can see which endpoints are under most stress.

rate(http_server_requests_seconds_count[1m]

as I use a curl script to hammer Sztab:

rksuma@Ramakrishnans-MacBook-Pro docker % for i in {1..200}; do curl -s -o /dev/null -w "%{http_code}\n" -b "JSESSIONID=0616D0AFC4A89C34ACCAC63A047376D8" http://sztab.local/api/issues; done 200 200 200 200
rk@tigase.net commented 3 weeks ago

Prometheus scraping layer complete. HTTP request rate monitoring demonstrated under curl load. Screenshots attached. Sztabina instrumentation tracked in SZ-144.
rk@tigase.net changed state to 'Closed' 3 weeks ago
Previous Value Current Value
In Progress

Closed
Login to comment

Previous Value	Current Value
In Progress	Closed

Type	New Feature
Priority	Normal
Assignee	rk@tigase.net
Version	1.10.1
Sprints	n/a
Customer	n/a

Issue Votes (0)

Watchers (2)

Reference

SZ-143

Problem

Approach

Tasks

Metrics exposed by Spring Boot Actuator

Addendum — Prometheus scraping layer