exception uploading statistics (#836)

Eric Dziewa opened 7 years ago

The new 7.2.0 installation on tpub cannot upload statistics. I have these cronjobs running on tpub, and home on my work computer.

They only ever report a connection error when Tigase (t2, t6) is actually down for a reboot.

curl -X POST --connect-timeout 60 --max-time 120 -sS --data-binary '<test/>' --header 'Content-Type: application/xml' http://192.95.36.80:8080/rest/stats/upload

curl -X POST --connect-timeout 60 --max-time 120 -sS --data-binary '<test/>' --header 'Content-Type: application/xml' http://198.27.120.208:8080/rest/stats/upload

If you view those 2 URLs in your browser they are different. One says Jetty the other does not.

I reverted the installation on tpub to 7.1.0 and the error went away.

However I don't think this is tied to 7.2.0 explicitly. One of our customers has reported this.

They use 7.1, I went to check on cluster-c and the error appears there too.

2017-06-02 16:00:07.472 [Timer-1]          StatisticsUploader$1.run()              SEVERE:   exception uploading statistics
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
    at tigase.stats.collector.provider.StatisticsUploader.a(SourceFile:50)
    at tigase.stats.collector.provider.StatisticsUploader$1.run(SourceFile:109)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

Artur Hefczyc commented 7 years ago

I can see that HTTPS (SSL) is used for connection to upload statistics. So, maybe, the SSL certificate is accepted on some machines and is not accepted on other installations? If the ca-root is not well recognized this could lead to random problems.

Eric Dziewa commented 7 years ago

Now that you've mentioned certificate it gave me an idea. Apache wasn't running on t6. Which is strange because it's in the autostart list. Is working fine now.

Artur Hefczyc commented 7 years ago

Why does it matter? Apache is not used for stats upload, is it?

By the way %Eric , please write some notes while you work on the installation upgrade, anything which you encounter, any issues, doubts, anything which is not straightforward during the installation and upgrade. Then, hand it over to %Daniel. He can then prepare some kind of hints and helpers on 7.2.0 installation and maybe he can also include some of this information in documentation.

Eric Dziewa commented 7 years ago

Apache handles serving the LetsEncrypt cert on port 443. Then acts as a proxt to localhost 8080.

Artur Hefczyc commented 7 years ago

Ok, understand now.

Referenced from commit 1 year ago

Added configuration option for maxInactivityTime in seconds: for inactivity time set to 15 minutes we need - comp-name/max-inactivity-time[L]=900, Fixes #836

Artur Hefczyc committed 1 decade ago

f25c3a08

Type	Bug
Priority	Major
Assignee	Artur Hefczyc
RedmineID	5616
Version	tigase-server-8.0.0

	Eric Dziewa opened 7 years ago The new 7.2.0 installation on tpub cannot upload statistics. I have these cronjobs running on tpub, and home on my work computer. They only ever report a connection error when Tigase (t2, t6) is actually down for a reboot. `curl -X POST --connect-timeout 60 --max-time 120 -sS --data-binary '<test/>' --header 'Content-Type: application/xml' http://192.95.36.80:8080/rest/stats/upload curl -X POST --connect-timeout 60 --max-time 120 -sS --data-binary '<test/>' --header 'Content-Type: application/xml' http://198.27.120.208:8080/rest/stats/upload` If you view those 2 URLs in your browser they are different. One says Jetty the other does not. I reverted the installation on tpub to 7.1.0 and the error went away. However I don't think this is tied to 7.2.0 explicitly. One of our customers has reported this. They use 7.1, I went to check on cluster-c and the error appears there too. 2017-06-02 16:00:07.472 [Timer-1] StatisticsUploader$1.run() SEVERE: exception uploading statistics java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) at tigase.stats.collector.provider.StatisticsUploader.a(SourceFile:50) at tigase.stats.collector.provider.StatisticsUploader$1.run(SourceFile:109) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
	Artur Hefczyc commented 7 years ago I can see that HTTPS (SSL) is used for connection to upload statistics. So, maybe, the SSL certificate is accepted on some machines and is not accepted on other installations? If the ca-root is not well recognized this could lead to random problems.
	Eric Dziewa commented 7 years ago Now that you've mentioned certificate it gave me an idea. Apache wasn't running on t6. Which is strange because it's in the autostart list. Is working fine now.
	Artur Hefczyc commented 7 years ago Why does it matter? Apache is not used for stats upload, is it? By the way %Eric , please write some notes while you work on the installation upgrade, anything which you encounter, any issues, doubts, anything which is not straightforward during the installation and upgrade. Then, hand it over to %Daniel. He can then prepare some kind of hints and helpers on 7.2.0 installation and maybe he can also include some of this information in documentation.
	Eric Dziewa commented 7 years ago Apache handles serving the LetsEncrypt cert on port 443. Then acts as a proxt to localhost 8080.
	Artur Hefczyc commented 7 years ago Ok, understand now.
	Referenced from commit 1 year ago Added configuration option for maxInactivityTime in seconds: for inactivity time set to 15 minutes we need - comp-name/max-inactivity-time[L]=900, Fixes #836 Artur Hefczyc committed 1 decade ago f25c3a08