Sometimes the HTTP code stops sending anything to the client. It will accept connections but doesn't send any data after. I noticed one of our health check cronjobs was establishing connections but never finishing as it waited for data. This isn't a problem with network because the same thing happens using localhost. The only way to fix this is to restart Tigase Server.
eric@t2 ~/ # nc -v localhost 8080
Connection to localhost 8080 port [tcp/http-alt] succeeded!
get /admin/ HTTP/1.1
GET /admin/ HTTP/1.0
^C
Andrzej Wójcik (Tigase) commented 9 years ago
I looked into this and this issue is caused by HttpServer embedded in Java which is used by default in recent versions of Tigase XMPP Server as we do not want to have Jetty as default HTTP server as this is another dependency. Before we used Jetty HTTP server which scaled well due to usage of NIO.
Now as I looked how HttpServer implementation processes data it looks like it rather BIO version which by default is using only 1 thread. Due to this fact each started request, ie. opened TCP connection with only:
GET / HTTP/1.1\r\n
will block this thread until TCP connection is closed or all data of request will be received by server and processed. Which means that some broken HTTP client or dummy TCP port scanner or even huge load may lead to this issue.
As it is not possible to change this behavior but it is possible to change used executor implementation we may use other approach, in which we may provide pool of ie. 4 threads to speed up processing and to process one request while other thread will wait for data and also add timeout after which connection which request was not finished would be stopped and TCP connection would be closed to deal with buggy HTTP clients.
%kobit , %eric - any suggestions about number of threads which should be used by default and default allowed timeout for single HTTP request?
I think we should go with at least 4 threads and 60 seconds for timeout but I'm open for suggestions.
Andrzej Wójcik (Tigase) commented 9 years ago
As I explained in previous comment solution for this was to change executor so more than one there would be used by HTTP API to process incoming HTTP requests and to limit time of single HTTP request.
I set default values to 4 threads and 60 seconds for single request timeout.
This paramters can be changed by entries in etc/init.properties file:
which will set number of used threads to 8 and request timout to 120s (120000ms)
This will solve issue with single request blocking whole server until restart as 1 request will block 1 thread only for up to 60 seconds by default but if in default configuration more that 4 requests will appear and each one will block processing for 60 seconds then every new request will be queued and executed after this default 60s timeout (as will need to wait for previous requests to complete execution).
Alternative solution is to use Tigase HTTP API - Jetty HTTP Server subproject which will perform better in this case as it is using non-blocking architecture.
Artur Hefczyc commented 9 years ago
Eric, please confirm if the problem is solves and close the ticket.
Sometimes the HTTP code stops sending anything to the client. It will accept connections but doesn't send any data after. I noticed one of our health check cronjobs was establishing connections but never finishing as it waited for data. This isn't a problem with network because the same thing happens using localhost. The only way to fix this is to restart Tigase Server.