SSLProtocolException: Input record too big: max = 16709 len = 19190 (#1361)

Wojciech Kapcia (Tigase) opened 2 years ago

[2023-06-26 15:57:05:159] [INFO    ] [      pool-11-thread-1 ] LicenceChecker.b()               : Licence OK
javax.net.ssl.SSLProtocolException: Input record too big: max = 16709 len = 19190
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
	at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
	at tigase.io.JcaTLSWrapper.unwrap(JcaTLSWrapper.java:279)
	at tigase.io.TLSIO.decodeData(TLSIO.java:349)
	at tigase.io.TLSIO.processHandshake(TLSIO.java:327)
	at tigase.server.xmppclient.C2SIOService.extractTlsHandshakeData(C2SIOService.java:182)
	at tigase.server.xmppclient.C2SIOService.handleMalformedInput(C2SIOService.java:152)
	at tigase.net.IOService.readData(IOService.java:809)
	at tigase.xmpp.XMPPIOService.processSocketData(XMPPIOService.java:486)
	at tigase.net.IOService.call(IOService.java:212)
	at tigase.xmpp.XMPPIOService.call(XMPPIOService.java:155)
	at tigase.xmpp.XMPPIOService.call(XMPPIOService.java:54)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
javax.net.ssl.SSLProtocolException: Input record too big: max = 16709 len = 57065
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
	at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
	at tigase.io.JcaTLSWrapper.unwrap(JcaTLSWrapper.java:279)
	at tigase.io.TLSIO.decodeData(TLSIO.java:349)
	at tigase.io.TLSIO.processHandshake(TLSIO.java:327)
	at tigase.server.xmppclient.C2SIOService.extractTlsHandshakeData(C2SIOService.java:182)
	at tigase.server.xmppclient.C2SIOService.handleMalformedInput(C2SIOService.java:152)
	at tigase.net.IOService.readData(IOService.java:809)
	at tigase.xmpp.XMPPIOService.processSocketData(XMPPIOService.java:486)
	at tigase.net.IOService.call(IOService.java:212)
	at tigase.xmpp.XMPPIOService.call(XMPPIOService.java:155)
	at tigase.xmpp.XMPPIOService.call(XMPPIOService.java:54)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Activities

Andrzej Wójcik (Tigase) changed fields 2 years ago
Name Previous Value Current Value
Assignee

wojtek

wojtek, andrzej.wojcik
Andrzej Wójcik (Tigase) commented 2 years ago

As this issue is more common now (at least a few each day) and I've introduced C2SIOService.handleMalformedInput(C2SIOService.java:152) method, I took a look on this issue.

This method is responsible for handling data sent after <starttls/> by the client, ie. when client sends in a single TCP frame <starttls/> followed by SSL/TLS handshake packet. After a detailed review, I've found out that all issues/exceptions are reported from line 152 of C2SIOService.handleMalformedInput(). This call is done only for handling old SSLv2 Client Hello.

Currently, most of the clients do now use SSLv2 Client Hello (nor they use SSLv2 at all), but some old software still may be using this handshake for compatibility reasons. (This handshake may be used to establish even nerver SSLv3 or even TLSv1.3).

It looks like, when in the "plain" or already TLS encrypted connection happens issue with UTF-8 malformation, Tigase checks if that is not a handshake for TLS (initial packet) by checking TLS/SSL packet header. For SSv2 test was very simple and taking into account possible content that is parsed it could cause false positives and cause SSL handshake to be executed (or rather plaintext data was processed by SSLEngine once again causing issue).

To fix that, I've increased logging of data in extractTlsHandshakeData() method to allow us check what actually was fed to SSL engine and improved verification of SSLv3 handshake header (checked if length of the header does not exceed 16709 bytes).

I think that this fix will be OK. If not, we may want to consider dropping support for SSLv2 Client Hello.
Andrzej Wójcik (Tigase) changed state to 'In Progress' 2 years ago
Previous Value Current Value
Open

In Progress
Andrzej Wójcik (Tigase) changed state to 'In QA' 2 years ago
Previous Value Current Value
In Progress

In QA
Wojciech Kapcia (Tigase) commented 1 year ago

I assume this is completed and the warnings from our installations are just due version with decreased logging not deployed yet and this can be closed?
Andrzej Wójcik (Tigase) commented 1 year ago

Yes, you are correct. Version with this change was not deployed at sure.im/tigase.im.
Wojciech Kapcia (Tigase) batch edited 1 year ago
Name Previous Value Current Value
Iterations
empty
Candidate for next minor release
Wojciech Kapcia (Tigase) changed fields 1 year ago
Name Previous Value Current Value
Version

Candidate for next minor release

tigase-server-8.4.0
Wojciech Kapcia (Tigase) changed state to 'Closed' 1 year ago
Previous Value Current Value
In QA

Closed
Wojciech Kapcia (Tigase) added to iteration "tigase-server-8.4.0" 1 year ago
Wojciech Kapcia (Tigase) removed from iteration "Candidate for next minor release" 1 year ago
Login to comment

Name	Previous Value	Current Value
Assignee	wojtek	wojtek, andrzej.wojcik

Previous Value	Current Value
Open	In Progress

Previous Value	Current Value
In Progress	In QA

Name	Previous Value	Current Value
Iterations	empty	Candidate for next minor release

Name	Previous Value	Current Value
Version	Candidate for next minor release	tigase-server-8.4.0

Previous Value	Current Value
In QA	Closed

Type	Bug
Priority	Normal
Assignee	Andrzej Wójcik (Tigase)
Version	tigase-server-8.4.0

Iterations

tigase-server-8.4.0 Closed

Issue Votes (0)

Watchers (3)

Reference

tigase/_server/server-core#1361