8369950: TLS connection to IPv6 address fails with BCJSSE due to IllegalArgumentException

[sercher]

Hi all,

Let me propose a fix and a test case for JDK-8369950.

The failure reproduces with BCJSSE provider and all implementations of SSLSocker other than SSLSocketImpl.

In the test case an anonymous wrapper is used, over the standard SSLSocketImpl, to simulate an external JSSE provider. The test case shows the same behavior as in BCJSSE case (failure due to non-LDH ASCII characters in the SNI host name).

The fix avoids constructing SNIHostName when the URL host name is an IPv4 or IPv6 literal address. Other than that, all other FQDN host names that have invalid characters (non-LDH ASCII characters) still produce that exception.

SNIHostName, as defined in

jdk/src/java.base/share/classes/javax/net/ssl/SNIHostName.java

Lines 44 to 66 in 873f8a6

	* As described in section 3, "Server Name Indication", of
	* <A HREF="http://www.ietf.org/rfc/rfc6066.txt">TLS Extensions (RFC 6066)</A>,
	* "HostName" contains the fully qualified DNS hostname of the server, as
	* understood by the client. The encoded server name value of a hostname is
	* represented as a byte string using ASCII encoding without a trailing dot.
	* This allows the support of Internationalized Domain Names (IDN) through
	* the use of A-labels (the ASCII-Compatible Encoding (ACE) form of a valid
	* string of Internationalized Domain Names for Applications (IDNA)) defined
	* in <A HREF="http://www.ietf.org/rfc/rfc5890.txt">RFC 5890</A>.
	* <P>
	* Note that {@code SNIHostName} objects are immutable.
	*
	* @spec https://www.rfc-editor.org/info/rfc5890
	* RFC 5890: Internationalized Domain Names for Applications (IDNA):
	* Definitions and Document Framework
	* @spec https://www.rfc-editor.org/info/rfc6066
	* RFC 6066: Transport Layer Security (TLS) Extensions: Extension Definitions
	* @see SNIServerName
	* @see StandardConstants#SNI_HOST_NAME
	*
	* @since 1.8
	*/
	public final class SNIHostName extends SNIServerName {

has the fully qualified DNS hostname of the server. As follows from the section 3, "Server Name Indication", RFC 6066, Literal IPv4 and IPv6 addresses are not permitted in "HostName".

This mirrors the behavior of SSLSocketImpl, that avoids constructing the SNIHostName from literal addresses. Please see

jdk/src/java.base/share/classes/sun/security/ssl/Utilities.java

Lines 110 to 116 in 873f8a6

	if (hostname != null && hostname.indexOf('.') > 0 &&
	!hostname.endsWith(".") &&
	!IPAddressUtil.isIPv4LiteralAddress(hostname) &&
	!IPAddressUtil.isIPv6LiteralAddress(hostname)) {
	try {
	return new SNIHostName(hostname);

Testing:

• standard jtreg tests goups (tier1-4) no regressions
• the new test passes with the fix and fails otherwise
• passes also with BCJSSE in FIPS and standard mode

BCJSSE standard

STDOUT:
STDERR:
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSecurityProperty
INFORMATION: Found boolean security property [keystore.type.compat]: true
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFORMATION: Found string security property [jdk.tls.disabledAlgorithms]: SSLv3, TLSv1, TLSv1.1, DTLSv1.0, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, ECDH, TLS_RSA_*, rsa_pkcs1_sha1 usage HandshakeSignature, ecdsa_sha1 usage HandshakeSignature, dsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': rsa_pkcs1_sha1 usage HandshakeSignature
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': ecdsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': dsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFORMATION: Found string security property [jdk.certpath.disabledAlgorithms]: MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224, SHA1 usage SignedJAR & denyAfter 2019-01-01
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 jdkCA & usage TLSServer
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 usage SignedJAR & denyAfter 2019-01-01
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
INFORMATION: Found string system property [java.home]: /Users/sercher/repos/jdk/build/macosx-x86_64-server-release/images/jdk
Dez. 01, 2025 2:44:02 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyHandshakeBeginning
INFORMATION: [server #1 @193b6d73] accepting connection from 0:0:0:0:0:0:0:1:56197
Dez. 01, 2025 2:44:03 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyHandshakeComplete
INFORMATION: [server #1 @193b6d73] established connection with 0:0:0:0:0:0:0:1:56197
Dez. 01, 2025 2:44:08 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyConnectionClosed
INFORMATION: [server #1 @193b6d73] disconnected from 0:0:0:0:0:0:0:1:56197
STATUS:Passed.

BCJSSE FIPS

STDOUT:
STDERR:
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSecurityProperty
INFORMATION: Found boolean security property [keystore.type.compat]: true
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFORMATION: Found string security property [jdk.tls.disabledAlgorithms]: SSLv3, TLSv1, TLSv1.1, DTLSv1.0, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, ECDH, TLS_RSA_*, rsa_pkcs1_sha1 usage HandshakeSignature, ecdsa_sha1 usage HandshakeSignature, dsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': rsa_pkcs1_sha1 usage HandshakeSignature
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': ecdsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.tls.disabledAlgorithms': dsa_sha1 usage HandshakeSignature
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFORMATION: Found string security property [jdk.certpath.disabledAlgorithms]: MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224, SHA1 usage SignedJAR & denyAfter 2019-01-01
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 jdkCA & usage TLSServer
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNUNG: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 usage SignedJAR & denyAfter 2019-01-01
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
INFORMATION: Found string system property [java.home]: /Users/sercher/repos/jdk/build/macosx-x86_64-server-release/images/jdk
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyHandshakeBeginning
INFORMATION: [server #1 @4d1e9767] accepting connection from 0:0:0:0:0:0:0:1:56184
Dez. 01, 2025 2:41:32 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyHandshakeComplete
INFORMATION: [server #1 @4d1e9767] established connection with 0:0:0:0:0:0:0:1:56184
Dez. 01, 2025 2:41:37 PM org.bouncycastle.jsse.provider.ProvTlsServer notifyConnectionClosed
INFORMATION: [server #1 @4d1e9767] disconnected from 0:0:0:0:0:0:0:1:56184
STATUS:Passed.