Inline ...
Hello Joakim,
Firstly, this is the setup (Scala) code for the SslContextFactory
etc.:
/***** start code *****/
val sslContextFactory = new SslContextFactory
val ciphers = Array(
"TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
"TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 -
IE7/8/9 - Android <= 4.3 */
"TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
"TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
"TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR)
sslContextFactory.setUseCipherSuitesOrder(true)
Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)
sslContextFactory.setIncludeCipherSuites(ciphers:_*)
Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.
sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
"SSL_DHE_RSA_WITH_DES_CBC_SHA",
"SSL_DHE_DSS_WITH_DES_CBC_SHA",
"SSL_RSA_EXPORT_WITH_RC4_40_MD5",
"SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
"SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
"SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")
Use defaults, none of those Cipher suites would be selected anyway due to default Cipher Suite exclusions in SslContextFactory AND the JVM disabled cipher suites.
/** Default Excluded Cipher Suite List */
private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
// Exclude weak / insecure ciphers
"^.*_(MD5|SHA|SHA1)$",
// Exclude ciphers that don't support forward secrecy
"^TLS_RSA_.*$",
// The following exclusions are present to cleanup known bad cipher
// suites that may be accidentally included via include patterns.
// The default enabled cipher list in Java will not include these
// (but they are available in the supported list).
"^SSL_.*$",
"^.*_NULL_.*$",
"^.*_anon_.*$"
};
sslContextFactory.addExcludeProtocols("SSLv3")
This is the in the default excluded protocols values, and also excluded/disabled at the JVM level, don't set it.
/** Default Excluded Protocols List */
private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL", "SSLv2", "SSLv2Hello", "SSLv3"};
sslContextFactory.setRenegotiationAllowed(false)
sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")
This is only relevant for a client initiating a connection to a destination machine with TLS.
Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols are disabled in so many places (User-Agents, Browsers, OS level, Java VM level, Server level, various hardware intermediates) now its pointless to attempt to support them. If this is still important to you, make a different connector for legacy behaviors.
sslContextFactory.setKeyStorePath(keyStore)
sslContextFactory.setKeyStorePassword(keyPassword)
sslContextFactory.setKeyManagerPassword(keyPassword)
sslContextFactory.setKeyStoreType(keyStoreType)
val httpsConfig = new HttpConfiguration
httpsConfig.setSecureScheme("https")
httpsConfig.setSecurePort(port)
httpsConfig.setSendXPoweredBy(false)
httpsConfig.setSendServerVersion(false)
httpsConfig.setOutputBufferSize(32768)
httpsConfig.addCustomizer(new SecureRequestCustomizer)
val http1 = new HttpConnectionFactory(httpsConfig)
val connector =
{
if (httpVersion != "http/2")
{
val ssl = new
SslConnectionFactory(sslContextFactory,"http/1.1")
val http = new HttpConnectionFactory(httpsConfig)
new ServerConnector(server,ssl,http)
}
else
{
val http2 = new
HTTP2ServerConnectionFactory(httpsConfig)
val alpn = new ALPNServerConnectionFactory
alpn.setDefaultProtocol(http1.getProtocol)
Use alpn.setDefaultProtocol(http1.getDefaultProtocol())
val ssl = new
SslConnectionFactory(sslContextFactory,alpn.getProtocol)
new ServerConnector(server,ssl,alpn,http2,http1)
}
}
if (address != null) connector.setHost(address)
connector.setPort(port)
connector.setIdleTimeout(1000L * idleTimeout)
server.addConnector(connector)
/***** end code *****/
Then for the ThreadPool part:
/***** start code *****/
val pool = new
QueuedThreadPool(maxThreads,minThreads,idleTimeout,new
ArrayBlockingQueue(queueSize))
val server = new Server(pool)
/***** end code *****/
HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better then a raw Executor with regards to HTTP/2.
Threads are 10-100, idletimeouts I have tried are from 1000 to
20000, queueSize is now 500 but I have tried both much smaller and
larger values.
I have no idea what values would be sensible. The number of
concurrent users is relatively low: up to a couple of hundred,
rarely more. Users can do requests that take longish to complete
(sometimes up to 30 or even 60 seconds) but those are very rare. The
majority is relatively quick (< 50ms). Current load on the
problematic servers is very low (perhaps 10 users) and as said: low
memory and CPU close to zero.
No problems with HTTP/1.1 with this code/settings, problematic with
HTTP/2. Strangely enough some servers seem fine (or users just did
not report any problems yet).
Start by using a default QueuedThreadPool, no configuration, no min thread, no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.
Any help would be welcome.
Kind regards,
Silvio
Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2
When it comes to the ALPN suppport, you have to worry about what JVM runtime you are on first.
If you are using Java 8 then you have to align your alpn-boot version to the specific Java 8 JVM you are using.
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn work properly.
However, if you are using Java 11 then you can use the jetty-alpn-java-server artifact instead (no bootclasspath requirement) to get ALPN behaviors required for HTTP/2.
You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which will work the same regardless of Java 8 or Java 11.
When it comes to too restrictive of an SslContextFactory we have the following advice:
* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like 2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working, then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know what the current state of your SslContextFactory is (including all selected cipher suites and protocols)
We have an example codebase that shows how to start an HTTP/2 server connector.
You can check the Jenkins project for their usage of HTTP/2 on Embedded Jetty as well. (their self executing WAR file does just this)
You could also run h2spec against your server and see if its behaving properly.
- Joakim
On 20-11-18 16:06, Joakim Erdfelt
wrote:
How do you enable HTTP/2 in your embedded
application?
Also, If you have configured your SslContextFactory, what
does this configuration look like?
Lastly, what does your ThreadPool / Executor configuration
look like?
Hello all,
We are running Jetty 9.4.14 embedded in an application that
runs on a
number of Ubuntu 18.10 servers. Last night we switched from
HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users
can not
access the application or are experiencing extremely long
latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default
setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing
this for
years now and each time we switched from HTTP1.1 to HTTP2
things started
falling apart. In the past we had reports of people not being
able to
connect at all receiving spurious messages from their browsers
(some of
which are undoubtedly outdated). We expected things to have
improved by
now and decided to try again, with the above result.
Are there additional things we need to take care of when using
HTTP2
when compared to HTTP1.1? More threads, different queue size,
more file
handles etc?
Any pointers would be very much appreciated.
Kind regards,
Silvio
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
