Re: [jetty-users] SSL Assertion

[Josh Spiegel <joshlakewg@xxxxxxxxx>]

Simone,

We *think* we may have found the issue.  It is hard to say with certainty since the symptom was somewhat intermittent.

However, in our own code, we were pulling and releasing buffers from the Jetty's pool.  Sometimes, we would switch the buffer to little-endian:

   buffer.order(ByteOrder.LITTLE_ENDIAN);

Before, when we released this buffer, we weren't switching it back to big endian.  Within Jetty, it calls BufferUtil.clear()
https://github.com/eclipse/jetty.project/blob/4c98990cd93e2f36032121d2ce76598b1c85ad1d/jetty-util/src/main/java/org/eclipse/jetty/util/BufferUtil.java#L175

Which resets most of the buffer state except for the bigEndian/nativeByteOrder members.  My theory is that SslConnection eventually got one of these little_endian buffers and got confused.  

We modified our own code to reset the byte order before releasing the buffer.  Since doing this, the issue hasn't reproduced. 

I think one could argue that Jetty should reset the byte order when the buffer is released.  However, not sure if you want an issue for this or not (let me know).

Thanks,

Josh

On Tue, Dec 7, 2021 at 8:44 AM Josh Spiegel <joshlakewg@xxxxxxxxx> wrote:

Thanks Simone,

Let me see if I can get an environment setup where I am able to catch this in the debugger.  After that, I will file a ticket and give you whatever information you need.

Thanks,

Josh

On Tue, Dec 7, 2021 at 4:10 AM Simone Bordet <simone.bordet@xxxxxxxxx> wrote:

Hi,

On Mon, Dec 6, 2021 at 11:18 PM Josh Spiegel <joshlakewg@xxxxxxxxx> wrote:
>
> Hi,
>
> Intermittently during stress testing, we are hitting this assertion (IllegalStateException):
>
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-io/src/main/java/org/eclipse/jetty/io/ssl/SslConnection.java#L1152
>
> I modified this line of code to include additional information:
>
>   case BUFFER_OVERFLOW: {
>
>      if (!flushed)
>
>        return result = false;
>
>      // It's possible that SSLSession.packetBufferSize has been expanded
>
>      int newSize = getPacketBufferSize();
>
>      if (packetBufferSize < newSize)
>
>      {
>
>        releaseEncryptedOutputBuffer();
>
>        continue;
>
>      }
>
>      throw new IllegalStateException(" Unexpected wrap result " + wrap + " " + packetBufferSize + " " + newSize + " " + _encryptedOutput.toString());
>
>
> The augmented exception message looks like this:
>
>  SEVERE: java.lang.IllegalStateException: Unexpected wrap result BUFFER_OVERFLOW 16709 16709 java.nio.HeapByteBuffer[pos=17408 lim=17408 cap=17408]
>         at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1154)
>         at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
>         at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
>         at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
>
> Has anyone seen this before?  Is the idea that this is a bug in the underlying library as packetBufferSize should have increased after the BUFFER_OVERFLOW event?

Can you please open an issue with the information in this email, and
continue the discussion there?
https://github.com/eclipse/jetty.project/issues

This is not expected, but we need a lot more information about OS, TLS
provider, etc.
Let's continue on the issue.

Thanks!
--
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz