[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jetty-dev] 8.1.2.v20120308 - rare mixup of requests
|
Hi Per,
surely we do care about bugs in jetty. :) My schedule today is pretty
tight, I will send you a longer answer. But if you could try switching
the connector to SelectChannelConnector in the meanwhile would be good.
I wonder why solr is using the blocking one by default.
There should be a config file for jetty where the connectors are being
setup (usually jetty.xml in our standard distribution).
In there there's a section setting up the connectors, simply switch the
class in yours to:
"org.eclipse.jetty.server.nio.SelectChannelConnector". Then it should
look similar to this one.
36 <Call name="addConnector">
37 <Arg>
38 <New
class="org.eclipse.jetty.server.nio.SelectChannelConnector">
39 <Set name="host"><Property name="jetty.host" /></Set>
40 <Set name="port"><Property name="jetty.port"
default="8080"/></Set>
41 <Set name="maxIdleTime">300000</Set>
42 <Set name="Acceptors">2</Set>
43 <Set name="statsOn">false</Set>
44 <Set name="confidentialPort">8443</Set>
45 <Set name="lowResourcesConnections">20000</Set>
46 <Set name="lowResourcesMaxIdleTime">5000</Set>
47 </New>
48 </Arg>
49 </Call>
Cheers,
Thomas
On 10/24/12 10:19 AM, Per Steffensen wrote:
Thomas Becker skrev:
Hi Per,
Hi
First of all, thanks for your quick response! Nice that someone really
seem to care and want to actively participate in doing something about
it. Have to say that this is not the case on all OSS projects.
this is not a known bug that has been fixed, but looks like a severe
issue to investigate.
Agree!
It doesn't look like a jvm issue as according to your description
stuff doesn't seem to get mixed up on the wire.
That was also my conclusion, but Im not a Jetty expert, and havnt
really dived very deep into it yet, so I wasnt sure. Just quickly
wanted to mention my observations and interpretation in its current
state, so that I could get reactions from Jetty community. Didnt want
do go very deep into it, if you already knew all about the problem and
had a fix.
Is there a specific reason that you use a blocking connector?
Wasnt really aware that I had taken such a decision :-) Im just using
Solr "out of the box". It ships with Jetty including a default/easy
way of starting Solr running on Jetty. I basically just use that, so
havnt done anything to configure Jetty og make any other specific
decissions on how it should run.
Could you please try a SelectChannelConnector and see if the problem
still occurs?
Yes certainly! But, in order to do it quicly, I probably would like
you to tell me what I should do in order to acheive this. As I
mentioned, I havnet really done any configuring or anything to Jetty
before. I basically just looked a little at the code and decided to
modify StreamEndPoint to do a little logging for me in order to see
what it receives from the wire. This way I saw that "received from
wire" seems to be fine, but "handed over to Filter/Servlet" seems to
be mixed up with content from other requests.
That information would give us some hints about where to look first.
Can you try to create a small webapp (maybe + simple loadtest) that
reproduces the issue?
I can say that I definitely want to participate actively in narrowing
in on the "bug" to have it fixed. Creating a small webapp that
reproduces the problem "on your side" will certainly be a nice way
forward, but if it is the best Im a little bit in doubt. We have
several "unit-test"-sized tests that do a fair amount on concurrent
requesting and we never came across the problem it such setups. So
maybe this problem requires "bigger" setups or something - Im not
really sure about the "properties" that need to hold in order for this
to occur. Maybe another way forward is that you provide we with
additional logging (or something) that I could insert into the code,
run the "performance/endurance" test that (sometimes) produces this
problem again, and provide the logging to you to evaluate in
corrporation with me. Is that a path that could fit you? If we get
closer to the cause of the problem following this path, we might be
able to reproduce in a smaller ad-hoc test for quicker "turnaround" on
potential fixes. Right now I just very much in doubt if I will be able
to reproduce this problem in a fairly small "test" to hand over to you.
Cheers,
Thomas
Regards, Per Steffensen
On 10/23/12 5:55 PM, Per Steffensen wrote:
Hi
I have found a bug in Jetty 8.1.2.v20120308.
Running with many concurrent clients sending requests to Solr
(web-app running on top of Jetty) sometimes (very rarely) the
content of the HTTP requests are mixed up when handed over by Jetty
as HttpServletRequest-objects to the doFilter method of a
javax.servlet.Filter configured in web.xml. When "hacking"
org.eclipse.jetty.io.bio.StreamEndPoint to log something about what
is actually received over the wire, requests does not seem to be
mixed up "yet". So it is something that goes wrong in Jetty
somewhere between receiving the data over sockets and handing the
(parsed) request over to a filter (or servlet, probably) as a
HttpServletRequest.
A little about what I mean with "mixed up":
* Data in request A: abcdefghijklmnopqrstuvwxyz
* Data in request B: 1234567890987654321
Well the requests are longer in my concrete cases, and thats
probably important, but not when just trying to illustrate
Data read by consecutive "fill"-calls in
org.eclipse.jetty.io.bio.StreamEndPoint - e.g:
Regarding request A:
* abcdefghij
* klmnopqrstu
* vwxyz
Regarding request B:
* 12345678909876
* 54321
Seems ok
Data "behind" HttpServletRequest-object handed over to filter/servlet:
Request A: abcdefghijklmnopqrstuvw345678909876
WUT!!!!
Closer investigations show that it is "always" (well I do not have
much statistical material) the last part of the data read in the
last "fill"-call by the "first request" (request A in this example)
that is "magically" replaced by something from a "later request"
(request B in this example) when the data reaches the
filter/servlet. And that the size of the correct part of the data
read in the last "fill"-call by request A is equal to the size of
data read in the "violating" "fill"-call (request B) which is before
the data the gets mixed from request B into request A. Never mind if
you didnt understand (I almost did not :-) ) - let me illustrate by
example. E.g. lets say (as the example above also suggests) that it
is the last "fill"-call regarding request A that gets mixed up with
data from the first "fill"-call regarding request B. The "end
result" seen in request A can be:
* abcdefghijklmnopqrstuv2345678909876
* abcdefghijklmnopqrstuvw345678909876
* abcdefghijklmnopqrstuvwx45678909876
* abcdefghijklmnopqrstuvwxy5678909876
But not e.g.
* abcdefghijklmnopqrstuvw5678909876
To me it really seems like org.eclipse.jetty.io.bio.StreamEndPoint
reuse (/share) byte-arrays among requests, and that a byte-array
used by request A will be "released" for usage by request B before
request A has finished reading from it - maybe
org.eclipse.jetty.io.bio.StreamEndPoint is finished using the
byte-array for request A stuff, but layers higher in Jetty might not
have ready from the byte-array yet to get the data for the
request-A-HttpServletRequest-object. But what do I know?
I can deepen the explanation and I can provide more "evidence" or
details about exactly how I revealed this bug. But first of all I
just want to know:
* Is this a known bug?
* Has it been fixed?
Regards, Per Steffensen
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev