[jetty-users] Jetty Client 9 for Iudex crawler

[David Kellum <dek94@xxxxxxxxxxxxx>]

I'm working on upgrading use of Jetty Client in the Iūdex web crawler to 9.0.0.M3.  Firstly, despite the work of absorbing a rewrite, client 9.x looks to simplify several aspects of my integration.  I like the callback pure-interfaces and move to use nio ByteBuffers (which matches my internals.)  Thanks very much for the open source!

I have couple of questions below, mostly related to the changes of client 9.x vs client 7.x  My current Client code is here:

https://github.com/dekellum/iudex/blob/jetty-9/iudex-jetty-httpclient/src/main/java/iudex/jettyhttpclient/Client.java

Timeouts

Client 7.x had settings for timeout, soTimeout, connectTimeout and idleTimeout.  Client 9.x only has idleTimeout and connectTimeout. My timeout related integration tests do appear to work: is idleTimeout essentially used as an soTimeout and (catch all) timeout in client 9.x?

Retry

Client 7.x had a setting for maxRetries.  Is retry as a feature no longer supported in Client 9.x? Any plans to add it?

Cookies

I think I have may have a use case for reuse of the new Cookie support, however, this being a crawler I need more control over it.  Is there a way to control what cookies are sent on a per-request basis while still using Jetty's pooled connections?  In other words I would like to introspect cookies from a response, possibly filtering, and then apply these to a subsequent request to the same registration-level domain but possibly via a different connection.   I believe this is not completely unlike how browsers behave.

Short of this, is there a way to disable cookie storage and sending entirely, with pooled connections?

Thanks in advance,
David