Re: [jetty-users] SolrJ/Solr: HTTP protocol violation: Authentication ch

[Shawn Heisey <eclipse@xxxxxxxxxxxx>]

On 5/31/23 17:48, Bill Ross wrote:
> Can you swap in another httpclient to test? I assume swapping jetty 
> server would be too much, given something works. :-)

I can't do anything about the Jetty server without upgrading Solr.  I 
really want to get them upgraded, but it's not up to me.

I tried to use the legacy SolrJ clients that utilize Apache HttpClient 
4.x, but for an unknown reason I was not able to get those clients to 
work.  I am not using any http client directly, I use SolrJ.  Layers 
upon layers.  I am completely shielded from any direct interaction with 
the Jetty client by SolrJ.

>  From faster result on smaller batch size: are you monitoring memory 
> use? I'd try even smaller, looking at the perf profile for clues.

The queries gathering documents from the source are faster with a filter 
in place, so they feed data to the queue faster.  I think this is 
probably because it is only sorting a few million documents for each 
document batch instead of the full 30 million.

I was running my program with a 1GB heap.  With a queue size of 100000 
or 150000, that worked well.

I later bumped the queue size to 200000 and had to bump the heap because 
I got OOME.  The space is consumed by the SolrInputDocument objects on 
the queue.  I set the heap to 2GB for the 200K queue size.  Now the max 
queue size is 500K and the heap is 5GB.  A larger queue evens out the 
transfer of data from the query thread to the indexing threads and keeps 
the migration from stalling.

I'm using ZGC to optimize for latency.  The code is compiled for Java 11.

Thanks,
Shawn