We're running 3 legs of MQ in a cluster backed by MySQL. Our primary clients are Payara servers running MDBs.
I don't have the exact version, it's a fairly recent 5.x. Running on recent JDK 8.
It's under heavy load, and has not been without drama. Each instance has 6G of RAM dedicated to it.
We've recently dropped the traffic by 75%, but even still we have issues keeping up with draining the queues. It seems certain that it's cheaper to add messages to the system than to take them out. Trying both extra clients and batching client transactions (vs committing each message). I didn't see much of any benefit over batching the transactions (that is, read several, and "commit up to here" type of work).
But our current curiosity is lingering transactions.
Currently, one of our legs is showing 55000 COMMITTED transactions lingering there.
This is an improvement to the other day, where we had 1.7M COMMITTED (not COMPLETED) transactions.
Only one leg shows these, the rest show 0.
We've resorted to clearing these out via direct SQL:
delete from MQTXN41Cmqcluster2
where type = '2' and state = '6'
And then restarting the leg, but that's a bailing wire solution. We couldn't restart the leg with the 1.7M (we can, it just takes hours, which isn't really acceptable for us).
So, we dug around and came up with that SQL. But we don't know what the consequences of that is.
We're curious what these are, why they're not being reaped, any consequences of our actions. Why are they only manifesting on one leg of the cluster?
Thanks.
Regards,
Will Hartung
