This does look like a configuration issue. You'd need to compare
the configuration with OpenMQ documentation on HA
(enchanced-cluster) configuration - for exmaples be sure all
brokers have unique brokerIDs and all brokers use the same
configuration for common cluster properties. The HA cluster
heartbeat does use UDP.
Hope this helps.
--Amy
Thank you, Amy.
So, what I ended up doing was I removed the 3rd broker from the cluster using imqdbmgr:
$ imqdbmgr remove bkr -n brokerngsharemail3
Once we restarted Node 3, everything connected up. Which is great. I did this because on the same machines, I stood up a parallel cluster on different ports, and they all connected up swimmingly (so it wasn't a network issue, it was something else). I stabbed in the dark and tried removing the broker from the DB, and now the cluster in connecting up.
What we're getting now, however are some warnings:
[#|2019-12-26T15:01:32.768-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B3229]: Transaction acknowledgement [22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872, [consumer:1093089674314572800, type=NONE]]TUID=1093089674321704192 processing failed: [B1261]: Transaction acknowledgement could not be added because message 22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872[[consumer:1093089674314572800, type=NONE]:[consumer:0, type=CLIENT_ACKNOWLEDGE]]TUID=1093089674321704192 not found in destination:
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B1261]: Transaction acknowledgement could not be added because message 22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872[[consumer:1093089674314572800, type=NONE]:[consumer:0, type=CLIENT_ACKNOWLEDGE]]TUID=1093089674321704192 not found in destination
|#]
[#|2019-12-26T15:01:32.792-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B4307]: END_TRANSACTION(58)[XAFlag=0x4000000:TMSUCCESS]: The transaction [XID=6170706C69616E63652C7365727665722C503130302C0056000000962871446170706C69616E63652C7365727665722C50313030] has been marked as rollback-only [state=FAILED from STARTED(1)]: TUID=1093089674321704192 Xid=6170706C69616E63652C7365727665722C503130302C0056000000962871446170706C69616E63652C7365727665722C50313030
|#]
and
[#|2019-12-26T15:01:32.106-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B2117]: Message acknowledgement failed from mq://
10.40.60.22:7777/?instName=imqbroker&brokerID=brokerngsharemail2&brokerSessionUID=8974670318866050048&ha=true&storeSessionUID=4295148795086427904:
ackStatus = NOT_FOUND(404)
Reason = [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted
AckType = MSG_TXN_PREPARE
MessageBrokerSession = 8974670318866050048
MessageStoreSession = 4295148795086427904
TransactionID = 1093089674321526784
notfound = 1
SysMessageID = 22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038
ConsumerUID = 1093089674314572800
|#]
[#|2019-12-26T15:01:32.126-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING Preparing transaction 1093089674321526784 failed from mq://
10.40.60.22:7777/?instName=imqbroker&brokerID=brokerngsharemail2&brokerSessionUID=8974670318866050048&ha=true&storeSessionUID=4295148795086427904: [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted
[22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]
|#]
[#|2019-12-26T15:01:32.136-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING com.sun.messaging.jmq.jmsserver.util.AckEntryNotFoundException: [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted: TUID=1093089674321526784 Xid=6170706C69616E63652C7365727665722C503130302C0043000000962871446170706C69616E63652C7365727665722C50313030
|#]
Nodes 2 and 3 are barking quite a bit about this, Node 1 just a little bit.
Nodes 1 and 2 were long established, we're just adding 3 in to the cluster. But we haven't reset the cluster, we haven't downed the infrastructure. We basically lit up node 3 and hope that it would seamlessly integrate in with no drama.
Is this some housekeeping thing going on? Nodes 1 and 2 don't bark about this, never have (at least not regularly). Do I need to worry about this?
Thanks
Regards,
Will Hartung
