Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [openmq-dev] OpenMQ Clustering connection issue

The WARNING log messages can be ignored as they basically indicates there were takeover(s) happened - messages/transactions owned by one broker were taken over by another broker due to a broker shutdown or one broker unable to communicate with other brokers for a period of time.

-Amy

On 1/13/20 4:35 PM, Will Hartung wrote:


On Sun, Jan 12, 2020 at 5:31 PM Amy Kang <amy.kang@xxxxxxxxxx> wrote:

This does look like a configuration issue.  You'd need to compare the configuration with OpenMQ documentation on HA (enchanced-cluster) configuration  - for exmaples be sure all brokers have unique brokerIDs and all brokers use the same configuration for common cluster properties.  The HA cluster heartbeat does use UDP.

Hope this helps.

--Amy

Thank you, Amy.

So, what I ended up doing was I removed the 3rd broker from the cluster using imqdbmgr:

    $ imqdbmgr remove bkr -n brokerngsharemail3

Once we restarted Node 3, everything connected up. Which is great. I did this because on the same machines, I stood up a parallel cluster on different ports, and they all connected up swimmingly (so it wasn't a network issue, it was something else). I stabbed in the dark and tried removing the broker from the DB, and now the cluster in connecting up.

What we're getting now, however are some warnings:

[#|2019-12-26T15:01:32.768-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B3229]: Transaction acknowledgement [22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872, [consumer:1093089674314572800, type=NONE]]TUID=1093089674321704192 processing failed: [B1261]: Transaction acknowledgement could not be added because message 22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872[[consumer:1093089674314572800, type=NONE]:[consumer:0, type=CLIENT_ACKNOWLEDGE]]TUID=1093089674321704192 not found in destination:
com.sun.messaging.jmq.jmsserver.util.BrokerException: [B1261]: Transaction acknowledgement could not be added because message 22265172-127.0.0.1(dd:31:fb:60:3b:39)-47872-1577401272872[[consumer:1093089674314572800, type=NONE]:[consumer:0, type=CLIENT_ACKNOWLEDGE]]TUID=1093089674321704192 not found in destination
|#]


[#|2019-12-26T15:01:32.792-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B4307]: END_TRANSACTION(58)[XAFlag=0x4000000:TMSUCCESS]: The transaction [XID=6170706C69616E63652C7365727665722C503130302C0056000000962871446170706C69616E63652C7365727665722C50313030] has been marked as rollback-only [state=FAILED from STARTED(1)]: TUID=1093089674321704192 Xid=6170706C69616E63652C7365727665722C503130302C0056000000962871446170706C69616E63652C7365727665722C50313030
|#]

and

[#|2019-12-26T15:01:32.106-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING [B2117]: Message acknowledgement failed from mq://10.40.60.22:7777/?instName=imqbroker&brokerID=brokerngsharemail2&brokerSessionUID=8974670318866050048&ha=true&storeSessionUID=4295148795086427904:
        ackStatus = NOT_FOUND(404)
        Reason = [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted
        AckType = MSG_TXN_PREPARE
        MessageBrokerSession = 8974670318866050048
        MessageStoreSession = 4295148795086427904
        TransactionID = 1093089674321526784
        notfound = 1
                SysMessageID = 22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038
                ConsumerUID = 1093089674314572800

|#]


[#|2019-12-26T15:01:32.126-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING Preparing transaction 1093089674321526784 failed from mq://10.40.60.22:7777/?instName=imqbroker&brokerID=brokerngsharemail2&brokerSessionUID=8974670318866050048&ha=true&storeSessionUID=4295148795086427904: [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted
        [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]
|#]


[#|2019-12-26T15:01:32.136-0800|WARNING|5.1|imq.log.Logger|_ThreadID=98;_ThreadName=Thread-jms[64];|WARNING com.sun.messaging.jmq.jmsserver.util.AckEntryNotFoundException: [22264545-127.0.0.1(87:e5:36:64:61:b9)-34896-1577401271038:[consumer:1093089674314572800, type=NONE]]TID=1093089674321526784 not found, maybe rerouted: TUID=1093089674321526784 Xid=6170706C69616E63652C7365727665722C503130302C0043000000962871446170706C69616E63652C7365727665722C50313030
|#]

Nodes 2 and 3 are barking quite a bit about this, Node 1 just a little bit.

Nodes 1 and 2 were long established, we're just adding 3 in to the cluster. But we haven't reset the cluster, we haven't downed the infrastructure. We basically lit up node 3 and hope that it would seamlessly integrate in with no drama.

Is this some housekeeping thing going on? Nodes 1 and 2 don't bark about this, never have (at least not regularly). Do I need to worry about this?

Thanks

Regards,

Will Hartung


Back to the top