| Re: [mosquitto-dev] mosquitto-dev Digest, Vol 20, Issue 7 |
Send mosquitto-dev mailing list submissions to
mosquitto-dev@xxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
or, via email, send a message with subject or body 'help' to
mosquitto-dev-request@xxxxxxxxxxx
You can reach the person managing the list at
mosquitto-dev-owner@xxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of mosquitto-dev digest..."
Today's Topics:
1. bridges for dynamically assigned hosts. (Duncan Law)
2. Re: bridges for dynamically assigned hosts. (Roger Light)
----------------------------------------------------------------------
Message: 1
Date: Wed, 19 Aug 2015 00:45:28 +0100
From: Duncan Law <mrdunk@xxxxxxxxx>
To: mosquitto-dev@xxxxxxxxxxx
Subject: [mosquitto-dev] bridges for dynamically assigned hosts.
Message-ID:
<CABtVNKX4N6VqM-RHK9ieYLFPR6JPNXvLDQZ-31x9NsPbM2QZ4w@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"
TLDR: my thoughts about dynamic clustering of brokers.
hi, first post!
i'm enjoying playing with Mosquitto but
i'm looking for a little advice on the bast way to achieve the following:
my current hobby project has many small embedded nodes acting as MQTT
clients.
a subset of these nodes are also brokers running on OpenWRT.
brokers advertise them selves via mDNS.
clients pick a broker at random from those currently online.
the brokers will all bridge to each other so clients connected to different
brokers can communicate.
so far, so good.
where things become less obvious:
once this is all working i don't want to have to configure anything.
i would like the broker nodes to dynamically build the bridges with their
neighbors.
currently under Mosquitto it's not possible to change the broker
information of a running instance so
when new broker nodes come online i have no way to update the existing
brokers bridge records.
possible solution 1:
re-generate mosquitto.conf periodically with the updated broker list and
restart the mosquitto binary.
this is less drastic than it sounds because all clients already re-try
other brokers in the event a broker drops off the network.
but it's an ugly solution. we can do better.
possible solution 2:
a binary running on *all* the broker nodes that acts as the bridge
functionality.
something like this:
- builds a list of all brokers on the network from mDNS information.
- subscribes to "#/" on the broker and forwards all topics to all other
brokers (having modified the topic with a "/broker/" identifier).
- re-writes any incoming topics with the "/broker/" identifier in the
topic (these will have been sent by other bridge programs) removing
the "/broker/" identifier. it will need to keep a temporary record of these
topics so it knows not to re-send when they are re-published.
this should work well but re-implementing the bridge functionality seems
pointless when the Mosquitto broker already has most of what we need...
also, it's not possible to ask Mosquitto what topics are currently
subscribed on a broker so i'll have to send *everything* *everywhere*
unless we explicitly tell it which topics... (the current bridge
implementation doesn't do any better though.)
possible solution 3:
modify the Mosquito codebase to allow dynamic updates of Bridges.
can any one tell me if this is on the roadmap for Mosquitto?
it would be useful for anyone trying to dynamically scale a cluster of
Mosquitto servers without re-starting jobs.
presuming nobody has done any work on this already,
what would people's thoughts on an interface be?
re-writing mosquitto.conf and performing a SIGHUP would be closest to the
current model...
can anyone with knowledge of the codebase suggest whether there is anything
fundamental to the current bridging code that means it can't be modified
while Mosquitto is running?
obviously we'll need to disable a bridge whle working on it. probably
easiest to remove it completely, modify, then bring it up again from
scratch...
other tangents:
has anybody considered the possibility of modifying running config using an
interface similar to "$SYS/#" ?
security would need to be handled carefully... but mosquitto.conf shows we
can limit certain topics to certain users.
no reason something like this would have to be part of the main binary; a
separate binary could subscribe to the control topic and build a new
configuration file according to the results.
anyway, thoughts, suggestions, etc welcome.
dunk.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/mosquitto-dev/attachments/20150819/64e31847/attachment.html>
------------------------------
Message: 2
Date: Wed, 19 Aug 2015 13:04:16 +0100
From: Roger Light <roger@xxxxxxxxxx>
To: General development discussions for the mosquitto project
<mosquitto-dev@xxxxxxxxxxx>
Subject: Re: [mosquitto-dev] bridges for dynamically assigned hosts.
Message-ID:
<CAH7zdyeoBwCVaByB1656Sx==t01VZ-7H_g=3Cxgbxg+gSxgi6g@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset=UTF-8
Hi Duncan,
Your solutions 1 and 2 would work, but as you say they aren't ideal.
Solution 3 would be great. Reloading the bridge config on a SIGHUP
would be the most straightforward approach for sure. The reason this
isn't already done is the panic attacks I had trying to consider all
of the insane things users might try and do, and having to cope with
them :) In principle it's not that bad though. If you'd like to take a
look at it, I'd be very happy to receive a patch. I would do something
like load the bridge config into an array, then compare to the
existing bridges, disconnecting or creating bridges as appropriate.
Any bridges that already exist (based on the connection name) would
ideally be kept connected and just the topics/other changed - but
you'd need to see what parameters had changed to determine whether or
not that was possible.
Runtime control like $SYS - yes, I've considered it. Again, I'd be
happy to have that functionality but defining the interface requires
some thought.
Cheers,
Roger
On Wed, Aug 19, 2015 at 12:45 AM, Duncan Law <mrdunk@xxxxxxxxx> wrote:
> TLDR: my thoughts about dynamic clustering of brokers.
>
> hi, first post!
> i'm enjoying playing with Mosquitto but
> i'm looking for a little advice on the bast way to achieve the following:
>
> my current hobby project has many small embedded nodes acting as MQTT
> clients.
> a subset of these nodes are also brokers running on OpenWRT.
> brokers advertise them selves via mDNS.
> clients pick a broker at random from those currently online.
> the brokers will all bridge to each other so clients connected to different
> brokers can communicate.
>
> so far, so good.
>
> where things become less obvious:
> once this is all working i don't want to have to configure anything.
> i would like the broker nodes to dynamically build the bridges with their
> neighbors.
>
> currently under Mosquitto it's not possible to change the broker information
> of a running instance so
> when new broker nodes come online i have no way to update the existing
> brokers bridge records.
>
>
> possible solution 1:
> re-generate mosquitto.conf periodically with the updated broker list and
> restart the mosquitto binary.
> this is less drastic than it sounds because all clients already re-try other
> brokers in the event a broker drops off the network.
> but it's an ugly solution. we can do better.
>
>
>
> possible solution 2:
> a binary running on *all* the broker nodes that acts as the bridge
> functionality.
> something like this:
> - builds a list of all brokers on the network from mDNS information.
> - subscribes to "#/" on the broker and forwards all topics to all other
> brokers (having modified the topic with a "/broker/" identifier).
> - re-writes any incoming topics with the "/broker/" identifier in the topic
> (these will have been sent by other bridge programs) removing the "/broker/"
> identifier. it will need to keep a temporary record of these topics so it
> knows not to re-send when they are re-published.
>
> this should work well but re-implementing the bridge functionality seems
> pointless when the Mosquitto broker already has most of what we need...
> also, it's not possible to ask Mosquitto what topics are currently
> subscribed on a broker so i'll have to send *everything* *everywhere* unless
> we explicitly tell it which topics... (the current bridge implementation
> doesn't do any better though.)
>
>
>
> possible solution 3:
> modify the Mosquito codebase to allow dynamic updates of Bridges.
> can any one tell me if this is on the roadmap for Mosquitto?
> it would be useful for anyone trying to dynamically scale a cluster of
> Mosquitto servers without re-starting jobs.
>
> presuming nobody has done any work on this already,
> what would people's thoughts on an interface be?
> re-writing mosquitto.conf and performing a SIGHUP would be closest to the
> current model...
>
> can anyone with knowledge of the codebase suggest whether there is anything
> fundamental to the current bridging code that means it can't be modified
> while Mosquitto is running?
> obviously we'll need to disable a bridge whle working on it. probably
> easiest to remove it completely, modify, then bring it up again from
> scratch...
>
>
>
> other tangents:
> has anybody considered the possibility of modifying running config using an
> interface similar to "$SYS/#" ?
> security would need to be handled carefully... but mosquitto.conf shows we
> can limit certain topics to certain users.
> no reason something like this would have to be part of the main binary; a
> separate binary could subscribe to the control topic and build a new
> configuration file according to the results.
>
>
> anyway, thoughts, suggestions, etc welcome.
> dunk.
>
> _______________________________________________
> mosquitto-dev mailing list
> mosquitto-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
------------------------------
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
End of mosquitto-dev Digest, Vol 20, Issue 7
********************************************