Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mosquitto-dev] Cassandra as a retained message store
Hi Michael

This sound promising, especially when we add zmq in the mix and Cassandra - is the plan to cluster cluster the zmq interface as well?


On Thu, Dec 11, 2014 at 11:19 PM, Laing, Michael <michael.laing@xxxxxxxxxxx> wrote:
I have built a proof-of-concept C interface to my POC retained store Cassandra and Postgres drivers.

It is full duplex, non-blocking and asynchronous using zmq.

I am thinking that I will clone the mosquitto 1.4 repo and incorporate my interface as a POC.

My approach will be to send non-blocking PULLs from mosquitto upon subscription and non-blocking PUSHes upon publish (with retain).

For responses, I will add a polled item to the main polling loop and pull up to 100 or so messages per loop - I'll figure out how to send each directly to the client id.

Does that sound reasonable? It's been a decade or so since I did any C work so I am a bit rusty.

ml


On Wed, Dec 3, 2014 at 5:33 PM, Laing, Michael <michael.laing@xxxxxxxxxxx> wrote:
Just an update.

I have increased the wildcard level span to 9 by combining inversion and indexing strategies. It is still very fast and retrieves all results in a single query.

To scale, it requires one or 2 more strategies to increase the cardinality of the partition keys - the 'row' in the big table model. In the current nytfabrik, we use the 0 level as an index into metadata that lets us do this, in effect moving some levels into the partition key and, sometimes, a rarely wildcarded level as a hash into a partition key shard.

For contrast, I have done the same thing using Postgres - much easier, of course, as the db does the query planning. Also fast.

But now I am thinking about how to implement these in mosquitto. They could be embedded, but maybe it would be more flexible to use something like zmq to asynchronously stream queries and responses. Then I could do my Cassandra and Postgres backends in python - others may want to do mongo or redis, etc., using their language of choice.

Cheers,

Michael

On Mon, Dec 1, 2014 at 2:07 AM, Oegma2 <oegma2@xxxxxxxxx> wrote:
Hi Michael

I will definitely be interested to hear more ^_^

On Thu, Nov 27, 2014 at 10:43 AM, Roger Light <roger@xxxxxxxxxx> wrote:
Hi Michael,

I'd definitely be interested to hear more.

Regards,

Roger


On Wed, Nov 26, 2014 at 3:23 PM, Laing, Michael
<michael.laing@xxxxxxxxxxx> wrote:
> I am experimenting with MQTT as a way to extend the nytfabrik at the edge,
> complementing and perhaps replacing our custom WebSocket/SockJS protocol.
>
> We use RabbitMQ with topic routing on the nytfabrik global backbone mesh, so
> we are familiar with wildcarding, subscription, etc.
>
> And we have lots of 'retained' messages - 10's of millions growing to 100's
> of millions.
>
> This is no problem for Cassandra but, as we look to support MQTT, we need a
> technical solution for query with wildcards on retained messages.
>
> Cassandra has a limited query language but can be very fast. I was looking
> for a way to run 'close to the metal' (no additional layers) and now have a
> working prototype architecture and algorithms for store and query that are
> simple, fast and which scale. So I think that technical blocker is overcome
> for our usage cases.
>
> Anyway, if this is of interest, I'll keep this list informed, and maybe
> solicit some advice.
>
> We're not sure yet how we'll add MQTT to the nytfabrik, but most likely we
> would remix mosquitto with our own code, package it up, and run it under
> python on our autoscaling edge instances.
>
> Cheers,
>
> Michael
>
> _______________________________________________
> mosquitto-dev mailing list
> mosquitto-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev


_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev



_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev

Back to the top