Re: [hono-dev] Changes in settlement of telemetry and dispatch router co

["Hudalla Kai (INST/ECS4)" <kai.hudalla@xxxxxxxxxxxx>]

Hey Gordon,

thanks for the detailed reply :-) See below


On 20.09.2017 16:36, Gordon Sim wrote:
> On 14/09/17 13:58, Hudalla Kai (INST/ECS4) wrote:
>> We are currently sending telemetry messages to multicast addresses
>> pre-settled. Doing so, we are seeing many (telemetry) messages being
>> dropped when we run our JMeter tests, i.e. we often receive less
>> messages than we have sent. Hono Messaging doesn't seem to drop the
>> messages (we can tell because of the metrics) so we assume that the
>> Dispatch Router drops them. The problem here seems to be that the
>> Dispatch Router starts to discard messages aggressively, if the
>> upstream senders send more messages than the downstream consumers can
>> handle (i.e. if any of the downstream consumers is not providing
>> enough credits). The number of credits the Dispatch Router hands out
>> to Hono Messaging seems to not be linked to the credits the DR
>> receives from the downstream consumers. Note that this is true for
>> both multicast as well as balanced target addresses.
>>
>> So, if we want to make sure that the Protocol Adapters only get as
>> many credits as the DR can forward to its downstream consumers, we
>> need to send telemetry messages unsettled, because only this way the
>> DR is able to provide end-to-end flow control. However, if we do so,
>> we also need to make sure that the telemetry addresses are balanced
>> (not multicast), as Marc pointed out.
>
> Do you want to allow multiple independent subscribers to the telemetry
> stream? If so, and if you want at-least-once, you need to route them
> through a broker topic. You can do this by establishing link routes
> for that address, configured to route the links to the broker hosting
> the topic.
>
This is indeed an interesting question. Since Hono was never meant to be
another message broker, I think that the decision to set up a system
like proposed by you, should be up to the user/admin. The AMQP network
is not per se considered to be a part of Hono itself. So I guess our
approach should be to allow Hono to be used within as many system
designs as possible. The question therefore is more: do we prevent Hono
from being used in the way you describe if we send Telemetry messages
unsettled? Or would this even be a requirement?
Personally, I do not think that the use case of multiple independent
subscribers of telemetry messages should be implemented using brokers
but instead using something like Apache Kafka that is hooked up with
Hono ...
> In general, if your consumers can't keep up with the rate at which
> telemetry messages are produced, there is really only two things you
> can do: queue the messages up somewhere or drop some of them.
>
> Queuing the messages up somewhere only makes sense if the situation is
> temporary, and the consumers will catch up (meaning messages have to
> actually be processed faster than they are produced for some amount of
> time, to allow the backlog to be cleared). If queuing makes sense, it
> can be done either at the producers (by throttling them) or by the
> messaging infrastructure. The router deliberately only has a fixed
> (and relatively small) amount of buffer space. For handling larger
> backlogs, a broker is a better choice.
>
> The decision to drop messages could likewise be taken either at the
> producers (providing they receive some indication of the problem) or
> by the messaging infrastructure. For telemetry data, the producers
> might reduce the frequency at which they send updates, or more
> aggressively batch any data that needs to be archived.
>
> However, where it is not desirable (or not possible) to put the
> responsibility for this onto the producers the messaging
> infrastructure could be given hints about how to drop messages to
> minimise the impact, e.g. ttls on message with data that becomes stale
> (this saves the consumers having to process these stale messages, and
> allows them more chance to catch up with the recent data) or
> 'last-value' type semantics, where a new update effectively removes
> any older messages. At present the router doesn't do anything like
> this (modifying it to use ttl when dropping messages should be
> possible however), but brokers do.
I guess our intent with changing the disposition handling for Telemetry
is exactly to allow for back pressure being pushed all the way upstream
to the protocol adapters and devices so that we can prevent the
(unnecessary) messages from being sent in the first place (instead of
having to dispose of them somewhere in between the sender and consumer).
> _______________________________________________
> hono-dev mailing list
> hono-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/hono-dev

-- 
Mit freundlichen Grüßen / Best regards

Kai Hudalla
Chief Software Architect

Bosch Software Innovations GmbH
Schöneberger Ufer 89-91
10785 Berlin
GERMANY
www.bosch-si.com

Registered Office: Berlin, Registration Court: Amtsgericht Charlottenburg; HRB 148411 B
Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing Directors: Dr.-Ing. Rainer Kallenbach, Michael Hahn