Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

["Hudalla Kai (INST/ECS4)" <kai.hudalla@xxxxxxxxxxxx>]

see below

On 14.09.2017 10:12, Paolo Patierno wrote:

Hi Karsten,

regarding the future field gateways scenario I think that for this use case, some data aggregation will be done at the edge and data information will be sent using the payload even with device information. I don't see a field gateway publishing to different topics with device information there (i.e. publish on /telemetry/tenant_id/device1, /telemetry/tenant_id/device2, ...). I see the field gateway publishing always on a single telemetry address with aggregated data in the payload even with device ids (info). But I'm thinking aloud ...

that is one scenario, another one is that the gateway publishes data "on behalf of" the connected devices but to a downstream consumer this would still look like the devices have published their data themselves

I agree that even using client-id for a different use case could be a little bit confusing; regarding the connection "stealing" we already agreed in the past to not handle it.

From you experience I have understood that devices have two ids : technical and logical. A device MUST know the technical one but it COULD NOT know the logical one.

Afaik the authentication is needed only if device provides the technical one, is that right ? Using the logical one means no authentication is required (even because the logical one is recovered by the adapter during the device authentication which provides username (technical id) and password).

it is the other way around: if devices are required to authenticate, they MUST have been provisioned with a set of credentials (e.g. a usernme/password pair). since there are multiple ways of authenticating a client using arbitrary mechanisms, we should not assume nor require the credentials to contain the logical ID (the ID the device is registered under in the Device Registration service). During authentication the device therefore presents its credentials which are then verified using the information stored in the Credentials service. As part of a successful authentication we have also resolved the logical ID that belong to the authenticated device.

However, if no authentication is required, we still need to determine the device's logical ID. Since no authentication is taking place, the device needs to convey this information in some other way. It could either do so during connection establishment (as you seem to propose) or as part of the topic address (the way we have been doing it all the time so far). However, in this case the device MUST know its logical ID (and tenant), there is no way around that.

So to summarize the problem here is to distinguish the following use case : 

- device provides username with technical id but no password, just because the password is not needed for authentication

- device provides username with logical id and no password, just because no authentication is needed

Assuming that you are proposing that the device ALWAYS provides username/password in its CONNECT packet,
I would rather not abuse these fields for conveying information other than required for authentication.

Having different adapters configured in different ways could be awkward ...

Paolo Patierno

Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT

Microsoft Azure Advisor 

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

---

From: hono-dev-bounces@xxxxxxxxxxx <hono-dev-bounces@xxxxxxxxxxx> on behalf of Frank Karsten (INST/ECS4) <Karsten.Frank2@xxxxxxxxxxxx>
Sent: Thursday, September 14, 2017 7:54 AM
To: hono developer discussions
Subject: Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

 

Hi Paolo,

 

I agree to your point to avoid having different topic addresses for different use cases – wasn’t 100% glad with it, either J

 

So, let us stick strictly to only support “telemetry”/”event” as topic address then – in all cases.

 

My thoughts on that strategy:

 

-        I skip the point that some MQTT clients might not support username/password – seems to be too pessimistic. So we just demand that MQTT clients support that, fine for me.

 

-        Shorter PUBLISH messages : good point, I think that is very reasonable for single devices.

Could imagine that in the future we might need clients (e.g. field gateways) that deliver messages for a bunch of devices and then we might need the more detailed topic structure again. But this would be another discussion.

 

 

-        Use clientId for the case where no password is required: not sure about that, doesn’t that introduce the discussion again with session stealing? When we never use the clientId for such purposes anymore, we never run into problems here. My opinion: I would rather not use the clientId for Hono specific purposes.

 

How about this approach to the scenario: the device provides it’s logicalId in the username, but this fact needs to be specifically configured in the MQTT adapter. Then all devices connected to that adapter need to do that. So no special usage of clientId anymore. And if at the same time we have other devices, we need more (differently configured) MQTT adapters.

 

 

Mit freundlichen Grüßen / Best regards

Karsten Frank

Senior Software Developer

 

Bosch Software Innovations GmbH
Development Core Products (INST/ECS6-Be)
Schöneberger Ufer 89-91
10785 Berlin
GERMANY
www.bosch-si.de
www.blog.bosch-si.com

 

Von: hono-dev-bounces@xxxxxxxxxxx [mailto:hono-dev-bounces@xxxxxxxxxxx] Im Auftrag von Paolo Patierno
Gesendet: Donnerstag, 14. September 2017 09:18
An: hono developer discussions <hono-dev@xxxxxxxxxxx>
Betreff: Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

 

Hi Karsten,

the main project providing all MQTT clients is Eclipse Paho and afaik all of them support username and password.

 

I got your point about device providing identity (username) but not password. Maybe in such case could we use client-id as "logical" id when no authentication is required ?

 

My point is that we should not change the topic address for different use cases (device needs authentication or not), having /telemetry in some cases and /telemetry/tenant_id/device_id in others.

Having a short one is really better even because the topic address is in each PUBLISH message while identity/authentication information are needed only on CONNECT (then using username, password and client-id).

 

Paolo Patierno

Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT

Microsoft Azure Advisor 

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

 

---

From: hono-dev-bounces@xxxxxxxxxxx <hono-dev-bounces@xxxxxxxxxxx> on behalf of Frank Karsten (INST/ECS4) <Karsten.Frank2@xxxxxxxxxxxx>
Sent: Wednesday, September 13, 2017 3:26 PM
To: hono developer discussions
Subject: Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

 

Thanks for your thoughts, Paolo :

 

Yes, that is correct:

the proposal uses different topic addresses for authenticated and unauthenticated devices.

 

Your idea makes sense regarding the simplification (or unification) of topic addresses, but to me new questions then arise:

 

-        maybe not all MQTT clients support “username” and “password” ? I do not know if this might be the case, at least this change would force ALL MQTT client to use the username/password pair.

 

-        to not set any password might be a valid use case for normal authentication? Like “devices have to tell their identity, but they do not need a password”. This would be prevented then, since “no password” suddenly has a semantic.

 

-        It does not really simplify the implementation for not authenticated devices, just changes where they have to put their information (tenantId and logical device id).

 

WDYT?

 

Generally, in the authenticated use case with devices not knowing their logicalId, it is not only a simplification, but otherwise just not possible with the previous scheme (that required the logicalId in the topic).

 

I just implemented a first version of it, and it “feels” much better for the device to only say “I want to send telemetry data downstream, and you know to which Hono topic this has to be sent” J

 

Mit freundlichen Grüßen / Best regards

Karsten Frank

Von: hono-dev-bounces@xxxxxxxxxxx [mailto:hono-dev-bounces@xxxxxxxxxxx] Im Auftrag von Paolo Patierno
Gesendet: Mittwoch, 13. September 2017 17:08
An: hono-dev@xxxxxxxxxxx
Betreff: Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

 

In my understanding about this discussion we are searching for a different way to conveying information like tenant_id and device_id (logical or technical) in order to simplify the topic address for telemetry (i.e. using just /telemetry) and not having such information there.

It seems to me that the proposed solution comes to use different topic address if the device needs to be authenticated or not.

Maybe could we use just /telemetry as topic address and then playing with "username" and "password" field in MQTT ?

 

1. username = device_id@tenant_id, password = <password> --> the device_id is the "technical id" and the Credentials API are used on the adapter for getting the "logical" one and authenticate device with provided password.
2. username = device_id@tenant_id, no password --> the device_is is already the "logical" one and no authentication is needed

 

even in this case the MQTT client id isn't needed anymore.

Is there something I'm missing here ?

 

Paolo Patierno

Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT

Microsoft Azure Advisor 

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

 

---

From: hono-dev-bounces@xxxxxxxxxxx <hono-dev-bounces@xxxxxxxxxxx> on behalf of Hudalla Kai (INST/ECS4) <kai.hudalla@xxxxxxxxxxxx>
Sent: Wednesday, September 13, 2017 9:23 AM
To: hono-dev@xxxxxxxxxxx
Subject: Re: [hono-dev] Topic structure for protocol adapters (MQTT and REST)

 

Thanks for putting this together, Karsten :-)

+1 for using this approach

Some (additional) remarks from my side:

I think it is important to understand that we would like Hono to support all of the following types of devices:

1) Devices that need not/can not authenticate to the protocol adapter, e.g. because they are deployed to a strictly private network
2) Devices that do authenticate to the protocol adapter using arbitrary credentials that the device has been provisioned with e.g. during the production process and which
2.1) also know their logical Hono ID
2.2) do not/should not know their logical Hono ID

Using the approach described by Karsten we should be able to easily support all these cases.
Devices of type 1 clearly need to provide the tenant and logical device ID information as part of the topic name (this is what we do today already)
For devices of type 2 the protocol adapter can resolve the logical device ID using the Credentials service. When the device publishes to e.g. "telemetry", the protocol adapter simply uses the resolved device ID when forwarding the data to Hono Messaging. In cases where the device publishes to "telemetry/$TENANT/$DEVICE", the adapter makes sure that the $TENANT and the $DEVICE match the resolved device ID.

NB For the MQTT adapter this means that we no longer require the device to provide its logical device ID in the Client Identifier of its CONNECT packet.

Kai

On 13.09.2017 10:14, Frank Karsten (INST/ECS4) wrote:

Hello to all,

 

after having introduced credentials for the MQTT adapter and soon for the REST adapter as well,

it became obvious that the current structure of topics is not intuitive and even contradictory to the

basic concept of the Credentials API:

not forcing devices to know their "logical device id" when communicating with the adapters.

 

Thus I like to discuss a new concept for the topic structure based on discussions between @sophokles78 (Kai) and @sysexcontrol (me):

 

   publish data to "telemetry" instead of "telemetry/tenantId/logicalDeviceId".

 

 

 

** What is the problem?**

 

Currently, we defined the topic structure "[telemetry|event]/tenantId/deviceId", both for MQTT and for REST. This is fine for the unauthenticated

case, since Hono needs to know where to send data to when a device wants to publish data.

But now we have the credentials API integrated into the MQTT adapter and it became obvious that this topic structure causes problems (being outlined in the following lines).

 

For simplicity, let's focus on how this works for MQTT (REST will be discussed later).

 

The steps for a device are:

 

1.) A device authenticates itself by providing some kind of secret, for example "authId@tenantId/password".

The "authId" (or name it "technicalDeviceId") is usually not the same as the "logicalDeviceId" that is used inside of Hono and the solutions (see Credentials API).

 

- the device opens a MQTT connection and provides the "authId@tenantId/password" which is checked. If not successful, the connection is closed.

- in multi-tenancy configurations, the authId contains the tenantId as well (like "sensor1@myTenant")

 

 

2.) After successful authentication the device wants to publish data. But the question is : to what topic ?

 

- it needs to use the topic "telemetry/tenantId/logicalDeviceId".

- before the publish is accepted, the logicalDeviceId is taken to check with the device registry if the device is enabled

 

3.) And these are the problems:

 

- for publishing, the device needs to know the logicalDeviceId (fatal, since it contradicts to the concept to decouple the device from the logicalDeviceId)

 

- it needs to fill in the same tenant again that was necessary during authentication already (not fatal, but annoying, it did tell Hono the tenantId already, why again?)

 

 

 

** What would be the better solution? **

 

In typical production installations, devices are authenticated first, and Hono knows all it needs for letting the device just publish it’s data.

 

 

So a MQTT device would :

 

- authenticate itself and gets an open connection

- publishes it's data only to "telemetry" or "event" (no need for tenantId or even logicalDeviceId anymore)

- the check with the device registry for the device being enabled still is necessary

 

There is just no necessity anymore for it to fill in already known data: the authenticated device just publishes to "telemetry" or "event". That's it.

 

 

 

** What is the topic structure for installations without authentication? **

 

This use case was the reason for the current topic structure, so there would be no changes to that:

- "[telemetry|event]/tenantId/logicalDeviceId" would still have to be used

 

** How do we handle the same topic for the REST adapter? **

 

For REST I would like to postpone the detailed proposal. It will mainly be a consequence of what we have discussed here (there would be several changes for simplification as well).

Just to not let this point to become even lengthier as it already is.

 

 

 

 

**What is your feedback to that proposal?**

 

Note that it is not only an improvement, partially it is a pure necessity, since the logicalDeviceId must not be known to the device and we need to change that.

 

 

Looking forward to a discussion, feedback, ideas!

 

Thanks,

 

sysexcontrol (Karsten)

 

 

_______________________________________________

hono-dev mailing list

hono-dev@xxxxxxxxxxx

To change your delivery options, retrieve your password, or unsubscribe from this list, visit

https://dev.eclipse.org/mailman/listinfo/hono-dev

 

-- 

Mit freundlichen Grüßen / Best regards

Kai Hudalla

Chief Software Architect

Bosch Software Innovations GmbH

Schöneberger Ufer 89-91

10785 Berlin

GERMANY

www.bosch-si.com

Registered Office: Berlin, Registration Court: Amtsgericht Charlottenburg; HRB 148411 B

Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing Directors: Dr.-Ing. Rainer Kallenbach, Michael Hahn

_______________________________________________
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev

-- 
Mit freundlichen Grüßen / Best regards

Kai Hudalla
Chief Software Architect

Bosch Software Innovations GmbH
Schöneberger Ufer 89-91
10785 Berlin
GERMANY
www.bosch-si.com

Registered Office: Berlin, Registration Court: Amtsgericht Charlottenburg; HRB 148411 B
Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing Directors: Dr.-Ing. Rainer Kallenbach, Michael Hahn